jim/URLNotesGrabberCore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: track TL.db schema notes in the repo

Browse Source 
TL.db.md documents the live schema: the three content tables and their
row counts, the '.' placeholder convention the crawler writes instead of
NULL, the two incompatible date formats in Blogs.DateAdded, and the
access paths that matter on the 1.19M-row Notes table.

It also covers Blogs.IsDeleted, which Rolodex adds by ALTER TABLE and
this crawler must not write.

The file was sitting untracked next to the database it describes.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jim
2026-07-29 09:54:20 -05:00
co-authored by Claude Opus 5
parent d6637266b7
commit a14debd5ed
1 changed files with 290 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
URLNotesGrabberCORE/TL.db.md
+290
View File
@@ -0,0 +1,290 @@
# `TL.db` — schema notes
The SQLite database behind **URLNotesGrabberCORE** and its sibling crawlers, and the one
[Rolodex](https://git.basso.land/jim/Rolodex) reads.
Everything below was read out of the live file, not inferred from code. Counts are as of
**2026-07-29**; re-run the queries at the bottom to refresh them.
- Journal mode: **WAL** — `TL.db-wal` and `TL.db-shm` live beside the file and are part of
the database. Copying `TL.db` alone gives you whatever was last checkpointed, not the
current state.
- Page size: 4096.
---
## The three content tables
| Table | Rows | What it is |
|---|--:|---|
| `Blogs` | 144,367 | The crawl registry — one row per known blog, plus crawl-state flags |
| `Posts` | 14,589 | Stored post content. Only 3,602 blogs actually have any |
| `Notes` | 1,189,604 | The engagement graph: `NoteBlogName` acted on `(RootBlogName, PostID)` |
The engagement graph is the interesting part. 31,888 distinct blogs appear as engagers —
far more than the 3,602 that have stored posts — which is what makes this a social graph
rather than a post archive.
### `Blogs`
```sql
CREATE TABLE "Blogs" (
"BlogName" TEXT,
"HasBeenOutput" INTEGER DEFAULT 0,
"IsActive" INTEGER DEFAULT 1,
"DateAdded" TEXT NOT NULL DEFAULT '12/24/25',
"ByLikes" INTEGER NOT NULL DEFAULT 0,
"LikesPulled" INTEGER NOT NULL DEFAULT 0,
"LikesCursor" INTEGER DEFAULT 0,
"DateModified" TEXT,
"DateCreated" TEXT,
LikesNewestTimestamp INTEGER DEFAULT 0,
LikesLastRefreshed INTEGER DEFAULT 0,
LikesLastNewCount INTEGER DEFAULT 0,
TTFolderPath TEXT,
PRIMARY KEY("BlogName")
);
```
`BlogName` is the primary key, so it is the only indexed way in. There is no index on any
flag or date — filtering or sorting on those scans all 144k rows, which is affordable
here and is not on `Notes`.
Flag distribution: `IsActive = 1` on 144,366 of 144,367 rows (effectively always true, so
the Active filter is close to a no-op today), `HasBeenOutput = 1` on 5,369, `ByLikes = 1`
on 2.
The columns after `DateCreated` were added later by `ALTER TABLE`, which is why they carry
no quoting in the stored DDL. That is the normal way this schema grows — and it is how
[`IsDeleted`](#blogsisdeleted--added-by-rolodex) gets there too, so expect that column in
the DDL of any database Rolodex has opened.
**`DateAdded` is not written consistently.** 126,423 rows hold ISO `yyyy-MM-dd HH:mm:ss`;
17,944 hold US-format `M/d/yy` from a bulk import. As text those two sort into different
parts of the table, so anything ordering or range-filtering on this column has to
normalise first — see `DateSql` in Rolodex.
### `Posts`
```sql
CREATE TABLE "Posts" (
"BlogName" TEXT,
"PostID" INTEGER,
"HasNotesGathered" INTEGER DEFAULT 0,
"reblogURL" TEXT,
"NotFound" INTEGER DEFAULT 0,
"PostDate" TEXT,
"NotesGatheredDateTime" INTEGER NOT NULL DEFAULT 1729746000,
"HasImage" INTEGER NOT NULL DEFAULT 0,
"PostURL" TEXT,
"Slug" TEXT,
"ReblogKey" TEXT,
"ReblogName" TEXT,
"Summary" TEXT,
"Quote" TEXT,
"Body" TEXT,
"Tags" TEXT,
"Link" TEXT,
"PhotoURL" TEXT,
"PhotoCaption" TEXT,
"DownloadedFiles" TEXT,
"AudioCaption" TEXT,
"Question" TEXT,
"Answer" TEXT,
"Title" TEXT,
"ByLikes" INTEGER NOT NULL DEFAULT 0,
"RootBlogName" TEXT,
"RootURL" TEXT,
"DateModified" TEXT,
"DateCreated" TEXT,
PostType TEXT,
PRIMARY KEY("BlogName","PostID")
);
```
**The key is `(BlogName, PostID)`, not `PostID`.** This matters more than it looks: 325
post IDs exist under more than one blog, so an ID on its own is both ambiguous *and*
unindexed. Any lookup should carry the blog name, and a batch lookup should group by blog
so it stays on the leading column of the key.
Notable:
- **`PostType` is `NULL` on all 14,589 rows.** The column exists but nothing has ever
populated it. Treat it as unpopulated rather than as a type discriminator.
- `HasImage = 1` on 14,268 rows — nearly all of them. It records that the post *had* a
picture, not that a usable URL was kept, so it is not a reliable predictor that anything
will render.
- `PhotoURL` is largely unused; in practice the image markup lives inside `Body`.
- `NotFound = 1` on 4,663 rows — posts that have since been deleted upstream.
- The content columns (`Body`, `Quote`, `Question`, `Answer`, …) are the heavy ones. List
views should not select them.
### `Notes`
```sql
CREATE TABLE "Notes" (
"RootBlogName" TEXT,
"PostID" INTEGER,
"NoteBlogName" TEXT,
"TimeStamp" INTEGER,
"Type" TEXT,
"replyText" TEXT DEFAULT '.',
"DatetimeCrawled" TEXT DEFAULT '2/12/26 12am',
"DateModified" TEXT,
"DateCreated" TEXT,
PRIMARY KEY("RootBlogName","PostID","TimeStamp","Type","NoteBlogName")
);
CREATE INDEX "Notes_idx_06e01ae3" ON "Notes" ("TimeStamp" DESC);
CREATE INDEX "ix_NoteBlogName01" ON "Notes" ("NoteBlogName");
```
One row per engagement event. `TimeStamp` is **unix seconds** — unlike every date column
elsewhere in the schema, which are text.
| `Type` | Rows | Share |
|---|--:|--:|
| `like` | 947,955 | 79.7% |
| `reblog` | 224,323 | 18.9% |
| `reply` | 15,201 | 1.3% |
| `posted` | 2,106 | 0.2% |
| `post_attribution` | 19 | — |
At 1.19M rows this is the table that dictates how the whole database has to be queried:
- **Nothing should run an unbounded `SELECT` or a bare `COUNT(*)` here.** A count scans
the lot on every call.
- The only fast access paths are the primary key's leading columns (`RootBlogName`, then
`PostID`) and `ix_NoteBlogName01` on `NoteBlogName`. "Notes received by a blog" and
"notes given by a blog" are both cheap; almost nothing else is.
- Ordering by anything but `TimeStamp` is a full sort of whatever the filters leave.
- `replyText` is `'.'` on 1,174,706 rows — only `reply` notes carry real text.
### Referential integrity
There are no foreign keys, and the tables do not perfectly agree:
- 4 `Posts` rows name a blog with no `Blogs` row.
- 15 of the 31,888 distinct engagers have no `Blogs` row.
So a name appearing in `Notes` or `Posts` is not a guarantee that the registry knows about
it. Joins from those tables back to `Blogs` should tolerate a miss.
---
## The `'.'` placeholder convention
**The crawler writes a single dot into text columns it has no value for, rather than
`NULL`.** This is the single most surprising thing about the schema and it affects every
consumer.
| Column | `'.'` rows |
|---|--:|
| `Notes.replyText` | 1,174,706 |
| `Posts.Title` | 13,144 |
| `Posts.Body` | 172 |
Any query whose output reaches a human should collapse it:
```sql
NULLIF(NULLIF(SomeColumn, '.'), '') AS SomeColumn
```
Empty string turns up too, hence the double `NULLIF`. Not every column is affected —
`Blogs.TTFolderPath` and `Blogs.DateModified` currently have zero dot rows — but new
columns tend to acquire them, so treat cleaning as the default for any text column
rendered to a user.
---
## Supporting tables
Crawler bookkeeping. Rolodex ignores all of these.
| Table | Rows | What it is |
|---|--:|---|
| `DailyAPICount` | 133 | `(Date TEXT PK, APICount INTEGER)` — per-day API call tally against the rate limit |
| `ApiKeyPoolState` | 2 | `(KeyName TEXT PK, RetryUntil INTEGER)` — per-key backoff; `RetryUntil` is unix seconds |
| `ApiKeyPoolMeta` | 1 | `(Id PK CHECK (Id = 1), LastIndex)` — round-robin cursor. Singleton by check constraint |
| `CollectRunState` | 1 | `(Id PK CHECK (Id = 1), RunCutoff, RunComplete, RunStarted, RunCompletedAt)` — resume state for an interrupted collection run. Also a singleton |
---
## `Blogs.IsDeleted` — added by Rolodex
```sql
ALTER TABLE Blogs ADD COLUMN IsDeleted INTEGER NOT NULL DEFAULT 0;
```
**This column is not written by the crawler.** Rolodex adds it automatically at startup if
it is missing, and uses it to hide a blog from its own UI without destroying anything:
- `0` — the blog is live. `NOT NULL DEFAULT 0` means every existing row, and every row the
crawler creates afterwards, starts here.
- `1` — somebody removed the blog through Rolodex.
Removal is a soft delete and nothing else changes: the row keeps every crawl flag it had,
and its `Posts` and `Notes` rows are untouched. Restoring is the same `UPDATE` setting the
column back to `0`.
**Do not write `NULL` here.** Rolodex reads the column through `COALESCE(IsDeleted, 0)`
precisely so a NULL cannot strand a row — without it a NULL satisfies neither the live
test nor its negation, and the blog would disappear from the registry *and* from the
removed list with no way back through the UI. The `NOT NULL` declaration is the real
guard; the `COALESCE` is the belt to its braces.
### What other tools need to know
1. **Adding the column is safe for `SELECT`, and for `INSERT` that names its columns.** It
is *not* safe for `INSERT INTO Blogs VALUES (...)` without a column list — that form
breaks the moment column count changes. If any crawler does this, give it an explicit
column list.
2. **The crawler should leave the column alone.** Writing to it would silently un-remove or
remove blogs behind Rolodex's back.
3. **Re-crawling a removed blog will not bring it back.** An `INSERT OR REPLACE` on the
`Blogs` row *would*, by resetting the column to its default `0`. If that matters,
prefer an `UPDATE` of the specific columns, or `INSERT … ON CONFLICT DO UPDATE SET`
naming only the crawl columns.
4. **A tool that lists blogs should decide whether it cares.** Rolodex filters
`WHERE COALESCE(IsDeleted, 0) = 0` everywhere a blog surfaces. A crawler probably
should not — a blog hidden from a browsing UI is not necessarily one to stop crawling.
That is a deliberate choice, not an oversight.
Rolodex degrades if the column is absent — a read-only or `immutable=1` deployment cannot
take the `ALTER` — by falling back to its previous behaviour of showing every blog. So
removing the column is a supported way to back the feature out:
```sql
ALTER TABLE Blogs DROP COLUMN IsDeleted;
```
---
## Reproducing the numbers
```sql
SELECT 'Blogs', COUNT(*) FROM Blogs
UNION ALL SELECT 'Posts', COUNT(*) FROM Posts
UNION ALL SELECT 'Notes', COUNT(*) FROM Notes;
-- note type mix
SELECT Type, COUNT(*) FROM Notes GROUP BY Type ORDER BY 2 DESC;
-- the two date shapes in Blogs.DateAdded
SELECT CASE WHEN DateAdded LIKE '____-__-__%' THEN 'ISO' ELSE 'US' END, COUNT(*)
FROM Blogs GROUP BY 1;
-- post IDs that are ambiguous without a blog name
SELECT COUNT(*) FROM (
SELECT PostID FROM Posts GROUP BY PostID HAVING COUNT(DISTINCT BlogName) > 1);
-- rows that reference a blog the registry does not have
SELECT COUNT(*) FROM Posts p
WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = p.BlogName);
```
Open the file read-only so an inspection can never disturb a running crawl:
```bash
sqlite3 "file:TL.db?mode=ro" ".schema"
```