diff --git a/URLNotesGrabberCORE/TL.db.md b/URLNotesGrabberCORE/TL.db.md new file mode 100644 index 0000000..41a3107 --- /dev/null +++ b/URLNotesGrabberCORE/TL.db.md @@ -0,0 +1,290 @@ +# `TL.db` — schema notes + +The SQLite database behind **URLNotesGrabberCORE** and its sibling crawlers, and the one +[Rolodex](https://git.basso.land/jim/Rolodex) reads. + +Everything below was read out of the live file, not inferred from code. Counts are as of +**2026-07-29**; re-run the queries at the bottom to refresh them. + +- Journal mode: **WAL** — `TL.db-wal` and `TL.db-shm` live beside the file and are part of + the database. Copying `TL.db` alone gives you whatever was last checkpointed, not the + current state. +- Page size: 4096. + +--- + +## The three content tables + +| Table | Rows | What it is | +|---|--:|---| +| `Blogs` | 144,367 | The crawl registry — one row per known blog, plus crawl-state flags | +| `Posts` | 14,589 | Stored post content. Only 3,602 blogs actually have any | +| `Notes` | 1,189,604 | The engagement graph: `NoteBlogName` acted on `(RootBlogName, PostID)` | + +The engagement graph is the interesting part. 31,888 distinct blogs appear as engagers — +far more than the 3,602 that have stored posts — which is what makes this a social graph +rather than a post archive. + +### `Blogs` + +```sql +CREATE TABLE "Blogs" ( + "BlogName" TEXT, + "HasBeenOutput" INTEGER DEFAULT 0, + "IsActive" INTEGER DEFAULT 1, + "DateAdded" TEXT NOT NULL DEFAULT '12/24/25', + "ByLikes" INTEGER NOT NULL DEFAULT 0, + "LikesPulled" INTEGER NOT NULL DEFAULT 0, + "LikesCursor" INTEGER DEFAULT 0, + "DateModified" TEXT, + "DateCreated" TEXT, + LikesNewestTimestamp INTEGER DEFAULT 0, + LikesLastRefreshed INTEGER DEFAULT 0, + LikesLastNewCount INTEGER DEFAULT 0, + TTFolderPath TEXT, + PRIMARY KEY("BlogName") +); +``` + +`BlogName` is the primary key, so it is the only indexed way in. There is no index on any +flag or date — filtering or sorting on those scans all 144k rows, which is affordable +here and is not on `Notes`. + +Flag distribution: `IsActive = 1` on 144,366 of 144,367 rows (effectively always true, so +the Active filter is close to a no-op today), `HasBeenOutput = 1` on 5,369, `ByLikes = 1` +on 2. + +The columns after `DateCreated` were added later by `ALTER TABLE`, which is why they carry +no quoting in the stored DDL. That is the normal way this schema grows — and it is how +[`IsDeleted`](#blogsisdeleted--added-by-rolodex) gets there too, so expect that column in +the DDL of any database Rolodex has opened. + +**`DateAdded` is not written consistently.** 126,423 rows hold ISO `yyyy-MM-dd HH:mm:ss`; +17,944 hold US-format `M/d/yy` from a bulk import. As text those two sort into different +parts of the table, so anything ordering or range-filtering on this column has to +normalise first — see `DateSql` in Rolodex. + +### `Posts` + +```sql +CREATE TABLE "Posts" ( + "BlogName" TEXT, + "PostID" INTEGER, + "HasNotesGathered" INTEGER DEFAULT 0, + "reblogURL" TEXT, + "NotFound" INTEGER DEFAULT 0, + "PostDate" TEXT, + "NotesGatheredDateTime" INTEGER NOT NULL DEFAULT 1729746000, + "HasImage" INTEGER NOT NULL DEFAULT 0, + "PostURL" TEXT, + "Slug" TEXT, + "ReblogKey" TEXT, + "ReblogName" TEXT, + "Summary" TEXT, + "Quote" TEXT, + "Body" TEXT, + "Tags" TEXT, + "Link" TEXT, + "PhotoURL" TEXT, + "PhotoCaption" TEXT, + "DownloadedFiles" TEXT, + "AudioCaption" TEXT, + "Question" TEXT, + "Answer" TEXT, + "Title" TEXT, + "ByLikes" INTEGER NOT NULL DEFAULT 0, + "RootBlogName" TEXT, + "RootURL" TEXT, + "DateModified" TEXT, + "DateCreated" TEXT, + PostType TEXT, + PRIMARY KEY("BlogName","PostID") +); +``` + +**The key is `(BlogName, PostID)`, not `PostID`.** This matters more than it looks: 325 +post IDs exist under more than one blog, so an ID on its own is both ambiguous *and* +unindexed. Any lookup should carry the blog name, and a batch lookup should group by blog +so it stays on the leading column of the key. + +Notable: + +- **`PostType` is `NULL` on all 14,589 rows.** The column exists but nothing has ever + populated it. Treat it as unpopulated rather than as a type discriminator. +- `HasImage = 1` on 14,268 rows — nearly all of them. It records that the post *had* a + picture, not that a usable URL was kept, so it is not a reliable predictor that anything + will render. +- `PhotoURL` is largely unused; in practice the image markup lives inside `Body`. +- `NotFound = 1` on 4,663 rows — posts that have since been deleted upstream. +- The content columns (`Body`, `Quote`, `Question`, `Answer`, …) are the heavy ones. List + views should not select them. + +### `Notes` + +```sql +CREATE TABLE "Notes" ( + "RootBlogName" TEXT, + "PostID" INTEGER, + "NoteBlogName" TEXT, + "TimeStamp" INTEGER, + "Type" TEXT, + "replyText" TEXT DEFAULT '.', + "DatetimeCrawled" TEXT DEFAULT '2/12/26 12am', + "DateModified" TEXT, + "DateCreated" TEXT, + PRIMARY KEY("RootBlogName","PostID","TimeStamp","Type","NoteBlogName") +); + +CREATE INDEX "Notes_idx_06e01ae3" ON "Notes" ("TimeStamp" DESC); +CREATE INDEX "ix_NoteBlogName01" ON "Notes" ("NoteBlogName"); +``` + +One row per engagement event. `TimeStamp` is **unix seconds** — unlike every date column +elsewhere in the schema, which are text. + +| `Type` | Rows | Share | +|---|--:|--:| +| `like` | 947,955 | 79.7% | +| `reblog` | 224,323 | 18.9% | +| `reply` | 15,201 | 1.3% | +| `posted` | 2,106 | 0.2% | +| `post_attribution` | 19 | — | + +At 1.19M rows this is the table that dictates how the whole database has to be queried: + +- **Nothing should run an unbounded `SELECT` or a bare `COUNT(*)` here.** A count scans + the lot on every call. +- The only fast access paths are the primary key's leading columns (`RootBlogName`, then + `PostID`) and `ix_NoteBlogName01` on `NoteBlogName`. "Notes received by a blog" and + "notes given by a blog" are both cheap; almost nothing else is. +- Ordering by anything but `TimeStamp` is a full sort of whatever the filters leave. +- `replyText` is `'.'` on 1,174,706 rows — only `reply` notes carry real text. + +### Referential integrity + +There are no foreign keys, and the tables do not perfectly agree: + +- 4 `Posts` rows name a blog with no `Blogs` row. +- 15 of the 31,888 distinct engagers have no `Blogs` row. + +So a name appearing in `Notes` or `Posts` is not a guarantee that the registry knows about +it. Joins from those tables back to `Blogs` should tolerate a miss. + +--- + +## The `'.'` placeholder convention + +**The crawler writes a single dot into text columns it has no value for, rather than +`NULL`.** This is the single most surprising thing about the schema and it affects every +consumer. + +| Column | `'.'` rows | +|---|--:| +| `Notes.replyText` | 1,174,706 | +| `Posts.Title` | 13,144 | +| `Posts.Body` | 172 | + +Any query whose output reaches a human should collapse it: + +```sql +NULLIF(NULLIF(SomeColumn, '.'), '') AS SomeColumn +``` + +Empty string turns up too, hence the double `NULLIF`. Not every column is affected — +`Blogs.TTFolderPath` and `Blogs.DateModified` currently have zero dot rows — but new +columns tend to acquire them, so treat cleaning as the default for any text column +rendered to a user. + +--- + +## Supporting tables + +Crawler bookkeeping. Rolodex ignores all of these. + +| Table | Rows | What it is | +|---|--:|---| +| `DailyAPICount` | 133 | `(Date TEXT PK, APICount INTEGER)` — per-day API call tally against the rate limit | +| `ApiKeyPoolState` | 2 | `(KeyName TEXT PK, RetryUntil INTEGER)` — per-key backoff; `RetryUntil` is unix seconds | +| `ApiKeyPoolMeta` | 1 | `(Id PK CHECK (Id = 1), LastIndex)` — round-robin cursor. Singleton by check constraint | +| `CollectRunState` | 1 | `(Id PK CHECK (Id = 1), RunCutoff, RunComplete, RunStarted, RunCompletedAt)` — resume state for an interrupted collection run. Also a singleton | + +--- + +## `Blogs.IsDeleted` — added by Rolodex + +```sql +ALTER TABLE Blogs ADD COLUMN IsDeleted INTEGER NOT NULL DEFAULT 0; +``` + +**This column is not written by the crawler.** Rolodex adds it automatically at startup if +it is missing, and uses it to hide a blog from its own UI without destroying anything: + +- `0` — the blog is live. `NOT NULL DEFAULT 0` means every existing row, and every row the + crawler creates afterwards, starts here. +- `1` — somebody removed the blog through Rolodex. + +Removal is a soft delete and nothing else changes: the row keeps every crawl flag it had, +and its `Posts` and `Notes` rows are untouched. Restoring is the same `UPDATE` setting the +column back to `0`. + +**Do not write `NULL` here.** Rolodex reads the column through `COALESCE(IsDeleted, 0)` +precisely so a NULL cannot strand a row — without it a NULL satisfies neither the live +test nor its negation, and the blog would disappear from the registry *and* from the +removed list with no way back through the UI. The `NOT NULL` declaration is the real +guard; the `COALESCE` is the belt to its braces. + +### What other tools need to know + +1. **Adding the column is safe for `SELECT`, and for `INSERT` that names its columns.** It + is *not* safe for `INSERT INTO Blogs VALUES (...)` without a column list — that form + breaks the moment column count changes. If any crawler does this, give it an explicit + column list. +2. **The crawler should leave the column alone.** Writing to it would silently un-remove or + remove blogs behind Rolodex's back. +3. **Re-crawling a removed blog will not bring it back.** An `INSERT OR REPLACE` on the + `Blogs` row *would*, by resetting the column to its default `0`. If that matters, + prefer an `UPDATE` of the specific columns, or `INSERT … ON CONFLICT DO UPDATE SET` + naming only the crawl columns. +4. **A tool that lists blogs should decide whether it cares.** Rolodex filters + `WHERE COALESCE(IsDeleted, 0) = 0` everywhere a blog surfaces. A crawler probably + should not — a blog hidden from a browsing UI is not necessarily one to stop crawling. + That is a deliberate choice, not an oversight. + +Rolodex degrades if the column is absent — a read-only or `immutable=1` deployment cannot +take the `ALTER` — by falling back to its previous behaviour of showing every blog. So +removing the column is a supported way to back the feature out: + +```sql +ALTER TABLE Blogs DROP COLUMN IsDeleted; +``` + +--- + +## Reproducing the numbers + +```sql +SELECT 'Blogs', COUNT(*) FROM Blogs +UNION ALL SELECT 'Posts', COUNT(*) FROM Posts +UNION ALL SELECT 'Notes', COUNT(*) FROM Notes; + +-- note type mix +SELECT Type, COUNT(*) FROM Notes GROUP BY Type ORDER BY 2 DESC; + +-- the two date shapes in Blogs.DateAdded +SELECT CASE WHEN DateAdded LIKE '____-__-__%' THEN 'ISO' ELSE 'US' END, COUNT(*) +FROM Blogs GROUP BY 1; + +-- post IDs that are ambiguous without a blog name +SELECT COUNT(*) FROM ( + SELECT PostID FROM Posts GROUP BY PostID HAVING COUNT(DISTINCT BlogName) > 1); + +-- rows that reference a blog the registry does not have +SELECT COUNT(*) FROM Posts p +WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = p.BlogName); +``` + +Open the file read-only so an inspection can never disturb a running crawl: + +```bash +sqlite3 "file:TL.db?mode=ro" ".schema" +```