From ab36085ba863112bb186c998da488294c98c2bf7 Mon Sep 17 00:00:00 2001 From: jim Date: Fri, 7 Aug 2026 20:54:24 -0500 Subject: [PATCH] feat(db)!: replace blog names and note types in Notes with integer IDs BREAKING CHANGE: Notes.RootBlogName, Notes.NoteBlogName and Notes.Type no longer exist. They are RootBlogId, NoteBlogId and TypeId, resolved through two new lookup tables. Every Notes query in this repo and in Rolodex fails against a migrated database until rewritten. Neither application is ported yet. TumblThree is unaffected -- it touches only Blogs. Takes TL.db from 207 MB to 148 MB (-29%); cumulative with this morning's WITHOUT ROWID change, 267 MB to 148 MB (-45%). The names were text repeated across 1.18M rows, in the table and again in every index over it. BlogNames(BlogId, BlogName) 20,430 rows, the ID authority NoteTypes(TypeId, Type) 5 rows, a table rather than a CHECK so a new type is an INSERT not a migration Blogs.BlogId new, additive, NULL on the 168,202 blogs with no notes Blogs.BlogId exists so Notes reaches Blogs in one integer hop instead of going through BlogNames and ending in the text comparison this change was meant to remove. It costs 2 MB and is purely additive, which is what leaves TumblThree untouched. BlogNames is built from Notes rather than from Blogs, deliberately: 12 engagers have no registry row, and sourcing it from Blogs would have dropped their notes through the migration's inner joins. Proven lossless before and after applying to the live file: the old text shape was reconstructed from the new schema and diffed against the pre-migration database in both directions. Zero rows differed either way across all 1,182,333 rows and all ten columns. integrity_check ok, journal_mode still wal. A view-plus-INSTEAD-OF-triggers compatibility shim was built and measured first. It worked completely -- reads, INSERT OR IGNORE dedup, both apps' update paths, cross-table transactions -- but cost 194 ms to 321 ms on Rolodex's unfiltered Notes page, and a clean break was chosen over carrying it. TL.db.md gains a "Porting to the integer schema" section: column mapping and the old-to-new form of every query shape the two applications use, including the INSERT-OR-IGNORE-into-BlogNames-first pattern for notes naming a blog that has no ID yet. Roughly 14 call sites in DataAccess.cs, 16 in RolodexRepository.cs. Every documented snippet was executed against the live file. Also flags that the duplicate-key error string DataAccess.cs matches on at two sites now names the new columns and will no longer match. Unrelated corrections found while refreshing the counts, all of which had drifted on their own: Posts.PostType is no longer NULL on every row but populated on 20,679 of 22,468, which invalidates the stated reason both this document and Rolodex derive post type from content instead of reading it; the Posts.IsActive and Notes.IsActive columns described as "not in this database yet" both exist; and the registry is 188,620 blogs, not 144,367. Co-Authored-By: Claude Opus 5 --- URLNotesGrabberCORE/TL.db.md | 441 +++++++++++++++++++++++++++++------ normalize-notes.sql | 143 ++++++++++++ 2 files changed, 512 insertions(+), 72 deletions(-) create mode 100644 normalize-notes.sql diff --git a/URLNotesGrabberCORE/TL.db.md b/URLNotesGrabberCORE/TL.db.md index d89c843..3161eb8 100644 --- a/URLNotesGrabberCORE/TL.db.md +++ b/URLNotesGrabberCORE/TL.db.md @@ -4,12 +4,25 @@ The SQLite database behind **URLNotesGrabberCORE** and its sibling crawlers, and [Rolodex](https://git.basso.land/jim/Rolodex) reads. Everything below was read out of the live file, not inferred from code. Counts are as of -**2026-07-29**; re-run the queries at the bottom to refresh them. +**2026-08-07**; re-run the queries at the bottom to refresh them. - Journal mode: **WAL** — `TL.db-wal` and `TL.db-shm` live beside the file and are part of the database. Copying `TL.db` alone gives you whatever was last checkpointed, not the current state. -- Page size: 4096. +- Page size: 4096. File size: 148 MB. + +> ### ⚠ Breaking change, 2026-08-07: `Notes` holds integer IDs, not names +> +> `Notes.RootBlogName`, `Notes.NoteBlogName` and `Notes.Type` **no longer exist**. They +> are now `RootBlogId`, `NoteBlogId` and `TypeId`, resolved through the new `BlogNames` +> and `NoteTypes` tables. Any query naming the old columns fails outright. +> +> There is no compatibility view. See [porting to the integer +> schema](#porting-to-the-integer-schema) for the old-to-new translation of every query +> shape the applications use. +> +> Applied by `../normalize-notes.sql`, which took the file from 207 MB to 148 MB. An +> earlier change the same day (`../shrink-db.sql`) took it from 267 MB to 207 MB. --- @@ -17,13 +30,20 @@ Everything below was read out of the live file, not inferred from code. Counts a | Table | Rows | What it is | |---|--:|---| -| `Blogs` | 144,367 | The crawl registry — one row per known blog, plus crawl-state flags | -| `Posts` | 14,589 | Stored post content. Only 3,602 blogs actually have any | -| `Notes` | 1,189,604 | The engagement graph: `NoteBlogName` acted on `(RootBlogName, PostID)` | +| `Blogs` | 188,620 | The crawl registry — one row per known blog, plus crawl-state flags | +| `Posts` | 22,468 | Stored post content. Only 3,867 blogs actually have any | +| `Notes` | 1,182,333 | The engagement graph: `NoteBlogId` acted on `(RootBlogId, PostID)` | -The engagement graph is the interesting part. 31,888 distinct blogs appear as engagers — -far more than the 3,602 that have stored posts — which is what makes this a social graph -rather than a post archive. +…supported by two lookup tables that exist only to keep `Notes` small: + +| Table | Rows | What it is | +|---|--:|---| +| `BlogNames` | 20,430 | `BlogId` ⇄ `BlogName`. The ID authority for everything in `Notes` | +| `NoteTypes` | 5 | `TypeId` ⇄ `Type`. `like`, `reblog`, `reply`, `posted`, `post_attribution` | + +The engagement graph is the interesting part. 20,311 distinct blogs appear as engagers — +far more than the 3,867 that have stored posts — which is what makes this a social graph +rather than a post archive. Only 2,771 blogs appear as the *root* of a note. ### `Blogs` @@ -42,23 +62,45 @@ CREATE TABLE "Blogs" ( LikesLastRefreshed INTEGER DEFAULT 0, LikesLastNewCount INTEGER DEFAULT 0, TTFolderPath TEXT, + BlogId INTEGER, PRIMARY KEY("BlogName") ); + +CREATE INDEX ix_Blogs_BlogId ON Blogs (BlogId); ``` -`BlogName` is the primary key, so it is the only indexed way in. There is no index on any -flag or date — filtering or sorting on those scans all 144k rows, which is affordable -here and is not on `Notes`. +`BlogName` is the primary key, so it is the only indexed way in by name. There is no index +on any flag or date — filtering or sorting on those scans all 188k rows, which is +affordable here and is not on `Notes`. -Flag distribution: `IsActive = 1` on 144,366 of 144,367 rows, `HasBeenOutput = 1` on -5,369, `ByLikes = 1` on 2. `IsActive` carries a second meaning as of Rolodex — see +**`BlogId` is new as of 2026-08-07 and is the join key to `Notes`.** It exists so that +`Notes` can reach `Blogs` in a single integer hop rather than going through `BlogNames` +and ending in a text comparison: + +```sql +-- what you want +FROM Blogs B JOIN Notes N ON N.NoteBlogId = B.BlogId + +-- not this +FROM Blogs B JOIN BlogNames BN ON BN.BlogName = B.BlogName + JOIN Notes N ON N.NoteBlogId = BN.BlogId +``` + +**`BlogId` is NULL on 168,202 of 188,620 rows** — every blog that has never appeared in a +note. That is the large majority, and it is not an error: the registry is far bigger than +the engagement graph. An inner join on `BlogId` therefore silently drops those blogs, +which is usually what you want for engagement queries and is wrong for registry listings. + +Flag distribution: `IsActive = 1` on 188,601 of 188,620 rows, `HasBeenOutput = 1` on +5,059, `ByLikes = 1` on 2. `IsActive` carries a second meaning as of Rolodex — see [`Blogs.IsActive`](#blogsisactive--now-written-by-two-applications) below. The columns after `DateCreated` were added later by `ALTER TABLE`, which is why they carry -no quoting in the stored DDL. That is the normal way this schema grows. +no quoting in the stored DDL. That is the normal way this schema grows, and `BlogId` is +the newest example. -**`DateAdded` is not written consistently.** 126,423 rows hold ISO `yyyy-MM-dd HH:mm:ss`; -17,944 hold US-format `M/d/yy` from a bulk import. As text those two sort into different +**`DateAdded` is not written consistently.** 170,677 rows hold ISO `yyyy-MM-dd HH:mm:ss`; +17,943 hold US-format `M/d/yy` from a bulk import. As text those two sort into different parts of the table, so anything ordering or range-filtering on this column has to normalise first — see `DateSql` in Rolodex. @@ -100,54 +142,73 @@ CREATE TABLE "Posts" ( ); ``` -**The key is `(BlogName, PostID)`, not `PostID`.** This matters more than it looks: 325 +**The key is `(BlogName, PostID)`, not `PostID`.** This matters more than it looks: 345 post IDs exist under more than one blog, so an ID on its own is both ambiguous *and* unindexed. Any lookup should carry the blog name, and a batch lookup should group by blog so it stays on the leading column of the key. Notable: -- **`PostType` is `NULL` on all 14,589 rows.** The column exists but nothing has ever - populated it. Treat it as unpopulated rather than as a type discriminator. -- `HasImage = 1` on 14,268 rows — nearly all of them. It records that the post *had* a - picture, not that a usable URL was kept, so it is not a reliable predictor that anything - will render. +- **`PostType` is now mostly populated: 20,679 of 22,468 rows, leaving 1,789 `NULL`.** + This reverses what earlier revisions of this document said — the column really was empty + on every row, and something has since started writing it. Anything that treated it as + permanently unset, or derived the type from post content instead, should be re-examined + against the live data. Rolodex still derives it. +- `HasImage = 1` on 14,026 rows. It records that the post *had* a picture, not that a + usable URL was kept, so it is not a reliable predictor that anything will render. - `PhotoURL` is largely unused; in practice the image markup lives inside `Body`. -- `NotFound = 1` on 4,663 rows — posts that have since been deleted upstream. +- `NotFound = 1` on 4,712 rows — posts that have since been deleted upstream. - The content columns (`Body`, `Quote`, `Question`, `Answer`, …) are the heavy ones. List views should not select them. ### `Notes` ```sql -CREATE TABLE "Notes" ( - "RootBlogName" TEXT, - "PostID" INTEGER, - "NoteBlogName" TEXT, - "TimeStamp" INTEGER, - "Type" TEXT, - "replyText" TEXT DEFAULT '.', - "DatetimeCrawled" TEXT DEFAULT '2/12/26 12am', - "DateModified" TEXT, - "DateCreated" TEXT, - IsActive INTEGER NOT NULL DEFAULT 1, - PRIMARY KEY("RootBlogName","PostID","TimeStamp","Type","NoteBlogName") +CREATE TABLE Notes ( + RootBlogId INTEGER NOT NULL, + PostID INTEGER NOT NULL, + NoteBlogId INTEGER NOT NULL, + TimeStamp INTEGER NOT NULL, + TypeId INTEGER NOT NULL, + replyText TEXT, + DatetimeCrawled TEXT, + DateModified TEXT, + DateCreated TEXT, + IsActive INTEGER NOT NULL DEFAULT 1, + PRIMARY KEY (RootBlogId, PostID, TimeStamp, TypeId, NoteBlogId) ) WITHOUT ROWID; -CREATE INDEX "ix_NoteBlogName01" ON "Notes" ("NoteBlogName"); +CREATE INDEX ix_Notes_NoteBlogId ON Notes (NoteBlogId); ``` -**`WITHOUT ROWID`, since 2026-08-07.** The rows live in the primary key's b-tree -rather than in a rowid table with a separate key index beside it. Nothing about the -SQL surface changes — same columns, same types, same constraint — but two -consequences are worth knowing before adding an index here: +**Integer IDs since 2026-08-07 — this is the breaking change.** `RootBlogName`, +`NoteBlogName` and `Type` are gone, replaced by `RootBlogId`, `NoteBlogId` and `TypeId`. +Resolve them through [`BlogNames`](#blognames) and [`NoteTypes`](#notetypes), or join +straight to `Blogs` on `BlogId`. The old names were text repeated on 1.18 million rows, +in the table *and* in every index over it; the swap took the file from 207 MB to 148 MB. + +The **primary key column order is deliberately unchanged**, so the leading-prefix access +patterns callers already depend on still hold: `(RootBlogId)` and `(RootBlogId, PostID)` +remain cheap prefixes, exactly as `(RootBlogName)` and `(RootBlogName, PostID)` were. + +Two nulls-and-defaults differences from the old DDL, both intentional: + +- `replyText` and `DatetimeCrawled` **no longer carry column defaults**. The old table + defaulted them to `'.'` and `'2/12/26 12am'`, which is how 1.1M rows acquired + placeholder values nobody wrote. New rows now get `NULL` unless a writer supplies + something. The crawler names both columns explicitly, so its behaviour is unchanged. +- The five key columns are now `NOT NULL`. They always were in practice. + +**`WITHOUT ROWID`, since earlier the same day.** The rows live in the primary key's +b-tree rather than in a rowid table with a separate key index beside it. Two consequences +matter before adding an index here: - There is no `rowid` on this table. `SELECT rowid FROM Notes` is an error, and no code in any of the three apps relied on it. - A secondary index carries the whole five-column primary key as its row reference - instead of a compact rowid, so indexes on this table are **expensive**. - `ix_NoteBlogName01` costs 58 MB, up from 25 MB before the conversion. It earns - that: Rolodex filters on `NoteBlogName` and the crawler joins on it. + instead of a compact rowid, so indexes here are **expensive** — though far less so + than before, now that the key is five integers rather than three integers and two + strings. `ix_Notes_NoteBlogId` costs 27 MB; its text predecessor cost 58 MB. **`Notes_idx_06e01ae3` on `TimeStamp DESC` was dropped at the same time.** It cost 14 MB as a rowid index and would have cost 58 MB after the conversion. It was worth @@ -174,37 +235,247 @@ as "unknown" rather than assuming `NULL` is now the only such marker. One row per engagement event. `TimeStamp` is **unix seconds** — unlike every date column elsewhere in the schema, which are text. -| `Type` | Rows | Share | -|---|--:|--:| -| `like` | 947,955 | 79.7% | -| `reblog` | 224,323 | 18.9% | -| `reply` | 15,201 | 1.3% | -| `posted` | 2,106 | 0.2% | -| `post_attribution` | 19 | — | - -At 1.19M rows this is the table that dictates how the whole database has to be queried: +At 1.18M rows this is the table that dictates how the whole database has to be queried: - **Nothing should run an unbounded `SELECT` or a bare `COUNT(*)` here.** A count scans the lot on every call. -- The only fast access paths are the primary key's leading columns (`RootBlogName`, then - `PostID`) and `ix_NoteBlogName01` on `NoteBlogName`. "Notes received by a blog" and +- The only fast access paths are the primary key's leading columns (`RootBlogId`, then + `PostID`) and `ix_Notes_NoteBlogId` on `NoteBlogId`. "Notes received by a blog" and "notes given by a blog" are both cheap; almost nothing else is. - **Every** ordering here is a full sort of whatever the filters leave, `TimeStamp` - included. That was already true in practice of the default `TimeStamp` order, whose - tiebreakers forced a sort even while `Notes_idx_06e01ae3` existed; since that index - was dropped on 2026-08-07 it is true unconditionally. Filter first, then sort. -- `replyText` is `'.'` on 1,167,464 rows — only `reply` notes carry real text. + included. Filter first, then sort. +- `replyText` is `'.'` on 1,167,464 rows — only `reply` notes carry real text. Those + dots are inherited from the old column default; new rows get `NULL` instead. + +**Resolve IDs by filtering the lookup, not by scanning `Notes`.** The lookup tables are +tiny and uniquely indexed, so pushing a name predicate into them costs nothing and lets +the `Notes` index do the work: + +```sql +-- good: BlogNames resolves the name, then the index is searched +SELECT * FROM Notes + WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = ?); + +-- also good, same plan +SELECT n.* FROM Notes n + JOIN BlogNames b ON b.BlogId = n.NoteBlogId + WHERE b.BlogName = ?; +``` + +### `BlogNames` + +```sql +CREATE TABLE BlogNames ( + BlogId INTEGER PRIMARY KEY, + BlogName TEXT NOT NULL UNIQUE +); +``` + +20,430 rows — every name appearing in `Notes` as either participant, and nothing else. +This is the **ID authority**: `Notes.RootBlogId` and `Notes.NoteBlogId` both point here, +and `Blogs.BlogId` is a copy of the value for the blogs that have one. + +**12 of these names have no `Blogs` row.** The registry has never been a superset of the +engagement graph and still is not, so resolving an ID through `Blogs` rather than +`BlogNames` will occasionally find nothing. Use `BlogNames` when you need the name itself +and `Blogs` when you need registry columns. + +IDs are assigned by SQLite and are **stable**: they are stored in over a million `Notes` +rows. Never renumber them. A blog that is renamed upstream should get a new row, not an +edit to an existing one, unless every `Notes` reference is migrated with it. + +### `NoteTypes` + +```sql +CREATE TABLE NoteTypes ( + TypeId INTEGER PRIMARY KEY, + Type TEXT NOT NULL UNIQUE +); +``` + +| `TypeId` | `Type` | Rows | Share | +|--:|---|--:|--:| +| 1 | `like` | 945,167 | 79.9% | +| 2 | `reblog` | 219,203 | 18.5% | +| 3 | `reply` | 15,345 | 1.3% | +| 4 | `posted` | 2,617 | 0.2% | +| 5 | `post_attribution` | 1 | — | + +The set is fixed in practice, but it is a table rather than a `CHECK` constraint so that +adding a type is an `INSERT` and not a schema migration. **The IDs above are stored in +`Notes` and must not be reassigned.** + +Five rows means the lookup is effectively free; write `t.Type = 'reblog'` and let SQLite +resolve it, or hardcode the ID if you prefer — both are fine, but hardcoding ties your +code to this table's contents, so prefer the join in anything long-lived. + +--- + +## Porting to the integer schema + +Everything here was checked against the live 148 MB file. There are roughly 14 affected +call sites in `DataAccess.cs` and 16 in `RolodexRepository.cs`. TumblThree needs no +changes — its single statement touches `Blogs.IsActive` and `BlogName` only. + +### Column mapping + +| Was | Is now | Resolve via | +|---|---|---| +| `Notes.RootBlogName` | `Notes.RootBlogId` | `BlogNames.BlogId` → `.BlogName` | +| `Notes.NoteBlogName` | `Notes.NoteBlogId` | `BlogNames.BlogId` → `.BlogName` | +| `Notes.Type` | `Notes.TypeId` | `NoteTypes.TypeId` → `.Type` | +| `ix_NoteBlogName01` | `ix_Notes_NoteBlogId` | — | + +`PostID`, `TimeStamp`, `replyText`, `DatetimeCrawled`, `DateModified`, `DateCreated` and +`IsActive` are unchanged. + +### Filtering by a blog name + +```sql +-- was +WHERE NoteBlogName = @Name + +-- now, either form; both search ix_Notes_NoteBlogId after a unique-index lookup +WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @Name) +-- or +JOIN BlogNames b ON b.BlogId = n.NoteBlogId WHERE b.BlogName = @Name +``` + +Measured 73 ms against 63 ms for the old text form on the busiest blog — the extra hop is +a unique-index probe on a 20k-row table and does not show. + +### Joining `Notes` to `Blogs` + +This is the join to get right; it is the most common shape in both applications. + +```sql +-- was +FROM Blogs B INNER JOIN Notes N ON N.NoteBlogName = B.BlogName + +-- now: one integer hop, using the new Blogs.BlogId +FROM Blogs B INNER JOIN Notes N ON N.NoteBlogId = B.BlogId +``` + +Do **not** route this through `BlogNames` — that adds a hop and ends in the text +comparison the change was meant to remove. + +### Selecting a name back out + +```sql +-- was +SELECT NoteBlogName AS blogName, COUNT(*) FROM Notes ... GROUP BY NoteBlogName + +-- now +SELECT bn.BlogName AS blogName, COUNT(*) + FROM Notes n JOIN BlogNames bn ON bn.BlogId = n.NoteBlogId + ... GROUP BY bn.BlogName +``` + +Group by `n.NoteBlogId` instead of `bn.BlogName` when you only need the name for display — +grouping on the integer is cheaper and the name comes along for free. + +### Filtering by type + +```sql +-- was +WHERE type IN ('reblog', 'reply', 'posted') + +-- now +WHERE TypeId IN (SELECT TypeId FROM NoteTypes WHERE Type IN ('reblog','reply','posted')) +-- or, equivalently +JOIN NoteTypes t ON t.TypeId = n.TypeId WHERE t.Type IN ('reblog','reply','posted') +``` + +`WHERE TypeId IN (2,3,4)` also works and is marginally faster, but hardcodes this table's +contents into application code. Prefer the lookup outside of hot paths. + +Note the negated form needs care: `type NOT IN ('reblog','reply','posted')` becomes +`TypeId NOT IN (SELECT TypeId FROM NoteTypes WHERE Type IN (...))`, which is correct only +because `TypeId` is `NOT NULL`. + +### Inserting a note + +The crawler must ensure both names have IDs first. `INSERT OR IGNORE` on `BlogNames` is +the whole of it — no read-back, no round trip, safe to run every time: + +```sql +INSERT OR IGNORE INTO BlogNames (BlogName) VALUES (@rootBlogName); +INSERT OR IGNORE INTO BlogNames (BlogName) VALUES (@noteBlogName); + +INSERT OR IGNORE INTO Notes + (RootBlogId, PostID, NoteBlogId, TimeStamp, TypeId, + DatetimeCrawled, DateModified, DateCreated) +SELECT (SELECT BlogId FROM BlogNames WHERE BlogName = @rootBlogName), + @PostID, + (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName), + @TimeStamp, + (SELECT TypeId FROM NoteTypes WHERE Type = @Type), + @DatetimeCrawled, @DateModified, @DateCreated; +``` + +Verified: a genuinely new note inserts, and re-running the identical statement inserts 0. +Run all three statements in one transaction so a crash cannot leave a name registered +with no note. + +**The duplicate-key error message has changed.** `DataAccess.cs` compares against the +literal string + +``` +UNIQUE constraint failed: Notes.RootBlogName, Notes.PostID, Notes.TimeStamp, Notes.Type, Notes.NoteBlogName +``` + +at two call sites to decide whether to swallow an exception. SQLite now emits the *new* +column names, so those comparisons no longer match and real errors will surface where +they used to be silently ignored — or vice versa. Both sites need updating. + +### Updating notes + +Predicates translate the same way. The reply-matching update, which cannot use an index +on `TimeStamp` either before or after: + +```sql +-- now +UPDATE Notes SET replyText = @replyText, DateModified = @dateModified + WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName) + AND ABS(TimeStamp - @TimeStamp) <= 5 + AND TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply') + AND (replyText IS NULL OR replyText = '' OR replyText = '.') + AND (replyText IS NULL OR replyText <> @replyText); +``` + +Rolodex's soft-delete updates need no change beyond the `WHERE` clause — they set +`IsActive`, which is untouched. + +### Three traps + +**`Blogs.BlogId` is NULL on 168,202 of 188,620 rows.** Any inner join on it silently drops +every blog that has never appeared in a note. Correct for engagement queries; wrong for +registry listings, which need a `LEFT JOIN` or no join at all. + +**12 names in `BlogNames` have no `Blogs` row.** Resolving an ID to a name through `Blogs` +will occasionally find nothing. Use `BlogNames` for names and `Blogs` for registry columns. + +**IDs are stable and must stay so.** `BlogNames.BlogId` and `NoteTypes.TypeId` are stored +in over a million `Notes` rows. Never renumber. A blog renamed upstream gets a new row, +not an edited one, unless every `Notes` reference migrates with it. + +--- ### Referential integrity There are no foreign keys, and the tables do not perfectly agree: - 4 `Posts` rows name a blog with no `Blogs` row. -- 15 of the 31,888 distinct engagers have no `Blogs` row. +- 12 of the 20,430 names in `BlogNames` have no `Blogs` row. So a name appearing in `Notes` or `Posts` is not a guarantee that the registry knows about it. Joins from those tables back to `Blogs` should tolerate a miss. +The integer schema does not fix this and was not meant to. `BlogNames` is deliberately +built from `Notes` rather than from `Blogs`, precisely so that the 12 unregistered +engagers keep their IDs and their rows. Had it been built from the registry, those notes +would have been dropped by the migration's inner joins. + --- ## The `'.'` placeholder convention @@ -216,9 +487,13 @@ consumer. | Column | `'.'` rows | |---|--:| | `Notes.replyText` | 1,167,464 | -| `Posts.Title` | 13,144 | +| `Posts.Title` | 12,562 | | `Posts.Body` | 172 | +`Notes.replyText` and `Notes.DatetimeCrawled` **no longer carry column defaults** as of +the integer migration, so new note rows get `NULL` rather than a placeholder. The dots +already in `replyText` were not rewritten — cleaning is still required on read. + Any query whose output reaches a human should collapse it: ```sql @@ -251,9 +526,12 @@ Crawler bookkeeping. Rolodex ignores all of these. `DataAccess.cs` joins on it to decide what to collect: ```sql -SELECT NoteBlogName, count(*) FROM notes -INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName -WHERE blogs.IsActive = @isActive AND ... +-- as it will read after the integer-schema port; see the porting guide above +SELECT bn.BlogName, count(*) + FROM Notes n + JOIN Blogs b ON b.BlogId = n.NoteBlogId + JOIN BlogNames bn ON bn.BlogId = n.NoteBlogId + WHERE b.IsActive = @isActive AND ... ``` Nothing inside the crawler *writes* it — it is an input, set from outside. @@ -299,12 +577,16 @@ under its Posts and Notes pages. Removing a blog hides the blog, not what it col --- -## `Posts.IsActive` and `Notes.IsActive` — optional, and not in this database yet +## `Posts.IsActive` and `Notes.IsActive` — present, and written from outside -The same flag is being extended to the two content tables, with the same meaning: `0` is -removed, anything else — including `NULL` — is live. **Neither column exists in the live -`TL.db` as of 2026-07-29**; the DDL quoted above for `Posts` and `Notes` is complete. Like -`Blogs.IsActive`, they are written from outside this crawler. +The same flag extends to the two content tables, with the same meaning: `0` is removed, +anything else — including `NULL` — is live. **Both columns now exist in the live `TL.db`** +and are included in the DDL quoted above. As of 2026-08-07, `Posts.IsActive = 0` on 5,900 +rows and `Notes.IsActive = 0` on none. Like `Blogs.IsActive`, they are written from +outside this crawler. + +On `Notes` the column is `INTEGER NOT NULL DEFAULT 1`, so a `NULL` cannot occur there; +`Posts` and `Blogs` are laxer, which is why the predicate below still uses `COALESCE`. The crawler therefore treats both as optional, and as nothing it owns: @@ -344,11 +626,19 @@ handled: ```sql SELECT 'Blogs', COUNT(*) FROM Blogs -UNION ALL SELECT 'Posts', COUNT(*) FROM Posts -UNION ALL SELECT 'Notes', COUNT(*) FROM Notes; +UNION ALL SELECT 'Posts', COUNT(*) FROM Posts +UNION ALL SELECT 'Notes', COUNT(*) FROM Notes +UNION ALL SELECT 'BlogNames', COUNT(*) FROM BlogNames; --- note type mix -SELECT Type, COUNT(*) FROM Notes GROUP BY Type ORDER BY 2 DESC; +-- note type mix (joins NoteTypes; Notes.Type no longer exists) +SELECT t.Type, COUNT(*) + FROM Notes n JOIN NoteTypes t ON t.TypeId = n.TypeId + GROUP BY t.Type ORDER BY 2 DESC; + +-- how much of the registry participates in the engagement graph +SELECT COUNT(*) FILTER (WHERE BlogId IS NOT NULL) AS with_notes, + COUNT(*) FILTER (WHERE BlogId IS NULL) AS without_notes + FROM Blogs; -- the two date shapes in Blogs.DateAdded SELECT CASE WHEN DateAdded LIKE '____-__-__%' THEN 'ISO' ELSE 'US' END, COUNT(*) @@ -361,6 +651,13 @@ SELECT COUNT(*) FROM ( -- rows that reference a blog the registry does not have SELECT COUNT(*) FROM Posts p WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = p.BlogName); + +SELECT COUNT(*) FROM BlogNames bn +WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = bn.BlogName); + +-- space by object, to see where the file actually goes +SELECT name, SUM(pgsize)/1024/1024 AS mb + FROM dbstat GROUP BY name ORDER BY SUM(pgsize) DESC; ``` Open the file read-only so an inspection can never disturb a running crawl: diff --git a/normalize-notes.sql b/normalize-notes.sql new file mode 100644 index 0000000..5602564 --- /dev/null +++ b/normalize-notes.sql @@ -0,0 +1,143 @@ +-- normalize-notes.sql +-- Replaces the repeated blog-name and type TEXT in Notes with integer IDs. +-- Reduces TL.db from ~207 MB to ~148 MB (-29%). +-- +-- THIS IS A BREAKING SCHEMA CHANGE. There is no compatibility layer. Every +-- query in URLNotesGrabberCORE and Rolodex that names Notes.RootBlogName, +-- Notes.NoteBlogName or Notes.Type stops working the moment this runs, and +-- stays broken until those queries are rewritten. This was a deliberate choice +-- over a view-plus-triggers shim, which was measured to work but cost 194 ms -> +-- 321 ms on Rolodex's unfiltered Notes page. +-- +-- TumblThree is unaffected. It touches only Blogs, and the column added to +-- Blogs here is additive. +-- +-- HOW TO RUN (DB Browser for SQLite): +-- 1. Stop all three apps. Pause NextCloud sync. +-- 2. Back up TL.db. +-- 3. Execute SQL, paste this file, run. Then Write Changes. +-- 4. Tools > Compact Database (VACUUM). Nothing shrinks until this finishes. + +-------------------------------------------------------------------------- +-- The shape this produces +-------------------------------------------------------------------------- +-- BlogNames(BlogId, BlogName) the ID authority: every name appearing in +-- Notes as either participant. 20,430 rows. +-- 12 of these have no Blogs row -- the +-- registry has never been a superset of the +-- engagement graph, and still is not. +-- +-- NoteTypes(TypeId, Type) 5 rows. Fixed set, but written as a table +-- rather than a CHECK so a new type is an +-- INSERT and not a schema migration. +-- +-- Notes(...Id columns...) integer FKs in place of text. WITHOUT ROWID, +-- same 5-column key in the same column order. +-- +-- Blogs.BlogId NEW additive column. Lets Notes join Blogs in +-- one integer hop instead of going through +-- BlogNames and comparing text at the end. +-- NULL on the 168,202 blogs with no notes. + +PRAGMA foreign_keys = off; + +BEGIN; + +-------------------------------------------------------------------------- +-- STEP 1: the ID authority +-------------------------------------------------------------------------- +CREATE TABLE BlogNames ( + BlogId INTEGER PRIMARY KEY, + BlogName TEXT NOT NULL UNIQUE +); + +INSERT INTO BlogNames (BlogName) + SELECT RootBlogName FROM Notes + UNION + SELECT NoteBlogName FROM Notes; + +-------------------------------------------------------------------------- +-- STEP 2: the type lookup +-------------------------------------------------------------------------- +CREATE TABLE NoteTypes ( + TypeId INTEGER PRIMARY KEY, + Type TEXT NOT NULL UNIQUE +); + +-- IDs are assigned explicitly and must stay stable: they are stored in Notes. +INSERT INTO NoteTypes (TypeId, Type) VALUES + (1, 'like'), + (2, 'reblog'), + (3, 'reply'), + (4, 'posted'), + (5, 'post_attribution'); + +-------------------------------------------------------------------------- +-- STEP 3: rebuild Notes with integer keys +-------------------------------------------------------------------------- +-- Column order of the primary key is unchanged from the text version, so the +-- leading-column access patterns callers already rely on still hold: +-- (RootBlogId) and (RootBlogId, PostID) remain cheap prefixes. +CREATE TABLE NotesN ( + RootBlogId INTEGER NOT NULL, + PostID INTEGER NOT NULL, + NoteBlogId INTEGER NOT NULL, + TimeStamp INTEGER NOT NULL, + TypeId INTEGER NOT NULL, + replyText TEXT, + DatetimeCrawled TEXT, + DateModified TEXT, + DateCreated TEXT, + IsActive INTEGER NOT NULL DEFAULT 1, + PRIMARY KEY (RootBlogId, PostID, TimeStamp, TypeId, NoteBlogId) +) WITHOUT ROWID; + +-- Inner joins are safe here: BlogNames was just built from these very columns, +-- and NoteTypes covers all 5 values present. A row that failed to match would +-- be silently dropped, which is what the row-count check at the bottom is for. +INSERT INTO NotesN +SELECT r.BlogId, n.PostID, b.BlogId, n.TimeStamp, t.TypeId, + n.replyText, n.DatetimeCrawled, n.DateModified, n.DateCreated, n.IsActive + FROM Notes n + JOIN BlogNames r ON r.BlogName = n.RootBlogName + JOIN BlogNames b ON b.BlogName = n.NoteBlogName + JOIN NoteTypes t ON t.Type = n.Type; + +DROP TABLE Notes; +ALTER TABLE NotesN RENAME TO Notes; + +-- Replaces ix_NoteBlogName01. Renamed because it indexes a different column now. +CREATE INDEX ix_Notes_NoteBlogId ON Notes (NoteBlogId); + +-------------------------------------------------------------------------- +-- STEP 4: give Blogs the matching id +-------------------------------------------------------------------------- +-- Additive: no existing column changes, so TumblThree's +-- "UPDATE Blogs SET IsActive = 0 ... WHERE BlogName = ?" is untouched. +ALTER TABLE Blogs ADD COLUMN BlogId INTEGER; + +UPDATE Blogs + SET BlogId = (SELECT bn.BlogId FROM BlogNames bn WHERE bn.BlogName = Blogs.BlogName); + +CREATE INDEX ix_Blogs_BlogId ON Blogs (BlogId); + +COMMIT; + +-------------------------------------------------------------------------- +-- STEP 5: Write Changes, then Tools > Compact Database +-------------------------------------------------------------------------- +-- From the CLI instead: sqlite3 TL.db "VACUUM;" + +-------------------------------------------------------------------------- +-- VERIFY +-------------------------------------------------------------------------- +-- PRAGMA integrity_check; -- expect: ok +-- SELECT COUNT(*) FROM Notes; -- expect: 1182333, unchanged +-- SELECT COUNT(*) FROM BlogNames; -- expect: 20430 +-- SELECT COUNT(*) FROM Blogs WHERE BlogId IS NOT NULL; -- expect: 20418 +-- +-- Losslessness was proven before this ran, by reconstructing the old text shape +-- from the new schema and diffing it against the original both ways: +-- SELECT COUNT(*) FROM (SELECT * FROM old.Notes EXCEPT SELECT * FROM Rebuilt); +-- SELECT COUNT(*) FROM (SELECT * FROM Rebuilt EXCEPT SELECT * FROM old.Notes); +-- Both returned 0 across all 1,182,333 rows and all 10 columns.