jim/URLNotesGrabberCore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(db): make Blogs.BlogId the only blog ID and drop BlogNames

Browse Source 
Blogs.BlogId was a one-time copy of BlogNames and nothing kept it
current: 12,238 blogs first seen after 2026-08-07 had a BlogNames ID
but a NULL Blogs.BlogId, so GetBlogs' join on BlogId silently skipped
them and their 23,148 notes.

- retire-blognames.sql: stub Blogs rows for the 17 unregistered note
  participants, backfill IDs (none renumbered), make ix_Blogs_BlogId
  UNIQUE, drop BlogNames, and add triggers that stop a Blogs row with
  a BlogId from being deleted, renamed or renumbered
- AddNote registers both blogs via RegisterBlog (Blogs row + MAX+1 ID)
  and every query resolves names through Blogs instead of BlogNames
- verify-db-schema.sql reports a DB that still has BlogNames (1e)
- Update TL.db.md, AGENTS.md and the DB Browser saved queries

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jim
2026-09-28 11:33:04 -05:00
co-authored by Claude Opus 5.5
parent c8c43c4918
commit 8fe2ffeb96
8 changed files with 751 additions and 232 deletions
Download Patch File Download Diff File Expand all files Collapse all files
URLNotesGrabberCORE/TL.db.md
+112 -94
View File
@@ -24,21 +24,43 @@ Everything below was read out of the live file, not inferred from code. Counts a
> Applied by `../normalize-notes.sql`, which took the file from 207 MB to 148 MB. An
> earlier change the same day (`../shrink-db.sql`) took it from 267 MB to 207 MB.
> ### ⚠ Breaking change, 2026-09-28: `BlogNames` is gone; `Blogs.BlogId` is the only ID authority
>
> The IDs in `Notes` used to live in a `BlogNames` table, with a copy in `Blogs.BlogId`.
> Nothing kept the copy current, so by 2026-09-28 12,238 blogs first seen after the
> migration had `Blogs.BlogId = NULL`. Every `Notes`-to-`Blogs` join on `BlogId` silently
> skipped them and their 23,148 notes, which kept them out of `GetBlogs`.
>
> `../retire-blognames.sql` fixed this by giving every note participant a `Blogs` row,
> backfilling the IDs (none renumbered), making `ix_Blogs_BlogId` unique, and **dropping
> `BlogNames`**. There is no compatibility view: any query naming it fails with
> `no such table: BlogNames`. Two triggers now protect the IDs.
>
> **Porting an app:** replace `BlogNames` with `Blogs` everywhere. The columns you used,
> `BlogId` and `BlogName`, exist there with the same meaning. A name lookup
> (`SELECT BlogId FROM Blogs WHERE BlogName = ?`) is a primary-key probe, and an ID
> lookup or join (`JOIN Blogs b ON b.BlogId = n.NoteBlogId`) uses the unique
> `ix_Blogs_BlogId`. Every ID in `Notes` resolves to exactly one `Blogs` row. `Blogs.BlogId`
> is **no longer** a stale copy, so any code or docs that distrust it can drop that
> caveat. Never write `BlogId` or `BlogName` on a row that has an ID, and never delete such
> a row: the triggers reject all three. See [`Blogs`](#blogs).
---
## The three content tables
| Table | Rows | What it is |
|---|--:|---|
| `Blogs` | 188,620 | The crawl registry — one row per known blog, plus crawl-state flags |
| `Blogs` | 198,560 | The crawl registry, one row per known blog plus crawl-state flags. Also the ID authority for blogs in `Notes` |
| `Posts` | 22,468 | Stored post content. Only 3,867 blogs actually have any |
| `Notes` | 1,182,333 | The engagement graph: `NoteBlogId` acted on `(RootBlogId, PostID)` |
| `Notes` | 1,234,830 | The engagement graph: `NoteBlogId` acted on `(RootBlogId, PostID)` |
…supported by two lookup tables that exist only to keep `Notes` small:
(`Blogs` and `Notes` counts as of 2026-09-28; the rest as of 2026-08-07.)
…supported by one lookup table that exists only to keep `Notes` small:
| Table | Rows | What it is |
|---|--:|---|
| `BlogNames` | 20,430 | `BlogId` ⇄ `BlogName`. The ID authority for everything in `Notes` |
| `NoteTypes` | 5 | `TypeId` ⇄ `Type`. `like`, `reblog`, `reply`, `posted`, `post_attribution` |
The engagement graph is the interesting part. 20,311 distinct blogs appear as engagers —
@@ -66,27 +88,47 @@ CREATE TABLE "Blogs" (
PRIMARY KEY("BlogName")
);
CREATE INDEX ix_Blogs_BlogId ON Blogs (BlogId);
CREATE UNIQUE INDEX ix_Blogs_BlogId ON Blogs (BlogId);
CREATE TRIGGER trg_Blogs_BlogId_NoDelete -- no DELETE of a row that has a BlogId
CREATE TRIGGER trg_Blogs_BlogId_Immutable -- no change to its BlogId or BlogName
```
`BlogName` is the primary key, so it is the only indexed way in by name. There is no index
on any flag or date — filtering or sorting on those scans all 188k rows, which is
on any flag or date. Filtering or sorting on those scans the whole table, which is
affordable here and is not on `Notes`.
**`BlogId` is new as of 2026-08-07 and is the join key to `Notes`.** It exists so that
`Notes` can reach `Blogs` in a single integer hop rather than going through `BlogNames`
and ending in a text comparison:
**`BlogId` is the ID that `Notes.RootBlogId` and `Notes.NoteBlogId` store, and `Blogs` is
the only place it lives** (since 2026-09-28; see the banner at the top). The join to
`Notes` is one integer hop on the unique index:
```sql
-- what you want
FROM Blogs B JOIN Notes N ON N.NoteBlogId = B.BlogId
-- not this
FROM Blogs B JOIN BlogNames BN ON BN.BlogName = B.BlogName
JOIN Notes N ON N.NoteBlogId = BN.BlogId
```
**`BlogId` is NULL on 168,202 of 188,620 rows** — every blog that has never appeared in a
**Every blog that appears in `Notes` has a `Blogs` row with a `BlogId`.** `AddNote`
guarantees it through `RegisterBlog`, which runs in the note's own transaction:
```sql
INSERT OR IGNORE INTO Blogs (BlogName, DateAdded, DateModified, DateCreated)
VALUES (@name, @now, @now, @now);
UPDATE Blogs SET BlogId = (SELECT IFNULL(MAX(BlogId), 0) + 1 FROM Blogs)
WHERE BlogName = @name AND BlogId IS NULL;
```
- Unlike `AddBlog`, this does **not** skip names containing `deact`. A note by a
deactivated blog still needs an ID, so such blogs now get registry rows too, with the
usual defaults (`HasBeenOutput = 0`, `IsActive` left at its default).
- Assigning a `BlogId` is bookkeeping, so it **does not move `DateModified`**.
- `MAX(BlogId) + 1` is safe only because an ID can never be freed. The two triggers see
to that: deleting a row that has a `BlogId`, or changing its `BlogId` or `BlogName`,
aborts. Remove a blog with `IsActive = 0` instead. A blog renamed upstream gets a new
row. Rows with no `BlogId` can still be deleted or renamed freely.
- `INSERT OR REPLACE` on `Blogs` gets around the delete trigger (SQLite does not fire
delete triggers for REPLACE unless `recursive_triggers` is on), and it would wipe the
`BlogId`. It was already forbidden because it resets `IsActive`. Do not use it.
**`BlogId` is NULL on 165,887 of 198,560 rows**, every blog that has never appeared in a
note. That is the large majority, and it is not an error: the registry is far bigger than
the engagement graph. An inner join on `BlogId` therefore silently drops those blogs,
which is usually what you want for engagement queries and is wrong for registry listings.
@@ -187,8 +229,7 @@ CREATE INDEX ix_Notes_NoteBlogId ON Notes (NoteBlogId);
**Integer IDs since 2026-08-07 — this is the breaking change.** `RootBlogName`,
`NoteBlogName` and `Type` are gone, replaced by `RootBlogId`, `NoteBlogId` and `TypeId`.
Resolve them through [`BlogNames`](#blognames) and [`NoteTypes`](#notetypes), or join
straight to `Blogs` on `BlogId`. The old names were text repeated on 1.18 million rows,
Resolve blog IDs through `Blogs.BlogId` and types through [`NoteTypes`](#notetypes). The old names were text repeated on 1.18 million rows,
in the table *and* in every index over it; the swap took the file from 207 MB to 148 MB.
The **primary key column order is deliberately unchanged**, so the leading-prefix access
@@ -251,43 +292,21 @@ At 1.18M rows this is the table that dictates how the whole database has to be q
- `replyText` is `'.'` on 1,167,464 rows — only `reply` notes carry real text. Those
dots are inherited from the old column default; new rows get `NULL` instead.
**Resolve IDs by filtering the lookup, not by scanning `Notes`.** The lookup tables are
tiny and uniquely indexed, so pushing a name predicate into them costs nothing and lets
the `Notes` index do the work:
**Resolve IDs by filtering `Blogs`, not by scanning `Notes`.** A name predicate on `Blogs`
is a primary-key probe, so pushing it there costs nothing and lets the `Notes` index do
the work:
```sql
-- good: BlogNames resolves the name, then the index is searched
-- good: Blogs resolves the name, then the index is searched
SELECT * FROM Notes
WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = ?);
WHERE NoteBlogId = (SELECT BlogId FROM Blogs WHERE BlogName = ?);
-- also good, same plan
SELECT n.* FROM Notes n
JOIN BlogNames b ON b.BlogId = n.NoteBlogId
JOIN Blogs b ON b.BlogId = n.NoteBlogId
WHERE b.BlogName = ?;
```
### `BlogNames`
```sql
CREATE TABLE BlogNames (
BlogId INTEGER PRIMARY KEY,
BlogName TEXT NOT NULL UNIQUE
);
```
20,430 rows — every name appearing in `Notes` as either participant, and nothing else.
This is the **ID authority**: `Notes.RootBlogId` and `Notes.NoteBlogId` both point here,
and `Blogs.BlogId` is a copy of the value for the blogs that have one.
**12 of these names have no `Blogs` row.** The registry has never been a superset of the
engagement graph and still is not, so resolving an ID through `Blogs` rather than
`BlogNames` will occasionally find nothing. Use `BlogNames` when you need the name itself
and `Blogs` when you need registry columns.
IDs are assigned by SQLite and are **stable**: they are stored in over a million `Notes`
rows. Never renumber them. A blog that is renamed upstream should get a new row, not an
edit to an existing one, unless every `Notes` reference is migrated with it.
### `NoteTypes`
```sql
@@ -317,6 +336,9 @@ code to this table's contents, so prefer the join in anything long-lived.
## Porting to the integer schema
> Written for the 2026-08-07 change, and updated for 2026-09-28: wherever this section
> once said `BlogNames`, it now says `Blogs`. `BlogNames` no longer exists.
Everything here was checked against the live 148 MB file. There were 14 affected call
sites in `DataAccess.cs` and 16 in `RolodexRepository.cs`. TumblThree needs no changes —
its single statement touches `Blogs.IsActive` and `BlogName` only.
@@ -333,8 +355,8 @@ the result.
| Was | Is now | Resolve via |
|---|---|---|
| `Notes.RootBlogName` | `Notes.RootBlogId` | `BlogNames.BlogId` → `.BlogName` |
| `Notes.NoteBlogName` | `Notes.NoteBlogId` | `BlogNames.BlogId` → `.BlogName` |
| `Notes.RootBlogName` | `Notes.RootBlogId` | `Blogs.BlogId` → `.BlogName` |
| `Notes.NoteBlogName` | `Notes.NoteBlogId` | `Blogs.BlogId` → `.BlogName` |
| `Notes.Type` | `Notes.TypeId` | `NoteTypes.TypeId` → `.Type` |
| `ix_NoteBlogName01` | `ix_Notes_NoteBlogId` | — |
@@ -347,14 +369,14 @@ the result.
-- was
WHERE NoteBlogName = @Name
-- now, either form; both search ix_Notes_NoteBlogId after a unique-index lookup
WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @Name)
-- now, either form; both search ix_Notes_NoteBlogId after a primary-key lookup
WHERE NoteBlogId = (SELECT BlogId FROM Blogs WHERE BlogName = @Name)
-- or
JOIN BlogNames b ON b.BlogId = n.NoteBlogId WHERE b.BlogName = @Name
JOIN Blogs b ON b.BlogId = n.NoteBlogId WHERE b.BlogName = @Name
```
Measured 73 ms against 63 ms for the old text form on the busiest blog — the extra hop is
a unique-index probe on a 20k-row table and does not show.
Measured at 73 ms against 63 ms for the old text form on the busiest blog, when the lookup
was still `BlogNames`. The extra hop is one index probe and does not show.
### Joining `Notes` to `Blogs`
@@ -364,12 +386,17 @@ This is the join to get right; it is the most common shape in both applications.
-- was
FROM Blogs B INNER JOIN Notes N ON N.NoteBlogName = B.BlogName
-- now: one integer hop, using the new Blogs.BlogId
-- now: one integer hop, using Blogs.BlogId
FROM Blogs B INNER JOIN Notes N ON N.NoteBlogId = B.BlogId
```
Do **not** route this through `BlogNames` — that adds a hop and ends in the text
comparison the change was meant to remove.
Joining `Notes` to `Posts` also goes through `Blogs`, since `Posts` has only a name:
```sql
FROM Posts P
JOIN Blogs RB ON RB.BlogName = P.BlogName
JOIN Notes N ON N.RootBlogId = RB.BlogId AND N.PostID = P.PostID
```
### Selecting a name back out
@@ -378,12 +405,12 @@ comparison the change was meant to remove.
SELECT NoteBlogName AS blogName, COUNT(*) FROM Notes ... GROUP BY NoteBlogName
-- now
SELECT bn.BlogName AS blogName, COUNT(*)
FROM Notes n JOIN BlogNames bn ON bn.BlogId = n.NoteBlogId
... GROUP BY bn.BlogName
SELECT b.BlogName AS blogName, COUNT(*)
FROM Notes n JOIN Blogs b ON b.BlogId = n.NoteBlogId
... GROUP BY b.BlogName
```
Group by `n.NoteBlogId` instead of `bn.BlogName` when you only need the name for display —
Group by `n.NoteBlogId` instead of `b.BlogName` when you only need the name for display —
grouping on the integer is cheaper and the name comes along for free.
### Filtering by type
@@ -407,27 +434,25 @@ because `TypeId` is `NOT NULL`.
### Inserting a note
The crawler must ensure both names have IDs first. `INSERT OR IGNORE` on `BlogNames` is
the whole of it — no read-back, no round trip, safe to run every time:
The crawler must ensure both blogs have IDs first: run the `RegisterBlog` pair shown under
[`Blogs`](#blogs) for each name. No read-back, no round trip, and safe to run every time.
Then:
```sql
INSERT OR IGNORE INTO BlogNames (BlogName) VALUES (@rootBlogName);
INSERT OR IGNORE INTO BlogNames (BlogName) VALUES (@noteBlogName);
INSERT OR IGNORE INTO Notes
(RootBlogId, PostID, NoteBlogId, TimeStamp, TypeId,
DatetimeCrawled, DateModified, DateCreated)
SELECT (SELECT BlogId FROM BlogNames WHERE BlogName = @rootBlogName),
SELECT (SELECT BlogId FROM Blogs WHERE BlogName = @rootBlogName),
@PostID,
(SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName),
(SELECT BlogId FROM Blogs WHERE BlogName = @noteBlogName),
@TimeStamp,
(SELECT TypeId FROM NoteTypes WHERE Type = @Type),
@DatetimeCrawled, @DateModified, @DateCreated;
```
Verified: a genuinely new note inserts, and re-running the identical statement inserts 0.
Run all three statements in one transaction so a crash cannot leave a name registered
with no note.
Run the registrations and the insert in one transaction so a crash cannot leave a blog
registered with no note.
**The duplicate-key error message has changed.** `DataAccess.cs` compares against the
literal string
@@ -452,7 +477,7 @@ on `TimeStamp` either before or after:
```sql
-- now
UPDATE Notes SET replyText = @replyText, DateModified = @dateModified
WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName)
WHERE NoteBlogId = (SELECT BlogId FROM Blogs WHERE BlogName = @noteBlogName)
AND ABS(TimeStamp - @TimeStamp) <= 5
AND TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')
AND (replyText IS NULL OR replyText = '' OR replyText = '.')
@@ -462,18 +487,15 @@ UPDATE Notes SET replyText = @replyText, DateModified = @dateModified
Rolodex's soft-delete updates need no change beyond the `WHERE` clause — they set
`IsActive`, which is untouched.
### Three traps
### Two traps
**`Blogs.BlogId` is NULL on 168,202 of 188,620 rows.** Any inner join on it silently drops
**`Blogs.BlogId` is NULL on 165,887 of 198,560 rows.** Any inner join on it silently drops
every blog that has never appeared in a note. Correct for engagement queries; wrong for
registry listings, which need a `LEFT JOIN` or no join at all.
**12 names in `BlogNames` have no `Blogs` row.** Resolving an ID to a name through `Blogs`
will occasionally find nothing. Use `BlogNames` for names and `Blogs` for registry columns.
**IDs are stable and must stay so.** `BlogNames.BlogId` and `NoteTypes.TypeId` are stored
in over a million `Notes` rows. Never renumber. A blog renamed upstream gets a new row,
not an edited one, unless every `Notes` reference migrates with it.
**IDs are stable and must stay so.** `Blogs.BlogId` and `NoteTypes.TypeId` are stored in
over a million `Notes` rows. Never renumber. A blog renamed upstream gets a new row, not
an edited one. The `Blogs` triggers reject both.
---
@@ -481,16 +503,12 @@ not an edited one, unless every `Notes` reference migrates with it.
There are no foreign keys, and the tables do not perfectly agree:
- 4 `Posts` rows name a blog with no `Blogs` row.
- 12 of the 20,430 names in `BlogNames` have no `Blogs` row.
So a name appearing in `Notes` or `Posts` is not a guarantee that the registry knows about
it. Joins from those tables back to `Blogs` should tolerate a miss.
The integer schema does not fix this and was not meant to. `BlogNames` is deliberately
built from `Notes` rather than from `Blogs`, precisely so that the 12 unregistered
engagers keep their IDs and their rows. Had it been built from the registry, those notes
would have been dropped by the migration's inner joins.
- 4 `Posts` rows name a blog with no `Blogs` row, so joins from `Posts` back to `Blogs`
should tolerate a miss.
- `Notes` is covered: every `RootBlogId` and `NoteBlogId` resolves to a `Blogs` row.
`retire-blognames.sql` checked this before committing, and `RegisterBlog` keeps it true.
Before 2026-09-28, 12 to 17 note participants had no registry row. They now have stub
rows.
---
@@ -542,11 +560,10 @@ Crawler bookkeeping. Rolodex ignores all of these.
`DataAccess.cs` joins on it to decide what to collect:
```sql
-- shape only; the ported GetBlogs joins Blogs directly on BlogId and needs no BlogNames hop
SELECT bn.BlogName, count(*)
-- shape only
SELECT b.BlogName, count(*)
FROM Notes n
JOIN Blogs b ON b.BlogId = n.NoteBlogId
JOIN BlogNames bn ON bn.BlogId = n.NoteBlogId
JOIN Blogs b ON b.BlogId = n.NoteBlogId
WHERE b.IsActive = @isActive AND ...
```
@@ -644,7 +661,7 @@ handled:
SELECT 'Blogs', COUNT(*) FROM Blogs
UNION ALL SELECT 'Posts', COUNT(*) FROM Posts
UNION ALL SELECT 'Notes', COUNT(*) FROM Notes
UNION ALL SELECT 'BlogNames', COUNT(*) FROM BlogNames;
UNION ALL SELECT 'Blogs with a BlogId', COUNT(*) FROM Blogs WHERE BlogId IS NOT NULL;
-- note type mix (joins NoteTypes; Notes.Type no longer exists)
SELECT t.Type, COUNT(*)
@@ -668,8 +685,9 @@ SELECT COUNT(*) FROM (
SELECT COUNT(*) FROM Posts p
WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = p.BlogName);
SELECT COUNT(*) FROM BlogNames bn
WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = bn.BlogName);
-- note participants with no Blogs.BlogId (expect 0; anything else is the pre-2026-09-28 drift)
SELECT COUNT(*) FROM (SELECT DISTINCT NoteBlogId AS Id FROM Notes) n
WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogId = n.Id);
-- space by object, to see where the file actually goes
SELECT name, SUM(pgsize)/1024/1024 AS mb