BREAKING CHANGE: Notes.RootBlogName, Notes.NoteBlogName and Notes.Type no
longer exist. They are RootBlogId, NoteBlogId and TypeId, resolved through two
new lookup tables. Every Notes query in this repo and in Rolodex fails against
a migrated database until rewritten. Neither application is ported yet.
TumblThree is unaffected -- it touches only Blogs.
Takes TL.db from 207 MB to 148 MB (-29%); cumulative with this morning's
WITHOUT ROWID change, 267 MB to 148 MB (-45%). The names were text repeated
across 1.18M rows, in the table and again in every index over it.
BlogNames(BlogId, BlogName) 20,430 rows, the ID authority
NoteTypes(TypeId, Type) 5 rows, a table rather than a CHECK so a new
type is an INSERT not a migration
Blogs.BlogId new, additive, NULL on the 168,202 blogs with
no notes
Blogs.BlogId exists so Notes reaches Blogs in one integer hop instead of going
through BlogNames and ending in the text comparison this change was meant to
remove. It costs 2 MB and is purely additive, which is what leaves TumblThree
untouched.
BlogNames is built from Notes rather than from Blogs, deliberately: 12 engagers
have no registry row, and sourcing it from Blogs would have dropped their notes
through the migration's inner joins.
Proven lossless before and after applying to the live file: the old text shape
was reconstructed from the new schema and diffed against the pre-migration
database in both directions. Zero rows differed either way across all 1,182,333
rows and all ten columns. integrity_check ok, journal_mode still wal.
A view-plus-INSTEAD-OF-triggers compatibility shim was built and measured
first. It worked completely -- reads, INSERT OR IGNORE dedup, both apps' update
paths, cross-table transactions -- but cost 194 ms to 321 ms on Rolodex's
unfiltered Notes page, and a clean break was chosen over carrying it.
TL.db.md gains a "Porting to the integer schema" section: column mapping and
the old-to-new form of every query shape the two applications use, including
the INSERT-OR-IGNORE-into-BlogNames-first pattern for notes naming a blog that
has no ID yet. Roughly 14 call sites in DataAccess.cs, 16 in
RolodexRepository.cs. Every documented snippet was executed against the live
file. Also flags that the duplicate-key error string DataAccess.cs matches on
at two sites now names the new columns and will no longer match.
Unrelated corrections found while refreshing the counts, all of which had
drifted on their own: Posts.PostType is no longer NULL on every row but
populated on 20,679 of 22,468, which invalidates the stated reason both this
document and Rolodex derive post type from content instead of reading it; the
Posts.IsActive and Notes.IsActive columns described as "not in this database
yet" both exist; and the registry is 188,620 blogs, not 144,367.
Co-Authored-By: Claude Opus 5 <[email protected]>
The file was already tight -- freelist 0 pages, and a plain VACUUM reclaimed
nothing -- so the saving had to come from schema rather than compaction.
Profiled with dbstat and measured every step on copies of the live file.
Three changes to Notes, applied 2026-08-07:
- Rebuild as WITHOUT ROWID (-32 MB). The 5-column composite primary key was
stored twice: once in the table, once in a 62 MB autoindex existing only to
map key -> rowid. Keying the table b-tree on the primary key itself drops the
second copy. ix_NoteBlogName01 grows 25 -> 58 MB in exchange, since a
secondary index on such a table carries the whole primary key instead of a
rowid; net -32 MB.
- Drop Notes_idx_06e01ae3 on TimeStamp DESC (-14 MB). Barely earned its keep as
a rowid index and would have cost 58 MB after the conversion, cancelling the
entire exercise. The crawler's only TimeStamp filter (>= 1535778000) excludes
786 of 1,182,333 rows; Rolodex's default Notes sort carries a three-column
tiebreaker forcing a full sort regardless; the reply-matching UPDATE uses
ABS(TimeStamp - ?) <= 5, which no index on the column can serve. Cost is one
path: Rolodex's Notes page with a date-range filter, 60 ms -> 164 ms.
- Null the DatetimeCrawled placeholder (-13 MB). 1,148,077 rows stored the
literal DDL default '2/12/26 12am', a backfill marker rather than a crawl
time. UI-neutral: Rolodex reads the column through DateSql.Sortable, whose
CASE matches neither format, so those rows already rendered as an em dash.
No application code changed. The schema keeps the same tables, columns, types
and constraints; WITHOUT ROWID is a storage-layout change behind the same SQL
surface, and no consumer referenced rowid on Notes.
Verified against all three consumers on the live file: integrity_check ok, row
counts unchanged (1182333 / 22468 / 188620), journal_mode still wal, crawler
INSERT OR IGNORE still dedupes, Rolodex's NoteBlogName filter still uses
ix_NoteBlogName01, and exactly as many rows read as null through Sortable after
the change as before it. TumblThree touches only Blogs, which is untouched.
Deliberately not done: nulling Notes.DateCreated (a further -12 MB). Unlike
DatetimeCrawled its value parses as a real date, so Rolodex displays and sorts
by it; nulling would turn visible dates into em dashes.
Note that DEFAULT '2/12/26 12am' remains on the column, so any writer inserting
a note without naming it reintroduces the placeholder. Consumer-side date
normalisation must stay.
Co-Authored-By: Claude Opus 5 <[email protected]>
Both columns carry the meaning Blogs.IsActive has: 0 = removed by another
tool, anything else (including NULL) = live. Neither exists in the live
TL.db yet, and both are added from outside this crawler, so the code has
to work on databases either side of the change - naming a missing column
is a hard SQLite error.
HasIsActiveColumn asks PRAGMA table_info once per table per database path
and caches it; AndIsActive/WhereIsActive return "COALESCE(IsActive, 1) = 1"
or an empty string. Every read that selects posts or notes now carries the
filter: GetPosts (both branches, including the per-blog count subquery),
GetReplies, GetRepliesWithMissingText, GetRepliesWithFilledText,
GetAllPostTextColumns, GetAllPostsForBlog, GetPost, GetPostByIdAnyBlog,
and the engagement queries that count or join Notes - GetBlogs,
GetBlogsAll and both note-joining variants of GetBlogsForLikes.
The LEFT JOIN Notes in GetPosts is left alone on purpose: nothing is
selected from it and it can neither add nor remove a row.
LegacyPostsDbImporter is left alone too - it reads a foreign legacy
schema.
Writes were already safe and are documented rather than changed: no
INSERT column list names IsActive, no UPDATE sets it, MapPrefixToColumn
cannot map to it, and there is no INSERT OR REPLACE on Posts or Notes for
a column default to be reset by. Re-crawling a removed row refreshes its
content and leaves the flag at 0. As with Blogs, exclusion belongs at
selection, so the update paths stay keyed on rows the caller already
chose.
Verified against three synthetic databases - no IsActive columns, columns
present with a removed post and its notes, and columns present but NULL -
by running every affected reader: the queries are valid in all three, the
removed rows drop out only where the columns exist, NULL reads as live,
and AddPost/AddNote/UpsertPostFromTextFile/UpdatePostContentFields leave
an IsActive = 0 row at 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
Replaces the Blogs.IsDeleted section. Rolodex adds no column of its own;
it reuses the crawler's existing IsActive flag, so removing a blog in the
UI also stops it being collected.
Co-Authored-By: Claude Opus 5 <[email protected]>
TL.db.md documents the live schema: the three content tables and their
row counts, the '.' placeholder convention the crawler writes instead of
NULL, the two incompatible date formats in Blogs.DateAdded, and the
access paths that matter on the 1.19M-row Notes table.
It also covers Blogs.IsDeleted, which Rolodex adds by ALTER TABLE and
this crawler must not write.
The file was sitting untracked next to the database it describes.
Co-Authored-By: Claude Opus 5 <[email protected]>
jimandClaude Opus 5