URLNotesGrabberCore/URLNotesGrabberCORE/TL.db.md

# `TL.db` — schema notes

The SQLite database behind **URLNotesGrabberCORE** and its sibling crawlers, and the one
[Rolodex](https://git.basso.land/jim/Rolodex) reads.

Everything below was read out of the live file, not inferred from code. Counts are as of
**2026-08-07**; re-run the queries at the bottom to refresh them.

- Journal mode: **WAL** — `TL.db-wal` and `TL.db-shm` live beside the file and are part of
  the database. Copying `TL.db` alone gives you whatever was last checkpointed, not the
  current state.
- Page size: 4096. File size: 148 MB.

> ### ⚠ Breaking change, 2026-08-07: `Notes` holds integer IDs, not names
>
> `Notes.RootBlogName`, `Notes.NoteBlogName` and `Notes.Type` **no longer exist**. They
> are now `RootBlogId`, `NoteBlogId` and `TypeId`, resolved through the new `BlogNames`
> and `NoteTypes` tables. Any query naming the old columns fails outright.
>
> There is no compatibility view. See [porting to the integer
> schema](#porting-to-the-integer-schema) for the old-to-new translation of every query
> shape the applications use.
>
> Applied by `../normalize-notes.sql`, which took the file from 207 MB to 148 MB. An
> earlier change the same day (`../shrink-db.sql`) took it from 267 MB to 207 MB.

---

## The three content tables

| Table | Rows | What it is |
|---|--:|---|
| `Blogs` | 188,620 | The crawl registry — one row per known blog, plus crawl-state flags |
| `Posts` | 22,468 | Stored post content. Only 3,867 blogs actually have any |
| `Notes` | 1,182,333 | The engagement graph: `NoteBlogId` acted on `(RootBlogId, PostID)` |

…supported by two lookup tables that exist only to keep `Notes` small:

| Table | Rows | What it is |
|---|--:|---|
| `BlogNames` | 20,430 | `BlogId` ⇄ `BlogName`. The ID authority for everything in `Notes` |
| `NoteTypes` | 5 | `TypeId` ⇄ `Type`. `like`, `reblog`, `reply`, `posted`, `post_attribution` |

The engagement graph is the interesting part. 20,311 distinct blogs appear as engagers —
far more than the 3,867 that have stored posts — which is what makes this a social graph
rather than a post archive. Only 2,771 blogs appear as the *root* of a note.

### `Blogs`

```sql
CREATE TABLE "Blogs" (
    "BlogName"              TEXT,
    "HasBeenOutput"         INTEGER DEFAULT 0,
    "IsActive"              INTEGER DEFAULT 1,
    "DateAdded"             TEXT NOT NULL DEFAULT '12/24/25',
    "ByLikes"               INTEGER NOT NULL DEFAULT 0,
    "LikesPulled"           INTEGER NOT NULL DEFAULT 0,
    "LikesCursor"           INTEGER DEFAULT 0,
    "DateModified"          TEXT,
    "DateCreated"           TEXT,
    LikesNewestTimestamp    INTEGER DEFAULT 0,
    LikesLastRefreshed      INTEGER DEFAULT 0,
    LikesLastNewCount       INTEGER DEFAULT 0,
    TTFolderPath            TEXT,
    BlogId                  INTEGER,
    PRIMARY KEY("BlogName")
);

CREATE INDEX ix_Blogs_BlogId ON Blogs (BlogId);
```

`BlogName` is the primary key, so it is the only indexed way in by name. There is no index
on any flag or date — filtering or sorting on those scans all 188k rows, which is
affordable here and is not on `Notes`.

**`BlogId` is new as of 2026-08-07 and is the join key to `Notes`.** It exists so that
`Notes` can reach `Blogs` in a single integer hop rather than going through `BlogNames`
and ending in a text comparison:

```sql
-- what you want
FROM Blogs B JOIN Notes N ON N.NoteBlogId = B.BlogId

-- not this
FROM Blogs B JOIN BlogNames BN ON BN.BlogName = B.BlogName
             JOIN Notes N      ON N.NoteBlogId = BN.BlogId
```

**`BlogId` is NULL on 168,202 of 188,620 rows** — every blog that has never appeared in a
note. That is the large majority, and it is not an error: the registry is far bigger than
the engagement graph. An inner join on `BlogId` therefore silently drops those blogs,
which is usually what you want for engagement queries and is wrong for registry listings.

Flag distribution: `IsActive = 1` on 188,601 of 188,620 rows, `HasBeenOutput = 1` on
5,059, `ByLikes = 1` on 2. `IsActive` carries a second meaning as of Rolodex — see
[`Blogs.IsActive`](#blogsisactive--now-written-by-two-applications) below.

The columns after `DateCreated` were added later by `ALTER TABLE`, which is why they carry
no quoting in the stored DDL. That is the normal way this schema grows, and `BlogId` is
the newest example.

**`DateAdded` is not written consistently.** 170,677 rows hold ISO `yyyy-MM-dd HH:mm:ss`;
17,943 hold US-format `M/d/yy` from a bulk import. As text those two sort into different
parts of the table, so anything ordering or range-filtering on this column has to
normalise first — see `DateSql` in Rolodex.

### `Posts`

```sql
CREATE TABLE "Posts" (
    "BlogName"              TEXT,
    "PostID"                INTEGER,
    "HasNotesGathered"      INTEGER DEFAULT 0,
    "reblogURL"             TEXT,
    "NotFound"              INTEGER DEFAULT 0,
    "PostDate"              TEXT,
    "NotesGatheredDateTime" INTEGER NOT NULL DEFAULT 1729746000,
    "HasImage"              INTEGER NOT NULL DEFAULT 0,
    "PostURL"               TEXT,
    "Slug"                  TEXT,
    "ReblogKey"             TEXT,
    "ReblogName"            TEXT,
    "Summary"               TEXT,
    "Quote"                 TEXT,
    "Body"                  TEXT,
    "Tags"                  TEXT,
    "Link"                  TEXT,
    "PhotoURL"              TEXT,
    "PhotoCaption"          TEXT,
    "DownloadedFiles"       TEXT,
    "AudioCaption"          TEXT,
    "Question"              TEXT,
    "Answer"                TEXT,
    "Title"                 TEXT,
    "ByLikes"               INTEGER NOT NULL DEFAULT 0,
    "RootBlogName"          TEXT,
    "RootURL"               TEXT,
    "DateModified"          TEXT,
    "DateCreated"           TEXT,
    PostType                TEXT,
    PRIMARY KEY("BlogName","PostID")
);
```

**The key is `(BlogName, PostID)`, not `PostID`.** This matters more than it looks: 345
post IDs exist under more than one blog, so an ID on its own is both ambiguous *and*
unindexed. Any lookup should carry the blog name, and a batch lookup should group by blog
so it stays on the leading column of the key.

Notable:

- **`PostType` is now mostly populated: 20,679 of 22,468 rows, leaving 1,789 `NULL`.**
  This reverses what earlier revisions of this document said — the column really was empty
  on every row, and something has since started writing it. Anything that treated it as
  permanently unset, or derived the type from post content instead, should be re-examined
  against the live data. Rolodex still derives it.
- `HasImage = 1` on 14,026 rows. It records that the post *had* a picture, not that a
  usable URL was kept, so it is not a reliable predictor that anything will render.
- `PhotoURL` is largely unused; in practice the image markup lives inside `Body`.
- `NotFound = 1` on 4,712 rows — posts that have since been deleted upstream.
- The content columns (`Body`, `Quote`, `Question`, `Answer`, …) are the heavy ones. List
  views should not select them.

### `Notes`

```sql
CREATE TABLE Notes (
    RootBlogId      INTEGER NOT NULL,
    PostID          INTEGER NOT NULL,
    NoteBlogId      INTEGER NOT NULL,
    TimeStamp       INTEGER NOT NULL,
    TypeId          INTEGER NOT NULL,
    replyText       TEXT,
    DatetimeCrawled TEXT,
    DateModified    TEXT,
    DateCreated     TEXT,
    IsActive        INTEGER NOT NULL DEFAULT 1,
    PRIMARY KEY (RootBlogId, PostID, TimeStamp, TypeId, NoteBlogId)
) WITHOUT ROWID;

CREATE INDEX ix_Notes_NoteBlogId ON Notes (NoteBlogId);
```

**Integer IDs since 2026-08-07 — this is the breaking change.** `RootBlogName`,
`NoteBlogName` and `Type` are gone, replaced by `RootBlogId`, `NoteBlogId` and `TypeId`.
Resolve them through [`BlogNames`](#blognames) and [`NoteTypes`](#notetypes), or join
straight to `Blogs` on `BlogId`. The old names were text repeated on 1.18 million rows,
in the table *and* in every index over it; the swap took the file from 207 MB to 148 MB.

The **primary key column order is deliberately unchanged**, so the leading-prefix access
patterns callers already depend on still hold: `(RootBlogId)` and `(RootBlogId, PostID)`
remain cheap prefixes, exactly as `(RootBlogName)` and `(RootBlogName, PostID)` were.

Two nulls-and-defaults differences from the old DDL, both intentional:

- `replyText` and `DatetimeCrawled` **no longer carry column defaults**. The old table
  defaulted them to `'.'` and `'2/12/26 12am'`, which is how 1.1M rows acquired
  placeholder values nobody wrote. New rows now get `NULL` unless a writer supplies
  something. The crawler names both columns explicitly, so its behaviour is unchanged.
- The five key columns are now `NOT NULL`. They always were in practice.

**`WITHOUT ROWID`, since earlier the same day.** The rows live in the primary key's
b-tree rather than in a rowid table with a separate key index beside it. Two consequences
matter before adding an index here:

- There is no `rowid` on this table. `SELECT rowid FROM Notes` is an error, and no
  code in any of the three apps relied on it.
- A secondary index carries the whole five-column primary key as its row reference
  instead of a compact rowid, so indexes here are **expensive** — though far less so
  than before, now that the key is five integers rather than three integers and two
  strings. `ix_Notes_NoteBlogId` costs 27 MB; its text predecessor cost 58 MB.

**`Notes_idx_06e01ae3` on `TimeStamp DESC` was dropped at the same time.** It cost
14 MB as a rowid index and would have cost 58 MB after the conversion. It was worth
neither: the crawler's only `TimeStamp` filter (`>= 1535778000`) excludes 786 rows
of 1.18M, Rolodex's default Notes sort carries a three-column tiebreaker that forces
a full sort regardless, and the reply-matching `UPDATE` uses `ABS(TimeStamp - ?) <= 5`,
which no index on `TimeStamp` can serve. The one path that got slower is Rolodex's
Notes page with a date-range filter: 60 ms to 164 ms.

See `../shrink-db.sql` for the full rationale and the applied result.

**`DatetimeCrawled` is `NULL` on 1,148,077 rows, and that is the honest value.** Those
rows previously stored the literal string `'2/12/26 12am'` — this column's own DDL
default, written as a bulk backfill placeholder rather than as a crawl time. They were
set to `NULL` on 2026-08-07, which is what consumers already displayed them as: the
string parses as a date in neither format this schema writes.

Note the trap: **the `DEFAULT '2/12/26 12am'` clause is still in the DDL above.** Any
`INSERT` that omits this column writes the placeholder straight back. The crawler names
it explicitly on every insert, so nothing reintroduces it today, but a new writer that
forgets to would — which is why consumers should keep treating an unparseable value here
as "unknown" rather than assuming `NULL` is now the only such marker.

One row per engagement event. `TimeStamp` is **unix seconds** — unlike every date column
elsewhere in the schema, which are text.

At 1.18M rows this is the table that dictates how the whole database has to be queried:

- **Nothing should run an unbounded `SELECT` or a bare `COUNT(*)` here.** A count scans
  the lot on every call.
- The only fast access paths are the primary key's leading columns (`RootBlogId`, then
  `PostID`) and `ix_Notes_NoteBlogId` on `NoteBlogId`. "Notes received by a blog" and
  "notes given by a blog" are both cheap; almost nothing else is.
- **Every** ordering here is a full sort of whatever the filters leave, `TimeStamp`
  included. Filter first, then sort.
- `replyText` is `'.'` on 1,167,464 rows — only `reply` notes carry real text. Those
  dots are inherited from the old column default; new rows get `NULL` instead.

**Resolve IDs by filtering the lookup, not by scanning `Notes`.** The lookup tables are
tiny and uniquely indexed, so pushing a name predicate into them costs nothing and lets
the `Notes` index do the work:

```sql
-- good: BlogNames resolves the name, then the index is searched
SELECT * FROM Notes
 WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = ?);

-- also good, same plan
SELECT n.* FROM Notes n
  JOIN BlogNames b ON b.BlogId = n.NoteBlogId
 WHERE b.BlogName = ?;
```

### `BlogNames`

```sql
CREATE TABLE BlogNames (
    BlogId   INTEGER PRIMARY KEY,
    BlogName TEXT NOT NULL UNIQUE
);
```

20,430 rows — every name appearing in `Notes` as either participant, and nothing else.
This is the **ID authority**: `Notes.RootBlogId` and `Notes.NoteBlogId` both point here,
and `Blogs.BlogId` is a copy of the value for the blogs that have one.

**12 of these names have no `Blogs` row.** The registry has never been a superset of the
engagement graph and still is not, so resolving an ID through `Blogs` rather than
`BlogNames` will occasionally find nothing. Use `BlogNames` when you need the name itself
and `Blogs` when you need registry columns.

IDs are assigned by SQLite and are **stable**: they are stored in over a million `Notes`
rows. Never renumber them. A blog that is renamed upstream should get a new row, not an
edit to an existing one, unless every `Notes` reference is migrated with it.

### `NoteTypes`

```sql
CREATE TABLE NoteTypes (
    TypeId INTEGER PRIMARY KEY,
    Type   TEXT NOT NULL UNIQUE
);
```

| `TypeId` | `Type` | Rows | Share |
|--:|---|--:|--:|
| 1 | `like` | 945,167 | 79.9% |
| 2 | `reblog` | 219,203 | 18.5% |
| 3 | `reply` | 15,345 | 1.3% |
| 4 | `posted` | 2,617 | 0.2% |
| 5 | `post_attribution` | 1 | — |

The set is fixed in practice, but it is a table rather than a `CHECK` constraint so that
adding a type is an `INSERT` and not a schema migration. **The IDs above are stored in
`Notes` and must not be reassigned.**

Five rows means the lookup is effectively free; write `t.Type = 'reblog'` and let SQLite
resolve it, or hardcode the ID if you prefer — both are fine, but hardcoding ties your
code to this table's contents, so prefer the join in anything long-lived.

---

## Porting to the integer schema

Everything here was checked against the live 148 MB file. There were 14 affected call
sites in `DataAccess.cs` and 16 in `RolodexRepository.cs`. TumblThree needs no changes —
its single statement touches `Blogs.IsActive` and `BlogName` only.

**`DataAccess.cs` is ported.** All 14 sites now read the integer schema, `AddNote`
registers names and types before inserting, and `verify-db-schema.sql` reports a
pre-migration file rather than letting the app fail on it. `RolodexRepository.cs` lives in
the [Rolodex](https://git.basso.land/jim/Rolodex) repository and is not covered by that
work. One site was dropped rather than translated: the `LEFT JOIN Notes` in `GetPosts`
selected nothing and was collapsed by the query's own `GROUP BY`, so it could not affect
the result.

### Column mapping

| Was | Is now | Resolve via |
|---|---|---|
| `Notes.RootBlogName` | `Notes.RootBlogId` | `BlogNames.BlogId` → `.BlogName` |
| `Notes.NoteBlogName` | `Notes.NoteBlogId` | `BlogNames.BlogId` → `.BlogName` |
| `Notes.Type` | `Notes.TypeId` | `NoteTypes.TypeId` → `.Type` |
| `ix_NoteBlogName01` | `ix_Notes_NoteBlogId` | — |

`PostID`, `TimeStamp`, `replyText`, `DatetimeCrawled`, `DateModified`, `DateCreated` and
`IsActive` are unchanged.

### Filtering by a blog name

```sql
-- was
WHERE NoteBlogName = @Name

-- now, either form; both search ix_Notes_NoteBlogId after a unique-index lookup
WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @Name)
-- or
JOIN BlogNames b ON b.BlogId = n.NoteBlogId WHERE b.BlogName = @Name
```

Measured 73 ms against 63 ms for the old text form on the busiest blog — the extra hop is
a unique-index probe on a 20k-row table and does not show.

### Joining `Notes` to `Blogs`

This is the join to get right; it is the most common shape in both applications.

```sql
-- was
FROM Blogs B INNER JOIN Notes N ON N.NoteBlogName = B.BlogName

-- now: one integer hop, using the new Blogs.BlogId
FROM Blogs B INNER JOIN Notes N ON N.NoteBlogId = B.BlogId
```

Do **not** route this through `BlogNames` — that adds a hop and ends in the text
comparison the change was meant to remove.

### Selecting a name back out

```sql
-- was
SELECT NoteBlogName AS blogName, COUNT(*) FROM Notes ... GROUP BY NoteBlogName

-- now
SELECT bn.BlogName AS blogName, COUNT(*)
  FROM Notes n JOIN BlogNames bn ON bn.BlogId = n.NoteBlogId
 ... GROUP BY bn.BlogName
```

Group by `n.NoteBlogId` instead of `bn.BlogName` when you only need the name for display —
grouping on the integer is cheaper and the name comes along for free.

### Filtering by type

```sql
-- was
WHERE type IN ('reblog', 'reply', 'posted')

-- now
WHERE TypeId IN (SELECT TypeId FROM NoteTypes WHERE Type IN ('reblog','reply','posted'))
-- or, equivalently
JOIN NoteTypes t ON t.TypeId = n.TypeId WHERE t.Type IN ('reblog','reply','posted')
```

`WHERE TypeId IN (2,3,4)` also works and is marginally faster, but hardcodes this table's
contents into application code. Prefer the lookup outside of hot paths.

Note the negated form needs care: `type NOT IN ('reblog','reply','posted')` becomes
`TypeId NOT IN (SELECT TypeId FROM NoteTypes WHERE Type IN (...))`, which is correct only
because `TypeId` is `NOT NULL`.

### Inserting a note

The crawler must ensure both names have IDs first. `INSERT OR IGNORE` on `BlogNames` is
the whole of it — no read-back, no round trip, safe to run every time:

```sql
INSERT OR IGNORE INTO BlogNames (BlogName) VALUES (@rootBlogName);
INSERT OR IGNORE INTO BlogNames (BlogName) VALUES (@noteBlogName);

INSERT OR IGNORE INTO Notes
       (RootBlogId, PostID, NoteBlogId, TimeStamp, TypeId,
        DatetimeCrawled, DateModified, DateCreated)
SELECT (SELECT BlogId FROM BlogNames WHERE BlogName = @rootBlogName),
       @PostID,
       (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName),
       @TimeStamp,
       (SELECT TypeId FROM NoteTypes WHERE Type = @Type),
       @DatetimeCrawled, @DateModified, @DateCreated;
```

Verified: a genuinely new note inserts, and re-running the identical statement inserts 0.
Run all three statements in one transaction so a crash cannot leave a name registered
with no note.

**The duplicate-key error message has changed.** `DataAccess.cs` compares against the
literal string

```
UNIQUE constraint failed: Notes.RootBlogName, Notes.PostID, Notes.TimeStamp, Notes.Type, Notes.NoteBlogName
```

at two call sites to decide whether to swallow an exception. SQLite now emits the *new*
column names, so those comparisons no longer match and real errors will surface where
they used to be silently ignored — or vice versa.

Both sites now go through `IsNotesDuplicateKey` in `DataAccess.cs`, which matches on
`UNIQUE constraint failed` plus `Notes.` rather than on the column list. A literal
comparison is what broke here; the next rename should not break it again.

### Updating notes

Predicates translate the same way. The reply-matching update, which cannot use an index
on `TimeStamp` either before or after:

```sql
-- now
UPDATE Notes SET replyText = @replyText, DateModified = @dateModified
 WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName)
   AND ABS(TimeStamp - @TimeStamp) <= 5
   AND TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')
   AND (replyText IS NULL OR replyText = '' OR replyText = '.')
   AND (replyText IS NULL OR replyText <> @replyText);
```

Rolodex's soft-delete updates need no change beyond the `WHERE` clause — they set
`IsActive`, which is untouched.

### Three traps

**`Blogs.BlogId` is NULL on 168,202 of 188,620 rows.** Any inner join on it silently drops
every blog that has never appeared in a note. Correct for engagement queries; wrong for
registry listings, which need a `LEFT JOIN` or no join at all.

**12 names in `BlogNames` have no `Blogs` row.** Resolving an ID to a name through `Blogs`
will occasionally find nothing. Use `BlogNames` for names and `Blogs` for registry columns.

**IDs are stable and must stay so.** `BlogNames.BlogId` and `NoteTypes.TypeId` are stored
in over a million `Notes` rows. Never renumber. A blog renamed upstream gets a new row,
not an edited one, unless every `Notes` reference migrates with it.

---

### Referential integrity

There are no foreign keys, and the tables do not perfectly agree:

- 4 `Posts` rows name a blog with no `Blogs` row.
- 12 of the 20,430 names in `BlogNames` have no `Blogs` row.

So a name appearing in `Notes` or `Posts` is not a guarantee that the registry knows about
it. Joins from those tables back to `Blogs` should tolerate a miss.

The integer schema does not fix this and was not meant to. `BlogNames` is deliberately
built from `Notes` rather than from `Blogs`, precisely so that the 12 unregistered
engagers keep their IDs and their rows. Had it been built from the registry, those notes
would have been dropped by the migration's inner joins.

---

## The `'.'` placeholder convention

**The crawler writes a single dot into text columns it has no value for, rather than
`NULL`.** This is the single most surprising thing about the schema and it affects every
consumer.

| Column | `'.'` rows |
|---|--:|
| `Notes.replyText` | 1,167,464 |
| `Posts.Title` | 12,562 |
| `Posts.Body` | 172 |

`Notes.replyText` and `Notes.DatetimeCrawled` **no longer carry column defaults** as of
the integer migration, so new note rows get `NULL` rather than a placeholder. The dots
already in `replyText` were not rewritten — cleaning is still required on read.

Any query whose output reaches a human should collapse it:

```sql
NULLIF(NULLIF(SomeColumn, '.'), '') AS SomeColumn
```

Empty string turns up too, hence the double `NULLIF`. Not every column is affected —
`Blogs.TTFolderPath` and `Blogs.DateModified` currently have zero dot rows — but new
columns tend to acquire them, so treat cleaning as the default for any text column
rendered to a user.

---

## Supporting tables

Crawler bookkeeping. Rolodex ignores all of these.

| Table | Rows | What it is |
|---|--:|---|
| `DailyAPICount` | 133 | `(Date TEXT PK, APICount INTEGER)` — per-day API call tally against the rate limit |
| `ApiKeyPoolState` | 2 | `(KeyName TEXT PK, RetryUntil INTEGER)` — per-key backoff; `RetryUntil` is unix seconds |
| `ApiKeyPoolMeta` | 1 | `(Id PK CHECK (Id = 1), LastIndex)` — round-robin cursor. Singleton by check constraint |
| `CollectRunState` | 1 | `(Id PK CHECK (Id = 1), RunCutoff, RunComplete, RunStarted, RunCompletedAt)` — resume state for an interrupted collection run. Also a singleton |

---

## `Blogs.IsActive` — now written by two applications

`IsActive` has always been the crawler's work-selection flag. `GetBlogs` in
`DataAccess.cs` joins on it to decide what to collect:

```sql
-- shape only; the ported GetBlogs joins Blogs directly on BlogId and needs no BlogNames hop
SELECT bn.BlogName, count(*)
  FROM Notes n
  JOIN Blogs     b  ON b.BlogId  = n.NoteBlogId
  JOIN BlogNames bn ON bn.BlogId = n.NoteBlogId
 WHERE b.IsActive = @isActive AND ...
```

Nothing inside the crawler *writes* it — it is an input, set from outside.

**Rolodex is now one of the things that sets it.** Removing a blog through the Rolodex UI
runs exactly this:

```sql
UPDATE Blogs SET IsActive = 0 WHERE BlogName = ?;
```

Rolodex adds no column and changes no schema. It reuses this flag because the two meanings
were judged to be one decision: a blog you do not want in the browsing UI is a blog you do
not want to keep crawling. Removal therefore stops collection, and the Rolodex
confirmation screen says so before anyone commits.

- `1` (or absent/NULL) — live. Crawled, and visible in Rolodex.
- `0` — removed. Not crawled, hidden from the Rolodex registry, dashboard counts and
  engagement rollups.

Restoring is the same `UPDATE` with a `1`. Nothing is destroyed either way: the blog's
`Posts` and `Notes` rows are never touched, and Rolodex deliberately keeps showing them
under its Posts and Notes pages. Removing a blog hides the blog, not what it collected.

### What other tools need to know

1. **Setting `IsActive = 0` now also hides the blog from Rolodex**, and setting it back to
   `1` makes it reappear. If another tool deactivates blogs in bulk, it is also removing
   them from the browsing UI — which may be exactly right, but it is no longer a
   crawler-only decision.
2. **Re-crawling a removed blog will not bring it back**, since nothing in the crawler
   writes the flag. An `INSERT OR REPLACE` on the `Blogs` row *would*, by resetting it to
   the column default of `1`. Prefer an `UPDATE` of the specific columns, or
   `INSERT … ON CONFLICT DO UPDATE SET` naming only the columns being refreshed.
3. **NULL is treated as live.** The column is `INTEGER DEFAULT 1` with no `NOT NULL`, so
   Rolodex reads it through `COALESCE(IsActive, 1)`. A NULL therefore leaves the blog
   visible rather than stranding it outside both the registry and the removed list, where
   no screen could reach it. Write `0` or `1`, not NULL.
4. **Backing the feature out is a configuration change, not a migration.** Because there is
   no Rolodex-owned column, setting `Rolodex__EnableBlogDeletion=false` is the whole of it;
   there is nothing to drop. Any blogs already at `IsActive = 0` simply go back to being
   ordinary inactive blogs.

---

## `Posts.IsActive` and `Notes.IsActive` — present, and written from outside

The same flag extends to the two content tables, with the same meaning: `0` is removed,
anything else — including `NULL` — is live. **Both columns now exist in the live `TL.db`**
and are included in the DDL quoted above. As of 2026-08-07, `Posts.IsActive = 0` on 5,900
rows and `Notes.IsActive = 0` on none. Like `Blogs.IsActive`, they are written from
outside this crawler.

On `Notes` the column is `INTEGER NOT NULL DEFAULT 1`, so a `NULL` cannot occur there;
`Posts` and `Blogs` are laxer, which is why the predicate below still uses `COALESCE`.

The crawler therefore treats both as optional, and as nothing it owns:

- **It never writes them.** No `INSERT` column list names `IsActive`, no `UPDATE` sets it,
  and `MapPrefixToColumn` — the only place a column name is chosen at runtime — cannot map
  to it. Re-crawling a removed post or note refreshes its content and leaves the flag at
  `0`. There is no `INSERT OR REPLACE` on `Posts` or `Notes` for a default to be reset by.
- **It filters on them only when they exist.** `HasIsActiveColumn` in `DataAccess.cs` asks
  `PRAGMA table_info` once per table per database path and caches the answer; the filter
  is `COALESCE(IsActive, 1) = 1`, and it is dropped entirely when the column is absent.
  Naming a missing column is a hard SQLite error, so this is what lets one build run
  against databases on both sides of the change. The cache lives for the process — adding
  the columns to a live database takes effect on the next run.

Every read that selects posts or notes carries the filter: `GetPosts`, `GetReplies`,
`GetRepliesWithMissingText`, `GetRepliesWithFilledText`, `GetAllPostTextColumns`,
`GetAllPostsForBlog`, `GetPost`, `GetPostByIdAnyBlog`, and the engagement queries that
count or join `Notes` (`GetBlogs`, `GetBlogsAll`, `GetBlogsForLikes`). The one deliberate
omission is the `LEFT JOIN Notes` in `GetPosts`: nothing is selected from it and it can
neither add nor remove a row, so filtering it would buy nothing.

`LegacyPostsDbImporter` is unfiltered too — it reads a foreign legacy database whose
`Posts` table is not this schema.

Two consequences worth stating plainly, both inherited from how `Blogs.IsActive` is
handled:

1. **Removal hides a row; it does not freeze it.** The write paths are keyed on a post the
   caller already selected, so an ingest or a correction run still overwrites the content
   of a removed post. Only selection is filtered.
2. **`NULL` is live.** Write `0` or `1`, not `NULL`, but a `NULL` leaves the row visible
   rather than stranding it.

---

## Reproducing the numbers

```sql
SELECT 'Blogs', COUNT(*) FROM Blogs
UNION ALL SELECT 'Posts',     COUNT(*) FROM Posts
UNION ALL SELECT 'Notes',     COUNT(*) FROM Notes
UNION ALL SELECT 'BlogNames', COUNT(*) FROM BlogNames;

-- note type mix (joins NoteTypes; Notes.Type no longer exists)
SELECT t.Type, COUNT(*)
  FROM Notes n JOIN NoteTypes t ON t.TypeId = n.TypeId
 GROUP BY t.Type ORDER BY 2 DESC;

-- how much of the registry participates in the engagement graph
SELECT COUNT(*) FILTER (WHERE BlogId IS NOT NULL) AS with_notes,
       COUNT(*) FILTER (WHERE BlogId IS NULL)     AS without_notes
  FROM Blogs;

-- the two date shapes in Blogs.DateAdded
SELECT CASE WHEN DateAdded LIKE '____-__-__%' THEN 'ISO' ELSE 'US' END, COUNT(*)
FROM Blogs GROUP BY 1;

-- post IDs that are ambiguous without a blog name
SELECT COUNT(*) FROM (
    SELECT PostID FROM Posts GROUP BY PostID HAVING COUNT(DISTINCT BlogName) > 1);

-- rows that reference a blog the registry does not have
SELECT COUNT(*) FROM Posts p
WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = p.BlogName);

SELECT COUNT(*) FROM BlogNames bn
WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = bn.BlogName);

-- space by object, to see where the file actually goes
SELECT name, SUM(pgsize)/1024/1024 AS mb
  FROM dbstat GROUP BY name ORDER BY SUM(pgsize) DESC;
```

Open the file read-only so an inspection can never disturb a running crawl:

```bash
sqlite3 "file:TL.db?mode=ro" ".schema"
```