feat: honor optional Posts.IsActive and Notes.IsActive

Both columns carry the meaning Blogs.IsActive has: 0 = removed by another
tool, anything else (including NULL) = live. Neither exists in the live
TL.db yet, and both are added from outside this crawler, so the code has
to work on databases either side of the change - naming a missing column
is a hard SQLite error.

HasIsActiveColumn asks PRAGMA table_info once per table per database path
and caches it; AndIsActive/WhereIsActive return "COALESCE(IsActive, 1) = 1"
or an empty string. Every read that selects posts or notes now carries the
filter: GetPosts (both branches, including the per-blog count subquery),
GetReplies, GetRepliesWithMissingText, GetRepliesWithFilledText,
GetAllPostTextColumns, GetAllPostsForBlog, GetPost, GetPostByIdAnyBlog,
and the engagement queries that count or join Notes - GetBlogs,
GetBlogsAll and both note-joining variants of GetBlogsForLikes.

The LEFT JOIN Notes in GetPosts is left alone on purpose: nothing is
selected from it and it can neither add nor remove a row.
LegacyPostsDbImporter is left alone too - it reads a foreign legacy
schema.

Writes were already safe and are documented rather than changed: no
INSERT column list names IsActive, no UPDATE sets it, MapPrefixToColumn
cannot map to it, and there is no INSERT OR REPLACE on Posts or Notes for
a column default to be reset by. Re-crawling a removed row refreshes its
content and leaves the flag at 0. As with Blogs, exclusion belongs at
selection, so the update paths stay keyed on rows the caller already
chose.

Verified against three synthetic databases - no IsActive columns, columns
present with a removed post and its notes, and columns present but NULL -
by running every affected reader: the queries are valid in all three, the
removed rows drop out only where the columns exist, NULL reads as live,
and AddPost/AddNote/UpsertPostFromTextFile/UpdatePostContentFields leave
an IsActive = 0 row at 0.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jim
2026-07-29 14:21:09 -05:00
co-authored by Claude Opus 5
parent eded5271ea
commit 05ec465f74
3 changed files with 155 additions and 20 deletions
+41
View File
@@ -262,6 +262,47 @@ under its Posts and Notes pages. Removing a blog hides the blog, not what it col
---
## `Posts.IsActive` and `Notes.IsActive` — optional, and not in this database yet
The same flag is being extended to the two content tables, with the same meaning: `0` is
removed, anything else — including `NULL` — is live. **Neither column exists in the live
`TL.db` as of 2026-07-29**; the DDL quoted above for `Posts` and `Notes` is complete. Like
`Blogs.IsActive`, they are written from outside this crawler.
The crawler therefore treats both as optional, and as nothing it owns:
- **It never writes them.** No `INSERT` column list names `IsActive`, no `UPDATE` sets it,
and `MapPrefixToColumn` — the only place a column name is chosen at runtime — cannot map
to it. Re-crawling a removed post or note refreshes its content and leaves the flag at
`0`. There is no `INSERT OR REPLACE` on `Posts` or `Notes` for a default to be reset by.
- **It filters on them only when they exist.** `HasIsActiveColumn` in `DataAccess.cs` asks
`PRAGMA table_info` once per table per database path and caches the answer; the filter
is `COALESCE(IsActive, 1) = 1`, and it is dropped entirely when the column is absent.
Naming a missing column is a hard SQLite error, so this is what lets one build run
against databases on both sides of the change. The cache lives for the process — adding
the columns to a live database takes effect on the next run.
Every read that selects posts or notes carries the filter: `GetPosts`, `GetReplies`,
`GetRepliesWithMissingText`, `GetRepliesWithFilledText`, `GetAllPostTextColumns`,
`GetAllPostsForBlog`, `GetPost`, `GetPostByIdAnyBlog`, and the engagement queries that
count or join `Notes` (`GetBlogs`, `GetBlogsAll`, `GetBlogsForLikes`). The one deliberate
omission is the `LEFT JOIN Notes` in `GetPosts`: nothing is selected from it and it can
neither add nor remove a row, so filtering it would buy nothing.
`LegacyPostsDbImporter` is unfiltered too — it reads a foreign legacy database whose
`Posts` table is not this schema.
Two consequences worth stating plainly, both inherited from how `Blogs.IsActive` is
handled:
1. **Removal hides a row; it does not freeze it.** The write paths are keyed on a post the
caller already selected, so an ingest or a correction run still overwrites the content
of a removed post. Only selection is filtered.
2. **`NULL` is live.** Write `0` or `1`, not `NULL`, but a `NULL` leaves the row visible
rather than stranding it.
---
## Reproducing the numbers
```sql