feat: honor optional Posts.IsActive and Notes.IsActive
Both columns carry the meaning Blogs.IsActive has: 0 = removed by another tool, anything else (including NULL) = live. Neither exists in the live TL.db yet, and both are added from outside this crawler, so the code has to work on databases either side of the change - naming a missing column is a hard SQLite error. HasIsActiveColumn asks PRAGMA table_info once per table per database path and caches it; AndIsActive/WhereIsActive return "COALESCE(IsActive, 1) = 1" or an empty string. Every read that selects posts or notes now carries the filter: GetPosts (both branches, including the per-blog count subquery), GetReplies, GetRepliesWithMissingText, GetRepliesWithFilledText, GetAllPostTextColumns, GetAllPostsForBlog, GetPost, GetPostByIdAnyBlog, and the engagement queries that count or join Notes - GetBlogs, GetBlogsAll and both note-joining variants of GetBlogsForLikes. The LEFT JOIN Notes in GetPosts is left alone on purpose: nothing is selected from it and it can neither add nor remove a row. LegacyPostsDbImporter is left alone too - it reads a foreign legacy schema. Writes were already safe and are documented rather than changed: no INSERT column list names IsActive, no UPDATE sets it, MapPrefixToColumn cannot map to it, and there is no INSERT OR REPLACE on Posts or Notes for a column default to be reset by. Re-crawling a removed row refreshes its content and leaves the flag at 0. As with Blogs, exclusion belongs at selection, so the update paths stay keyed on rows the caller already chose. Verified against three synthetic databases - no IsActive columns, columns present with a removed post and its notes, and columns present but NULL - by running every affected reader: the queries are valid in all three, the removed rows drop out only where the columns exist, NULL reads as live, and AddPost/AddNote/UpsertPostFromTextFile/UpdatePostContentFields leave an IsActive = 0 row at 0. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
@@ -142,6 +142,9 @@ ORDER BY et.tbl;
|
||||
-- 1c. EXTRA / UNEXPECTED COLUMNS: present in the DB but not in the expected
|
||||
-- list above. Informational only -- e.g. a NEWER backup, or a column this
|
||||
-- script's expected-list hasn't been updated for. Not an error by itself.
|
||||
-- Posts.IsActive and Notes.IsActive are listed here and NOT in 1a on
|
||||
-- purpose: they are written by other tools, the app only reads them when
|
||||
-- present, and it must not be told to add them. See TL.db.md.
|
||||
WITH expected(tbl, col) AS (
|
||||
VALUES
|
||||
('Posts','BlogName'),('Posts','PostID'),('Posts','HasNotesGathered'),('Posts','reblogURL'),
|
||||
@@ -150,14 +153,14 @@ WITH expected(tbl, col) AS (
|
||||
('Posts','Quote'),('Posts','Body'),('Posts','Tags'),('Posts','Link'),('Posts','PhotoURL'),
|
||||
('Posts','PhotoCaption'),('Posts','DownloadedFiles'),('Posts','AudioCaption'),('Posts','Question'),
|
||||
('Posts','Answer'),('Posts','Title'),('Posts','ByLikes'),('Posts','RootBlogName'),('Posts','RootURL'),
|
||||
('Posts','DateModified'),('Posts','DateCreated'),('Posts','PostType'),
|
||||
('Posts','DateModified'),('Posts','DateCreated'),('Posts','PostType'),('Posts','IsActive'),
|
||||
('Blogs','BlogName'),('Blogs','HasBeenOutput'),('Blogs','IsActive'),('Blogs','DateAdded'),
|
||||
('Blogs','ByLikes'),('Blogs','DateModified'),('Blogs','DateCreated'),('Blogs','LikesPulled'),
|
||||
('Blogs','LikesCursor'),('Blogs','LikesNewestTimestamp'),('Blogs','LikesLastRefreshed'),
|
||||
('Blogs','LikesLastNewCount'),('Blogs','TTFolderPath'),
|
||||
('Notes','RootBlogName'),('Notes','PostID'),('Notes','NoteBlogName'),('Notes','TimeStamp'),
|
||||
('Notes','Type'),('Notes','DatetimeCrawled'),('Notes','DateModified'),('Notes','DateCreated'),
|
||||
('Notes','replyText'),
|
||||
('Notes','replyText'),('Notes','IsActive'),
|
||||
('DailyAPICount','Date'),('DailyAPICount','APICount'),
|
||||
('ApiKeyPoolState','KeyName'),('ApiKeyPoolState','RetryUntil'),
|
||||
('ApiKeyPoolMeta','Id'),('ApiKeyPoolMeta','LastIndex')
|
||||
|
||||
Reference in New Issue
Block a user