jim/URLNotesGrabberCore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(db)!: port DataAccess to the Notes integer schema

Browse Source 
Notes.RootBlogName/NoteBlogName/Type became RootBlogId/NoteBlogId/TypeId
on 2026-08-07, resolved through the new BlogNames and NoteTypes tables.
There is no compatibility view, so every affected statement is a hard cut.

All 14 call sites in DataAccess.cs are ported:

- Notes->Blogs joins go through Blogs.BlogId in one integer hop; the
  Notes->Posts join in GetRepliesWithFilledText is the only one that must
  route through BlogNames, since Posts carries no BlogId
- AddNote registers both blog names and the note type with INSERT OR
  IGNORE before inserting, in one transaction committed before the console
  sleep. Registering the type matters: an unseen type would resolve to
  NULL and fail NOT NULL on TypeId, silently losing the note
- The LEFT JOIN Notes in GetPosts is dropped rather than translated. It
  selected nothing, could not remove a row, and its duplicates were
  collapsed by the query's own GROUP BY
- Duplicate-key detection moves to IsNotesDuplicateKey, matching the
  constraint and table instead of an exact column list. The old literal
  string is what broke on this rename
- EnsureReplyTextColumnExists drops DEFAULT '.', matching the migrated
  schema: new rows get NULL, not a placeholder nobody wrote

verify-db-schema.sql gains BlogNames, NoteTypes, Blogs.BlogId and the new
Notes columns, plus query 1d naming a pre-migration file and pointing at
normalize-notes.sql. Blogs.BlogId is deliberately not auto-fixable -- an
added-but-empty column makes engagement joins return zero rows silently.

Verified against the live 148 MB file: query plans hit the intended
indexes, and the BlogId join matches an independent name-resolved
formulation exactly on all 4,267 GetBlogs and 2,637 GetBlogsForLikes rows.

RolodexRepository.cs (16 sites) lives in the Rolodex repo and is not
covered here.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jim
2026-08-07 21:23:13 -05:00
co-authored by Claude Opus 5
parent c3cf89c3f1
commit b31d5842cc
4 changed files with 277 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files
verify-db-schema.sql
+54 -9
View File
@@ -79,18 +79,35 @@ WITH expected(tbl, col, alter_stmt) AS (
('Blogs','LikesLastRefreshed', 'ALTER TABLE Blogs ADD COLUMN LikesLastRefreshed INTEGER DEFAULT 0;'),
('Blogs','LikesLastNewCount', 'ALTER TABLE Blogs ADD COLUMN LikesLastNewCount INTEGER DEFAULT 0;'),
('Blogs','TTFolderPath', 'ALTER TABLE Blogs ADD COLUMN TTFolderPath TEXT;'),
-- Blogs.BlogId (2026-08-07) is the single-hop join key into Notes. Deliberately NOT
-- auto-fixable: an added-but-empty BlogId makes every engagement join return zero
-- rows silently, which is worse than the hard error a missing column gives.
('Blogs','BlogId', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
-- Notes (base columns: manual review if missing)
('Notes','RootBlogName', 'MANUAL REVIEW - base/PK column missing'),
-- Integer IDs since 2026-08-07. RootBlogName/NoteBlogName/Type are GONE, not renamed
-- in place -- a backup that still has them needs normalize-notes.sql, not an ALTER.
-- Query 1d below reports exactly that case.
('Notes','RootBlogId', 'MANUAL REVIEW - see query 1d: pre-2026-08-07 name schema, or damaged'),
('Notes','PostID', 'MANUAL REVIEW - base/PK column missing'),
('Notes','NoteBlogName', 'MANUAL REVIEW - base/PK column missing'),
('Notes','NoteBlogId', 'MANUAL REVIEW - see query 1d: pre-2026-08-07 name schema, or damaged'),
('Notes','TimeStamp', 'MANUAL REVIEW - base/PK column missing'),
('Notes','Type', 'MANUAL REVIEW - base/PK column missing'),
('Notes','TypeId', 'MANUAL REVIEW - see query 1d: pre-2026-08-07 name schema, or damaged'),
('Notes','DatetimeCrawled', 'MANUAL REVIEW - base column missing'),
('Notes','DateModified', 'MANUAL REVIEW - base column missing'),
('Notes','DateCreated', 'MANUAL REVIEW - base column missing'),
-- Notes (additive migration column, auto-fixable)
('Notes','replyText', 'ALTER TABLE Notes ADD COLUMN replyText TEXT DEFAULT ''.'';'),
-- No DEFAULT: the migrated schema dropped it, so new rows get NULL rather than a
-- placeholder. EnsureReplyTextColumnExists in DataAccess.cs adds it the same way.
('Notes','replyText', 'ALTER TABLE Notes ADD COLUMN replyText TEXT;'),
-- BlogNames / NoteTypes (the lookup tables Notes resolves its IDs through, 2026-08-07).
-- Not auto-fixable: an empty BlogNames does not mean "add the table", it means the
-- Notes rows have nothing to resolve against. Rebuild with normalize-notes.sql.
('BlogNames','BlogId', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
('BlogNames','BlogName', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
('NoteTypes','TypeId', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
('NoteTypes','Type', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
-- DailyAPICount (base columns)
('DailyAPICount','Date', 'MANUAL REVIEW - base/PK column missing'),
@@ -106,6 +123,8 @@ actual(tbl, col) AS (
SELECT 'Posts', name FROM pragma_table_info('Posts')
UNION ALL SELECT 'Blogs', name FROM pragma_table_info('Blogs')
UNION ALL SELECT 'Notes', name FROM pragma_table_info('Notes')
UNION ALL SELECT 'BlogNames', name FROM pragma_table_info('BlogNames')
UNION ALL SELECT 'NoteTypes', name FROM pragma_table_info('NoteTypes')
UNION ALL SELECT 'DailyAPICount', name FROM pragma_table_info('DailyAPICount')
UNION ALL SELECT 'ApiKeyPoolState', name FROM pragma_table_info('ApiKeyPoolState')
UNION ALL SELECT 'ApiKeyPoolMeta', name FROM pragma_table_info('ApiKeyPoolMeta')
@@ -127,7 +146,7 @@ ORDER BY (e.alter_stmt LIKE 'ALTER%') DESC, e.tbl, e.col;
-- 1b. MISSING TABLES: expected tables that don't exist at all in this DB.
-- Zero rows = good.
WITH expected_tables(tbl) AS (
VALUES ('Posts'),('Blogs'),('Notes'),('DailyAPICount'),
VALUES ('Posts'),('Blogs'),('Notes'),('BlogNames'),('NoteTypes'),('DailyAPICount'),
('ApiKeyPoolState'),('ApiKeyPoolMeta')
)
SELECT et.tbl AS missing_table
@@ -157,10 +176,12 @@ WITH expected(tbl, col) AS (
('Blogs','BlogName'),('Blogs','HasBeenOutput'),('Blogs','IsActive'),('Blogs','DateAdded'),
('Blogs','ByLikes'),('Blogs','DateModified'),('Blogs','DateCreated'),('Blogs','LikesPulled'),
('Blogs','LikesCursor'),('Blogs','LikesNewestTimestamp'),('Blogs','LikesLastRefreshed'),
('Blogs','LikesLastNewCount'),('Blogs','TTFolderPath'),
('Notes','RootBlogName'),('Notes','PostID'),('Notes','NoteBlogName'),('Notes','TimeStamp'),
('Notes','Type'),('Notes','DatetimeCrawled'),('Notes','DateModified'),('Notes','DateCreated'),
('Blogs','LikesLastNewCount'),('Blogs','TTFolderPath'),('Blogs','BlogId'),
('Notes','RootBlogId'),('Notes','PostID'),('Notes','NoteBlogId'),('Notes','TimeStamp'),
('Notes','TypeId'),('Notes','DatetimeCrawled'),('Notes','DateModified'),('Notes','DateCreated'),
('Notes','replyText'),('Notes','IsActive'),
('BlogNames','BlogId'),('BlogNames','BlogName'),
('NoteTypes','TypeId'),('NoteTypes','Type'),
('DailyAPICount','Date'),('DailyAPICount','APICount'),
('ApiKeyPoolState','KeyName'),('ApiKeyPoolState','RetryUntil'),
('ApiKeyPoolMeta','Id'),('ApiKeyPoolMeta','LastIndex')
@@ -169,6 +190,8 @@ actual(tbl, col) AS (
SELECT 'Posts', name FROM pragma_table_info('Posts')
UNION ALL SELECT 'Blogs', name FROM pragma_table_info('Blogs')
UNION ALL SELECT 'Notes', name FROM pragma_table_info('Notes')
UNION ALL SELECT 'BlogNames', name FROM pragma_table_info('BlogNames')
UNION ALL SELECT 'NoteTypes', name FROM pragma_table_info('NoteTypes')
UNION ALL SELECT 'DailyAPICount', name FROM pragma_table_info('DailyAPICount')
UNION ALL SELECT 'ApiKeyPoolState', name FROM pragma_table_info('ApiKeyPoolState')
UNION ALL SELECT 'ApiKeyPoolMeta', name FROM pragma_table_info('ApiKeyPoolMeta')
@@ -181,6 +204,25 @@ WHERE e.col IS NULL
ORDER BY a.tbl, a.col;
-- 1d. PRE-MIGRATION DATABASE: a backup from before 2026-08-07, when Notes still
-- stored names. Zero rows = good.
--
-- This is the one failure SECTION 2 cannot fix. Notes.RootBlogName /
-- NoteBlogName / Type were replaced by RootBlogId / NoteBlogId / TypeId
-- resolving through BlogNames and NoteTypes -- a data migration, not an
-- ADD COLUMN. There is no compatibility view, so the current code fails
-- outright ("no such column: RootBlogId") against such a file.
--
-- Fix: run normalize-notes.sql against a COPY of the backup, then re-run
-- SECTION 1. Do not hand-add the ID columns: they would be empty, and an
-- empty NoteBlogId is indistinguishable from a note by blog #0.
SELECT 'Notes still stores names -- run normalize-notes.sql on a copy' AS pre_migration_schema,
group_concat(name, ', ') AS legacy_columns_found
FROM pragma_table_info('Notes')
WHERE lower(name) IN ('rootblogname','noteblogname','type')
HAVING COUNT(*) > 0;
-- ============================================================================
-- SECTION 2 -- FIX (opt-in, additive only)
--
@@ -190,6 +232,9 @@ ORDER BY a.tbl, a.col;
-- "duplicate column name" error and changes nothing -- just run the flagged
-- subset. These are the 8 additive migration columns and nothing else; the
-- likes high-water-mark reset is intentionally NOT included.
--
-- Nothing here addresses query 1d. The Notes integer schema is a data migration
-- (normalize-notes.sql) and cannot be reached by adding columns.
-- ============================================================================
-- ALTER TABLE Posts ADD COLUMN PostType TEXT;
@@ -199,4 +244,4 @@ ORDER BY a.tbl, a.col;
-- ALTER TABLE Blogs ADD COLUMN LikesLastRefreshed INTEGER DEFAULT 0;
-- ALTER TABLE Blogs ADD COLUMN LikesLastNewCount INTEGER DEFAULT 0;
-- ALTER TABLE Blogs ADD COLUMN TTFolderPath TEXT;
-- ALTER TABLE Notes ADD COLUMN replyText TEXT DEFAULT '.';
-- ALTER TABLE Notes ADD COLUMN replyText TEXT;