Merge branch 'master' into claude/jovial-mayer-77c2b5

The INSERT named a DateCreated column that DailyAPICount (Date, APICount)
never had, so every call threw "no such column: DateCreated" into an
empty catch block. Today's row was never created and the tally sat idle
since 2026-04-13. Drop the column from the INSERT, and report the three
silent failure points (insert error, missing row after insert, update
matching zero rows) instead of swallowing them.

.github/copilot-instructions.md

@@ -22,18 +22,18 @@ This document provides essential context for AI agents working with URLNotesGrab
 ```powershell
 dotnet build
 dotnet run            # Process all files in input directory
-dotnet run -- -parse [blogname]  # Process specific blog
+dotnet run -- --parse [blogname]  # Process specific blog
-dotnet run -- -test [blogname] [postID]  # Test API for specific post
+dotnet run -- --test [blogname] [postID]  # Test API for specific post
 ```
 
 ### Command-Line Interface
-- `-parse [blogname]`: Parse text files for specific blog
+- `--parse [blogname]`: Parse text files for specific blog
-- `-test [blogname] [postID]`: Test API note collection
+- `--test [blogname] [postID]`: Test API note collection
-- `-posts`: Export post blogs to file
+- `--posts`: Export post blogs to file
-- `-blogs`: Export blog list to file
+- `--blogs`: Export blog list to file
-- `-collect`: Collect notes for all posts in DB
+- `--collect`: Collect notes for all posts in DB
-- `-blogsR`: Export reply blogs to file
+- `--blogsR`: Export reply blogs to file
-- `-blogsO [start] [stop]`: Export blogs within range
+- `--blogsO [start] [stop]`: Export blogs within range
 
 ## Project Conventions
 

.gitignore

@@ -360,4 +360,14 @@ MigrationBackup/
 .ionide/
 
 # Fody - auto-generated XML schema
-FodyWeavers.xsd
+FodyWeavers.xsd
+/URLNotesGrabberCORE/TL.db
+
+# Claude Code local settings + worktrees
+.claude/settings.local.json
+.claude/worktrees/
+
+/URLNotesGrabberCORE/TL.db
+/URLNotesGrabberCORE/tl.db-shm
+/URLNotesGrabberCORE/tl.db-wal
+/U/jim/documents/web copies/blogs

AGENTS.md

@@ -0,0 +1,85 @@
+# AGENTS.md
+
+## Project Overview
+- **Primary Language**: C# (.NET 8 Console Application)
+- **Key Libraries**: RestSharp, Newtonsoft.Json, System.Data.SQLite, Microsoft.Extensions.Configuration
+- **Purpose**: Tumblr API data harvester for collecting notes, posts, likes, and replies, storing results in SQLite.
+
+## Architectural Patterns
+- CLI entry point in `Program.cs` with workflow orchestration
+- `DataAccess.cs`: Database operations, `ApiKeyPool` (API key management), `APIAccess` (Tumblr client)
+- `ResponseNotes.cs`: Tumblr API response models
+- Round-robin API key rotation with rate-limit tracking
+- Automatic console color assignment per API key for output differentiation
+- No-argument mode (`Program.TraverseDirectory`) ingests `.txt` blog export files into `Posts` via `DataAccess.AddPost`. Recognized field prefixes live in `TraverseDirectoryFieldPrefixes`; `Body:` and `Downloaded files:` collect every following line up to the next recognized prefix (multi-line values). `RootURL` is populated from a `Reblog root url:` line the same way it's populated from the API-based `--likes` flow — both paths converge on `DataAccess.AddPost`'s `rootURL` parameter, which `UpdatePost` only overwrites when the incoming value is non-empty (existing `RootURL` is preserved otherwise)
+
+## Developer Guidelines
+
+### Code Formatting
+- 4-space indentation, no tabs, match existing C# style
+- PascalCase for public members, camelCase for locals
+- Minimize code comments unless explicitly requested
+- Use only existing project libraries; no new dependencies without confirmation
+- Match accessibility modifiers (`public` for models, `internal` for helpers)
+
+### Error Handling
+- Wrap file/network operations in `try-catch`
+- Log non-critical errors (e.g., config write failures) with `[Warning]` prefix
+- Preserve console color state: use save/restore pattern for temporary color changes
+- API rate limits must use `ApiKeyPool.MarkRateLimited()`/`MarkAvailable()`
+
+### API Failure Classification
+Tumblr sits behind a CDN that returns HTML error pages (403, 5xx) which never reach the API. These
+say nothing about the item being fetched, so they must not be recorded as per-item failures.
+
+- A response body that will not parse as JSON did not come from the API. Flag it with
+  `Root.transientFailure`, never as `FAILURE`
+- Transient failures retry in place (`TransientBackoffSeconds`) before the item is skipped; a skipped
+  item stays unmarked in the DB so a later launch retries it
+- `MaxConsecutiveTransient` consecutive transient failures aborts the pass rather than skipping
+  item-by-item against an edge that is refusing all traffic
+- Only call `ApiKeyPool.MarkAvailable()` on a response that actually reached the API. A transport or
+  CDN failure says nothing about the key's standing and must not clear its flag
+- Only a real HTTP 429 (or `meta.status == 429`) counts as a rate limit. Do not infer one from the
+  presence of `X-RateLimit-*` headers, which Tumblr sends on every response
+- Rate limiters must pace with `await AcquireAsync()`. `AttemptAcquire()` does not wait, so a
+  saturated window aborts the run instead of throttling it
+- Long-running commands return exit 3 when a pass ends incomplete (rate-limit pause, breaker trip, or
+  skipped items), so a caller can distinguish that from a clean run
+
+### `IsActive` Is Not Ours To Write
+`Blogs.IsActive`, `Posts.IsActive` and `Notes.IsActive` are removal flags set by other tools
+(Rolodex). `0` means removed; anything else, including `NULL`, means live. Full detail in
+`URLNotesGrabberCORE/TL.db.md`.
+
+- **Never write any `IsActive` column.** Not in an `INSERT` column list, not in an `UPDATE`,
+  and never via `INSERT OR REPLACE` on these tables — that resets the column default and
+  un-removes the row. Re-crawling a removed row must refresh its content and leave the flag
+  where it was
+- **Filter at selection, not at write.** Every query that *selects* posts, notes or blogs
+  excludes removed rows. Update statements stay keyed on a row the caller already selected;
+  filtering them would spend API quota and then fail to persist the result
+- `Posts.IsActive` and `Notes.IsActive` are **optional** — they are absent from databases
+  that predate them, and naming a missing column is a hard SQLite error. Compose the filter
+  with `AndIsActive`/`WhereIsActive` in `DataAccess.cs`, which return
+  `COALESCE(IsActive, 1) = 1` only when `HasIsActiveColumn` finds the column. `Blogs.IsActive`
+  is not optional and is filtered directly
+- Do not add these columns from this app, and do not add them to the missing-column list in
+  `verify-db-schema.sql`
+
+### Testing
+- No existing test suite; use xUnit if adding tests
+- Test critical logic: `ApiKeyPool` init, color parsing, config persistence
+- Avoid testing one-off CLI workflows
+
+### Git Commit Messages
+- Imperative mood ("Add feature" not "Added feature")
+- Prefix with type: `feat:`, `fix:`, `chore:`, `docs:`
+- Keep messages under 72 characters
+- Never commit sensitive data (API keys/tokens)
+
+### API Key Color Rules
+- Unconfigured keys auto-assign colors from a preset palette
+- Auto-assigned colors persist to `appsettings.json`
+- All output for an active key uses its assigned color
+- Temporary color changes (e.g., errors) must restore the key's color afterward

RERUN.sqbpro

@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8"?><sqlb_project><db path="C:/Users/jim/Nextcloud/C#/URLNotesGrabberCORE/URLNotesGrabberCORE/TL.db" readonly="0" foreign_keys="1" case_sensitive_like="0" temp_store="0" wal_autocheckpoint="1000" synchronous="2"/><attached/><window><main_tabs open="structure browser pragmas query" current="3"/></window><tab_structure><column_width id="0" width="300"/><column_width id="1" width="0"/><column_width id="2" width="100"/><column_width id="3" width="4305"/><column_width id="4" width="0"/><expanded_item id="0" parent="1"/><expanded_item id="1" parent="1"/><expanded_item id="2" parent="1"/><expanded_item id="3" parent="1"/></tab_structure><tab_browse><table title="Posts" custom_title="0" dock_id="4" table="4,5:mainPosts"/><dock_state state="000000ff00000000fd00000001000000020000077200000379fc0100000006fb000000160064006f0063006b00420072006f00770073006500310100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500320100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500330100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500350100000000000005f40000000000000000fb000000160064006f0063006b00420072006f00770073006500340100000000000007720000011700fffffffb000000160064006f0063006b00420072006f00770073006500340100000000000005f40000000000000000000007720000000000000004000000040000000800000008fc00000000"/><default_encoding codec=""/><browse_table_settings><table schema="main" name="ApiKeyPoolMeta" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="29"/><column index="2" value="64"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Blogs" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="257"/><column index="2" value="95"/><column index="3" value="54"/><column index="4" value="156"/><column index="5" value="51"/><column index="6" value="71"/><column index="7" value="85"/><column index="8" value="156"/><column index="9" value="156"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Posts" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="28" mode="1"/></sort><column_widths><column index="1" value="241"/><column index="2" value="148"/><column index="3" value="126"/><column index="4" value="300"/><column index="5" value="75"/><column index="6" value="187"/><column index="7" value="159"/><column index="8" value="75"/><column index="9" value="300"/><column index="10" value="300"/><column index="11" value="78"/><column index="12" value="249"/><column index="13" value="300"/><column index="14" value="53"/><column index="15" value="300"/><column index="16" value="300"/><column index="17" value="41"/><column index="18" value="75"/><column index="19" value="96"/><column index="20" value="300"/><column index="21" value="96"/><column index="22" value="300"/><column index="23" value="300"/><column index="24" value="42"/><column index="25" value="60"/><column index="26" value="218"/><column index="27" value="920"/><column index="28" value="156"/><column index="29" value="156"/></column_widths><filter_values><column index="24" value="=1"/><column index="28" value="&gt;2026-05-06 20:00:01"/></filter_values><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table></browse_table_settings></tab_browse><tab_sql><sql name="SQL 1">UPDATE Posts
+SET HasNotesGathered = 0
+WHERE (BlogName, PostID) IN (
+    SELECT p.BlogName, p.PostID
+    FROM Posts p
+    WHERE p.HasNotesGathered = 1
+	AND P.notesGatheredDatetime &lt; 1774294520
+    AND EXISTS (
+        SELECT 1 
+        FROM Notes n 
+        WHERE n.PostID = p.PostID 
+          AND n.RootBlogName = p.BlogName
+          --AND n.Type NOT IN ('reblog', 'reply')
+    )
+    ORDER BY P.PostDate ASC
+    --LIMIT 500
+);</sql><sql name="Mark Blogs">select *
+from Blogs
+--update blogs set HasBeenOutput = 1
+where HasBeenOutput = 0
+AND
+blogname in
+(
+'teaberrybee',
+'reddevilgoddesstoo',
+'waywardog13',
+'wzjustbrowsing-blog',
+'lewerta',
+'nudenymph',
+'caylachief'
+
+)</sql><sql name="New Notes">select RootBlogName, PostID, NoteBlogName || '.tumblr.com' as NoteBlogName, DatetimeCrawled, TimeStamp, type, RootBlogName || '.tumblr.com/post/' || postid, datetime(timestamp, 'unixepoch')
+from Notes
+where 
+DatetimeCrawled &gt; '2026-05-14 02:50:05' --and type like 'r%'
+order by DatetimeCrawled desc</sql><sql name="Pull Blogs*">SELECT distinct␍
+	'''' || blogname || ''',',
+	blogs.*
+	, blogname || '.tumblr.com'
+FROM
+	Blogs
+		inner JOIN
+	Notes on notes.noteBlogName = blogs.BlogName
+WHERE
+	HasBeenOutput = 0 and type = 'reblog'
+order by
+	Notes.Type desc,
+	DateAdded desc
+LIMIT 100;</sql><sql name="SQL 7">WITH ReplyCounts AS (
+    SELECT 
+        NoteBlogName, 
+        COUNT(DISTINCT replyText) AS DistinctReplyCount
+    FROM Notes
+	where replyText &lt;&gt; '.'
+    GROUP BY NoteBlogName
+)
+SELECT 
+    n.RootBlogName || '.tumblr.com/post/' || n.PostID AS PostURL, postid,
+    n.NoteBlogName,
+    n.replyText,
+    c.DistinctReplyCount
+FROM Notes n
+JOIN ReplyCounts c ON n.NoteBlogName = c.NoteBlogName
+where replyText &lt;&gt; '.' and type &lt;&gt; 'reply'
+--AND N.NoteBlogName NOT IN ( 'roadblocker21', 'thesaddemon666', 'edwardabbeyhoffman', 'tattedsoldier20', 'zomb-eh', 'animalistic13', 'indken', 'maccloud1592',
+--'moss-wizard', 'supertrucker12682', 'exploringthrupics', 'padeyepete' )
+order by c.DistinctReplyCount desc, n.NoteBlogName, n.DateModified desc, replyText, RootBlogName, PostID</sql><current_tab id="3"/></tab_sql></sqlb_project>

TL.sqbpro

@@ -1,19 +1,24 @@
-<?xml version="1.0" encoding="UTF-8"?><sqlb_project><db path="D:/NextCloud/C#/URLNotesGrabberCORE/URLNotesGrabberCORE/bin/Debug/net8.0/TL.db" readonly="1" foreign_keys="1" case_sensitive_like="0" temp_store="0" wal_autocheckpoint="1000" synchronous="2"/><attached/><window><main_tabs open="structure browser pragmas query" current="3"/></window><tab_structure><column_width id="0" width="300"/><column_width id="1" width="0"/><column_width id="2" width="100"/><column_width id="3" width="3571"/><column_width id="4" width="0"/><expanded_item id="0" parent="1"/><expanded_item id="1" parent="1"/><expanded_item id="2" parent="1"/><expanded_item id="3" parent="1"/></tab_structure><tab_browse><table title="Blogs" custom_title="0" dock_id="1" table="4,5:mainBlogs"/><dock_state state="000000ff00000000fd0000000100000002000005f40000031dfc0100000001fb000000160064006f0063006b00420072006f00770073006500310100000000000005f4000000fb00ffffff000002130000000000000004000000040000000800000008fc00000000"/><default_encoding codec=""/><browse_table_settings><table schema="main" name="Blogs" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="0" mode="1"/></sort><column_widths><column index="1" value="257"/><column index="2" value="48"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table></browse_table_settings></tab_browse><tab_sql><sql name="SQL 1*">SELECT	␍
+<?xml version="1.0" encoding="UTF-8"?><sqlb_project><db path="" readonly="1" foreign_keys="" case_sensitive_like="" temp_store="" wal_autocheckpoint="" synchronous=""/><attached/><window><main_tabs open="structure browser pragmas query" current="3"/></window><tab_structure><column_width id="0" width="300"/><column_width id="1" width="0"/><column_width id="2" width="100"/><column_width id="3" width="3571"/><column_width id="4" width="0"/></tab_structure><tab_browse><table title="." custom_title="0" dock_id="4" table="0,0:"/><dock_state state="000000ff00000000fd0000000100000002000005f40000030ffc0100000002fb000000160064006f0063006b00420072006f00770073006500310100000000000005f40000000000000000fb000000160064006f0063006b00420072006f00770073006500340100000000ffffffff0000011700ffffff000005f40000000000000004000000040000000800000008fc00000000"/><default_encoding codec=""/><browse_table_settings/></tab_browse><tab_sql><sql name="SQL 1">SELECT	
-	BlogName || '.tumblr.com/post/' || postID,␍
+	BlogName || '.tumblr.com/post/' || postID,
-	␍
+	
-	datetime(NotesGatheredDateTime, 'unixepoch'), *␍
+	datetime(NotesGatheredDateTime, 'unixepoch'), *
-FROM␍
+FROM
-	Posts␍
+	Posts
-WHERE␍
+WHERE
-	NotesGatheredDateTime &lt;&gt; 0␍
+	NotesGatheredDateTime &lt;&gt; 0
-ORDER BY␍
+ORDER BY
-	postdate desc</sql><sql name="SQL 2*">SELECT	␍
+	postdate desc</sql><sql name="SQL 2*">SELECT	
-	datetime(TimeStamp, 'unixepoch'),␍
+	datetime(TimeStamp, 'unixepoch'),
-	RootBlogName || '.tumblr.com/post/' || postid,␍
+	RootBlogName || '.tumblr.com/post/' || N.postid,
-	*,␍
+	*,
-	NoteBlogName || '.tumblr.com'␍
+	NoteBlogName || '.tumblr.com'
-FROM␍
+FROM
-	Notes␍
+	Notes N␍
-WHERE	RootBlogName NOT IN ('xlittle-ghost', 'glimmerin-darlin')␍
+	inner JOIN␍
-ORDER BY␍
+	Posts P on P.PostID = N.PostID and P.BlogName = N.RootBlogName
-	TimeStamp desc</sql><current_tab id="0"/></tab_sql></sqlb_project>
+WHERE	RootBlogName NOT IN ('xlittle-ghost', 'glimmerin-darlin', 'vvenus-child')␍
+		and type like 'r%'␍
+		and RootBlogName = 'zomb-eh'␍
+		and P.HasImage = 1
+ORDER BY
+	TimeStamp desc</sql><current_tab id="1"/></tab_sql></sqlb_project>

URLNotesGrabberCORE/CorrectMode.cs

@@ -0,0 +1,277 @@
+using Microsoft.Extensions.Configuration;
+
+namespace URLNotesGrabberCORE
+{
+    // Port of ThreeTxtFileHelper RunCorrectionMode (dry-run) and RunFullCorrectionMode (apply).
+    // Scans a BAK directory of .txt files, parses posts with multi-line field support,
+    // and either reports or applies content-column corrections to TL.db.Posts.
+    // Apply path only writes non-empty values (mirrors original ThreeTxtFileHelper semantics)
+    // and never touches engagement columns.
+    public static class CorrectMode
+    {
+        public static int Run(IConfiguration config, string[] args, bool applyChanges)
+        {
+            DataAccess.EnsureTTFileHelperColumnsExist();
+
+            // Resolve BAK path: explicit arg > appSettings:PathTTBackup > derive from PathTTRoot/PathInput
+            string? bakRootPath = args.Length > 0 ? args[0] : config["appSettings:PathTTBackup"];
+            if (string.IsNullOrWhiteSpace(bakRootPath))
+            {
+                string? root = config["appSettings:PathTTRoot"];
+                if (string.IsNullOrWhiteSpace(root)) root = config["appSettings:PathInput"];
+                if (!string.IsNullOrWhiteSpace(root))
+                    bakRootPath = root.TrimEnd('\\', '/') + "_BAK\\";
+            }
+
+            if (string.IsNullOrWhiteSpace(bakRootPath) || !Directory.Exists(bakRootPath))
+            {
+                Console.WriteLine($"BAK directory not found: {bakRootPath}");
+                return 1;
+            }
+
+            Console.WriteLine($"BAK source path: {bakRootPath}");
+            Console.WriteLine(applyChanges
+                ? "Correction mode: APPLY - non-empty fields from BAK overwrite DB columns\n"
+                : "Correction mode: Dry-run - reports multi-line field updates available\n");
+
+            string prefixesPath = config["appSettings:PathPrefixes"] ?? "prefixes.txt";
+            if (!Path.IsPathRooted(prefixesPath))
+                prefixesPath = Path.Combine(AppContext.BaseDirectory, prefixesPath);
+            if (!File.Exists(prefixesPath))
+            {
+                Console.WriteLine($"Prefixes file not found: {prefixesPath}");
+                return 1;
+            }
+            var allowedPrefixes = new HashSet<string>(File.ReadLines(prefixesPath), StringComparer.OrdinalIgnoreCase);
+
+            if (applyChanges)
+            {
+                Console.Write("WARNING: This will overwrite non-empty fields in matching posts from BAK files. Continue? (yes/no): ");
+                string? response = Console.ReadLine();
+                if (string.IsNullOrWhiteSpace(response) || !response.Equals("yes", StringComparison.OrdinalIgnoreCase))
+                {
+                    Console.WriteLine("Operation cancelled.");
+                    return 0;
+                }
+            }
+
+            var bakTxtFiles = new List<string>();
+            try
+            {
+                foreach (var dir in Directory.GetDirectories(bakRootPath, "*", SearchOption.AllDirectories))
+                    bakTxtFiles.AddRange(Directory.GetFiles(dir, "*.txt"));
+                bakTxtFiles.AddRange(Directory.GetFiles(bakRootPath, "*.txt"));
+            }
+            catch (Exception ex)
+            {
+                Console.WriteLine($"Error scanning BAK directory: {ex.Message}");
+                return 1;
+            }
+            Console.WriteLine($"Found {bakTxtFiles.Count} file(s) in BAK directory\n");
+
+            int totalPostsFound = 0;
+            int postsWithUpdates = 0;
+            int postsUpdated = 0;
+            int postsNotFound = 0;
+            var correctionLog = new List<string>();
+            var updateLog = new List<string>();
+
+            foreach (string bakFile in bakTxtFiles)
+            {
+                Console.WriteLine($"Processing BAK file: {Path.GetFileName(bakFile)}");
+                try
+                {
+                    var bakPosts = ParsePostsFromFile(bakFile, allowedPrefixes);
+                    Console.WriteLine($"  Found {bakPosts.Count} post(s) in this file");
+
+                    foreach (var (postId, bakData) in bakPosts)
+                    {
+                        totalPostsFound++;
+                        var dbPost = DataAccess.GetPostByIdAnyBlog(postId);
+                        if (dbPost == null)
+                        {
+                            postsNotFound++;
+                            continue;
+                        }
+
+                        if (applyChanges)
+                        {
+                            bool updated = DataAccess.UpdatePostContentFields(dbPost.BlogName, dbPost.PostId, bakData);
+                            if (updated)
+                            {
+                                postsUpdated++;
+                                updateLog.Add($"Post ID: {postId} - Updated from {Path.GetFileName(bakFile)}");
+                            }
+                        }
+                        else
+                        {
+                            var updateList = BuildDryRunDiff(bakData, dbPost);
+                            if (updateList.Count > 0)
+                            {
+                                postsWithUpdates++;
+                                correctionLog.Add($"\nPost ID: {postId}");
+                                correctionLog.Add($"  File: {Path.GetFileName(bakFile)}");
+                                correctionLog.Add($"  Fields to update:");
+                                correctionLog.AddRange(updateList);
+                            }
+                        }
+                    }
+                }
+                catch (Exception ex)
+                {
+                    Console.WriteLine($"  ERROR processing file: {ex.Message}");
+                }
+            }
+
+            if (applyChanges)
+            {
+                Console.WriteLine($"\n========== CORRECTION COMPLETE ==========");
+                Console.WriteLine($"Total posts found in BAK files: {totalPostsFound}");
+                Console.WriteLine($"Posts updated in database:      {postsUpdated}");
+                Console.WriteLine($"Posts not found in database:    {postsNotFound}");
+
+                string logPath = config["appSettings:PathCorrectionApplied"] ?? "correction_applied.txt";
+                try
+                {
+                    var logLines = new List<string>
+                    {
+                        $"Correction Applied: {DateTime.Now:yyyy-MM-dd HH:mm:ss}",
+                        $"Total posts found in BAK files: {totalPostsFound}",
+                        $"Posts updated in database:      {postsUpdated}",
+                        $"Posts not found in database:    {postsNotFound}",
+                        "",
+                        "Updated Posts:"
+                    };
+                    logLines.AddRange(updateLog);
+                    File.WriteAllLines(logPath, logLines);
+                    Console.WriteLine($"Update log saved to: {logPath}");
+                }
+                catch (Exception ex)
+                {
+                    Console.WriteLine($"Error writing log file: {ex.Message}");
+                }
+            }
+            else
+            {
+                Console.WriteLine($"\n========== CORRECTION REPORT (DRY RUN) ==========");
+                Console.WriteLine($"Total posts found in BAK files: {totalPostsFound}");
+                Console.WriteLine($"Posts with multi-line field updates available: {postsWithUpdates}");
+
+                if (correctionLog.Count > 0)
+                {
+                    string logPath = config["appSettings:PathCorrectionReport"] ?? "correction_report.txt";
+                    try
+                    {
+                        File.WriteAllLines(logPath, correctionLog);
+                        Console.WriteLine($"\nDetailed report saved to: {logPath}");
+                    }
+                    catch (Exception ex)
+                    {
+                        Console.WriteLine($"Error writing report file: {ex.Message}");
+                    }
+                }
+                else
+                {
+                    Console.WriteLine("\nNo multi-line field updates found.");
+                }
+
+                Console.WriteLine("\nDry-run complete. No database changes were made.");
+                Console.WriteLine("If updates look correct, re-run with `-correct -apply` to apply changes.");
+            }
+            return 0;
+        }
+
+        private static List<string> BuildDryRunDiff(Dictionary<string, string> bakData, TTPostRecord dbPost)
+        {
+            var updates = new List<string>();
+            foreach (var (fieldName, bakValue) in bakData)
+            {
+                if (string.IsNullOrWhiteSpace(bakValue)) continue;
+                string? currentValue = fieldName.ToLowerInvariant() switch
+                {
+                    "reblog url" => dbPost.ReblogUrl,
+                    "date" => dbPost.Date,
+                    "has image" => dbPost.HasImage,
+                    "post url" => dbPost.PostUrl,
+                    "slug" => dbPost.Slug,
+                    "reblog key" => dbPost.ReblogKey,
+                    "reblog name" => dbPost.ReblogName,
+                    "summary" => dbPost.Summary,
+                    "quote" => dbPost.Quote,
+                    "body" => dbPost.Body,
+                    "tags" => dbPost.Tags,
+                    "link" => dbPost.Link,
+                    "photo url" => dbPost.PhotoUrl,
+                    "photo caption" => dbPost.PhotoCaption,
+                    "downloaded files" => dbPost.DownloadedFiles,
+                    "audio caption" => dbPost.AudioCaption,
+                    "question" => dbPost.Question,
+                    "answer" => dbPost.Answer,
+                    "title" => dbPost.Title,
+                    _ => null
+                };
+                if (bakValue != currentValue && bakValue.Contains('\n'))
+                    updates.Add($"    {fieldName}: [MULTILINE]");
+            }
+            return updates;
+        }
+
+        private static Dictionary<string, Dictionary<string, string>> ParsePostsFromFile(string filePath, HashSet<string> allowedPrefixes)
+        {
+            var posts = new Dictionary<string, Dictionary<string, string>>(StringComparer.OrdinalIgnoreCase);
+            var lines = File.ReadAllLines(filePath);
+            int lineIndex = 0;
+            string currentPostId = "";
+            var currentPostData = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
+
+            while (lineIndex < lines.Length)
+            {
+                string line = lines[lineIndex];
+                string searchText = line.Length > 25 ? line.Substring(0, 25) : line;
+                int colonIndex = searchText.IndexOf(": ");
+
+                if (colonIndex > 0)
+                {
+                    string prefix = line.Substring(0, colonIndex).Trim();
+                    if (!string.IsNullOrWhiteSpace(prefix) && allowedPrefixes.Contains(prefix))
+                    {
+                        if (string.Equals(prefix, "Post ID", StringComparison.OrdinalIgnoreCase))
+                        {
+                            if (!string.IsNullOrWhiteSpace(currentPostId) && currentPostData.Count > 0)
+                                posts[currentPostId] = new Dictionary<string, string>(currentPostData, StringComparer.OrdinalIgnoreCase);
+                            currentPostId = line.Substring(colonIndex + 2).Trim();
+                            currentPostData.Clear();
+                            lineIndex++;
+                            continue;
+                        }
+
+                        var valueLines = new List<string> { line.Substring(colonIndex + 2).Trim() };
+                        int nextLineIndex = lineIndex + 1;
+                        while (nextLineIndex < lines.Length)
+                        {
+                            string nextLine = lines[nextLineIndex];
+                            string nextSearch = nextLine.Length > 25 ? nextLine.Substring(0, 25) : nextLine;
+                            int nextColon = nextSearch.IndexOf(": ");
+                            if (nextColon > 0)
+                            {
+                                string nextPrefix = nextLine.Substring(0, nextColon).Trim();
+                                if (!string.IsNullOrWhiteSpace(nextPrefix) && allowedPrefixes.Contains(nextPrefix))
+                                    break;
+                            }
+                            valueLines.Add(nextLine);
+                            nextLineIndex++;
+                        }
+                        currentPostData[prefix] = string.Join("\n", valueLines);
+                        lineIndex = nextLineIndex;
+                        continue;
+                    }
+                }
+                lineIndex++;
+            }
+            if (!string.IsNullOrWhiteSpace(currentPostId) && currentPostData.Count > 0)
+                posts[currentPostId] = new Dictionary<string, string>(currentPostData, StringComparer.OrdinalIgnoreCase);
+
+            return posts;
+        }
+    }
+}

URLNotesGrabberCORE/IngestMode.cs

@@ -0,0 +1,219 @@
+using System.Text.RegularExpressions;
+using Microsoft.Extensions.Configuration;
+
+namespace URLNotesGrabberCORE
+{
+    // Port of ThreeTxtFileHelper RunIngestMode. Scans a root folder for .txt files,
+    // parses Tumblr-export fields (multi-line aware, prefix-driven), upserts each
+    // post into TL.db.Posts via DataAccess.UpsertPostFromTextFile.
+    public static class IngestMode
+    {
+        public static int Run(IConfiguration config, string[] args)
+        {
+            string? targetBlog = args.Length > 0 ? args[0]?.Trim() : null;
+            if (string.IsNullOrWhiteSpace(targetBlog)) targetBlog = null;
+
+            string? rootPath = config["appSettings:PathTTRoot"];
+            if (string.IsNullOrWhiteSpace(rootPath))
+                rootPath = config["appSettings:PathInput"];
+
+            if (string.IsNullOrWhiteSpace(rootPath))
+            {
+                Console.WriteLine("Ingest: no root path configured. Set appSettings:PathTTRoot or appSettings:PathInput.");
+                return 1;
+            }
+
+            if (!Directory.Exists(rootPath))
+            {
+                Console.WriteLine($"Directory not found: {rootPath}");
+                return 1;
+            }
+
+            DataAccess.EnsureTTFileHelperColumnsExist();
+
+            string prefixesPath = config["appSettings:PathPrefixes"] ?? "prefixes.txt";
+            if (!Path.IsPathRooted(prefixesPath))
+                prefixesPath = Path.Combine(AppContext.BaseDirectory, prefixesPath);
+
+            if (!File.Exists(prefixesPath))
+            {
+                Console.WriteLine($"Prefixes file not found: {prefixesPath}");
+                return 1;
+            }
+
+            var allowedPrefixes = new HashSet<string>(File.ReadLines(prefixesPath), StringComparer.OrdinalIgnoreCase);
+            Console.WriteLine($"Loaded {allowedPrefixes.Count} prefixes from {prefixesPath}");
+
+            Console.WriteLine($"========== Ingest Settings ==========");
+            Console.WriteLine($"Root path: {rootPath}");
+            Console.WriteLine($"Blog filter: {(targetBlog == null ? "(all blogs)" : targetBlog)}");
+            Console.WriteLine($"=====================================");
+
+            var txtFiles = new List<string>();
+            try
+            {
+                var dirs = Directory.GetDirectories(rootPath, "*", SearchOption.AllDirectories);
+                Console.WriteLine($"Found {dirs.Length} directories under root.");
+                foreach (var dir in dirs)
+                {
+                    try { txtFiles.AddRange(Directory.GetFiles(dir, "*.txt")); }
+                    catch (Exception ex) { Console.WriteLine($"  Skipping {dir}: {ex.Message}"); }
+                }
+                txtFiles.AddRange(Directory.GetFiles(rootPath, "*.txt"));
+            }
+            catch (Exception ex)
+            {
+                Console.WriteLine($"Error scanning root: {ex.Message}");
+                return 1;
+            }
+
+            Console.WriteLine($"Processing {txtFiles.Count} .txt file(s)...");
+
+            int filesProcessed = 0;
+            int filesSkipped = 0;
+            int postsTouched = 0;
+            string? lastBlogFolder = null;
+
+            try
+            {
+                DataAccess.EnableImportModePragmas();
+                DataAccess.BeginImportSession();
+
+                foreach (string file in txtFiles)
+                {
+                    try
+                    {
+                        string rawBlogName = Path.GetFileName(Path.GetDirectoryName(file) ?? "unknown");
+                        string blogName = Regex.Replace(rawBlogName, @"_\d+$", "");
+                        string postType = Path.GetFileNameWithoutExtension(file);
+
+                        if (targetBlog != null && !string.Equals(blogName, targetBlog, StringComparison.OrdinalIgnoreCase))
+                        {
+                            filesSkipped++;
+                            continue;
+                        }
+
+                        filesProcessed++;
+
+                        if (lastBlogFolder != rawBlogName)
+                        {
+                            Console.WriteLine($"[{filesProcessed}/{txtFiles.Count}] >> entering folder: {rawBlogName}");
+                            lastBlogFolder = rawBlogName;
+                        }
+                        else if (filesProcessed % 50 == 0)
+                        {
+                            Console.WriteLine($"[{filesProcessed}/{txtFiles.Count}] {rawBlogName}/{Path.GetFileName(file)}");
+                        }
+
+                        string currentPostId = "";
+                        var currentPostData = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
+
+                        void Flush()
+                        {
+                            if (!string.IsNullOrWhiteSpace(currentPostId) && currentPostData.Count > 0)
+                            {
+                                UpsertPostFromParsedData(blogName, currentPostId, postType, currentPostData);
+                                postsTouched++;
+                            }
+                        }
+
+                        var lines = File.ReadAllLines(file);
+                        int lineIndex = 0;
+                        while (lineIndex < lines.Length)
+                        {
+                            string line = lines[lineIndex];
+                            string searchText = line.Length > 25 ? line.Substring(0, 25) : line;
+                            int colonIndex = searchText.IndexOf(": ");
+
+                            if (colonIndex > 0)
+                            {
+                                string prefix = line.Substring(0, colonIndex).Trim();
+                                if (!string.IsNullOrWhiteSpace(prefix) && allowedPrefixes.Contains(prefix))
+                                {
+                                    if (string.Equals(prefix, "Post ID", StringComparison.OrdinalIgnoreCase))
+                                    {
+                                        Flush();
+                                        currentPostId = line.Substring(colonIndex + 2).Trim();
+                                        currentPostData.Clear();
+                                        lineIndex++;
+                                        continue;
+                                    }
+
+                                    var valueLines = new List<string> { line.Substring(colonIndex + 2).Trim() };
+                                    int nextLineIndex = lineIndex + 1;
+                                    while (nextLineIndex < lines.Length)
+                                    {
+                                        string nextLine = lines[nextLineIndex];
+                                        string nextSearch = nextLine.Length > 25 ? nextLine.Substring(0, 25) : nextLine;
+                                        int nextColon = nextSearch.IndexOf(": ");
+                                        if (nextColon > 0)
+                                        {
+                                            string nextPrefix = nextLine.Substring(0, nextColon).Trim();
+                                            if (!string.IsNullOrWhiteSpace(nextPrefix) && allowedPrefixes.Contains(nextPrefix))
+                                                break;
+                                        }
+                                        valueLines.Add(nextLine);
+                                        nextLineIndex++;
+                                    }
+
+                                    currentPostData[prefix] = string.Join("\n", valueLines);
+                                    lineIndex = nextLineIndex;
+                                    continue;
+                                }
+                            }
+                            lineIndex++;
+                        }
+
+                        Flush();
+                    }
+                    catch (Exception ex)
+                    {
+                        Console.WriteLine($"  ERROR processing file {file}: {ex.Message}");
+                    }
+                }
+            }
+            finally
+            {
+                DataAccess.EndImportSession();
+                DataAccess.RestoreImportModePragmas();
+            }
+
+            Console.WriteLine($"\nIngest complete. Files processed: {filesProcessed}. Files skipped (blog filter): {filesSkipped}. Posts touched: {postsTouched}.");
+            return 0;
+        }
+
+        private static void UpsertPostFromParsedData(string blogName, string postId, string postType, Dictionary<string, string> data)
+        {
+            string? G(string key) => data.TryGetValue(key, out var v) ? v : null;
+            string? hasImageStr = G("Has Image");
+            bool hasImage = !string.IsNullOrWhiteSpace(hasImageStr)
+                && (hasImageStr.Equals("true", StringComparison.OrdinalIgnoreCase)
+                    || hasImageStr == "1"
+                    || hasImageStr.Equals("yes", StringComparison.OrdinalIgnoreCase));
+
+            DataAccess.UpsertPostFromTextFile(
+                blogName: blogName,
+                postID: postId,
+                reblogURL: G("reblog URL"),
+                postDate: G("Date"),
+                postURL: G("Post URL"),
+                slug: G("Slug"),
+                reblogKey: G("Reblog Key"),
+                reblogName: G("Reblog Name"),
+                summary: G("Summary"),
+                quote: G("Quote"),
+                body: G("Body"),
+                tags: G("Tags"),
+                link: G("Link"),
+                photoURL: G("Photo URL"),
+                photoCaption: G("Photo Caption"),
+                downloadedFiles: G("Downloaded Files"),
+                audioCaption: G("Audio Caption"),
+                question: G("Question"),
+                answer: G("Answer"),
+                title: G("Title"),
+                postType: postType,
+                hasImage: hasImage);
+        }
+    }
+}

URLNotesGrabberCORE/LegacyPostsDbImporter.cs

@@ -0,0 +1,147 @@
+using System.Data.SQLite;
+
+namespace URLNotesGrabberCORE
+{
+    // One-time migration: opens a legacy ThreeTxtFileHelper posts.db, copies its
+    // Blog + PostData rows into the merged TL.db via DataAccess.
+    // Conflict rule on (BlogName, PostId): ThreeTxtFileHelper wins on the 22 content
+    // columns + PostType + DateModified (handled inside UpsertPostFromTextFile).
+    // Engagement columns in TL.db (ByLikes, RootBlogName, RootURL, HasNotesGathered,
+    // NotFound, NotesGatheredDateTime) are preserved.
+    public static class LegacyPostsDbImporter
+    {
+        public static int Run(string legacyDbPath)
+        {
+            if (string.IsNullOrWhiteSpace(legacyDbPath))
+            {
+                Console.WriteLine("LegacyPostsDbImporter: path to legacy posts.db is required.");
+                return 1;
+            }
+
+            if (!File.Exists(legacyDbPath))
+            {
+                Console.WriteLine($"Legacy posts.db not found at: {legacyDbPath}");
+                return 1;
+            }
+
+            DataAccess.EnsureTTFileHelperColumnsExist();
+
+            Console.WriteLine($"Reading legacy posts.db: {legacyDbPath}");
+
+            int blogsCopied = 0;
+            int postsUpserted = 0;
+            int errors = 0;
+
+            try
+            {
+                using var src = new SQLiteConnection("Data Source=" + legacyDbPath + ";Read Only=True;");
+                src.Open();
+
+                // 1) Copy Blogs (BlogName + TTFolderPath)
+                using (var cmd = new SQLiteCommand("SELECT BlogName, TTFolderPath FROM Blogs", src))
+                using (var reader = cmd.ExecuteReader())
+                {
+                    while (reader.Read())
+                    {
+                        string blogName = reader.IsDBNull(0) ? string.Empty : reader.GetString(0);
+                        string? ttFolderPath = reader.IsDBNull(1) ? null : reader.GetString(1);
+                        if (string.IsNullOrWhiteSpace(blogName)) continue;
+                        try
+                        {
+                            DataAccess.SetBlogTTFolderPath(blogName, ttFolderPath);
+                            blogsCopied++;
+                        }
+                        catch (Exception ex)
+                        {
+                            errors++;
+                            Console.WriteLine($"  Blog copy failed for '{blogName}': {ex.Message}");
+                        }
+                    }
+                }
+                Console.WriteLine($"  Blogs copied: {blogsCopied}");
+
+                // 2) Copy Posts
+                try
+                {
+                    DataAccess.EnableImportModePragmas();
+                    DataAccess.BeginImportSession();
+
+                    string sql = @"SELECT BlogName, PostId, ReblogUrl, Date, HasImage, PostUrl, Slug,
+                                          ReblogKey, ReblogName, Summary, Quote, Body, Tags, Link,
+                                          PhotoUrl, PhotoCaption, DownloadedFiles, AudioCaption,
+                                          Question, Answer, Title, PostType
+                                   FROM Posts";
+
+                    using var cmd = new SQLiteCommand(sql, src);
+                    using var reader = cmd.ExecuteReader();
+                    while (reader.Read())
+                    {
+                        try
+                        {
+                            string blogName = reader.IsDBNull(0) ? string.Empty : reader.GetString(0);
+                            string postId = reader.IsDBNull(1) ? string.Empty : reader.GetString(1);
+                            if (string.IsNullOrWhiteSpace(blogName) || string.IsNullOrWhiteSpace(postId)) continue;
+
+                            string? hasImageRaw = reader.IsDBNull(4) ? null : reader.GetValue(4)?.ToString();
+                            bool hasImage = !string.IsNullOrWhiteSpace(hasImageRaw)
+                                && (hasImageRaw.Equals("true", StringComparison.OrdinalIgnoreCase)
+                                    || hasImageRaw == "1"
+                                    || hasImageRaw.Equals("yes", StringComparison.OrdinalIgnoreCase));
+
+                            DataAccess.UpsertPostFromTextFile(
+                                blogName: blogName,
+                                postID: postId,
+                                reblogURL: reader.IsDBNull(2) ? null : reader.GetString(2),
+                                postDate: reader.IsDBNull(3) ? null : reader.GetString(3),
+                                postURL: reader.IsDBNull(5) ? null : reader.GetString(5),
+                                slug: reader.IsDBNull(6) ? null : reader.GetString(6),
+                                reblogKey: reader.IsDBNull(7) ? null : reader.GetString(7),
+                                reblogName: reader.IsDBNull(8) ? null : reader.GetString(8),
+                                summary: reader.IsDBNull(9) ? null : reader.GetString(9),
+                                quote: reader.IsDBNull(10) ? null : reader.GetString(10),
+                                body: reader.IsDBNull(11) ? null : reader.GetString(11),
+                                tags: reader.IsDBNull(12) ? null : reader.GetString(12),
+                                link: reader.IsDBNull(13) ? null : reader.GetString(13),
+                                photoURL: reader.IsDBNull(14) ? null : reader.GetString(14),
+                                photoCaption: reader.IsDBNull(15) ? null : reader.GetString(15),
+                                downloadedFiles: reader.IsDBNull(16) ? null : reader.GetString(16),
+                                audioCaption: reader.IsDBNull(17) ? null : reader.GetString(17),
+                                question: reader.IsDBNull(18) ? null : reader.GetString(18),
+                                answer: reader.IsDBNull(19) ? null : reader.GetString(19),
+                                title: reader.IsDBNull(20) ? null : reader.GetString(20),
+                                postType: reader.IsDBNull(21) ? null : reader.GetString(21),
+                                hasImage: hasImage);
+                            postsUpserted++;
+                            if (postsUpserted % 500 == 0)
+                                Console.WriteLine($"  ... {postsUpserted} posts upserted");
+                        }
+                        catch (Exception ex)
+                        {
+                            errors++;
+                            if (errors < 20)
+                                Console.WriteLine($"  Post upsert error: {ex.Message}");
+                        }
+                    }
+                }
+                finally
+                {
+                    DataAccess.EndImportSession();
+                    DataAccess.RestoreImportModePragmas();
+                }
+
+                Console.WriteLine($"  Posts upserted: {postsUpserted}");
+            }
+            catch (Exception ex)
+            {
+                Console.WriteLine($"Fatal error reading legacy posts.db: {ex.Message}");
+                return 1;
+            }
+
+            Console.WriteLine($"\n========== Legacy import summary ==========");
+            Console.WriteLine($"Blogs copied:    {blogsCopied}");
+            Console.WriteLine($"Posts upserted:  {postsUpserted}");
+            Console.WriteLine($"Errors:          {errors}");
+            return errors == 0 ? 0 : 2;
+        }
+    }
+}

URLNotesGrabberCORE/OutputMode.cs

@@ -0,0 +1,136 @@
+using Microsoft.Extensions.Configuration;
+
+namespace URLNotesGrabberCORE
+{
+    // Port of ThreeTxtFileHelper RunOutputMode + WritePostToFile + RenameExistingTxtFilesToBak.
+    // For each Blog with a TTFolderPath, renames any existing .txt files in that folder to .bak,
+    // then writes one .txt per PostType containing all posts of that type (date-sorted, fixed
+    // field order). Reads from TL.db via DataAccess.GetAllPostsForBlog.
+    public static class OutputMode
+    {
+        public static int Run(IConfiguration config)
+        {
+            DataAccess.EnsureTTFileHelperColumnsExist();
+
+            var blogs = DataAccess.GetAllBlogsWithTTFolderPath();
+            Console.WriteLine($"Found {blogs.Count} blog(s) to process.");
+
+            foreach (var (blogName, ttFolderPath) in blogs)
+            {
+                Console.WriteLine($"\nProcessing blog: {blogName}");
+
+                if (string.IsNullOrWhiteSpace(ttFolderPath) || !Directory.Exists(ttFolderPath))
+                {
+                    Console.WriteLine($"  TTFolderPath does not exist or is not set. Skipping.");
+                    continue;
+                }
+
+                Console.WriteLine($"  TTFolderPath: {ttFolderPath}");
+
+                try
+                {
+                    foreach (var bakFile in Directory.GetFiles(ttFolderPath, "*.bak"))
+                        File.Delete(bakFile);
+                }
+                catch (Exception ex)
+                {
+                    Console.WriteLine($"  Error deleting .bak files: {ex.Message}");
+                }
+
+                RenameExistingTxtFilesToBak(ttFolderPath);
+
+                var posts = DataAccess.GetAllPostsForBlog(blogName);
+                Console.WriteLine($"  Found {posts.Count} post(s) for this blog.");
+
+                var grouped = posts.GroupBy(p => p.PostType ?? "Unknown");
+                foreach (var typeGroup in grouped)
+                {
+                    string postType = typeGroup.Key ?? "Unknown";
+                    string outputFilePath = Path.Combine(ttFolderPath, $"{postType}.txt");
+                    var ordered = typeGroup.OrderBy(p => p.Date).ToList();
+                    Console.WriteLine($"    Writing {ordered.Count} post(s) to {postType}.txt");
+
+                    using var writer = new StreamWriter(outputFilePath, false, System.Text.Encoding.UTF8);
+                    bool isFirst = true;
+                    foreach (var post in ordered)
+                    {
+                        if (!isFirst)
+                        {
+                            writer.WriteLine();
+                            writer.WriteLine();
+                        }
+                        WritePostToFile(writer, post);
+                        isFirst = false;
+                    }
+                }
+            }
+
+            Console.WriteLine("\nOutput mode complete.");
+            return 0;
+        }
+
+        private static void RenameExistingTxtFilesToBak(string folderPath)
+        {
+            try
+            {
+                foreach (var txtFile in Directory.GetFiles(folderPath, "*.txt"))
+                {
+                    string bakPath = Path.ChangeExtension(txtFile, ".bak");
+                    if (File.Exists(bakPath)) File.Delete(bakPath);
+                    File.Move(txtFile, bakPath, overwrite: true);
+                }
+            }
+            catch (Exception ex)
+            {
+                Console.WriteLine($"    Error renaming txt files to .bak: {ex.Message}");
+            }
+        }
+
+        private static void WritePostToFile(StreamWriter writer, TTPostRecord post)
+        {
+            var startColumns = new[] { "Post ID", "Date", "Post URL", "Slug", "Reblog Key", "Reblog URL", "Reblog Name", "Title", "Body" };
+            var endColumns = new[] { "Tags", "Downloaded Files" };
+
+            var columns = new Dictionary<string, string>();
+            if (!string.IsNullOrWhiteSpace(post.PostId)) columns["Post ID"] = post.PostId;
+            if (!string.IsNullOrWhiteSpace(post.Date)) columns["Date"] = post.Date!;
+            if (!string.IsNullOrWhiteSpace(post.PostUrl)) columns["Post URL"] = post.PostUrl!;
+            if (!string.IsNullOrWhiteSpace(post.Slug)) columns["Slug"] = post.Slug!;
+            if (!string.IsNullOrWhiteSpace(post.ReblogKey)) columns["Reblog Key"] = post.ReblogKey!;
+            if (!string.IsNullOrWhiteSpace(post.ReblogUrl)) columns["Reblog URL"] = post.ReblogUrl!;
+            if (!string.IsNullOrWhiteSpace(post.ReblogName)) columns["Reblog Name"] = post.ReblogName!;
+            if (!string.IsNullOrWhiteSpace(post.Title)) columns["Title"] = post.Title!;
+            if (!string.IsNullOrWhiteSpace(post.Body)) columns["Body"] = post.Body!;
+            if (!string.IsNullOrWhiteSpace(post.HasImage)) columns["Has Image"] = post.HasImage!;
+            if (!string.IsNullOrWhiteSpace(post.Summary)) columns["Summary"] = post.Summary!;
+            if (!string.IsNullOrWhiteSpace(post.Quote)) columns["Quote"] = post.Quote!;
+            if (!string.IsNullOrWhiteSpace(post.Link)) columns["Link"] = post.Link!;
+            if (!string.IsNullOrWhiteSpace(post.PhotoUrl)) columns["Photo URL"] = post.PhotoUrl!;
+            if (!string.IsNullOrWhiteSpace(post.PhotoCaption)) columns["Photo Caption"] = post.PhotoCaption!;
+            if (!string.IsNullOrWhiteSpace(post.AudioCaption)) columns["Audio Caption"] = post.AudioCaption!;
+            if (!string.IsNullOrWhiteSpace(post.Question)) columns["Question"] = post.Question!;
+            if (!string.IsNullOrWhiteSpace(post.Answer)) columns["Answer"] = post.Answer!;
+            if (!string.IsNullOrWhiteSpace(post.Tags)) columns["Tags"] = post.Tags!;
+            if (!string.IsNullOrWhiteSpace(post.DownloadedFiles)) columns["Downloaded Files"] = post.DownloadedFiles!;
+
+            foreach (var col in startColumns)
+            {
+                if (columns.ContainsKey(col))
+                {
+                    writer.WriteLine($"{col}: {columns[col]}");
+                    columns.Remove(col);
+                }
+            }
+
+            var remaining = columns.Keys.Where(k => !endColumns.Contains(k)).OrderBy(k => k).ToList();
+            foreach (var col in remaining)
+                writer.WriteLine($"{col}: {columns[col]}");
+
+            foreach (var col in endColumns)
+            {
+                if (columns.ContainsKey(col))
+                    writer.WriteLine($"{col}: {columns[col]}");
+            }
+        }
+    }
+}

URLNotesGrabberCORE/Properties/launchSettings.json

@@ -2,7 +2,7 @@
   "profiles": {
     "URLNotesGrabberCORE": {
       "commandName": "Project",
-      "commandLineArgs": "-collect 0"
+      "commandLineArgs": "--collect 1 --api4"
     }
   }
 }

URLNotesGrabberCORE/ResponseNotes.cs

@@ -5,12 +5,12 @@ using System.Text;
 using System.Threading.Tasks;
 using Newtonsoft.Json;
 
-
 namespace URLNotesGrabberCORE
 {
     internal class ResponseNotes
     {
     }
+
     // Root myDeserializedClass = JsonConvert.DeserializeObject<Root>(myJsonResponse);
     public class AvatarUrl
     {
@@ -42,7 +42,7 @@ namespace URLNotesGrabberCORE
     public class Note
     {
         public string type { get; set; }
-        public int timestamp { get; set; }
+        public long timestamp { get; set; }
         public string blog_name { get; set; }
         public string blog_uuid { get; set; }
         public string blog_url { get; set; }
@@ -51,6 +51,7 @@ namespace URLNotesGrabberCORE
         public AvatarUrl avatar_url { get; set; }
         public string post_id { get; set; }
         public string reblog_parent_blog_name { get; set; }
+        public string reply_text { get; set; }
     }
 
     public class QueryParams
@@ -58,6 +59,8 @@ namespace URLNotesGrabberCORE
         public string mode { get; set; }
         public string id { get; set; }
         public string before_timestamp { get; set; }
+        public string before { get; set; }
+        public string after { get; set; }
     }
 
     public class Response
@@ -82,5 +85,60 @@ namespace URLNotesGrabberCORE
         public int retryInSeconds { get; set; }
 
         public string rawJson { get; set; }
+
+        // The request never reached the Tumblr API (transport error, or an edge/CDN response with a
+        // non-JSON body). Says nothing about the post, so the caller should retry rather than fail it.
+        public bool transientFailure { get; set; }
+    }
+
+    // Classes for Posts API endpoint (for reply_text)
+    public class PostsResponse
+    {
+        public List<Post> posts { get; set; }
+    }
+
+    public class Post
+    {
+        public long id { get; set; }
+        public List<NoteDetail> notes { get; set; }
+    }
+
+    public class NoteDetail
+    {
+        public long timestamp { get; set; }
+        public string type { get; set; }
+        public string blog_name { get; set; }
+        public string reply_text { get; set; }
+    }
+
+    public class PostsRoot
+    {
+        public Meta meta { get; set; }
+        [JsonConverter(typeof(EmptyArrayOrObjectConverter<PostsResponse>))]
+        public PostsResponse response { get; set; }
+
+        public string statusCode { get; set; }
+
+        public int retryInSeconds { get; set; }
+
+        public string rawJson { get; set; }
+    }
+
+    public class LikesResponse
+    {
+        public List<dynamic> liked_posts { get; set; }
+        public int liked_count { get; set; }
+        public Links _links { get; set; }
+    }
+
+    public class LikesRoot
+    {
+        public Meta meta { get; set; }
+        [JsonConverter(typeof(EmptyArrayOrObjectConverter<LikesResponse>))]
+        public LikesResponse response { get; set; }
+
+        public string statusCode { get; set; }
+        public int retryInSeconds { get; set; }
+        public string rawJson { get; set; }
     }
 }

URLNotesGrabberCORE/RevertMode.cs

@@ -0,0 +1,110 @@
+using Microsoft.Extensions.Configuration;
+
+namespace URLNotesGrabberCORE
+{
+    // Inverse of OutputMode. Recursively walks the PathInput tree (the same directory tree the
+    // no-parameter run uses) and restores every *.bak back to its *.txt, first preserving the
+    // current *.txt as the next-free *.bkN. Consumes the *.bak (File.Move). Filesystem-only;
+    // does not read the DB. An optional blogname argument filters by path substring.
+    public static class RevertMode
+    {
+        public static int Run(IConfiguration config, string? blogFilter = null)
+        {
+            string? root = config["appSettings:PathInput"];
+            if (string.IsNullOrWhiteSpace(root) || !Directory.Exists(root))
+            {
+                Console.WriteLine($"PathInput is not set or does not exist: '{root}'. Nothing to revert.");
+                return 0;
+            }
+
+            Console.WriteLine($"Searching for .bak files under: {root}");
+
+            // Recursively collect every *.bak, optionally filtered by path substring (blogname).
+            var bakFiles = EnumerateBakFiles(root)
+                .Where(f => string.IsNullOrWhiteSpace(blogFilter)
+                            || f.IndexOf(blogFilter, StringComparison.OrdinalIgnoreCase) >= 0)
+                .ToList();
+
+            if (bakFiles.Count == 0)
+            {
+                Console.WriteLine("No .bak files found. Nothing to revert.");
+                return 0;
+            }
+
+            Console.Write($"WARNING: This will restore {bakFiles.Count} .bak file(s) over their .txt files. " +
+                          $"Current .txt files are preserved as the next-free .bkN. Continue? (yes/no): ");
+            string? response = Console.ReadLine();
+            if (string.IsNullOrWhiteSpace(response) || !response.Equals("yes", StringComparison.OrdinalIgnoreCase))
+            {
+                Console.WriteLine("Operation cancelled.");
+                return 0;
+            }
+
+            int restored = 0, backedUp = 0;
+            foreach (var bakFile in bakFiles)
+            {
+                try
+                {
+                    string txtPath = Path.ChangeExtension(bakFile, ".txt");
+
+                    if (File.Exists(txtPath))
+                    {
+                        string bkPath = NextFreeBkPath(txtPath);
+                        File.Move(txtPath, bkPath);
+                        backedUp++;
+                        Console.WriteLine($"  Backed up {Path.GetFileName(txtPath)} -> {Path.GetFileName(bkPath)}");
+                    }
+
+                    File.Move(bakFile, txtPath);
+                    restored++;
+                    Console.WriteLine($"  Restored {bakFile} -> {Path.GetFileName(txtPath)}");
+                }
+                catch (Exception ex)
+                {
+                    Console.WriteLine($"  Error reverting {bakFile}: {ex.Message}");
+                }
+            }
+
+            Console.WriteLine($"\nRevert mode complete. Restored {restored} file(s); backed up {backedUp} current .txt file(s).");
+            return 0;
+        }
+
+        // Recursively yields every *.bak path under root. Per-directory try/catch so an
+        // inaccessible folder doesn't abort the whole walk (mirrors TraverseDirectory).
+        private static IEnumerable<string> EnumerateBakFiles(string path)
+        {
+            string[] subDirs;
+            try { subDirs = Directory.GetDirectories(path); }
+            catch (Exception ex)
+            {
+                Console.WriteLine($"  Skipping '{path}': {ex.Message}");
+                yield break;
+            }
+
+            foreach (var dir in subDirs)
+                foreach (var bak in EnumerateBakFiles(dir))
+                    yield return bak;
+
+            string[] bakFiles;
+            try { bakFiles = Directory.GetFiles(path, "*.bak"); }
+            catch (Exception ex)
+            {
+                Console.WriteLine($"  Skipping files in '{path}': {ex.Message}");
+                yield break;
+            }
+
+            foreach (var bak in bakFiles)
+                yield return bak;
+        }
+
+        // Returns the lowest unused .bkN path for a given .txt file (.bk1, .bk2, ...).
+        private static string NextFreeBkPath(string txtFile)
+        {
+            for (int n = 1; ; n++)
+            {
+                string candidate = Path.ChangeExtension(txtFile, $".bk{n}");
+                if (!File.Exists(candidate)) return candidate;
+            }
+        }
+    }
+}

URLNotesGrabberCORE/TL.db.md

@@ -0,0 +1,333 @@
+# `TL.db` — schema notes
+
+The SQLite database behind **URLNotesGrabberCORE** and its sibling crawlers, and the one
+[Rolodex](https://git.basso.land/jim/Rolodex) reads.
+
+Everything below was read out of the live file, not inferred from code. Counts are as of
+**2026-07-29**; re-run the queries at the bottom to refresh them.
+
+- Journal mode: **WAL** — `TL.db-wal` and `TL.db-shm` live beside the file and are part of
+  the database. Copying `TL.db` alone gives you whatever was last checkpointed, not the
+  current state.
+- Page size: 4096.
+
+---
+
+## The three content tables
+
+| Table | Rows | What it is |
+|---|--:|---|
+| `Blogs` | 144,367 | The crawl registry — one row per known blog, plus crawl-state flags |
+| `Posts` | 14,589 | Stored post content. Only 3,602 blogs actually have any |
+| `Notes` | 1,189,604 | The engagement graph: `NoteBlogName` acted on `(RootBlogName, PostID)` |
+
+The engagement graph is the interesting part. 31,888 distinct blogs appear as engagers —
+far more than the 3,602 that have stored posts — which is what makes this a social graph
+rather than a post archive.
+
+### `Blogs`
+
+```sql
+CREATE TABLE "Blogs" (
+    "BlogName"              TEXT,
+    "HasBeenOutput"         INTEGER DEFAULT 0,
+    "IsActive"              INTEGER DEFAULT 1,
+    "DateAdded"             TEXT NOT NULL DEFAULT '12/24/25',
+    "ByLikes"               INTEGER NOT NULL DEFAULT 0,
+    "LikesPulled"           INTEGER NOT NULL DEFAULT 0,
+    "LikesCursor"           INTEGER DEFAULT 0,
+    "DateModified"          TEXT,
+    "DateCreated"           TEXT,
+    LikesNewestTimestamp    INTEGER DEFAULT 0,
+    LikesLastRefreshed      INTEGER DEFAULT 0,
+    LikesLastNewCount       INTEGER DEFAULT 0,
+    TTFolderPath            TEXT,
+    PRIMARY KEY("BlogName")
+);
+```
+
+`BlogName` is the primary key, so it is the only indexed way in. There is no index on any
+flag or date — filtering or sorting on those scans all 144k rows, which is affordable
+here and is not on `Notes`.
+
+Flag distribution: `IsActive = 1` on 144,366 of 144,367 rows, `HasBeenOutput = 1` on
+5,369, `ByLikes = 1` on 2. `IsActive` carries a second meaning as of Rolodex — see
+[`Blogs.IsActive`](#blogsisactive--now-written-by-two-applications) below.
+
+The columns after `DateCreated` were added later by `ALTER TABLE`, which is why they carry
+no quoting in the stored DDL. That is the normal way this schema grows.
+
+**`DateAdded` is not written consistently.** 126,423 rows hold ISO `yyyy-MM-dd HH:mm:ss`;
+17,944 hold US-format `M/d/yy` from a bulk import. As text those two sort into different
+parts of the table, so anything ordering or range-filtering on this column has to
+normalise first — see `DateSql` in Rolodex.
+
+### `Posts`
+
+```sql
+CREATE TABLE "Posts" (
+    "BlogName"              TEXT,
+    "PostID"                INTEGER,
+    "HasNotesGathered"      INTEGER DEFAULT 0,
+    "reblogURL"             TEXT,
+    "NotFound"              INTEGER DEFAULT 0,
+    "PostDate"              TEXT,
+    "NotesGatheredDateTime" INTEGER NOT NULL DEFAULT 1729746000,
+    "HasImage"              INTEGER NOT NULL DEFAULT 0,
+    "PostURL"               TEXT,
+    "Slug"                  TEXT,
+    "ReblogKey"             TEXT,
+    "ReblogName"            TEXT,
+    "Summary"               TEXT,
+    "Quote"                 TEXT,
+    "Body"                  TEXT,
+    "Tags"                  TEXT,
+    "Link"                  TEXT,
+    "PhotoURL"              TEXT,
+    "PhotoCaption"          TEXT,
+    "DownloadedFiles"       TEXT,
+    "AudioCaption"          TEXT,
+    "Question"              TEXT,
+    "Answer"                TEXT,
+    "Title"                 TEXT,
+    "ByLikes"               INTEGER NOT NULL DEFAULT 0,
+    "RootBlogName"          TEXT,
+    "RootURL"               TEXT,
+    "DateModified"          TEXT,
+    "DateCreated"           TEXT,
+    PostType                TEXT,
+    PRIMARY KEY("BlogName","PostID")
+);
+```
+
+**The key is `(BlogName, PostID)`, not `PostID`.** This matters more than it looks: 325
+post IDs exist under more than one blog, so an ID on its own is both ambiguous *and*
+unindexed. Any lookup should carry the blog name, and a batch lookup should group by blog
+so it stays on the leading column of the key.
+
+Notable:
+
+- **`PostType` is `NULL` on all 14,589 rows.** The column exists but nothing has ever
+  populated it. Treat it as unpopulated rather than as a type discriminator.
+- `HasImage = 1` on 14,268 rows — nearly all of them. It records that the post *had* a
+  picture, not that a usable URL was kept, so it is not a reliable predictor that anything
+  will render.
+- `PhotoURL` is largely unused; in practice the image markup lives inside `Body`.
+- `NotFound = 1` on 4,663 rows — posts that have since been deleted upstream.
+- The content columns (`Body`, `Quote`, `Question`, `Answer`, …) are the heavy ones. List
+  views should not select them.
+
+### `Notes`
+
+```sql
+CREATE TABLE "Notes" (
+    "RootBlogName"    TEXT,
+    "PostID"          INTEGER,
+    "NoteBlogName"    TEXT,
+    "TimeStamp"       INTEGER,
+    "Type"            TEXT,
+    "replyText"       TEXT DEFAULT '.',
+    "DatetimeCrawled" TEXT DEFAULT '2/12/26 12am',
+    "DateModified"    TEXT,
+    "DateCreated"     TEXT,
+    PRIMARY KEY("RootBlogName","PostID","TimeStamp","Type","NoteBlogName")
+);
+
+CREATE INDEX "Notes_idx_06e01ae3" ON "Notes" ("TimeStamp" DESC);
+CREATE INDEX "ix_NoteBlogName01"  ON "Notes" ("NoteBlogName");
+```
+
+One row per engagement event. `TimeStamp` is **unix seconds** — unlike every date column
+elsewhere in the schema, which are text.
+
+| `Type` | Rows | Share |
+|---|--:|--:|
+| `like` | 947,955 | 79.7% |
+| `reblog` | 224,323 | 18.9% |
+| `reply` | 15,201 | 1.3% |
+| `posted` | 2,106 | 0.2% |
+| `post_attribution` | 19 | — |
+
+At 1.19M rows this is the table that dictates how the whole database has to be queried:
+
+- **Nothing should run an unbounded `SELECT` or a bare `COUNT(*)` here.** A count scans
+  the lot on every call.
+- The only fast access paths are the primary key's leading columns (`RootBlogName`, then
+  `PostID`) and `ix_NoteBlogName01` on `NoteBlogName`. "Notes received by a blog" and
+  "notes given by a blog" are both cheap; almost nothing else is.
+- Ordering by anything but `TimeStamp` is a full sort of whatever the filters leave.
+- `replyText` is `'.'` on 1,174,706 rows — only `reply` notes carry real text.
+
+### Referential integrity
+
+There are no foreign keys, and the tables do not perfectly agree:
+
+- 4 `Posts` rows name a blog with no `Blogs` row.
+- 15 of the 31,888 distinct engagers have no `Blogs` row.
+
+So a name appearing in `Notes` or `Posts` is not a guarantee that the registry knows about
+it. Joins from those tables back to `Blogs` should tolerate a miss.
+
+---
+
+## The `'.'` placeholder convention
+
+**The crawler writes a single dot into text columns it has no value for, rather than
+`NULL`.** This is the single most surprising thing about the schema and it affects every
+consumer.
+
+| Column | `'.'` rows |
+|---|--:|
+| `Notes.replyText` | 1,174,706 |
+| `Posts.Title` | 13,144 |
+| `Posts.Body` | 172 |
+
+Any query whose output reaches a human should collapse it:
+
+```sql
+NULLIF(NULLIF(SomeColumn, '.'), '') AS SomeColumn
+```
+
+Empty string turns up too, hence the double `NULLIF`. Not every column is affected —
+`Blogs.TTFolderPath` and `Blogs.DateModified` currently have zero dot rows — but new
+columns tend to acquire them, so treat cleaning as the default for any text column
+rendered to a user.
+
+---
+
+## Supporting tables
+
+Crawler bookkeeping. Rolodex ignores all of these.
+
+| Table | Rows | What it is |
+|---|--:|---|
+| `DailyAPICount` | 133 | `(Date TEXT PK, APICount INTEGER)` — per-day API call tally against the rate limit |
+| `ApiKeyPoolState` | 2 | `(KeyName TEXT PK, RetryUntil INTEGER)` — per-key backoff; `RetryUntil` is unix seconds |
+| `ApiKeyPoolMeta` | 1 | `(Id PK CHECK (Id = 1), LastIndex)` — round-robin cursor. Singleton by check constraint |
+| `CollectRunState` | 1 | `(Id PK CHECK (Id = 1), RunCutoff, RunComplete, RunStarted, RunCompletedAt)` — resume state for an interrupted collection run. Also a singleton |
+
+---
+
+## `Blogs.IsActive` — now written by two applications
+
+`IsActive` has always been the crawler's work-selection flag. `GetBlogs` in
+`DataAccess.cs` joins on it to decide what to collect:
+
+```sql
+SELECT NoteBlogName, count(*) FROM notes
+INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName
+WHERE blogs.IsActive = @isActive AND ...
+```
+
+Nothing inside the crawler *writes* it — it is an input, set from outside.
+
+**Rolodex is now one of the things that sets it.** Removing a blog through the Rolodex UI
+runs exactly this:
+
+```sql
+UPDATE Blogs SET IsActive = 0 WHERE BlogName = ?;
+```
+
+Rolodex adds no column and changes no schema. It reuses this flag because the two meanings
+were judged to be one decision: a blog you do not want in the browsing UI is a blog you do
+not want to keep crawling. Removal therefore stops collection, and the Rolodex
+confirmation screen says so before anyone commits.
+
+- `1` (or absent/NULL) — live. Crawled, and visible in Rolodex.
+- `0` — removed. Not crawled, hidden from the Rolodex registry, dashboard counts and
+  engagement rollups.
+
+Restoring is the same `UPDATE` with a `1`. Nothing is destroyed either way: the blog's
+`Posts` and `Notes` rows are never touched, and Rolodex deliberately keeps showing them
+under its Posts and Notes pages. Removing a blog hides the blog, not what it collected.
+
+### What other tools need to know
+
+1. **Setting `IsActive = 0` now also hides the blog from Rolodex**, and setting it back to
+   `1` makes it reappear. If another tool deactivates blogs in bulk, it is also removing
+   them from the browsing UI — which may be exactly right, but it is no longer a
+   crawler-only decision.
+2. **Re-crawling a removed blog will not bring it back**, since nothing in the crawler
+   writes the flag. An `INSERT OR REPLACE` on the `Blogs` row *would*, by resetting it to
+   the column default of `1`. Prefer an `UPDATE` of the specific columns, or
+   `INSERT … ON CONFLICT DO UPDATE SET` naming only the columns being refreshed.
+3. **NULL is treated as live.** The column is `INTEGER DEFAULT 1` with no `NOT NULL`, so
+   Rolodex reads it through `COALESCE(IsActive, 1)`. A NULL therefore leaves the blog
+   visible rather than stranding it outside both the registry and the removed list, where
+   no screen could reach it. Write `0` or `1`, not NULL.
+4. **Backing the feature out is a configuration change, not a migration.** Because there is
+   no Rolodex-owned column, setting `Rolodex__EnableBlogDeletion=false` is the whole of it;
+   there is nothing to drop. Any blogs already at `IsActive = 0` simply go back to being
+   ordinary inactive blogs.
+
+---
+
+## `Posts.IsActive` and `Notes.IsActive` — optional, and not in this database yet
+
+The same flag is being extended to the two content tables, with the same meaning: `0` is
+removed, anything else — including `NULL` — is live. **Neither column exists in the live
+`TL.db` as of 2026-07-29**; the DDL quoted above for `Posts` and `Notes` is complete. Like
+`Blogs.IsActive`, they are written from outside this crawler.
+
+The crawler therefore treats both as optional, and as nothing it owns:
+
+- **It never writes them.** No `INSERT` column list names `IsActive`, no `UPDATE` sets it,
+  and `MapPrefixToColumn` — the only place a column name is chosen at runtime — cannot map
+  to it. Re-crawling a removed post or note refreshes its content and leaves the flag at
+  `0`. There is no `INSERT OR REPLACE` on `Posts` or `Notes` for a default to be reset by.
+- **It filters on them only when they exist.** `HasIsActiveColumn` in `DataAccess.cs` asks
+  `PRAGMA table_info` once per table per database path and caches the answer; the filter
+  is `COALESCE(IsActive, 1) = 1`, and it is dropped entirely when the column is absent.
+  Naming a missing column is a hard SQLite error, so this is what lets one build run
+  against databases on both sides of the change. The cache lives for the process — adding
+  the columns to a live database takes effect on the next run.
+
+Every read that selects posts or notes carries the filter: `GetPosts`, `GetReplies`,
+`GetRepliesWithMissingText`, `GetRepliesWithFilledText`, `GetAllPostTextColumns`,
+`GetAllPostsForBlog`, `GetPost`, `GetPostByIdAnyBlog`, and the engagement queries that
+count or join `Notes` (`GetBlogs`, `GetBlogsAll`, `GetBlogsForLikes`). The one deliberate
+omission is the `LEFT JOIN Notes` in `GetPosts`: nothing is selected from it and it can
+neither add nor remove a row, so filtering it would buy nothing.
+
+`LegacyPostsDbImporter` is unfiltered too — it reads a foreign legacy database whose
+`Posts` table is not this schema.
+
+Two consequences worth stating plainly, both inherited from how `Blogs.IsActive` is
+handled:
+
+1. **Removal hides a row; it does not freeze it.** The write paths are keyed on a post the
+   caller already selected, so an ingest or a correction run still overwrites the content
+   of a removed post. Only selection is filtered.
+2. **`NULL` is live.** Write `0` or `1`, not `NULL`, but a `NULL` leaves the row visible
+   rather than stranding it.
+
+---
+
+## Reproducing the numbers
+
+```sql
+SELECT 'Blogs', COUNT(*) FROM Blogs
+UNION ALL SELECT 'Posts', COUNT(*) FROM Posts
+UNION ALL SELECT 'Notes', COUNT(*) FROM Notes;
+
+-- note type mix
+SELECT Type, COUNT(*) FROM Notes GROUP BY Type ORDER BY 2 DESC;
+
+-- the two date shapes in Blogs.DateAdded
+SELECT CASE WHEN DateAdded LIKE '____-__-__%' THEN 'ISO' ELSE 'US' END, COUNT(*)
+FROM Blogs GROUP BY 1;
+
+-- post IDs that are ambiguous without a blog name
+SELECT COUNT(*) FROM (
+    SELECT PostID FROM Posts GROUP BY PostID HAVING COUNT(DISTINCT BlogName) > 1);
+
+-- rows that reference a blog the registry does not have
+SELECT COUNT(*) FROM Posts p
+WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = p.BlogName);
+```
+
+Open the file read-only so an inspection can never disturb a running crawl:
+
+```bash
+sqlite3 "file:TL.db?mode=ro" ".schema"
+```

URLNotesGrabberCORE/URLNotesGrabberCORE.csproj

@@ -29,6 +29,9 @@
       <CopyToOutputDirectory>Always</CopyToOutputDirectory>
     </None>
     <None Update="TL.db">
+      <CopyToOutputDirectory>Never</CopyToOutputDirectory>
+    </None>
+    <None Update="prefixes.txt">
       <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
     </None>
   </ItemGroup>

URLNotesGrabberCORE/UpdateBlogPathsRunner.cs

@@ -0,0 +1,70 @@
+using System.Text.Json;
+
+namespace URLNotesGrabberCORE
+{
+    // Port of ThreeTxtFileHelper/UpdateBlogPaths.cs. Reads .tumblr / .tmblrpriv metadata
+    // files from a root\Index folder and populates Blogs.TTFolderPath in TL.db.
+    public static class UpdateBlogPathsRunner
+    {
+        public static int Run(string rootPath)
+        {
+            if (string.IsNullOrWhiteSpace(rootPath))
+            {
+                Console.WriteLine("UpdateBlogPaths: rootPath is required.");
+                return 1;
+            }
+
+            DataAccess.EnsureTTFileHelperColumnsExist();
+
+            string indexPath = Path.Combine(rootPath, "Index");
+            if (!Directory.Exists(indexPath))
+            {
+                Console.WriteLine($"Index folder not found at: {indexPath}");
+                return 1;
+            }
+
+            Console.WriteLine($"Scanning Index folder: {indexPath}");
+
+            var blogFiles = Directory.GetFiles(indexPath, "*.tumblr")
+                .Concat(Directory.GetFiles(indexPath, "*.tmblrpriv"))
+                .ToList();
+
+            Console.WriteLine($"Found {blogFiles.Count} blog metadata files");
+
+            int updatedCount = 0;
+
+            foreach (var blogFile in blogFiles)
+            {
+                try
+                {
+                    string blogName = Path.GetFileNameWithoutExtension(blogFile);
+                    string jsonContent = File.ReadAllText(blogFile);
+                    using JsonDocument doc = JsonDocument.Parse(jsonContent);
+                    JsonElement root = doc.RootElement;
+
+                    if (root.TryGetProperty("FileDownloadLocation", out JsonElement locationElement))
+                    {
+                        string? fileDownloadLocation = locationElement.GetString();
+                        if (!string.IsNullOrWhiteSpace(fileDownloadLocation))
+                        {
+                            DataAccess.SetBlogTTFolderPath(blogName, fileDownloadLocation);
+                            updatedCount++;
+                            Console.WriteLine($"Updated {blogName}: {fileDownloadLocation}");
+                        }
+                    }
+                    else
+                    {
+                        Console.WriteLine($"No FileDownloadLocation found in {blogFile}");
+                    }
+                }
+                catch (Exception ex)
+                {
+                    Console.WriteLine($"Error processing {blogFile}: {ex.Message}");
+                }
+            }
+
+            Console.WriteLine($"\nUpdated {updatedCount} blogs with TTFolderPath");
+            return 0;
+        }
+    }
+}

URLNotesGrabberCORE/appsettings.json

@@ -1,4 +1,4 @@
-{
+{
   "appSettings": {
     "PathInput": "u:\\jim\\Documents\\Web Copies\\blogs\\",
     "PathOutput": "u:\\jim\\Documents\\Web Copies\\blogs\\GetNotes.txt",
@@ -6,13 +6,40 @@
     "PathOutputPosts": "u:\\jim\\Documents\\Web Copies\\blogs\\GetPosts.txt",
     "PathOutputBlogs": "u:\\jim\\Documents\\Web Copies\\blogs\\GetBlogs.txt",
     "PathOutputReplies": "u:\\jim\\Documents\\Web Copies\\blogs\\GetReplies.txt",
-    "PathDB": "u:\\jim\\Documents\\Web Copies\\blogs\\TL.db",
+    "PathOutputUrls": "u:\\jim\\Documents\\Web Copies\\blogs\\GetUrls.txt",
-    "ContainsList": "zombaee,zomb-eh,ahzombae,thebugandme,lovingbabybug,swarthyvillain,h4rdspot"
+    "PathDB": "..\\..\\..\\tl.db",
+    "ContainsList": "zombaee,zomb-eh,ahzombae,thebugandme,lovingbabybug,h4rdspot",
+    "PostIDToExclude": "730084470076686336,726309940037287936,639974102120235008,184045126218",
+    "EnableFileLogging": false,
+    "LogTraversalRecordImports": false,
+    "LikesRefreshCooldownDays": 7,
+    "PathTTRoot": "",
+    "PathTTBackup": "",
+    "PathPrefixes": "prefixes.txt",
+    "PathCorrectionReport": "correction_report.txt",
+    "PathCorrectionApplied": "correction_applied.txt"
   },
   "TumblrApi": {
     "ConsumerKey": "PtsBCGumcsgihyynxUh8b47Jfmi7uXhEIOU46bmhdsUSJ5mEP3",
     "ConsumerSecret": "sA8BwNVTVKqBRRJmHbAD6NuyKPJ3bb9cei2bYhMgqT8cLX8tSG",
     "OAuthToken": "HPJI6IijHoKN6WzBumG7KjS7g01iCu07jQsulpueKbWN1ZJ35J",
-    "OAuthTokenSecret": "ajkN0Z1kKrsJaIXZEDI8zLsjRSQxFkfgO1i5k78FuQYCQjSY7w"
+    "OAuthTokenSecret": "ajkN0Z1kKrsJaIXZEDI8zLsjRSQxFkfgO1i5k78FuQYCQjSY7w",
+    "PoolEnabled": true,
+    "Color": "Cyan"
+  },
+  "TumblrApi3": {
+    "ConsumerKey": "Jmoh13AS9hKBYsf939uENQsiWUuZJLFT3do4YK9P0bpspVuxkH",
+    "ConsumerSecret": "8LcHjuqpOS9gZZPaML1joT330w5NqyGdreSqSBazdAWCADzcms",
+    "OAuthToken": "Gm2esFTeb5LKsb4A2YJGrF2Udycfc0AF8afof2qNGyzNSq1qWM",
+    "OAuthTokenSecret": "Kfokor80s5fVue91OmNHidCdILZ9AnHtRplrct2P6Q75jgNTLa",
+    "PoolEnabled": false
+  },
+  "TumblrApi4": {
+    "ConsumerKey": "gOuydIEENkRmEvvuf57R69yPRb39FC0Egb9p6ntxWuCuFzlsV3",
+    "ConsumerSecret": "0my913slXAHgra4mEEYV98stJ3wXGjPTWJeviiZXwJ2AqzYtIF   ",
+    "OAuthToken": "JHk08YufSrsetTR1Ekuh10cPOzC6rjqU5WwVu3uV4vvLzp0w8w",
+    "OAuthTokenSecret": "1oryr4WMBxuDrt4vL4sJQs5qNagkH3KxwIf61M8SfvklTrhHSb",
+    "PoolEnabled": true,
+    "Color": "Yellow"
   }
 }

URLNotesGrabberCORE/prefixes.txt

@@ -0,0 +1,20 @@
+Post ID
+reblog URL
+Date
+Has Image
+Post URL
+Slug
+Reblog Key
+Reblog Name
+Summary
+Quote
+Body
+Tags
+Link
+Photo URL
+Photo Caption
+Downloaded Files
+Audio Caption
+Question
+Answer
+Title

verify-db-schema.sql

@@ -0,0 +1,202 @@
+-- ============================================================================
+-- verify-db-schema.sql
+--
+-- Purpose: Verify that a TL.db (e.g. a restored backup) has every column the
+--          current URLNotesGrabberCORE code expects. The app has NO startup
+--          migration: missing columns only get added when specific modes run,
+--          and a referenced-but-missing column causes a "no such column" crash.
+--
+-- How to use (DB Browser for SQLite):
+--   1. File > Open Database  ->  pick the restored backup.
+--   2. Execute SQL tab. Run SECTION 1 (it is read-only).
+--        * Zero rows from every query  = schema is fully aligned, you're done.
+--        * Rows in "MISSING COLUMNS"    = copy the run_this_to_fix text.
+--   3. If columns are missing: KEEP A COPY OF THE BACKUP FIRST, then go to
+--      SECTION 2, uncomment ONLY the ALTER lines that match the report, and run.
+--   4. Re-run SECTION 1 to confirm zero rows.
+--
+-- This script never UPDATEs/DELETEs/DROPs. In particular it deliberately does
+-- NOT replicate the likes-reset that the app's -likes migration performs
+-- (DataAccess.cs:375), so existing likes high-water marks are preserved.
+-- ============================================================================
+
+
+-- ============================================================================
+-- SECTION 1  --  VERIFICATION (read-only)
+-- ============================================================================
+
+-- Expected schema for the current code version.
+-- alter_stmt is a runnable ALTER for additively-fixable columns; for base
+-- columns it is a 'MANUAL REVIEW' note (a missing base column means the backup
+-- predates the table's creation or is damaged -- do not blindly auto-add).
+WITH expected(tbl, col, alter_stmt) AS (
+  VALUES
+    -- Posts (base columns: manual review if missing)
+    ('Posts','BlogName',              'MANUAL REVIEW - base/PK column missing'),
+    ('Posts','PostID',                'MANUAL REVIEW - base/PK column missing'),
+    ('Posts','HasNotesGathered',      'MANUAL REVIEW - base column missing'),
+    ('Posts','reblogURL',             'MANUAL REVIEW - base column missing'),
+    ('Posts','NotFound',              'MANUAL REVIEW - base column missing'),
+    ('Posts','PostDate',              'MANUAL REVIEW - base column missing'),
+    ('Posts','NotesGatheredDateTime', 'MANUAL REVIEW - base column missing'),
+    ('Posts','HasImage',              'MANUAL REVIEW - base column missing'),
+    ('Posts','PostURL',               'MANUAL REVIEW - base column missing'),
+    ('Posts','Slug',                  'MANUAL REVIEW - base column missing'),
+    ('Posts','ReblogKey',             'MANUAL REVIEW - base column missing'),
+    ('Posts','ReblogName',            'MANUAL REVIEW - base column missing'),
+    ('Posts','Summary',               'MANUAL REVIEW - base column missing'),
+    ('Posts','Quote',                 'MANUAL REVIEW - base column missing'),
+    ('Posts','Body',                  'MANUAL REVIEW - base column missing'),
+    ('Posts','Tags',                  'MANUAL REVIEW - base column missing'),
+    ('Posts','Link',                  'MANUAL REVIEW - base column missing'),
+    ('Posts','PhotoURL',              'MANUAL REVIEW - base column missing'),
+    ('Posts','PhotoCaption',          'MANUAL REVIEW - base column missing'),
+    ('Posts','DownloadedFiles',       'MANUAL REVIEW - base column missing'),
+    ('Posts','AudioCaption',          'MANUAL REVIEW - base column missing'),
+    ('Posts','Question',              'MANUAL REVIEW - base column missing'),
+    ('Posts','Answer',                'MANUAL REVIEW - base column missing'),
+    ('Posts','Title',                 'MANUAL REVIEW - base column missing'),
+    ('Posts','ByLikes',               'MANUAL REVIEW - base column missing'),
+    ('Posts','RootBlogName',          'MANUAL REVIEW - base column missing'),
+    ('Posts','RootURL',               'MANUAL REVIEW - base column missing'),
+    ('Posts','DateModified',          'MANUAL REVIEW - base column missing'),
+    ('Posts','DateCreated',           'MANUAL REVIEW - base column missing'),
+    -- Posts (additive migration column, auto-fixable)
+    ('Posts','PostType',              'ALTER TABLE Posts ADD COLUMN PostType TEXT;'),
+
+    -- Blogs (base columns: manual review if missing)
+    ('Blogs','BlogName',              'MANUAL REVIEW - base/PK column missing'),
+    ('Blogs','HasBeenOutput',         'MANUAL REVIEW - base column missing'),
+    ('Blogs','IsActive',              'MANUAL REVIEW - base column missing'),
+    ('Blogs','DateAdded',             'MANUAL REVIEW - base column missing'),
+    ('Blogs','ByLikes',               'MANUAL REVIEW - base column missing'),
+    ('Blogs','DateModified',          'MANUAL REVIEW - base column missing'),
+    ('Blogs','DateCreated',           'MANUAL REVIEW - base column missing'),
+    -- Blogs (additive migration columns, auto-fixable)
+    ('Blogs','LikesPulled',           'ALTER TABLE Blogs ADD COLUMN LikesPulled INTEGER DEFAULT 0;'),
+    ('Blogs','LikesCursor',           'ALTER TABLE Blogs ADD COLUMN LikesCursor INTEGER DEFAULT 0;'),
+    ('Blogs','LikesNewestTimestamp',  'ALTER TABLE Blogs ADD COLUMN LikesNewestTimestamp INTEGER DEFAULT 0;'),
+    ('Blogs','LikesLastRefreshed',    'ALTER TABLE Blogs ADD COLUMN LikesLastRefreshed INTEGER DEFAULT 0;'),
+    ('Blogs','LikesLastNewCount',     'ALTER TABLE Blogs ADD COLUMN LikesLastNewCount INTEGER DEFAULT 0;'),
+    ('Blogs','TTFolderPath',          'ALTER TABLE Blogs ADD COLUMN TTFolderPath TEXT;'),
+
+    -- Notes (base columns: manual review if missing)
+    ('Notes','RootBlogName',          'MANUAL REVIEW - base/PK column missing'),
+    ('Notes','PostID',                'MANUAL REVIEW - base/PK column missing'),
+    ('Notes','NoteBlogName',          'MANUAL REVIEW - base/PK column missing'),
+    ('Notes','TimeStamp',             'MANUAL REVIEW - base/PK column missing'),
+    ('Notes','Type',                  'MANUAL REVIEW - base/PK column missing'),
+    ('Notes','DatetimeCrawled',       'MANUAL REVIEW - base column missing'),
+    ('Notes','DateModified',          'MANUAL REVIEW - base column missing'),
+    ('Notes','DateCreated',           'MANUAL REVIEW - base column missing'),
+    -- Notes (additive migration column, auto-fixable)
+    ('Notes','replyText',             'ALTER TABLE Notes ADD COLUMN replyText TEXT DEFAULT ''.'';'),
+
+    -- DailyAPICount (base columns)
+    ('DailyAPICount','Date',          'MANUAL REVIEW - base/PK column missing'),
+    ('DailyAPICount','APICount',      'MANUAL REVIEW - base column missing'),
+
+    -- ApiKeyPoolState (created at runtime by EnsureApiKeyPoolTables; auto-fixable by re-running app, but safe to add)
+    ('ApiKeyPoolState','KeyName',     'MANUAL REVIEW - run app once to auto-create ApiKeyPool tables'),
+    ('ApiKeyPoolState','RetryUntil',  'MANUAL REVIEW - run app once to auto-create ApiKeyPool tables'),
+    ('ApiKeyPoolMeta','Id',           'MANUAL REVIEW - run app once to auto-create ApiKeyPool tables'),
+    ('ApiKeyPoolMeta','LastIndex',    'MANUAL REVIEW - run app once to auto-create ApiKeyPool tables')
+),
+actual(tbl, col) AS (
+  SELECT 'Posts',           name FROM pragma_table_info('Posts')
+  UNION ALL SELECT 'Blogs',           name FROM pragma_table_info('Blogs')
+  UNION ALL SELECT 'Notes',           name FROM pragma_table_info('Notes')
+  UNION ALL SELECT 'DailyAPICount',   name FROM pragma_table_info('DailyAPICount')
+  UNION ALL SELECT 'ApiKeyPoolState', name FROM pragma_table_info('ApiKeyPoolState')
+  UNION ALL SELECT 'ApiKeyPoolMeta',  name FROM pragma_table_info('ApiKeyPoolMeta')
+)
+
+-- 1a. MISSING COLUMNS: columns the code needs that the DB does not have.
+--     Zero rows = good. Otherwise copy run_this_to_fix into SECTION 2.
+SELECT
+  e.tbl              AS table_name,
+  e.col              AS missing_column,
+  e.alter_stmt       AS run_this_to_fix
+FROM expected e
+LEFT JOIN actual a
+  ON a.tbl = e.tbl AND lower(a.col) = lower(e.col)
+WHERE a.col IS NULL
+ORDER BY (e.alter_stmt LIKE 'ALTER%') DESC, e.tbl, e.col;
+
+
+-- 1b. MISSING TABLES: expected tables that don't exist at all in this DB.
+--     Zero rows = good.
+WITH expected_tables(tbl) AS (
+  VALUES ('Posts'),('Blogs'),('Notes'),('DailyAPICount'),
+         ('ApiKeyPoolState'),('ApiKeyPoolMeta')
+)
+SELECT et.tbl AS missing_table
+FROM expected_tables et
+WHERE NOT EXISTS (
+  SELECT 1 FROM sqlite_master
+  WHERE type = 'table' AND lower(name) = lower(et.tbl)
+)
+ORDER BY et.tbl;
+
+
+-- 1c. EXTRA / UNEXPECTED COLUMNS: present in the DB but not in the expected
+--     list above. Informational only -- e.g. a NEWER backup, or a column this
+--     script's expected-list hasn't been updated for. Not an error by itself.
+--     Posts.IsActive and Notes.IsActive are listed here and NOT in 1a on
+--     purpose: they are written by other tools, the app only reads them when
+--     present, and it must not be told to add them. See TL.db.md.
+WITH expected(tbl, col) AS (
+  VALUES
+    ('Posts','BlogName'),('Posts','PostID'),('Posts','HasNotesGathered'),('Posts','reblogURL'),
+    ('Posts','NotFound'),('Posts','PostDate'),('Posts','NotesGatheredDateTime'),('Posts','HasImage'),
+    ('Posts','PostURL'),('Posts','Slug'),('Posts','ReblogKey'),('Posts','ReblogName'),('Posts','Summary'),
+    ('Posts','Quote'),('Posts','Body'),('Posts','Tags'),('Posts','Link'),('Posts','PhotoURL'),
+    ('Posts','PhotoCaption'),('Posts','DownloadedFiles'),('Posts','AudioCaption'),('Posts','Question'),
+    ('Posts','Answer'),('Posts','Title'),('Posts','ByLikes'),('Posts','RootBlogName'),('Posts','RootURL'),
+    ('Posts','DateModified'),('Posts','DateCreated'),('Posts','PostType'),('Posts','IsActive'),
+    ('Blogs','BlogName'),('Blogs','HasBeenOutput'),('Blogs','IsActive'),('Blogs','DateAdded'),
+    ('Blogs','ByLikes'),('Blogs','DateModified'),('Blogs','DateCreated'),('Blogs','LikesPulled'),
+    ('Blogs','LikesCursor'),('Blogs','LikesNewestTimestamp'),('Blogs','LikesLastRefreshed'),
+    ('Blogs','LikesLastNewCount'),('Blogs','TTFolderPath'),
+    ('Notes','RootBlogName'),('Notes','PostID'),('Notes','NoteBlogName'),('Notes','TimeStamp'),
+    ('Notes','Type'),('Notes','DatetimeCrawled'),('Notes','DateModified'),('Notes','DateCreated'),
+    ('Notes','replyText'),('Notes','IsActive'),
+    ('DailyAPICount','Date'),('DailyAPICount','APICount'),
+    ('ApiKeyPoolState','KeyName'),('ApiKeyPoolState','RetryUntil'),
+    ('ApiKeyPoolMeta','Id'),('ApiKeyPoolMeta','LastIndex')
+),
+actual(tbl, col) AS (
+  SELECT 'Posts',           name FROM pragma_table_info('Posts')
+  UNION ALL SELECT 'Blogs',           name FROM pragma_table_info('Blogs')
+  UNION ALL SELECT 'Notes',           name FROM pragma_table_info('Notes')
+  UNION ALL SELECT 'DailyAPICount',   name FROM pragma_table_info('DailyAPICount')
+  UNION ALL SELECT 'ApiKeyPoolState', name FROM pragma_table_info('ApiKeyPoolState')
+  UNION ALL SELECT 'ApiKeyPoolMeta',  name FROM pragma_table_info('ApiKeyPoolMeta')
+)
+SELECT a.tbl AS table_name, a.col AS unexpected_column
+FROM actual a
+LEFT JOIN expected e
+  ON e.tbl = a.tbl AND lower(e.col) = lower(a.col)
+WHERE e.col IS NULL
+ORDER BY a.tbl, a.col;
+
+
+-- ============================================================================
+-- SECTION 2  --  FIX (opt-in, additive only)
+--
+-- Run ONLY the lines that query 1a flagged with an ALTER statement.
+-- KEEP A COPY OF THE BACKUP FIRST. SQLite has no "ADD COLUMN IF NOT EXISTS",
+-- so running an ALTER for a column that already exists throws a harmless
+-- "duplicate column name" error and changes nothing -- just run the flagged
+-- subset. These are the 8 additive migration columns and nothing else; the
+-- likes high-water-mark reset is intentionally NOT included.
+-- ============================================================================
+
+-- ALTER TABLE Posts ADD COLUMN PostType TEXT;
+-- ALTER TABLE Blogs ADD COLUMN LikesPulled INTEGER DEFAULT 0;
+-- ALTER TABLE Blogs ADD COLUMN LikesCursor INTEGER DEFAULT 0;
+-- ALTER TABLE Blogs ADD COLUMN LikesNewestTimestamp INTEGER DEFAULT 0;
+-- ALTER TABLE Blogs ADD COLUMN LikesLastRefreshed INTEGER DEFAULT 0;
+-- ALTER TABLE Blogs ADD COLUMN LikesLastNewCount INTEGER DEFAULT 0;
+-- ALTER TABLE Blogs ADD COLUMN TTFolderPath TEXT;
+-- ALTER TABLE Notes ADD COLUMN replyText TEXT DEFAULT '.';

vibe.bat

@@ -0,0 +1 @@
+ollama launch opencode --model qwen-coder-32k