Merge branch 'master' into claude/jovial-mayer-77c2b5

The INSERT named a DateCreated column that DailyAPICount (Date, APICount)
never had, so every call threw "no such column: DateCreated" into an
empty catch block. Today's row was never created and the tally sat idle
since 2026-04-13. Drop the column from the INSERT, and report the three
silent failure points (insert error, missing row after insert, update
matching zero rows) instead of swallowing them.

.github/copilot-instructions.md

@@ -22,18 +22,18 @@ This document provides essential context for AI agents working with URLNotesGrab
 ```powershell
 dotnet build
 dotnet run            # Process all files in input directory
-dotnet run -- -parse [blogname]  # Process specific blog
+dotnet run -- --parse [blogname]  # Process specific blog
-dotnet run -- -test [blogname] [postID]  # Test API for specific post
+dotnet run -- --test [blogname] [postID]  # Test API for specific post
 ```
 
 ### Command-Line Interface
-- `-parse [blogname]`: Parse text files for specific blog
+- `--parse [blogname]`: Parse text files for specific blog
-- `-test [blogname] [postID]`: Test API note collection
+- `--test [blogname] [postID]`: Test API note collection
-- `-posts`: Export post blogs to file
+- `--posts`: Export post blogs to file
-- `-blogs`: Export blog list to file
+- `--blogs`: Export blog list to file
-- `-collect`: Collect notes for all posts in DB
+- `--collect`: Collect notes for all posts in DB
-- `-blogsR`: Export reply blogs to file
+- `--blogsR`: Export reply blogs to file
-- `-blogsO [start] [stop]`: Export blogs within range
+- `--blogsO [start] [stop]`: Export blogs within range
 
 ## Project Conventions
 

AGENTS.md

@@ -11,6 +11,7 @@
 - `ResponseNotes.cs`: Tumblr API response models
 - Round-robin API key rotation with rate-limit tracking
 - Automatic console color assignment per API key for output differentiation
+- No-argument mode (`Program.TraverseDirectory`) ingests `.txt` blog export files into `Posts` via `DataAccess.AddPost`. Recognized field prefixes live in `TraverseDirectoryFieldPrefixes`; `Body:` and `Downloaded files:` collect every following line up to the next recognized prefix (multi-line values). `RootURL` is populated from a `Reblog root url:` line the same way it's populated from the API-based `--likes` flow — both paths converge on `DataAccess.AddPost`'s `rootURL` parameter, which `UpdatePost` only overwrites when the incoming value is non-empty (existing `RootURL` is preserved otherwise)
 
 ## Developer Guidelines
 
@@ -27,6 +28,45 @@
 - Preserve console color state: use save/restore pattern for temporary color changes
 - API rate limits must use `ApiKeyPool.MarkRateLimited()`/`MarkAvailable()`
 
+### API Failure Classification
+Tumblr sits behind a CDN that returns HTML error pages (403, 5xx) which never reach the API. These
+say nothing about the item being fetched, so they must not be recorded as per-item failures.
+
+- A response body that will not parse as JSON did not come from the API. Flag it with
+  `Root.transientFailure`, never as `FAILURE`
+- Transient failures retry in place (`TransientBackoffSeconds`) before the item is skipped; a skipped
+  item stays unmarked in the DB so a later launch retries it
+- `MaxConsecutiveTransient` consecutive transient failures aborts the pass rather than skipping
+  item-by-item against an edge that is refusing all traffic
+- Only call `ApiKeyPool.MarkAvailable()` on a response that actually reached the API. A transport or
+  CDN failure says nothing about the key's standing and must not clear its flag
+- Only a real HTTP 429 (or `meta.status == 429`) counts as a rate limit. Do not infer one from the
+  presence of `X-RateLimit-*` headers, which Tumblr sends on every response
+- Rate limiters must pace with `await AcquireAsync()`. `AttemptAcquire()` does not wait, so a
+  saturated window aborts the run instead of throttling it
+- Long-running commands return exit 3 when a pass ends incomplete (rate-limit pause, breaker trip, or
+  skipped items), so a caller can distinguish that from a clean run
+
+### `IsActive` Is Not Ours To Write
+`Blogs.IsActive`, `Posts.IsActive` and `Notes.IsActive` are removal flags set by other tools
+(Rolodex). `0` means removed; anything else, including `NULL`, means live. Full detail in
+`URLNotesGrabberCORE/TL.db.md`.
+
+- **Never write any `IsActive` column.** Not in an `INSERT` column list, not in an `UPDATE`,
+  and never via `INSERT OR REPLACE` on these tables — that resets the column default and
+  un-removes the row. Re-crawling a removed row must refresh its content and leave the flag
+  where it was
+- **Filter at selection, not at write.** Every query that *selects* posts, notes or blogs
+  excludes removed rows. Update statements stay keyed on a row the caller already selected;
+  filtering them would spend API quota and then fail to persist the result
+- `Posts.IsActive` and `Notes.IsActive` are **optional** — they are absent from databases
+  that predate them, and naming a missing column is a hard SQLite error. Compose the filter
+  with `AndIsActive`/`WhereIsActive` in `DataAccess.cs`, which return
+  `COALESCE(IsActive, 1) = 1` only when `HasIsActiveColumn` finds the column. `Blogs.IsActive`
+  is not optional and is filtered directly
+- Do not add these columns from this app, and do not add them to the missing-column list in
+  `verify-db-schema.sql`
+
 ### Testing
 - No existing test suite; use xUnit if adding tests
 - Test critical logic: `ApiKeyPool` init, color parsing, config persistence

URLNotesGrabberCORE/Program.cs

@@ -5,7 +5,6 @@ using Microsoft.Extensions.Configuration;
 using System.Configuration;
 using System.Threading;
 using Microsoft.Extensions.Diagnostics.Latency;
-using static System.Runtime.InteropServices.JavaScript.JSType;
 using System.Text.RegularExpressions;
 
 namespace URLNotesGrabberCORE
@@ -13,12 +12,27 @@ namespace URLNotesGrabberCORE
     internal class Program
     {
 
-        static void Main(string[] args)
+        static int Main(string[] args)
         {
             // Reset console color on exit (including Ctrl+C)
             Console.CancelKeyPress += (s, e) => Console.ResetColor();
             AppDomain.CurrentDomain.ProcessExit += (s, e) => Console.ResetColor();
 
+            try
+            {
+                return Run(args);
+            }
+            catch (Exception ex)
+            {
+                // Turn configuration errors / unguarded indexers into a quiet, deterministic exit code.
+                Console.Error.WriteLine(ex.Message);
+                return 1;
+            }
+        }
+
+        static int Run(string[] args)
+        {
+            int exitCode = 0;
             IConfiguration config = new ConfigurationBuilder()
                                     .SetBasePath(Directory.GetCurrentDirectory())
                                     .AddJsonFile("appsettings.json", optional: true, reloadOnChange: false)
@@ -33,31 +47,34 @@ namespace URLNotesGrabberCORE
             List<string> filteredArgs = new List<string>();
             for (int i = 0; i < args.Length; i++)
             {
-                if (string.Equals(args[i], "-force", StringComparison.OrdinalIgnoreCase) ||
+                if (args[i] == "--")
-                    string.Equals(args[i], "--force", StringComparison.OrdinalIgnoreCase))
+                {
+                    // POSIX end-of-options: everything after is a literal operand.
+                    for (int j = i + 1; j < args.Length; j++) filteredArgs.Add(args[j]);
+                    break;
+                }
+
+                if (string.Equals(args[i], "--force", StringComparison.OrdinalIgnoreCase))
                 {
                     forceIgnoreCooldown = true;
                     continue;
                 }
 
-                if (string.Equals(args[i], "-api3", StringComparison.OrdinalIgnoreCase) ||
+                if (string.Equals(args[i], "--api3", StringComparison.OrdinalIgnoreCase))
-                    string.Equals(args[i], "--api3", StringComparison.OrdinalIgnoreCase))
                 {
                     apiSectionName = "TumblrApi3";
                     apiExplicitlySet = true;
                     continue;
                 }
 
-                if (string.Equals(args[i], "-api4", StringComparison.OrdinalIgnoreCase) ||
+                if (string.Equals(args[i], "--api4", StringComparison.OrdinalIgnoreCase))
-                    string.Equals(args[i], "--api4", StringComparison.OrdinalIgnoreCase))
                 {
                     apiSectionName = "TumblrApi4";
                     apiExplicitlySet = true;
                     continue;
                 }
 
-                if (string.Equals(args[i], "-api", StringComparison.OrdinalIgnoreCase) ||
+                if (string.Equals(args[i], "--api", StringComparison.OrdinalIgnoreCase))
-                    string.Equals(args[i], "--api", StringComparison.OrdinalIgnoreCase))
                 {
                     if (i + 1 < args.Length && !string.IsNullOrWhiteSpace(args[i + 1]))
                     {
@@ -67,13 +84,12 @@ namespace URLNotesGrabberCORE
                     }
                     else
                     {
-                        Console.WriteLine("--Missing API section after -api/--api. Using default TumblrApi.--");
+                        Console.WriteLine("--Missing API section after --api. Using default TumblrApi.--");
                     }
                     continue;
                 }
 
-                if (string.Equals(args[i], "-start", StringComparison.OrdinalIgnoreCase) ||
+                if (string.Equals(args[i], "--start", StringComparison.OrdinalIgnoreCase))
-                    string.Equals(args[i], "--start", StringComparison.OrdinalIgnoreCase))
                 {
                     if (i + 1 < args.Length && !string.IsNullOrWhiteSpace(args[i + 1]))
                     {
@@ -82,7 +98,7 @@ namespace URLNotesGrabberCORE
                     }
                     else
                     {
-                        Console.WriteLine("--Missing blog name after -start/--start. Ignoring.--");
+                        Console.WriteLine("--Missing blog name after --start. Ignoring.--");
                     }
                     continue;
                 }
@@ -123,7 +139,10 @@ namespace URLNotesGrabberCORE
                 Console.SetOut(dualLogger);
             }
 
-            List<string> contains = settings.GetValue<string>("ContainsList").Split(',').ToList();
+            string? containsListSetting = settings.GetValue<string>("ContainsList");
+            if (string.IsNullOrEmpty(containsListSetting))
+                throw new InvalidOperationException("ContainsList is not configured in appsettings.json");
+            List<string> contains = containsListSetting.Split(',').ToList();
             bool logTraversalRecordImports = settings.GetValue("LogTraversalRecordImports", false);
 
             if (args.Length == 0) //Traverse folder structure to add posts and thus blogs to DB
@@ -147,57 +166,23 @@ namespace URLNotesGrabberCORE
                 switch (args[0])
                 {
                     case "-?":
-                        Console.WriteLine("\t Parse .txt files to find blogs");
+                    case "-h":
-
+                    case "--help":
-                        Console.WriteLine("-?\t Usage help");
+                        PrintHelp();
-
-                        Console.WriteLine("-parse\t Parse .txt files with specified blogname");
-
-                        Console.WriteLine("-test\t Calls API for given blogname and postID");
-
-                        Console.WriteLine("-posts\t For each Post in DB, write blogname to file");
-
-                        Console.WriteLine("-blogs\t For each Blog in DB, write blogname to file");
-
-                        Console.WriteLine("-collect [0|1] [datetime]\t Collect Notes from API. 1=only posts without notes. 0=full re-check of all posts: a single resumable pass (interrupt & relaunch to resume; stops when complete, retrigger for a new pass). Optional datetime overrides the cutoff and runs as a one-off (bypasses resume tracking).");
-
-                        Console.WriteLine("-blogsR\t For each Note that is a REPLY, write blogname to file ");
-
-                        Console.WriteLine("-blogsO\t For each Blog in DB, write blogname to file, but limit via a passed start and stop range ");
-
-                        Console.WriteLine("-replies\t Fetch and update missing reply text for all replies in database");
-
-                        Console.WriteLine("-likes\t Fetch likes: initial backfill for new blogs, incremental refresh for blogs past cooldown. Optional blog name forces single-blog run.");
-
-                        Console.WriteLine("-force\t (with -likes) Ignore cooldown and refresh every fully-backfilled blog");
-
-                        Console.WriteLine("-urldump\t Scan all posts' text columns and extract suspected URLs to configured file");
-
-                        Console.WriteLine("-api3\t Use TumblrApi3 settings from appsettings.json");
-
-                        Console.WriteLine("-api4\t Use TumblrApi4 settings from appsettings.json");
-
-                        Console.WriteLine("-start [blogname]\t Start traversal alphabetically at this blog name");
-
-                        Console.WriteLine("-api [section]\t Use a specific API settings section from appsettings.json (e.g. TumblrApi3)");
-
-                        Console.WriteLine("-ingest [blogname]\t Ingest Tumblr .txt exports from appSettings:PathTTRoot into TL.db (all blogs, or single blog if name given)");
-
-                        Console.WriteLine("-output\t Export posts from TL.db back to .txt files in each blog's TTFolderPath");
-
-                        Console.WriteLine("-revert [blogname]\t Recursively scan the PathInput tree and restore *.bak back to *.txt (current .txt saved as next-free .bkN); optional blogname filters by path substring");
-
-                        Console.WriteLine("-correct [bakPath]\t Dry-run: report multi-line field updates available from a BAK directory");
-
-                        Console.WriteLine("-correct -apply [bakPath]\t Apply BAK-file corrections to matching posts (prompts yes/no)");
-
-                        Console.WriteLine("-updatepaths [rootPath]\t Read .tumblr/.tmblrpriv metadata from <root>\\Index and set Blogs.TTFolderPath");
-
-                        Console.WriteLine("-importposts [path-to-posts.db]\t One-time migration: copy legacy ThreeTxtFileHelper posts.db rows into TL.db");
-
                         break;
 
-                    case "-parse":
+                    case "-V":
+                    case "--version":
+                        Console.WriteLine(System.Reflection.Assembly.GetExecutingAssembly().GetName().Version?.ToString() ?? "unknown");
+                        break;
+
+                    case "--parse":
+                        if (args.Length < 2)
+                        {
+                            Console.WriteLine("Usage: --parse <blogname>");
+                            exitCode = 2;
+                            break;
+                        }
                         string blogNameToParse = args[1];
                         int postsAdded = 0;
                         try
@@ -212,23 +197,23 @@ namespace URLNotesGrabberCORE
                         Console.WriteLine($"Total posts added: {postsAdded}");
                         break;
                     
-                    case "-test":
+                    case "--test":
                         Console.WriteLine("Test command not implemented");
                         break;
 
-                    case "-post":
+                    case "--post":
                         TraverseDirectoryForCorruption(settings.GetValue<string>("PathInput"), settings.GetValue<string>("PathOutputBlogs"), contains);
                         break;
 
-                    case "-posts": //write post's blogs to file
+                    case "--posts": //write post's blogs to file
                         WritePostBlogsToFile(settings.GetValue<string>("PathOutputPosts"));
                         break;
 
-                    case "-blogs": //write blogs to file
+                    case "--blogs": //write blogs to file
                         WriteBlogsToFile(settings.GetValue<string>("PathOutputBlogs"));
                         break;
 
-                    case "-collect": //collect notes from all posts
+                    case "--collect": //collect notes from all posts
                         bool withoutNotesOnly = true;
                         DateTime? beforeDate = DateTime.Now;
                         bool explicitDateSupplied = false;
@@ -236,6 +221,7 @@ namespace URLNotesGrabberCORE
                         if (args.Length < 2)
                         {
                             Console.WriteLine("--Expected WITHOUTNOTESONLY (0, 1) [OPTIONAL: BEFOREDATE]--");
+                            exitCode = 2;
                             break;
                         }
 
@@ -268,6 +254,7 @@ namespace URLNotesGrabberCORE
                             else
                             {
                                 Console.WriteLine($"ERROR: Invalid date format '{args[2]}'");
+                                exitCode = 2;
                                 break;
                             }
                         }
@@ -294,17 +281,17 @@ namespace URLNotesGrabberCORE
                             managedCollectRun = true;
                         }
 
-                        CollectNotes(settings.GetValue<string>("PathOutput"), withoutNotesOnly, beforeDate, managedCollectRun).GetAwaiter().GetResult();
+                        exitCode = CollectNotes(settings.GetValue<string>("PathOutput"), withoutNotesOnly, beforeDate, managedCollectRun).GetAwaiter().GetResult();
                         break;
 
-                    case "-blogsR": //collect notes from all posts
+                    case "--blogsR": //collect notes from all posts
                         WriteBlogsToFile(settings.GetValue<string>("PathOutputBlogs"), true);
                         break;
 
-                    case "-blogsO": //collect notes from all posts
+                    case "--blogsO": //collect notes from all posts
                         int from = 1, to = 999999, top = 100;
 
-                        if (args[1] is not null && args[2] is not null && args[3] is not null)
+                        if (args.Length >= 4 && args[1] is not null && args[2] is not null && args[3] is not null)
                         {
                             from = int.Parse(args[1]);
                             to = int.Parse(args[2]);
@@ -313,14 +300,16 @@ namespace URLNotesGrabberCORE
                         else
                         {
                             Console.WriteLine("--Expected FROM TO--");
+                            exitCode = 2;
+                            break;
                         }
                         WriteBlogsToFile(settings.GetValue<string>("PathOutputBlogs"), false, from, to, top);
                         break;
 
-                    case "-bop": //collect notes from all posts
+                    case "--bop": //collect notes from all posts
                         from = 1; to = 999999; top = 100;
 
-                        if (args[1] is not null && args[2] is not null && args[3] is not null)
+                        if (args.Length >= 4 && args[1] is not null && args[2] is not null && args[3] is not null)
                         {
                             from = int.Parse(args[1]);
                             to = int.Parse(args[2]);
@@ -329,66 +318,70 @@ namespace URLNotesGrabberCORE
                         else
                         {
                             Console.WriteLine("--Expected FROM TO--");
+                            exitCode = 2;
+                            break;
                         }
                         WriteBlogsToFileAll(settings.GetValue<string>("PathOutputBlogs"), false, from, to, top);
                         break;
 
-                    case "-replies": //update reply text
+                    case "--replies": //update reply text
                         CollectMissingReplyText().GetAwaiter().GetResult();
                         break;
 
-                    case "-likes":
+                    case "--likes":
                         string likeBlog = args.Length > 1 ? args[1] : null;
                         int cooldownDays = settings.GetValue("LikesRefreshCooldownDays", 7);
                         CollectLikes(likeBlog, contains, cooldownDays, forceIgnoreCooldown).GetAwaiter().GetResult();
                         break;
 
-                    case "-urldump":
+                    case "--urldump":
                         DumpUrls(settings.GetValue<string>("PathOutputUrls"));
                         break;
 
-                    case "-ingest":
+                    case "--ingest":
-                        IngestMode.Run(config, args.Skip(1).ToArray());
+                        exitCode = IngestMode.Run(config, args.Skip(1).ToArray());
                         break;
 
-                    case "-output":
+                    case "--output":
-                        OutputMode.Run(config);
+                        exitCode = OutputMode.Run(config);
                         break;
 
-                    case "-revert":
+                    case "--revert":
-                        RevertMode.Run(config, args.Length > 1 ? args[1] : null);
+                        exitCode = RevertMode.Run(config, args.Length > 1 ? args[1] : null);
                         break;
 
-                    case "-correct":
+                    case "--correct":
                     {
-                        bool applyChanges = args.Skip(1).Any(a => string.Equals(a, "-apply", StringComparison.OrdinalIgnoreCase) || string.Equals(a, "--apply", StringComparison.OrdinalIgnoreCase));
+                        bool applyChanges = args.Skip(1).Any(a => string.Equals(a, "--apply", StringComparison.OrdinalIgnoreCase));
                         var correctArgs = args.Skip(1)
-                            .Where(a => !string.Equals(a, "-apply", StringComparison.OrdinalIgnoreCase) && !string.Equals(a, "--apply", StringComparison.OrdinalIgnoreCase))
+                            .Where(a => !string.Equals(a, "--apply", StringComparison.OrdinalIgnoreCase))
                             .ToArray();
-                        CorrectMode.Run(config, correctArgs, applyChanges);
+                        exitCode = CorrectMode.Run(config, correctArgs, applyChanges);
                         break;
                     }
 
-                    case "-updatepaths":
+                    case "--updatepaths":
                     {
                         string rootPath = args.Length > 1 ? args[1] : (settings.GetValue<string>("PathTTRoot") ?? settings.GetValue<string>("PathInput") ?? string.Empty);
-                        UpdateBlogPathsRunner.Run(rootPath);
+                        exitCode = UpdateBlogPathsRunner.Run(rootPath);
                         break;
                     }
 
-                    case "-importposts":
+                    case "--importposts":
                     {
                         if (args.Length < 2)
                         {
-                            Console.WriteLine("Usage: -importposts <path-to-legacy-posts.db>");
+                            Console.WriteLine("Usage: --importposts <path-to-legacy-posts.db>");
+                            exitCode = 2;
                             break;
                         }
-                        LegacyPostsDbImporter.Run(args[1]);
+                        exitCode = LegacyPostsDbImporter.Run(args[1]);
                         break;
                     }
 
                     default:
                         Console.WriteLine("** Unknown Command ** " + args[0]);
+                        exitCode = 2;
                         break;
                 }
             }
@@ -397,6 +390,69 @@ namespace URLNotesGrabberCORE
 
             System.Console.WriteLine("<fin>:/");
             //System.Console.ReadKey();
+            return exitCode;
+        }
+
+        static void PrintHelp()
+        {
+            Console.WriteLine("\t Parse .txt files to find blogs");
+
+            Console.WriteLine("-?, -h, --help\t Usage help");
+
+            Console.WriteLine("-V, --version\t Print the application version");
+
+            Console.WriteLine("--\t End of options: treat every following token as a literal operand");
+
+            Console.WriteLine("--parse\t Parse .txt files with specified blogname");
+
+            Console.WriteLine("--test\t Calls API for given blogname and postID");
+
+            Console.WriteLine("--post\t Traverse the input directory tree checking .txt files for corruption");
+
+            Console.WriteLine("--posts\t For each Post in DB, write blogname to file");
+
+            Console.WriteLine("--blogs\t For each Blog in DB, write blogname to file");
+
+            Console.WriteLine("--collect [0|1] [datetime]\t Collect Notes from API. 1=only posts without notes. 0=full re-check of all posts: a single resumable pass (interrupt & relaunch to resume; stops when complete, retrigger for a new pass). Optional datetime overrides the cutoff and runs as a one-off (bypasses resume tracking).");
+
+            Console.WriteLine("--blogsR\t For each Note that is a REPLY, write blogname to file ");
+
+            Console.WriteLine("--blogsO\t For each Blog in DB, write blogname to file, but limit via a passed start and stop range ");
+
+            Console.WriteLine("--bop [from] [to] [top]\t Write ALL blog names to file, limited by FROM TO TOP range arguments");
+
+            Console.WriteLine("--replies\t Fetch and update missing reply text for all replies in database");
+
+            Console.WriteLine("--likes\t Fetch likes: initial backfill for new blogs, incremental refresh for blogs past cooldown. Optional blog name forces single-blog run.");
+
+            Console.WriteLine("--force\t (with --likes) Ignore cooldown and refresh every fully-backfilled blog");
+
+            Console.WriteLine("--urldump\t Scan all posts' text columns and extract suspected URLs to configured file");
+
+            Console.WriteLine("--api3\t Use TumblrApi3 settings from appsettings.json");
+
+            Console.WriteLine("--api4\t Use TumblrApi4 settings from appsettings.json");
+
+            Console.WriteLine("--start [blogname]\t Start traversal alphabetically at this blog name");
+
+            Console.WriteLine("--api [section]\t Use a specific API settings section from appsettings.json (e.g. TumblrApi3)");
+
+            Console.WriteLine("--ingest [blogname]\t Ingest Tumblr .txt exports from appSettings:PathTTRoot into TL.db (all blogs, or single blog if name given)");
+
+            Console.WriteLine("--output\t Export posts from TL.db back to .txt files in each blog's TTFolderPath");
+
+            Console.WriteLine("--revert [blogname]\t Recursively scan the PathInput tree and restore *.bak back to *.txt (current .txt saved as next-free .bkN); optional blogname filters by path substring");
+
+            Console.WriteLine("--correct [bakPath]\t Dry-run: report multi-line field updates available from a BAK directory");
+
+            Console.WriteLine("--correct --apply [bakPath]\t Apply BAK-file corrections to matching posts (prompts yes/no)");
+
+            Console.WriteLine("--updatepaths [rootPath]\t Read .tumblr/.tmblrpriv metadata from <root>\\Index and set Blogs.TTFolderPath");
+
+            Console.WriteLine("--importposts [path-to-posts.db]\t One-time migration: copy legacy ThreeTxtFileHelper posts.db rows into TL.db");
+
+            Console.WriteLine();
+            Console.WriteLine("Exit status: 0 = success; 1 = unexpected error; 2 = usage error (unknown command or bad/missing arguments); 3 = incomplete (--collect paused on a rate limit, or skipped posts after transient API failures) - relaunch to resume");
         }
 
         static void WritePostBlogsToFile(string outPath)
@@ -526,16 +582,11 @@ namespace URLNotesGrabberCORE
 
         protected static string NormalizeBlogFolderName(string folderName)
         {
-            return folderName
+            // Archive tools suffix duplicate blog folders with _1, _2, ... _10 and beyond. Strip only a
-                .Replace("_1", "")
+            // trailing numeric suffix: unanchored substring removal ate the "_1" inside "_10" and left the
-                .Replace("_2", "")
+            // "0" welded to the name (zomb-eh_10 -> zomb-eh0), and mangled any blog whose real name
-                .Replace("_3", "")
+            // contains "_1". A blog name is never a prefix of itself plus "_<digits>", so this is safe.
-                .Replace("_4", "")
+            return System.Text.RegularExpressions.Regex.Replace(folderName, @"_\d+$", "");
-                .Replace("_5", "")
-                .Replace("_6", "")
-                .Replace("_7", "")
-                .Replace("_8", "")
-                .Replace("_9", "");
         }
 
         static async Task FetchAndStoreReplyText(string blogName, long postID, long timestamp)
@@ -791,7 +842,9 @@ namespace URLNotesGrabberCORE
 
                 RateLimiter limiter = new SlidingWindowRateLimiter(new SlidingWindowRateLimiterOptions
                 {
-                    PermitLimit = 300,
+                    // 1/sec average, matching --collect: the CDN reacts to aggregate traffic from the IP,
+                    // not to per-command rates.
+                    PermitLimit = 60,
                     QueueProcessingOrder = QueueProcessingOrder.OldestFirst,
                     QueueLimit = 1,
                     Window = TimeSpan.FromMinutes(1),
@@ -827,7 +880,9 @@ namespace URLNotesGrabberCORE
 
                     while (hasMoreLikes)
                     {
-                        using RateLimitLease lease = limiter.AttemptAcquire(1);
+                        // Wait for a permit rather than giving up on one: the limiter paces the loop, it is
+                        // not a failure condition. Only one acquire is ever pending, so QueueLimit = 1 suffices.
+                        using RateLimitLease lease = await limiter.AcquireAsync(1);
                         if (!lease.IsAcquired)
                         {
                             Console.WriteLine("!@@@@@@@ - Rate Limited Exceeded: No Lease Available");
@@ -1073,6 +1128,43 @@ if (shouldInsert)
             }
         }
 
+        // Backoff between in-place retries of a transient infrastructure failure. Most CDN 403s and edge
+        // 5xxs clear within a few seconds, so retrying here saves the post its single attempt for the pass.
+        static readonly int[] TransientBackoffSeconds = { 1, 4, 10 };
+
+        // Fetches one page, retrying transient failures in place. Rate limits are returned to the caller
+        // untouched — those are handled by pausing the whole run, not by retrying this post.
+        static async Task<Root> FetchNotesPage(Tuple<string, long, long, long> post, string beforeTimestamp)
+        {
+            Root response = null!;
+
+            for (int attempt = 0; ; attempt++)
+            {
+                var key = ApiKeyPool.GetCurrentKey();
+                response = await APIAccess.GrabNotes(key, post.Item1, post.Item2, beforeTimestamp);
+
+                if (response.statusCode == "TooManyRequests")
+                {
+                    ApiKeyPool.MarkRateLimited(key, response.retryInSeconds > 0 ? response.retryInSeconds : 60);
+                    return response;
+                }
+
+                if (!response.transientFailure)
+                {
+                    // Only a response that actually reached the API says anything about the key's standing.
+                    ApiKeyPool.MarkAvailable(key);
+                    return response;
+                }
+
+                if (attempt >= TransientBackoffSeconds.Length)
+                    return response;
+
+                int delay = TransientBackoffSeconds[attempt];
+                Console.WriteLine($"[Transient] retry {attempt + 1}/{TransientBackoffSeconds.Length} in {delay}s");
+                await Task.Delay(delay * 1000);
+            }
+        }
+
         static async Task<string> GrabNotes(Tuple<string, long, long, long> post)
         {
             try
@@ -1090,18 +1182,16 @@ if (shouldInsert)
                 string beforeTimestamp = post.Item3.ToString();
                 bool hasReplies = false;
                 const int maxPages = 500;
-                var key = ApiKeyPool.GetCurrentKey();
+                var response = await FetchNotesPage(post, beforeTimestamp);
-                var response = await APIAccess.GrabNotes(key, post.Item1, post.Item2, beforeTimestamp);
 
                 if (response.statusCode == "TooManyRequests")
-                {
-                    int retry = response.retryInSeconds > 0 ? response.retryInSeconds : 60;
-                    ApiKeyPool.MarkRateLimited(key, retry);
                     return "TooManyRequests";
-                }
 
-                if (response.meta?.status != 429)
+                if (response.transientFailure)
-                    ApiKeyPool.MarkAvailable(key);
+                {
+                    Console.WriteLine($"[Skip] {post.Item1}/{post.Item2} — {response.statusCode} after {TransientBackoffSeconds.Length} retries");
+                    return "Transient";
+                }
 
                 if (IsNotFound(response))
                 {
@@ -1110,19 +1200,6 @@ if (shouldInsert)
                     Thread.Sleep(1000);
                     return "NotFound";
                 }
-                if (response == null)
-                {
-                    Console.WriteLine("##### Response is null - API Failure? ###");
-                    return "FAILURE";
-                }
-                if (response.statusCode == "TooManyRequests")
-                {
-                    int retry = response.retryInSeconds > 0 ? response.retryInSeconds : 60;
-                    ApiKeyPool.MarkRateLimited(key, retry);
-
-                    ApiKeyPool.SleepUntilAnyAvailable(30);
-                    return response.statusCode;
-                }
 
                 // Pagination loop
                 while (true)
@@ -1172,16 +1249,15 @@ if (shouldInsert)
                         break;
                     }
 
-                    key = ApiKeyPool.GetCurrentKey();
+                    response = await FetchNotesPage(post, beforeTimestamp);
-                    response = await APIAccess.GrabNotes(key, post.Item1, post.Item2, beforeTimestamp);
                     if (response.statusCode == "TooManyRequests")
-                    {
-                        int retry = response.retryInSeconds > 0 ? response.retryInSeconds : 60;
-                        ApiKeyPool.MarkRateLimited(key, retry);
                         return "TooManyRequests";
+
+                    if (response.transientFailure)
+                    {
+                        Console.WriteLine($"[Skip] {post.Item1}/{post.Item2} — {response.statusCode} on page {page} after {TransientBackoffSeconds.Length} retries");
+                        return "Transient";
                     }
-                    if (response.meta?.status != 429)
-                        ApiKeyPool.MarkAvailable(key);
 
                     if (IsNotFound(response))
                     {
@@ -1212,7 +1288,11 @@ if (shouldInsert)
             return "UNKNOWN";
         }
 
-        static async Task CollectNotes(string outPath, bool withoutNotesOnly = true, DateTime? beforeDate = null, bool managedRun = false)
+        // Consecutive transient failures that mean the API edge is rejecting traffic wholesale rather than
+        // blipping on one post. Past this, skipping post-by-post would just hammer a closed door.
+        const int MaxConsecutiveTransient = 10;
+
+        static async Task<int> CollectNotes(string outPath, bool withoutNotesOnly = true, DateTime? beforeDate = null, bool managedRun = false)
         {
             List<Tuple<string, long, long, long>> posts = DataAccess.GetPosts(withoutNotesOnly, beforeDate);
 
@@ -1221,9 +1301,14 @@ if (shouldInsert)
             // post that keeps returning FAILURE/UNKNOWN. Successful/NotFound posts drop out via the DB filter anyway.
             HashSet<(string, long)> attempted = new HashSet<(string, long)>();
 
+            int skipped = 0;
+            int consecutiveTransient = 0;
+
             RateLimiter limiter = new SlidingWindowRateLimiter(new SlidingWindowRateLimiterOptions
             {
-                PermitLimit = 300,
+                // 1/sec average. Sustained higher rates draw CDN-level 403s that the API's own rate-limit
+                // headers never warn about, so this sits well under the per-key quota on purpose.
+                PermitLimit = 60,
                 QueueProcessingOrder = QueueProcessingOrder.OldestFirst,
                 QueueLimit = 1,
                 Window = TimeSpan.FromMinutes(1),
@@ -1248,19 +1333,35 @@ if (shouldInsert)
 
                         ApiKeyPool.SleepUntilAnyAvailable(30);
 
-                        string status;
+                        // Wait for a permit rather than giving up on one: the limiter paces the loop, it is
-
+                        // not a failure condition. Only one acquire is ever pending, so QueueLimit = 1 suffices.
-                        using RateLimitLease lease = limiter.AttemptAcquire(1);
+                        using RateLimitLease lease = await limiter.AcquireAsync(1);
-                        if (lease.IsAcquired)
+                        if (!lease.IsAcquired)
                         {
+                            Console.WriteLine("!@@@@@@@ - Rate Limited Exceeded: No Lease Available");
+                            return 3; // abort without completing the run so a later launch resumes
+                        }
+
                         Console.WriteLine("{0,32} - {1,15} - {2}", post.Item1, post.Item2, DateTimeOffset.FromUnixTimeSeconds(post.Item4).ToString());
-                            status = await GrabNotes(post);
+                        string status = await GrabNotes(post);
+
+                        if (status == "Transient")
+                        {
+                            // Retries in GrabNotes are already exhausted. Skip the post so the pass can make
+                            // progress; it stays unmarked in the DB, so the next launch picks it up again.
+                            attempted.Add((post.Item1, post.Item2));
+                            skipped++;
+                            consecutiveTransient++;
+
+                            if (consecutiveTransient >= MaxConsecutiveTransient)
+                            {
+                                Console.WriteLine($"[Abort] {consecutiveTransient} consecutive transient failures - the API edge is rejecting traffic. Pausing run; relaunch to resume. ({skipped} post(s) skipped)");
+                                return 3;
+                            }
                         }
                         else
                         {
-                            Console.WriteLine("!@@@@@@@ - Rate Limited Exceeded: No Lease Available");
+                            consecutiveTransient = 0;
-                            return; // throttle: abort without completing the run so a later launch resumes
-                        }
 
                             if (status == "Success")
                             {
@@ -1277,8 +1378,8 @@ if (shouldInsert)
                             {
                                 // Throttle, not a real per-post failure: don't consume this post's single attempt.
                                 // Abort the pass without completing so a later launch resumes against the same cutoff.
-                            Console.WriteLine("GrabNotes Result: TooManyRequests - pausing run; relaunch to resume.");
+                                Console.WriteLine($"GrabNotes Result: TooManyRequests - pausing run; relaunch to resume. ({skipped} post(s) skipped)");
-                            return;
+                                return 3;
                             }
                             else
                             {
@@ -1286,6 +1387,7 @@ if (shouldInsert)
                                 attempted.Add((post.Item1, post.Item2));
                                 Console.WriteLine("GrabNotes Result: " + status);
                             }
+                        }
 
                         // Re-fetch the updated list after processing the current post
                         posts = DataAccess.GetPosts(withoutNotesOnly, beforeDate);
@@ -1298,14 +1400,42 @@ if (shouldInsert)
                     DataAccess.CompleteCollectRun();
                     Console.WriteLine("Full re-check run complete.");
                 }
+
+                if (skipped > 0)
+                {
+                    Console.WriteLine($"Pass finished with {skipped} post(s) skipped after transient failures; relaunch to retry them.");
+                    return 3;
+                }
+
+                return 0;
             }
             catch (Exception ex)
             {
                 Console.WriteLine(ex.ToString());
+                return 1;
             }
         }
 
 
+        // Field prefixes TraverseDirectory recognizes as the start of a new record field.
+        // Used to know where a multi-line Body/Downloaded files value ends.
+        private static readonly string[] TraverseDirectoryFieldPrefixes = new[]
+        {
+            "Post id:", "Reblog url:", "Reblog name:", "Reblog root url:", "Downloaded files:",
+            "Reblog key:", "Date:", "Body:", "Post url:", "Answer:", "Audio Caption:", "Blog Name:",
+            "Link:", "Photo Caption:", "Photo url:", "Question:", "Quote:", "Slug:", "Summary:",
+            "Tags:", "Title:"
+        };
+
+        private static bool IsTraverseDirectoryFieldLine(string line)
+        {
+            foreach (var prefix in TraverseDirectoryFieldPrefixes)
+            {
+                if (line.StartsWith(prefix, StringComparison.OrdinalIgnoreCase)) return true;
+            }
+            return false;
+        }
+
         static void TraverseDirectory(string path, string outPath, List<string> contains, ref int postsAdded, string blogName = "", string startFromBlogName = "", bool logRecordImports = false)
         {
             DateTime directoryStart = DateTime.Now;
@@ -1342,8 +1472,10 @@ if (shouldInsert)
                                 var urls = new List<string>();
                                 var reblog = new ReblogRecord();
 
-                                foreach (string line in File.ReadLines(file))
+                                string[] fileLines = File.ReadAllLines(file);
+                                for (int lineIndex = 0; lineIndex < fileLines.Length; lineIndex++)
                                 {
+                                    string line = fileLines[lineIndex];
                                     if (line.StartsWith("Post id:", StringComparison.OrdinalIgnoreCase))
                                     {
                                         if (reblog.reblogName != "." && reblog.postID != "." && reblog.date != "." && reblog.reblogURL != "." && reblog.downloadedFiles == ".")
@@ -1364,7 +1496,7 @@ if (shouldInsert)
                                                 DataAccess.AddPost(curDir, long.Parse(reblog.postID), reblog.reblogURL, reblog.date, reblog.postURL, reblog.slug, reblog.reblogKey,
                                                                     reblog.reblogName, reblog.summary, reblog.quote, reblog.body, reblog.tags, reblog.link, reblog.photoURL,
                                                                     reblog.photoCaption, reblog.downloadedFiles, reblog.audioCaption, reblog.question, reblog.answer,
-                                                                    reblog.title, false);
+                                                                    reblog.title, false, rootURL: reblog.rootURL);
                                                 recordImportStopwatch.Stop();
 
                                                 postsAdded++;
@@ -1389,9 +1521,21 @@ if (shouldInsert)
                                     {
                                         reblog.reblogName = line.Substring(13).Trim();
                                     }
+                                    if (line.StartsWith(@"Reblog root url:", StringComparison.OrdinalIgnoreCase))
+                                    {
+                                        reblog.rootURL = line.Substring(16).Trim();
+                                    }
                                     if (line.StartsWith(@"Downloaded files:", StringComparison.OrdinalIgnoreCase))
                                     {
-                                        reblog.downloadedFiles = line.Substring(17).Trim();
+                                        var valueLines = new List<string> { line.Substring(17).Trim() };
+                                        int nextLineIndex = lineIndex + 1;
+                                        while (nextLineIndex < fileLines.Length && !IsTraverseDirectoryFieldLine(fileLines[nextLineIndex]))
+                                        {
+                                            valueLines.Add(fileLines[nextLineIndex]);
+                                            nextLineIndex++;
+                                        }
+                                        reblog.downloadedFiles = string.Join("\n", valueLines).Trim();
+                                        lineIndex = nextLineIndex - 1;
                                     }
                                     if (line.StartsWith(@"Reblog key:", StringComparison.OrdinalIgnoreCase))
                                     {
@@ -1403,7 +1547,15 @@ if (shouldInsert)
                                     }
                                     if (line.StartsWith(@"Body:", StringComparison.OrdinalIgnoreCase))
                                     {
-                                        reblog.body = line.Substring(6).Trim();
+                                        var valueLines = new List<string> { line.Substring(6).Trim() };
+                                        int nextLineIndex = lineIndex + 1;
+                                        while (nextLineIndex < fileLines.Length && !IsTraverseDirectoryFieldLine(fileLines[nextLineIndex]))
+                                        {
+                                            valueLines.Add(fileLines[nextLineIndex]);
+                                            nextLineIndex++;
+                                        }
+                                        reblog.body = string.Join("\n", valueLines).Trim();
+                                        lineIndex = nextLineIndex - 1;
                                     }
                                     if (line.StartsWith(@"Post url:", StringComparison.OrdinalIgnoreCase))
                                     {
@@ -1492,7 +1644,7 @@ if (shouldInsert)
                                             DataAccess.AddPost(curDir, long.Parse(reblog.postID), reblog.reblogURL, reblog.date, reblog.postURL, reblog.slug, reblog.reblogKey,
                                                                     reblog.reblogName, reblog.summary, reblog.quote, reblog.body, reblog.tags, reblog.link, reblog.photoURL,
                                                                     reblog.photoCaption, reblog.downloadedFiles, reblog.audioCaption, reblog.question, reblog.answer,
-                                                                    reblog.title, true);
+                                                                    reblog.title, true, rootURL: reblog.rootURL);
                                             recordImportStopwatch.Stop();
 
                                             postsAdded++;

URLNotesGrabberCORE/Properties/launchSettings.json

@@ -2,7 +2,7 @@
   "profiles": {
     "URLNotesGrabberCORE": {
       "commandName": "Project",
-      "commandLineArgs": "-collect 1 -api4"
+      "commandLineArgs": "--collect 1 --api4"
     }
   }
 }

URLNotesGrabberCORE/ResponseNotes.cs

@@ -85,6 +85,10 @@ namespace URLNotesGrabberCORE
         public int retryInSeconds { get; set; }
 
         public string rawJson { get; set; }
+
+        // The request never reached the Tumblr API (transport error, or an edge/CDN response with a
+        // non-JSON body). Says nothing about the post, so the caller should retry rather than fail it.
+        public bool transientFailure { get; set; }
     }
 
     // Classes for Posts API endpoint (for reply_text)

URLNotesGrabberCORE/TL.db.md

@@ -0,0 +1,333 @@
+# `TL.db` — schema notes
+
+The SQLite database behind **URLNotesGrabberCORE** and its sibling crawlers, and the one
+[Rolodex](https://git.basso.land/jim/Rolodex) reads.
+
+Everything below was read out of the live file, not inferred from code. Counts are as of
+**2026-07-29**; re-run the queries at the bottom to refresh them.
+
+- Journal mode: **WAL** — `TL.db-wal` and `TL.db-shm` live beside the file and are part of
+  the database. Copying `TL.db` alone gives you whatever was last checkpointed, not the
+  current state.
+- Page size: 4096.
+
+---
+
+## The three content tables
+
+| Table | Rows | What it is |
+|---|--:|---|
+| `Blogs` | 144,367 | The crawl registry — one row per known blog, plus crawl-state flags |
+| `Posts` | 14,589 | Stored post content. Only 3,602 blogs actually have any |
+| `Notes` | 1,189,604 | The engagement graph: `NoteBlogName` acted on `(RootBlogName, PostID)` |
+
+The engagement graph is the interesting part. 31,888 distinct blogs appear as engagers —
+far more than the 3,602 that have stored posts — which is what makes this a social graph
+rather than a post archive.
+
+### `Blogs`
+
+```sql
+CREATE TABLE "Blogs" (
+    "BlogName"              TEXT,
+    "HasBeenOutput"         INTEGER DEFAULT 0,
+    "IsActive"              INTEGER DEFAULT 1,
+    "DateAdded"             TEXT NOT NULL DEFAULT '12/24/25',
+    "ByLikes"               INTEGER NOT NULL DEFAULT 0,
+    "LikesPulled"           INTEGER NOT NULL DEFAULT 0,
+    "LikesCursor"           INTEGER DEFAULT 0,
+    "DateModified"          TEXT,
+    "DateCreated"           TEXT,
+    LikesNewestTimestamp    INTEGER DEFAULT 0,
+    LikesLastRefreshed      INTEGER DEFAULT 0,
+    LikesLastNewCount       INTEGER DEFAULT 0,
+    TTFolderPath            TEXT,
+    PRIMARY KEY("BlogName")
+);
+```
+
+`BlogName` is the primary key, so it is the only indexed way in. There is no index on any
+flag or date — filtering or sorting on those scans all 144k rows, which is affordable
+here and is not on `Notes`.
+
+Flag distribution: `IsActive = 1` on 144,366 of 144,367 rows, `HasBeenOutput = 1` on
+5,369, `ByLikes = 1` on 2. `IsActive` carries a second meaning as of Rolodex — see
+[`Blogs.IsActive`](#blogsisactive--now-written-by-two-applications) below.
+
+The columns after `DateCreated` were added later by `ALTER TABLE`, which is why they carry
+no quoting in the stored DDL. That is the normal way this schema grows.
+
+**`DateAdded` is not written consistently.** 126,423 rows hold ISO `yyyy-MM-dd HH:mm:ss`;
+17,944 hold US-format `M/d/yy` from a bulk import. As text those two sort into different
+parts of the table, so anything ordering or range-filtering on this column has to
+normalise first — see `DateSql` in Rolodex.
+
+### `Posts`
+
+```sql
+CREATE TABLE "Posts" (
+    "BlogName"              TEXT,
+    "PostID"                INTEGER,
+    "HasNotesGathered"      INTEGER DEFAULT 0,
+    "reblogURL"             TEXT,
+    "NotFound"              INTEGER DEFAULT 0,
+    "PostDate"              TEXT,
+    "NotesGatheredDateTime" INTEGER NOT NULL DEFAULT 1729746000,
+    "HasImage"              INTEGER NOT NULL DEFAULT 0,
+    "PostURL"               TEXT,
+    "Slug"                  TEXT,
+    "ReblogKey"             TEXT,
+    "ReblogName"            TEXT,
+    "Summary"               TEXT,
+    "Quote"                 TEXT,
+    "Body"                  TEXT,
+    "Tags"                  TEXT,
+    "Link"                  TEXT,
+    "PhotoURL"              TEXT,
+    "PhotoCaption"          TEXT,
+    "DownloadedFiles"       TEXT,
+    "AudioCaption"          TEXT,
+    "Question"              TEXT,
+    "Answer"                TEXT,
+    "Title"                 TEXT,
+    "ByLikes"               INTEGER NOT NULL DEFAULT 0,
+    "RootBlogName"          TEXT,
+    "RootURL"               TEXT,
+    "DateModified"          TEXT,
+    "DateCreated"           TEXT,
+    PostType                TEXT,
+    PRIMARY KEY("BlogName","PostID")
+);
+```
+
+**The key is `(BlogName, PostID)`, not `PostID`.** This matters more than it looks: 325
+post IDs exist under more than one blog, so an ID on its own is both ambiguous *and*
+unindexed. Any lookup should carry the blog name, and a batch lookup should group by blog
+so it stays on the leading column of the key.
+
+Notable:
+
+- **`PostType` is `NULL` on all 14,589 rows.** The column exists but nothing has ever
+  populated it. Treat it as unpopulated rather than as a type discriminator.
+- `HasImage = 1` on 14,268 rows — nearly all of them. It records that the post *had* a
+  picture, not that a usable URL was kept, so it is not a reliable predictor that anything
+  will render.
+- `PhotoURL` is largely unused; in practice the image markup lives inside `Body`.
+- `NotFound = 1` on 4,663 rows — posts that have since been deleted upstream.
+- The content columns (`Body`, `Quote`, `Question`, `Answer`, …) are the heavy ones. List
+  views should not select them.
+
+### `Notes`
+
+```sql
+CREATE TABLE "Notes" (
+    "RootBlogName"    TEXT,
+    "PostID"          INTEGER,
+    "NoteBlogName"    TEXT,
+    "TimeStamp"       INTEGER,
+    "Type"            TEXT,
+    "replyText"       TEXT DEFAULT '.',
+    "DatetimeCrawled" TEXT DEFAULT '2/12/26 12am',
+    "DateModified"    TEXT,
+    "DateCreated"     TEXT,
+    PRIMARY KEY("RootBlogName","PostID","TimeStamp","Type","NoteBlogName")
+);
+
+CREATE INDEX "Notes_idx_06e01ae3" ON "Notes" ("TimeStamp" DESC);
+CREATE INDEX "ix_NoteBlogName01"  ON "Notes" ("NoteBlogName");
+```
+
+One row per engagement event. `TimeStamp` is **unix seconds** — unlike every date column
+elsewhere in the schema, which are text.
+
+| `Type` | Rows | Share |
+|---|--:|--:|
+| `like` | 947,955 | 79.7% |
+| `reblog` | 224,323 | 18.9% |
+| `reply` | 15,201 | 1.3% |
+| `posted` | 2,106 | 0.2% |
+| `post_attribution` | 19 | — |
+
+At 1.19M rows this is the table that dictates how the whole database has to be queried:
+
+- **Nothing should run an unbounded `SELECT` or a bare `COUNT(*)` here.** A count scans
+  the lot on every call.
+- The only fast access paths are the primary key's leading columns (`RootBlogName`, then
+  `PostID`) and `ix_NoteBlogName01` on `NoteBlogName`. "Notes received by a blog" and
+  "notes given by a blog" are both cheap; almost nothing else is.
+- Ordering by anything but `TimeStamp` is a full sort of whatever the filters leave.
+- `replyText` is `'.'` on 1,174,706 rows — only `reply` notes carry real text.
+
+### Referential integrity
+
+There are no foreign keys, and the tables do not perfectly agree:
+
+- 4 `Posts` rows name a blog with no `Blogs` row.
+- 15 of the 31,888 distinct engagers have no `Blogs` row.
+
+So a name appearing in `Notes` or `Posts` is not a guarantee that the registry knows about
+it. Joins from those tables back to `Blogs` should tolerate a miss.
+
+---
+
+## The `'.'` placeholder convention
+
+**The crawler writes a single dot into text columns it has no value for, rather than
+`NULL`.** This is the single most surprising thing about the schema and it affects every
+consumer.
+
+| Column | `'.'` rows |
+|---|--:|
+| `Notes.replyText` | 1,174,706 |
+| `Posts.Title` | 13,144 |
+| `Posts.Body` | 172 |
+
+Any query whose output reaches a human should collapse it:
+
+```sql
+NULLIF(NULLIF(SomeColumn, '.'), '') AS SomeColumn
+```
+
+Empty string turns up too, hence the double `NULLIF`. Not every column is affected —
+`Blogs.TTFolderPath` and `Blogs.DateModified` currently have zero dot rows — but new
+columns tend to acquire them, so treat cleaning as the default for any text column
+rendered to a user.
+
+---
+
+## Supporting tables
+
+Crawler bookkeeping. Rolodex ignores all of these.
+
+| Table | Rows | What it is |
+|---|--:|---|
+| `DailyAPICount` | 133 | `(Date TEXT PK, APICount INTEGER)` — per-day API call tally against the rate limit |
+| `ApiKeyPoolState` | 2 | `(KeyName TEXT PK, RetryUntil INTEGER)` — per-key backoff; `RetryUntil` is unix seconds |
+| `ApiKeyPoolMeta` | 1 | `(Id PK CHECK (Id = 1), LastIndex)` — round-robin cursor. Singleton by check constraint |
+| `CollectRunState` | 1 | `(Id PK CHECK (Id = 1), RunCutoff, RunComplete, RunStarted, RunCompletedAt)` — resume state for an interrupted collection run. Also a singleton |
+
+---
+
+## `Blogs.IsActive` — now written by two applications
+
+`IsActive` has always been the crawler's work-selection flag. `GetBlogs` in
+`DataAccess.cs` joins on it to decide what to collect:
+
+```sql
+SELECT NoteBlogName, count(*) FROM notes
+INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName
+WHERE blogs.IsActive = @isActive AND ...
+```
+
+Nothing inside the crawler *writes* it — it is an input, set from outside.
+
+**Rolodex is now one of the things that sets it.** Removing a blog through the Rolodex UI
+runs exactly this:
+
+```sql
+UPDATE Blogs SET IsActive = 0 WHERE BlogName = ?;
+```
+
+Rolodex adds no column and changes no schema. It reuses this flag because the two meanings
+were judged to be one decision: a blog you do not want in the browsing UI is a blog you do
+not want to keep crawling. Removal therefore stops collection, and the Rolodex
+confirmation screen says so before anyone commits.
+
+- `1` (or absent/NULL) — live. Crawled, and visible in Rolodex.
+- `0` — removed. Not crawled, hidden from the Rolodex registry, dashboard counts and
+  engagement rollups.
+
+Restoring is the same `UPDATE` with a `1`. Nothing is destroyed either way: the blog's
+`Posts` and `Notes` rows are never touched, and Rolodex deliberately keeps showing them
+under its Posts and Notes pages. Removing a blog hides the blog, not what it collected.
+
+### What other tools need to know
+
+1. **Setting `IsActive = 0` now also hides the blog from Rolodex**, and setting it back to
+   `1` makes it reappear. If another tool deactivates blogs in bulk, it is also removing
+   them from the browsing UI — which may be exactly right, but it is no longer a
+   crawler-only decision.
+2. **Re-crawling a removed blog will not bring it back**, since nothing in the crawler
+   writes the flag. An `INSERT OR REPLACE` on the `Blogs` row *would*, by resetting it to
+   the column default of `1`. Prefer an `UPDATE` of the specific columns, or
+   `INSERT … ON CONFLICT DO UPDATE SET` naming only the columns being refreshed.
+3. **NULL is treated as live.** The column is `INTEGER DEFAULT 1` with no `NOT NULL`, so
+   Rolodex reads it through `COALESCE(IsActive, 1)`. A NULL therefore leaves the blog
+   visible rather than stranding it outside both the registry and the removed list, where
+   no screen could reach it. Write `0` or `1`, not NULL.
+4. **Backing the feature out is a configuration change, not a migration.** Because there is
+   no Rolodex-owned column, setting `Rolodex__EnableBlogDeletion=false` is the whole of it;
+   there is nothing to drop. Any blogs already at `IsActive = 0` simply go back to being
+   ordinary inactive blogs.
+
+---
+
+## `Posts.IsActive` and `Notes.IsActive` — optional, and not in this database yet
+
+The same flag is being extended to the two content tables, with the same meaning: `0` is
+removed, anything else — including `NULL` — is live. **Neither column exists in the live
+`TL.db` as of 2026-07-29**; the DDL quoted above for `Posts` and `Notes` is complete. Like
+`Blogs.IsActive`, they are written from outside this crawler.
+
+The crawler therefore treats both as optional, and as nothing it owns:
+
+- **It never writes them.** No `INSERT` column list names `IsActive`, no `UPDATE` sets it,
+  and `MapPrefixToColumn` — the only place a column name is chosen at runtime — cannot map
+  to it. Re-crawling a removed post or note refreshes its content and leaves the flag at
+  `0`. There is no `INSERT OR REPLACE` on `Posts` or `Notes` for a default to be reset by.
+- **It filters on them only when they exist.** `HasIsActiveColumn` in `DataAccess.cs` asks
+  `PRAGMA table_info` once per table per database path and caches the answer; the filter
+  is `COALESCE(IsActive, 1) = 1`, and it is dropped entirely when the column is absent.
+  Naming a missing column is a hard SQLite error, so this is what lets one build run
+  against databases on both sides of the change. The cache lives for the process — adding
+  the columns to a live database takes effect on the next run.
+
+Every read that selects posts or notes carries the filter: `GetPosts`, `GetReplies`,
+`GetRepliesWithMissingText`, `GetRepliesWithFilledText`, `GetAllPostTextColumns`,
+`GetAllPostsForBlog`, `GetPost`, `GetPostByIdAnyBlog`, and the engagement queries that
+count or join `Notes` (`GetBlogs`, `GetBlogsAll`, `GetBlogsForLikes`). The one deliberate
+omission is the `LEFT JOIN Notes` in `GetPosts`: nothing is selected from it and it can
+neither add nor remove a row, so filtering it would buy nothing.
+
+`LegacyPostsDbImporter` is unfiltered too — it reads a foreign legacy database whose
+`Posts` table is not this schema.
+
+Two consequences worth stating plainly, both inherited from how `Blogs.IsActive` is
+handled:
+
+1. **Removal hides a row; it does not freeze it.** The write paths are keyed on a post the
+   caller already selected, so an ingest or a correction run still overwrites the content
+   of a removed post. Only selection is filtered.
+2. **`NULL` is live.** Write `0` or `1`, not `NULL`, but a `NULL` leaves the row visible
+   rather than stranding it.
+
+---
+
+## Reproducing the numbers
+
+```sql
+SELECT 'Blogs', COUNT(*) FROM Blogs
+UNION ALL SELECT 'Posts', COUNT(*) FROM Posts
+UNION ALL SELECT 'Notes', COUNT(*) FROM Notes;
+
+-- note type mix
+SELECT Type, COUNT(*) FROM Notes GROUP BY Type ORDER BY 2 DESC;
+
+-- the two date shapes in Blogs.DateAdded
+SELECT CASE WHEN DateAdded LIKE '____-__-__%' THEN 'ISO' ELSE 'US' END, COUNT(*)
+FROM Blogs GROUP BY 1;
+
+-- post IDs that are ambiguous without a blog name
+SELECT COUNT(*) FROM (
+    SELECT PostID FROM Posts GROUP BY PostID HAVING COUNT(DISTINCT BlogName) > 1);
+
+-- rows that reference a blog the registry does not have
+SELECT COUNT(*) FROM Posts p
+WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = p.BlogName);
+```
+
+Open the file read-only so an inspection can never disturb a running crawl:
+
+```bash
+sqlite3 "file:TL.db?mode=ro" ".schema"
+```

verify-db-schema.sql

@@ -142,6 +142,9 @@ ORDER BY et.tbl;
 -- 1c. EXTRA / UNEXPECTED COLUMNS: present in the DB but not in the expected
 --     list above. Informational only -- e.g. a NEWER backup, or a column this
 --     script's expected-list hasn't been updated for. Not an error by itself.
+--     Posts.IsActive and Notes.IsActive are listed here and NOT in 1a on
+--     purpose: they are written by other tools, the app only reads them when
+--     present, and it must not be told to add them. See TL.db.md.
 WITH expected(tbl, col) AS (
   VALUES
     ('Posts','BlogName'),('Posts','PostID'),('Posts','HasNotesGathered'),('Posts','reblogURL'),
@@ -150,14 +153,14 @@ WITH expected(tbl, col) AS (
     ('Posts','Quote'),('Posts','Body'),('Posts','Tags'),('Posts','Link'),('Posts','PhotoURL'),
     ('Posts','PhotoCaption'),('Posts','DownloadedFiles'),('Posts','AudioCaption'),('Posts','Question'),
     ('Posts','Answer'),('Posts','Title'),('Posts','ByLikes'),('Posts','RootBlogName'),('Posts','RootURL'),
-    ('Posts','DateModified'),('Posts','DateCreated'),('Posts','PostType'),
+    ('Posts','DateModified'),('Posts','DateCreated'),('Posts','PostType'),('Posts','IsActive'),
     ('Blogs','BlogName'),('Blogs','HasBeenOutput'),('Blogs','IsActive'),('Blogs','DateAdded'),
     ('Blogs','ByLikes'),('Blogs','DateModified'),('Blogs','DateCreated'),('Blogs','LikesPulled'),
     ('Blogs','LikesCursor'),('Blogs','LikesNewestTimestamp'),('Blogs','LikesLastRefreshed'),
     ('Blogs','LikesLastNewCount'),('Blogs','TTFolderPath'),
     ('Notes','RootBlogName'),('Notes','PostID'),('Notes','NoteBlogName'),('Notes','TimeStamp'),
     ('Notes','Type'),('Notes','DatetimeCrawled'),('Notes','DateModified'),('Notes','DateCreated'),
-    ('Notes','replyText'),
+    ('Notes','replyText'),('Notes','IsActive'),
     ('DailyAPICount','Date'),('DailyAPICount','APICount'),
     ('ApiKeyPoolState','KeyName'),('ApiKeyPoolState','RetryUntil'),
     ('ApiKeyPoolMeta','Id'),('ApiKeyPoolMeta','LastIndex')