feat: fold the TTFolderPath refresh into --output

TL.db syncs between machines whose absolute paths differ, so a single
TTFolderPath column cannot be correct on both at once -- the stored paths are
only trustworthy on the machine that wrote them. That made --updatepaths a
mandatory prelude to every --output rather than the one-time setup step it
looks like.

--output now refreshes the column from the TumblThree Index metadata before
exporting. The root comes from the first non-flag argument, else
appSettings:PathTTRoot. With no root available it says so and exports whatever
TL.db already holds; a root whose Index folder is missing is a hard stop, since
silently exporting stale paths is the failure this change exists to prevent.
--norefresh skips the refresh for a pure export.

The scan is extracted from UpdateBlogPathsRunner.Run into a reusable Scan() that
returns counts instead of only printing them, so --updatepaths keeps its
per-file detail while --output prints a single summary line rather than a few
hundred lines ahead of the export.

Verified against a throwaway database: a stale cross-machine path is repaired
and the export lands in the correct local folder; no configured root warns and
continues (exit 0); a missing Index folder stops (exit 1); --norefresh skips the
refresh and exports (exit 0).

Co-Authored-By: Claude Opus 5 <[email protected]>

AGENTS.md

@@ -11,6 +11,7 @@
 - `ResponseNotes.cs`: Tumblr API response models
 - Round-robin API key rotation with rate-limit tracking
 - Automatic console color assignment per API key for output differentiation
+- No-argument mode (`Program.TraverseDirectory`) ingests `.txt` blog export files into `Posts` via `DataAccess.AddPost`. Recognized field prefixes live in `TraverseDirectoryFieldPrefixes`; `Body:` and `Downloaded files:` collect every following line up to the next recognized prefix (multi-line values). `RootURL` is populated from a `Reblog root url:` line the same way it's populated from the API-based `--likes` flow — both paths converge on `DataAccess.AddPost`'s `rootURL` parameter, which `UpdatePost` only overwrites when the incoming value is non-empty (existing `RootURL` is preserved otherwise)
 
 ## Developer Guidelines
 
@@ -27,6 +28,132 @@
 - Preserve console color state: use save/restore pattern for temporary color changes
 - API rate limits must use `ApiKeyPool.MarkRateLimited()`/`MarkAvailable()`
 
+### API Failure Classification
+Tumblr sits behind a CDN that returns HTML error pages (403, 5xx) which never reach the API. These
+say nothing about the item being fetched, so they must not be recorded as per-item failures.
+
+- A response body that will not parse as JSON did not come from the API. Flag it with
+  `Root.transientFailure`, never as `FAILURE`
+- Transient failures retry in place (`TransientBackoffSeconds`) before the item is skipped; a skipped
+  item stays unmarked in the DB so a later launch retries it
+- `MaxConsecutiveTransient` consecutive transient failures aborts the pass rather than skipping
+  item-by-item against an edge that is refusing all traffic
+- Only call `ApiKeyPool.MarkAvailable()` on a response that actually reached the API. A transport or
+  CDN failure says nothing about the key's standing and must not clear its flag
+- Only a real HTTP 429 (or `meta.status == 429`) counts as a rate limit. Do not infer one from the
+  presence of `X-RateLimit-*` headers, which Tumblr sends on every response
+- Rate limiters must pace with `await AcquireAsync()`. `AttemptAcquire()` does not wait, so a
+  saturated window aborts the run instead of throttling it
+- Long-running commands return exit 3 when a pass ends incomplete (rate-limit pause, breaker trip, or
+  skipped items), so a caller can distinguish that from a clean run
+
+### `IsActive` Is Not Ours To Write
+`Blogs.IsActive`, `Posts.IsActive` and `Notes.IsActive` are removal flags set by other tools
+(Rolodex). `0` means removed; anything else, including `NULL`, means live. Full detail in
+`URLNotesGrabberCORE/TL.db.md`.
+
+- **Never write any `IsActive` column.** Not in an `INSERT` column list, not in an `UPDATE`,
+  and never via `INSERT OR REPLACE` on these tables — that resets the column default and
+  un-removes the row. Re-crawling a removed row must refresh its content and leave the flag
+  where it was
+- **Filter at selection, not at write.** Every query that *selects* posts, notes or blogs
+  excludes removed rows. Update statements stay keyed on a row the caller already selected;
+  filtering them would spend API quota and then fail to persist the result
+- `Posts.IsActive` and `Notes.IsActive` are **optional** — they are absent from databases
+  that predate them, and naming a missing column is a hard SQLite error. Compose the filter
+  with `AndIsActive`/`WhereIsActive` in `DataAccess.cs`, which return
+  `COALESCE(IsActive, 1) = 1` only when `HasIsActiveColumn` finds the column. `Blogs.IsActive`
+  is not optional and is filtered directly
+- Do not add these columns from this app, and do not add them to the missing-column list in
+  `verify-db-schema.sql`
+
+### `DateModified` Tracks Real Changes Only
+`Blogs.DateModified`, `Posts.DateModified` and `Notes.DateModified` must move only when a
+column beside `DateModified` itself actually changed. Re-crawling or re-ingesting identical
+content has to leave the row — and its timestamp — untouched, or downstream consumers cannot
+tell a refreshed row from a rewritten one.
+
+- Enforce it in the `WHERE` clause, not in C#. Every `UPDATE` that sets `DateModified` ends
+  with an `AND (<col> <> @param OR ...)` term covering every column in its `SET` list, so
+  SQLite matches zero rows on a no-op and never writes
+- Compare NULL-safely: `IFNULL(col, '') <> IFNULL(@param, '')` for text,
+  `IFNULL(col, 0) <> @param` for integer flags. A bare `col <> @param` is NULL on a NULL
+  column and silently skips the row that most needs writing
+- Where NULL is not equivalent to the default, spell it out. The `HasBeenOutput = 0` stamps
+  use `(HasBeenOutput IS NULL OR HasBeenOutput <> 0)` because the selection queries test
+  `HasBeenOutput = 0`, which a NULL would never match
+- Dynamic `SET` lists (`UpdatePostContentFields`) build the guard alongside the assignments
+  so the two lists cannot drift apart
+- These statements now return 0 rows for "found but unchanged" as well as "not found".
+  Callers that read `ExecuteNonQuery()` must not treat 0 as "row missing"
+
+**`Posts.NotesGatheredDateTime` is crawl bookkeeping, not content.** It moves on every
+`-collect` pass and says nothing about the post, so it must never move `DateModified` on its
+own. `UpdatePostMarkNotesCollected` still writes it every pass but wraps the timestamp in
+`DateModified = CASE WHEN IFNULL(HasNotesGathered, 0) <> 1 THEN @dateModified ELSE
+DateModified END` — SQLite evaluates `SET` expressions against the pre-`UPDATE` row, so only
+the flag flipping counts as a modification. Use this shape for any column that has to be
+refreshed unconditionally without being a change. `Blogs.LikesLastRefreshed` is the
+deliberate exception: a refresh pass is treated as a real event on the blog row.
+
+**`Blogs.DateAdded` is write-once.** `AddBlog`'s `INSERT` is the only place that sets it. A
+new post arriving for a known blog reopens `HasBeenOutput` but must leave `DateAdded` alone —
+a new post is not a new blog, and rewriting the column both destroys the registration date
+and makes every insert look like a change.
+
+**`"."` in a `Posts` content field means "not supplied", not "empty".** `ReblogRecord`
+(`TraverseDirectory`'s parser for the local `.txt` export tree) and the `--likes` API path
+both default every content field to the literal string `"."` when their source has no value
+for it, then pass that straight to `UpdatePost`. A blog with two export folders in different
+field formats (a duplicate `_2` folder, or a Tumblr export whose field set changed over time)
+sends one record with a real `Title`/`Tags`/`Slug` and another with those fields `"."`
+because that format never had a line for them — and without a guard, re-importing both on
+every run flips the row back and forth forever, bumping `DateModified` on every pass even
+though the true content never changes.
+
+- Every content column in `UpdatePost`'s `SET` list is guarded the same way `RootBlogName`/
+  `RootURL` already were: `col = CASE WHEN @col = '.' THEN col ELSE @col END`. A `"."`
+  parameter leaves the existing value alone instead of overwriting it
+- The change-detection `WHERE` clause carries the same exception —
+  `(@col <> '.' AND IFNULL(col, '') <> @col) OR ...` — so a `"."`-only difference does not
+  make the statement fire at all, and `DateModified` stays put
+- Deliberately narrow: only the literal `"."` is the sentinel. An explicit empty string from
+  a real record still overwrites, same as before this fix. `postID`, `BlogName`, `hasImage`,
+  `ByLikes` are not part of this convention and are unaffected
+- If a new content field is added to `Posts`/`UpdatePost`, decide explicitly whether its
+  source can legitimately supply `"."` as "field absent" before deciding whether it needs
+  the same `CASE` treatment — don't assume every column needs it
+
+**`--ingest` (`UpsertPostFromTextFile`) uses `NULL`, not `"."`, for the same "field absent"
+convention, and reconciling exactly this kind of duplicate IS the feature's job.**
+`IngestMode` strips a trailing `_N` from the folder name before it ever reaches
+`UpsertPostFromTextFile`, so a duplicate export folder collapses onto the same `BlogName` on
+purpose — the whole point is to merge multiple differently-formatted files for the same post
+into one row. `IngestMode.G(key)` returns `null` (not `"."`) when a field's line is absent
+from a given file, `LegacyPostsDbImporter` passes `null` straight from a `NULL` source column,
+and files are walked in raw filesystem enumeration order — never sorted — so which file's call
+lands last for a given `(BlogName, PostID)` is arbitrary.
+
+- Before the fix, the `UPDATE` branch set every column unconditionally, so whichever file
+  processed last for a `PostID` would null out every field its own record didn't carry —
+  silently erasing real `Title`/`Slug`/`Tags`/… another file had, the opposite of what
+  `--ingest` exists to do. This is worse than the `"."` case above: that one only caused
+  churn (the two writes canceled out); this one loses data, and which posts lose which
+  fields depends on filesystem enumeration order
+- Same shape of fix, `NULL` instead of `"."` as the sentinel: `col = CASE WHEN @col IS NULL
+  THEN col ELSE @col END` in the `SET` list, `(@col IS NOT NULL AND IFNULL(col, '') <> @col)
+  OR ...` in the change-detection
+- Same narrow rule: only `NULL` (the field's line was never present in this file) is the
+  sentinel. `G()` already distinguishes this from "present but blank" — a dictionary miss is
+  `null`, an empty value after the prefix is `""` — so an explicitly blank field still
+  overwrites
+- `HasImage` is **not** guarded and remains a known gap: `IngestMode` always computes a
+  concrete `bool` (defaulting `false` when a file has no `Has Image:` line), so there is no
+  way for this function to tell "this format says no image" from "this format doesn't report
+  it at all" without changing the parameter to `bool?` and threading that through
+  `IngestMode`/`LegacyPostsDbImporter`. Fix this the same way if `--ingest` is observed
+  downgrading a post's `HasImage` from `1` to `0`
+
 ### Testing
 - No existing test suite; use xUnit if adding tests
 - Test critical logic: `ApiKeyPool` init, color parsing, config persistence

URLNotesGrabberCORE/DataAccess.cs

@@ -37,6 +37,7 @@ namespace URLNotesGrabberCORE
         public string reblogKey;
         public string reblogName;
         public string reblogURL;
+        public string rootURL;
         public string slug;
         public string summary;
         public string tags;
@@ -60,6 +61,7 @@ namespace URLNotesGrabberCORE
             reblogKey = ".";
             reblogName = ".";
             reblogURL = ".";
+            rootURL = ".";
             slug = ".";
             summary = ".";
             tags = ".";
@@ -78,6 +80,10 @@ namespace URLNotesGrabberCORE
         private static SQLiteConnection? _importConnection;
         private static HashSet<string>? _importBlogCache;
         private static readonly object _importSessionLock = new object();
+        // AddAPICount/UpdateAPICount run once per API call. A schema-level failure there repeats
+        // identically every time, so log each distinct message once instead of per call.
+        private static readonly HashSet<string> _apiCountFailuresLogged = new HashSet<string>();
+        private static readonly object _apiCountFailureLock = new object();
 
         static DataAccess()
         {
@@ -100,6 +106,10 @@ namespace URLNotesGrabberCORE
             }
         }
 
+        // The database every DataAccess call defaults to, exposed so modes can report
+        // which file they actually read when their results are surprising.
+        public static string GetActiveDbPath() => GetDefaultDbPath();
+
         private static string GetDefaultDbPath()
         {
             if (_cachedDbPath != null)
@@ -118,6 +128,91 @@ namespace URLNotesGrabberCORE
             return _cachedDbPath;
         }
 
+        #region IsActive
+
+        // Posts.IsActive and Notes.IsActive mean the same thing Blogs.IsActive does:
+        // 0 = removed elsewhere (Rolodex), anything else (including NULL) = live.
+        //
+        // This crawler is a reader of all three. It never writes any IsActive column --
+        // no INSERT lists it, no UPDATE sets it, and MapPrefixToColumn cannot map to it --
+        // so a row removed in Rolodex is never resurrected by a re-crawl.
+        //
+        // Unlike Blogs.IsActive, the Posts and Notes columns are optional: they are added
+        // from outside this app and are absent from databases that predate them. Naming a
+        // missing column is a hard SQLite error ("no such column"), so every read asks the
+        // schema first and simply drops the filter when the column is not there. The answer
+        // is cached per database path, so adding the columns to a live database takes effect
+        // on the next run.
+        private static readonly Dictionary<string, bool> _isActiveColumnCache =
+            new Dictionary<string, bool>(StringComparer.OrdinalIgnoreCase);
+        private static readonly object _isActiveColumnLock = new object();
+
+        private static bool HasIsActiveColumn(string table, string? DBPath)
+        {
+            DBPath ??= GetDefaultDbPath();
+            string cacheKey = DBPath + "|" + table;
+
+            lock (_isActiveColumnLock)
+            {
+                if (_isActiveColumnCache.TryGetValue(cacheKey, out bool cached))
+                    return cached;
+            }
+
+            bool exists = false;
+            try
+            {
+                using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+                connection.Open();
+
+                using SQLiteCommand command = new SQLiteCommand($"PRAGMA table_info({table});", connection);
+                using SQLiteDataReader reader = command.ExecuteReader();
+                while (reader.Read())
+                {
+                    if (reader.GetString(1).Equals("IsActive", StringComparison.OrdinalIgnoreCase))
+                    {
+                        exists = true;
+                        break;
+                    }
+                }
+            }
+            catch (Exception ex)
+            {
+                // Breakpoint here
+                // An unreadable schema is treated as "no column" so the caller's query still runs.
+                Console.WriteLine($"Error checking {table}.IsActive column: {ex.Message}");
+            }
+
+            lock (_isActiveColumnLock)
+            {
+                _isActiveColumnCache[cacheKey] = exists;
+            }
+
+            return exists;
+        }
+
+        /// <summary>
+        /// " AND COALESCE(alias.IsActive, 1) = 1" when the table carries the column, "" when it
+        /// does not. NULL is read as live, the same way Rolodex reads Blogs.IsActive.
+        /// </summary>
+        private static string AndIsActive(string table, string alias = "", string? DBPath = null)
+        {
+            if (!HasIsActiveColumn(table, DBPath)) return string.Empty;
+
+            string qualifier = string.IsNullOrEmpty(alias) ? string.Empty : alias + ".";
+            return $" AND COALESCE({qualifier}IsActive, 1) = 1";
+        }
+
+        /// <summary>
+        /// Same filter as <see cref="AndIsActive"/>, for a query that has no WHERE clause yet.
+        /// </summary>
+        private static string WhereIsActive(string table, string alias = "", string? DBPath = null)
+        {
+            string clause = AndIsActive(table, alias, DBPath);
+            return clause.Length == 0 ? string.Empty : " WHERE" + clause.Substring(" AND".Length);
+        }
+
+        #endregion IsActive
+
         public static string Q(string input)
         {
             return "'" + input.Replace("'", "''") + "'";
@@ -458,6 +553,8 @@ namespace URLNotesGrabberCORE
             {
                 if (ownsConnection) connection.Open();
 
+                // IsActive is deliberately absent from this column list: a post removed
+                // elsewhere must stay removed, so the crawler never writes that flag.
                 string sql = @"INSERT INTO Posts (
                                 BlogName,
                                 PostID,
@@ -505,7 +602,7 @@ namespace URLNotesGrabberCORE
                         {
                             if (ownsConnection) connection.Open();
 
-                            string updateSql = "UPDATE Posts SET hasImage = @hasImage, DateModified = @DateModified WHERE blogName = @blogName AND postID = @postID";
+                            string updateSql = "UPDATE Posts SET hasImage = @hasImage, DateModified = @DateModified WHERE blogName = @blogName AND postID = @postID AND IFNULL(hasImage, 0) <> @hasImage";
                             using SQLiteCommand updateCommand = new SQLiteCommand(updateSql, connection);
                             updateCommand.Parameters.AddWithValue("@hasImage", hasImage ? 1 : 0);
                             updateCommand.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
@@ -530,16 +627,17 @@ namespace URLNotesGrabberCORE
                     }
                 }
 
-                // Only update HasBeenOutput and DateAdded if a new post was inserted
+                // Only reopen the blog for output if a new post was inserted. DateAdded records
+                // when the blog first entered the registry and is never rewritten here -- a new
+                // post is not a new blog.
                 if (rowsInserted == 1)
                 {
                     try
                     {
-                        string updateBlogSql = "UPDATE Blogs SET HasBeenOutput = 0, DateAdded = @DateAdded, DateModified = @DateModified WHERE BlogName = @BlogName";
+                        string updateBlogSql = "UPDATE Blogs SET HasBeenOutput = 0, DateModified = @DateModified WHERE BlogName = @BlogName AND (HasBeenOutput IS NULL OR HasBeenOutput <> 0)";
                         using (var updateBlogCommand = new SQLiteCommand(updateBlogSql, connection))
                         {
                             updateBlogCommand.Parameters.AddWithValue("@BlogName", blogName);
-                            updateBlogCommand.Parameters.AddWithValue("@DateAdded", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
                             updateBlogCommand.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
                             updateBlogCommand.ExecuteNonQuery();
                         }
@@ -564,21 +662,40 @@ namespace URLNotesGrabberCORE
 
                 // Use INSERT OR IGNORE to avoid UNIQUE constraint errors when the date row already exists.
                 // Also explicitly initialize APICount to 0 in case the table has no default.
-                string sql = "INSERT OR IGNORE INTO DailyAPICount (Date, APICount, DateCreated) values(@date, 0, @DateCreated)";
+                //
+                // DailyAPICount is (Date TEXT PK, APICount INTEGER) — the crawler never creates or
+                // migrates this table, and no code reads a creation timestamp off it, so the insert
+                // names only those two columns. Naming a DateCreated column here used to throw
+                // "no such column: DateCreated" into a silent catch, which meant the day's row was
+                // never created and the tally sat at 0 for months.
+                string sql = "INSERT OR IGNORE INTO DailyAPICount (Date, APICount) values(@date, 0)";
                 using (SQLiteCommand command = new SQLiteCommand(sql, connection))
                 {
                     command.Parameters.AddWithValue("@date", DateTime.Today.ToShortDateString());
-                    command.Parameters.AddWithValue("@DateCreated", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
                     command.ExecuteNonQuery();
                 }
             }
             catch (Exception ex)
             {
                 // Breakpoint here
-                //Console.WriteLine(ex.Message);
+                ReportAPICountFailure($"Error creating the row for {DateTime.Today.ToShortDateString()}: {ex.Message}");
             }
         }
 
+        // Bookkeeping writes that fail identically on every API call would flood the console, but
+        // swallowing them entirely is what hid the DateCreated bug. Log each distinct message once.
+        // Messages embed today's date, so a date rollover reports afresh.
+        private static void ReportAPICountFailure(string message)
+        {
+            lock (_apiCountFailureLock)
+            {
+                if (!_apiCountFailuresLogged.Add(message))
+                    return;
+            }
+
+            Console.WriteLine($"[DailyAPICount] {message}");
+        }
+
         public static bool AddNote(string rootBlogName, string noteBlogName, long postID, long timestamp, string type, string? DBPath = null)
         {
             DBPath ??= GetDefaultDbPath();
@@ -591,6 +708,8 @@ namespace URLNotesGrabberCORE
             {
                 connection2.Open();
 
+                // INSERT OR IGNORE, and no IsActive in the column list: re-crawling a note
+                // that was removed elsewhere leaves the existing row -- and its flag -- alone.
                 string sql = "INSERT OR IGNORE INTO Notes (rootBlogName, noteBlogName, PostID, TimeStamp, Type, DatetimeCrawled, DateModified, DateCreated) values(@rootBlogName, @noteBlogName, @PostID, @TimeStamp, @Type, @DatetimeCrawled, @DateModified, @DateCreated)";
                 using (SQLiteCommand command = new SQLiteCommand(sql, connection2))
                 {
@@ -623,7 +742,9 @@ namespace URLNotesGrabberCORE
                     {
                         try
                         {
-                            string updateSql = "UPDATE Blogs SET HasBeenOutput = 0, DateModified = @DateModified WHERE BlogName = @BlogName";
+                            // HasBeenOutput IS NULL still counts as a change: the selection queries
+                            // test HasBeenOutput = 0, which a NULL would never match.
+                            string updateSql = "UPDATE Blogs SET HasBeenOutput = 0, DateModified = @DateModified WHERE BlogName = @BlogName AND (HasBeenOutput IS NULL OR HasBeenOutput <> 0)";
                             using (var updateCommand = new SQLiteCommand(updateSql, connection2))
                             {
                                 updateCommand.Parameters.AddWithValue("@BlogName", noteBlogName);
@@ -688,7 +809,7 @@ namespace URLNotesGrabberCORE
                           "        P.HasNotesGathered," + Environment.NewLine +
                           "        P.NotFound," + Environment.NewLine +
                           "        P.PostDate" + Environment.NewLine +
-                          "    FROM Posts P" + Environment.NewLine +
+                          "    FROM Posts P" + WhereIsActive("Posts", "P", DBPath) + Environment.NewLine +
                           ")," + Environment.NewLine +
                           "Unioned AS" + Environment.NewLine +
                           "(" + Environment.NewLine +
@@ -740,8 +861,8 @@ namespace URLNotesGrabberCORE
                                  "      LEFT OUTER JOIN " + Environment.NewLine +
                                  "   Notes ON Notes.RootBlogName = Posts.BlogName AND Notes.PostID = Posts.PostID " + Environment.NewLine +
                                  "      LEFT OUTER JOIN " + Environment.NewLine +
-                                 "   (      select BlogName, count(PostID) as CNT from Posts group by BlogName)  CNT on CNT.blogName = Posts.BlogName " +
-                                 "WHERE NotFound = 0 " + Environment.NewLine;
+                                 "   (      select BlogName, count(PostID) as CNT from Posts" + WhereIsActive("Posts", "", DBPath) + " group by BlogName)  CNT on CNT.blogName = Posts.BlogName " +
+                                 "WHERE NotFound = 0 " + AndIsActive("Posts", "Posts", DBPath) + Environment.NewLine;
 
                     if (beforeDate.HasValue)
                     {
@@ -850,7 +971,7 @@ namespace URLNotesGrabberCORE
             {
                 connection.Open();
 
-                string sql = "SELECT	distinct RootBlogName as blogName, postID FROM Notes WHERE Notes.type = 'reply' order by RootBlogName, PostID";
+                string sql = "SELECT	distinct RootBlogName as blogName, postID FROM Notes WHERE Notes.type = 'reply'" + AndIsActive("Notes", "Notes", DBPath) + " order by RootBlogName, PostID";
 
                 using (SQLiteCommand command = new SQLiteCommand(sql, connection))
                 {
@@ -894,7 +1015,7 @@ namespace URLNotesGrabberCORE
                              MAX(Notes.timestamp) as LatestTimestamp
                              FROM Notes 
                              WHERE Notes.type = 'reply'
-                             AND (Notes.replyText IS NULL OR Notes.replyText = '' OR Notes.replyText = '.')
+                             AND (Notes.replyText IS NULL OR Notes.replyText = '' OR Notes.replyText = '.')" + AndIsActive("Notes", "Notes", DBPath) + @"
                              GROUP BY Notes.RootBlogName, Notes.PostID
                              ORDER BY LatestTimestamp ASC
                              LIMIT @limit";
@@ -942,7 +1063,7 @@ namespace URLNotesGrabberCORE
                               INNER JOIN Notes N ON N.PostID = P.PostID AND N.RootBlogName = P.BlogName
                               WHERE P.NotFound = 0
                                 AND N.type = 'reply'
-                                AND (N.replyText IS NULL OR N.replyText = '' OR N.replyText = '.')
+                                AND (N.replyText IS NULL OR N.replyText = '' OR N.replyText = '.')" + AndIsActive("Posts", "P", DBPath) + AndIsActive("Notes", "N", DBPath) + @"
                               GROUP BY P.BlogName, P.PostID
                               ORDER BY LatestTimestamp ASC";
 
@@ -991,7 +1112,9 @@ namespace URLNotesGrabberCORE
             using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
             int count = 0;
 
-            try { AddAPICount(); } catch { }
+            // AddAPICount reports its own failures; this guard only stops a connection-level
+            // problem from taking down the read below.
+            try { AddAPICount(); } catch (Exception ex) { ReportAPICountFailure($"AddAPICount failed: {ex.Message}"); }
 
             try
             {
@@ -999,6 +1122,7 @@ namespace URLNotesGrabberCORE
 
                 string sql = "SELECT APICount FROM DailyAPICount WHERE [Date] = @date";
 
+                bool rowFound = false;
                 using (SQLiteCommand command = new SQLiteCommand(sql, connection))
                 {
                     command.Parameters.AddWithValue("@date", DateTime.Today.ToShortDateString());
@@ -1006,10 +1130,16 @@ namespace URLNotesGrabberCORE
                     {
                         while (reader.Read())
                         {
+                            rowFound = true;
                             count = reader.GetInt32(0); // Assuming Id is the first column
                         }
                     }
                 }
+
+                // A missing row means AddAPICount did not take. Returning a silent 0 here is what
+                // made the tally look merely idle rather than broken.
+                if (!rowFound)
+                    ReportAPICountFailure($"No row for {DateTime.Today.ToShortDateString()} after AddAPICount - reported count of 0 is not a real tally.");
             }
             catch (Exception ex)
             {
@@ -1038,7 +1168,8 @@ namespace URLNotesGrabberCORE
                                    COALESCE(LikesCursor, 0),
                                    COALESCE(LikesNewestTimestamp, 0)
                               FROM Blogs
-                             WHERE BlogName = @blog";
+                             WHERE BlogName = @blog
+                               AND IsActive = 1";
                 }
                 else if (ignoreCooldown)
                 {
@@ -1051,6 +1182,7 @@ namespace URLNotesGrabberCORE
                               INNER JOIN Notes N ON N.NoteBlogName = B.BlogName
                              WHERE N.TimeStamp >= 1535778000
                                AND N.rootBlogName = B.BlogName
+                               AND B.IsActive = 1" + AndIsActive("Notes", "N", DBPath) + @"
                              GROUP BY B.BlogName
                              ORDER BY MIN(N.Timestamp);";
                 }
@@ -1065,6 +1197,7 @@ namespace URLNotesGrabberCORE
                               INNER JOIN Notes N ON N.NoteBlogName = B.BlogName
                              WHERE N.TimeStamp >= 1535778000
                                AND N.rootBlogName = B.BlogName
+                               AND B.IsActive = 1" + AndIsActive("Notes", "N", DBPath) + @"
                                AND (
                                     B.LikesPulled = 0
                                  OR COALESCE(B.LikesLastRefreshed, 0)
@@ -1113,9 +1246,9 @@ namespace URLNotesGrabberCORE
                 connection.Open();
                 string sql = "";
                 if (reblogsOnly)
-                    sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive AND type IN ('reblog', 'reply', 'posted') AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top";
+                    sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive" + AndIsActive("Notes", "notes", DBPath) + " AND type IN ('reblog', 'reply', 'posted') AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top";
                 else
-                    sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive AND type NOT IN ('reblog', 'reply', 'posted') AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top";
+                    sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive" + AndIsActive("Notes", "notes", DBPath) + " AND type NOT IN ('reblog', 'reply', 'posted') AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top";
 
                 using (SQLiteCommand command = new SQLiteCommand(sql, connection))
                 {
@@ -1154,9 +1287,9 @@ namespace URLNotesGrabberCORE
                 connection.Open();
                 string sql = "";
                 if (reblogsOnly)
-                    sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive AND type IN ('reblog', 'reply', 'posted') AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top";
+                    sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive" + AndIsActive("Notes", "notes", DBPath) + " AND type IN ('reblog', 'reply', 'posted') AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top";
                 else
-                    sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top";
+                    sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive" + AndIsActive("Notes", "notes", DBPath) + " AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top";
 
                 using (SQLiteCommand command = new SQLiteCommand(sql, connection))
                 {
@@ -1188,7 +1321,7 @@ namespace URLNotesGrabberCORE
             DBPath ??= GetDefaultDbPath();
             using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
             connection.Open();
-            string sql = "SELECT BlogName, reblogURL, PostURL, Slug, ReblogKey, ReblogName, Summary, Quote, Body, Tags, Link, PhotoURL, PhotoCaption, DownloadedFiles, AudioCaption, Question, Answer, Title, RootBlogName, RootURL FROM Posts WHERE IFNULL(DownloadedFiles, '.') = '.'";
+            string sql = "SELECT BlogName, reblogURL, PostURL, Slug, ReblogKey, ReblogName, Summary, Quote, Body, Tags, Link, PhotoURL, PhotoCaption, DownloadedFiles, AudioCaption, Question, Answer, Title, RootBlogName, RootURL FROM Posts WHERE IFNULL(DownloadedFiles, '.') = '.'" + AndIsActive("Posts", "", DBPath);
             using (SQLiteCommand command = new SQLiteCommand(sql, connection))
             {
                 using (SQLiteDataReader reader = command.ExecuteReader())
@@ -1230,7 +1363,10 @@ namespace URLNotesGrabberCORE
                 connection.Open();
 
                 //string sql = "UPDATE Posts SET HasNotesGathered = 1, NotesGatheredDateTime = @notesGathered WHERE BlogName = @BlogName AND PostID = @PostID";
-                string sql = "UPDATE Posts SET HasNotesGathered = 1, NotesGatheredDateTime = @notesGathered, DateModified = @dateModified WHERE PostID = @PostID AND (IFNULL(HasNotesGathered, 0) <> 1 OR IFNULL(NotesGatheredDateTime, 0) <> @notesGathered)";
+                // NotesGatheredDateTime is crawl bookkeeping -- it moves on every pass and says
+                // nothing about the post itself, so only the HasNotesGathered flag flipping
+                // counts as a modification. The CASE reads the pre-UPDATE value of the flag.
+                string sql = "UPDATE Posts SET HasNotesGathered = 1, NotesGatheredDateTime = @notesGathered, DateModified = CASE WHEN IFNULL(HasNotesGathered, 0) <> 1 THEN @dateModified ELSE DateModified END WHERE PostID = @PostID AND (IFNULL(HasNotesGathered, 0) <> 1 OR IFNULL(NotesGatheredDateTime, 0) <> @notesGathered)";
                 using (SQLiteCommand command = new SQLiteCommand(sql, connection))
                 {
                     command.Parameters.AddWithValue("@notesGathered", DateTimeOffset.UtcNow.ToUnixTimeSeconds());
@@ -1432,49 +1568,60 @@ namespace URLNotesGrabberCORE
             {
                 if (ownsConnection) connection.Open();
 
+                // "." is TraverseDirectory/ReblogRecord's sentinel for "this field had no
+                // matching line in this particular export file" -- not an empty value. A blog
+                // with two export files in different formats (e.g. an "_2" duplicate folder, or
+                // a Tumblr export whose field set changed over time) sends one record with a real
+                // Title and another with Title = "." for the same PostID, and re-importing both
+                // on every run must not let the "not supplied" record blank out what the other
+                // one has. Every content field below is CASE-guarded the same way RootBlogName/
+                // RootURL already were, and the change-detection ignores "." too so a "."-only
+                // difference doesn't fire the UPDATE (and bump DateModified) on its own. Only "."
+                // is treated as the sentinel -- an explicit empty string from a real field still
+                // overwrites, same as before.
                 string sql = "UPDATE Posts SET ";
-                sql += "postDate = @postDate, ";
-                sql += "reblogURL = @reblogURL, ";
-                sql += "postURL = @postURL, ";
-                sql += "slug = @slug, ";
-                sql += "reblogKey = @reblogKey, ";
-                sql += "reblogName = @reblogName, ";
-                sql += "summary = @summary, ";
-                sql += "quote = @quote, ";
-                sql += "body = @body, ";
-                sql += "tags = @tags, ";
-                sql += "link = @link, ";
-                sql += "photoURL = @photoURL, ";
-                sql += "photoCaption = @photoCaption, ";
-                sql += "downloadedFiles = @downloadedFiles, ";
-                sql += "audioCaption = @audioCaption, ";
-                sql += "question = @question, ";
-                sql += "answer = @answer, ";
-                sql += "title = @title, ";
+                sql += "postDate = CASE WHEN @postDate = '.' THEN postDate ELSE @postDate END, ";
+                sql += "reblogURL = CASE WHEN @reblogURL = '.' THEN reblogURL ELSE @reblogURL END, ";
+                sql += "postURL = CASE WHEN @postURL = '.' THEN postURL ELSE @postURL END, ";
+                sql += "slug = CASE WHEN @slug = '.' THEN slug ELSE @slug END, ";
+                sql += "reblogKey = CASE WHEN @reblogKey = '.' THEN reblogKey ELSE @reblogKey END, ";
+                sql += "reblogName = CASE WHEN @reblogName = '.' THEN reblogName ELSE @reblogName END, ";
+                sql += "summary = CASE WHEN @summary = '.' THEN summary ELSE @summary END, ";
+                sql += "quote = CASE WHEN @quote = '.' THEN quote ELSE @quote END, ";
+                sql += "body = CASE WHEN @body = '.' THEN body ELSE @body END, ";
+                sql += "tags = CASE WHEN @tags = '.' THEN tags ELSE @tags END, ";
+                sql += "link = CASE WHEN @link = '.' THEN link ELSE @link END, ";
+                sql += "photoURL = CASE WHEN @photoURL = '.' THEN photoURL ELSE @photoURL END, ";
+                sql += "photoCaption = CASE WHEN @photoCaption = '.' THEN photoCaption ELSE @photoCaption END, ";
+                sql += "downloadedFiles = CASE WHEN @downloadedFiles = '.' THEN downloadedFiles ELSE @downloadedFiles END, ";
+                sql += "audioCaption = CASE WHEN @audioCaption = '.' THEN audioCaption ELSE @audioCaption END, ";
+                sql += "question = CASE WHEN @question = '.' THEN question ELSE @question END, ";
+                sql += "answer = CASE WHEN @answer = '.' THEN answer ELSE @answer END, ";
+                sql += "title = CASE WHEN @title = '.' THEN title ELSE @title END, ";
                 sql += "DateModified = @dateModified, ";
                 sql += "RootBlogName = CASE WHEN @rootBlogName IS NULL OR @rootBlogName = '' OR @rootBlogName = '.' THEN RootBlogName ELSE @rootBlogName END, ";
                 sql += "RootURL = CASE WHEN @rootURL IS NULL OR @rootURL = '' OR @rootURL = '.' THEN RootURL ELSE @rootURL END, ";
                 sql += "hasImage = @hasImage, ";
                 sql += "ByLikes = MAX(IFNULL(ByLikes, 0), @byLikes) ";
                 sql += " WHERE BlogName = @BlogName AND PostID = @PostID AND (";
-                sql += "IFNULL(postDate, '') <> @postDate OR ";
-                sql += "IFNULL(reblogURL, '') <> @reblogURL OR ";
-                sql += "IFNULL(postURL, '') <> @postURL OR ";
-                sql += "IFNULL(slug, '') <> @slug OR ";
-                sql += "IFNULL(reblogKey, '') <> @reblogKey OR ";
-                sql += "IFNULL(reblogName, '') <> @reblogName OR ";
-                sql += "IFNULL(summary, '') <> @summary OR ";
-                sql += "IFNULL(quote, '') <> @quote OR ";
-                sql += "IFNULL(body, '') <> @body OR ";
-                sql += "IFNULL(tags, '') <> @tags OR ";
-                sql += "IFNULL(link, '') <> @link OR ";
-                sql += "IFNULL(photoURL, '') <> @photoURL OR ";
-                sql += "IFNULL(photoCaption, '') <> @photoCaption OR ";
-                sql += "IFNULL(downloadedFiles, '') <> @downloadedFiles OR ";
-                sql += "IFNULL(audioCaption, '') <> @audioCaption OR ";
-                sql += "IFNULL(question, '') <> @question OR ";
-                sql += "IFNULL(answer, '') <> @answer OR ";
-                sql += "IFNULL(title, '') <> @title OR ";
+                sql += "(@postDate <> '.' AND IFNULL(postDate, '') <> @postDate) OR ";
+                sql += "(@reblogURL <> '.' AND IFNULL(reblogURL, '') <> @reblogURL) OR ";
+                sql += "(@postURL <> '.' AND IFNULL(postURL, '') <> @postURL) OR ";
+                sql += "(@slug <> '.' AND IFNULL(slug, '') <> @slug) OR ";
+                sql += "(@reblogKey <> '.' AND IFNULL(reblogKey, '') <> @reblogKey) OR ";
+                sql += "(@reblogName <> '.' AND IFNULL(reblogName, '') <> @reblogName) OR ";
+                sql += "(@summary <> '.' AND IFNULL(summary, '') <> @summary) OR ";
+                sql += "(@quote <> '.' AND IFNULL(quote, '') <> @quote) OR ";
+                sql += "(@body <> '.' AND IFNULL(body, '') <> @body) OR ";
+                sql += "(@tags <> '.' AND IFNULL(tags, '') <> @tags) OR ";
+                sql += "(@link <> '.' AND IFNULL(link, '') <> @link) OR ";
+                sql += "(@photoURL <> '.' AND IFNULL(photoURL, '') <> @photoURL) OR ";
+                sql += "(@photoCaption <> '.' AND IFNULL(photoCaption, '') <> @photoCaption) OR ";
+                sql += "(@downloadedFiles <> '.' AND IFNULL(downloadedFiles, '') <> @downloadedFiles) OR ";
+                sql += "(@audioCaption <> '.' AND IFNULL(audioCaption, '') <> @audioCaption) OR ";
+                sql += "(@question <> '.' AND IFNULL(question, '') <> @question) OR ";
+                sql += "(@answer <> '.' AND IFNULL(answer, '') <> @answer) OR ";
+                sql += "(@title <> '.' AND IFNULL(title, '') <> @title) OR ";
                 sql += "IFNULL(hasImage, 0) <> @hasImage OR ";
                 sql += "(@byLikes = 1 AND IFNULL(ByLikes, 0) = 0) OR ";
                 sql += "((@rootBlogName IS NOT NULL AND @rootBlogName <> '' AND @rootBlogName <> '.') AND IFNULL(RootBlogName, '') <> @rootBlogName) OR ";
@@ -1587,7 +1734,8 @@ namespace URLNotesGrabberCORE
                 string sql = @"UPDATE Blogs
                                   SET LikesNewestTimestamp = MAX(COALESCE(LikesNewestTimestamp, 0), @newest),
                                       DateModified = @modified
-                                WHERE BlogName = @name";
+                                WHERE BlogName = @name
+                                  AND COALESCE(LikesNewestTimestamp, 0) < @newest";
                 using (SQLiteCommand command = new SQLiteCommand(sql, connection))
                 {
                     command.Parameters.AddWithValue("@newest", newestTimestamp);
@@ -1649,7 +1797,11 @@ namespace URLNotesGrabberCORE
                 {
                     command.Parameters.AddWithValue("@APICount", APICount);
                     command.Parameters.AddWithValue("@date", DateTime.Today.ToShortDateString());
-                    command.ExecuteNonQuery();
+
+                    // No row for today means this UPDATE matched nothing and the increment was
+                    // thrown away, while the value returned below still looks like a real count.
+                    if (command.ExecuteNonQuery() == 0)
+                        ReportAPICountFailure($"UPDATE matched no row for {DateTime.Today.ToShortDateString()} - the count of {APICount} was not persisted.");
                 }
             }
             catch (Exception ex)
@@ -1679,7 +1831,7 @@ namespace URLNotesGrabberCORE
                 //string sql = "UPDATE Notes SET replyText = @replyText WHERE rootBlogName = @rootBlogName AND PostID = @PostID AND noteBlogName = @noteBlogName AND TimeStamp = @TimeStamp AND Type = 'reply'";
                 // Match on (noteBlogName, TimeStamp ±5s) only - a reply by a given blog at a given timestamp is the same reply across the original post and every reblog of it, so this fans out across reblog chains in one shot. Tolerance absorbs the ~1s drift between what -collect stored and what mode=conversation returns now.
                 // Only fan out to rows that match the SELECT criteria in GetRepliesWithFilledText (NULL/empty/legacy-'.'). Never overwrite '?' (confirmed-empty) or already-fetched text.
-                string sql = "UPDATE Notes SET replyText = @replyText, DateModified = @dateModified WHERE noteBlogName = @noteBlogName AND ABS(TimeStamp - @TimeStamp) <= 5 AND Type = 'reply' AND (replyText IS NULL OR replyText = '' OR replyText = '.')";
+                string sql = "UPDATE Notes SET replyText = @replyText, DateModified = @dateModified WHERE noteBlogName = @noteBlogName AND ABS(TimeStamp - @TimeStamp) <= 5 AND Type = 'reply' AND (replyText IS NULL OR replyText = '' OR replyText = '.') AND (replyText IS NULL OR replyText <> @replyText)";
                 using (SQLiteCommand command = new SQLiteCommand(sql, connection))
                 {
                     command.Parameters.AddWithValue("@replyText", replyText ?? "?");
@@ -1854,6 +2006,8 @@ namespace URLNotesGrabberCORE
             {
                 if (ownsConnection) connection.Open();
 
+                // As in AddPost, IsActive is never written -- neither here nor in the
+                // UPDATE below, which is why an ingest cannot un-remove a post.
                 string insertSql = @"INSERT INTO Posts (
                                 BlogName, PostID, reblogURL, PostDate, PostURL, Slug,
                                 ReblogKey, ReblogName, Summary, Quote, Body, Tags, Link,
@@ -1912,29 +2066,65 @@ namespace URLNotesGrabberCORE
 
                 if (rowsInserted == 0)
                 {
+                    // NULL is this function's sentinel for "this file's record had no line for
+                    // that field" (IngestMode's G(key) misses return null; LegacyPostsDbImporter
+                    // passes null straight from a NULL source column) -- it does not mean "clear
+                    // this field". --ingest's entire reason to exist is reconciling multiple
+                    // export files for the same (BlogName, PostID) -- IngestMode normalizes a
+                    // "_2"-suffixed duplicate folder onto the same blog name specifically so a
+                    // second, differently-formatted file for a post it already has gets merged in.
+                    // Files are walked in filesystem enumeration order, not sorted, so which
+                    // file's UpsertPostFromTextFile call runs last for a given PostID is
+                    // effectively arbitrary. An unconditional SET here would let whichever file
+                    // processed last silently null out every column its own record didn't carry,
+                    // erasing real content the other file had -- the opposite of "clean up". Each
+                    // column is CASE-guarded to keep the existing value when this call's parameter
+                    // is NULL, and the change-detection ignores a NULL-vs-real mismatch the same
+                    // way, so a partial record converges into the row instead of overwriting it.
                     string updateSql = @"UPDATE Posts SET
-                                reblogURL = @reblogURL,
-                                PostDate = @PostDate,
-                                PostURL = @PostURL,
-                                Slug = @Slug,
-                                ReblogKey = @ReblogKey,
-                                ReblogName = @ReblogName,
-                                Summary = @Summary,
-                                Quote = @Quote,
-                                Body = @Body,
-                                Tags = @Tags,
-                                Link = @Link,
-                                PhotoURL = @PhotoURL,
-                                PhotoCaption = @PhotoCaption,
-                                DownloadedFiles = @DownloadedFiles,
-                                AudioCaption = @AudioCaption,
-                                Question = @Question,
-                                Answer = @Answer,
-                                Title = @Title,
-                                PostType = @PostType,
+                                reblogURL = CASE WHEN @reblogURL IS NULL THEN reblogURL ELSE @reblogURL END,
+                                PostDate = CASE WHEN @PostDate IS NULL THEN PostDate ELSE @PostDate END,
+                                PostURL = CASE WHEN @PostURL IS NULL THEN PostURL ELSE @PostURL END,
+                                Slug = CASE WHEN @Slug IS NULL THEN Slug ELSE @Slug END,
+                                ReblogKey = CASE WHEN @ReblogKey IS NULL THEN ReblogKey ELSE @ReblogKey END,
+                                ReblogName = CASE WHEN @ReblogName IS NULL THEN ReblogName ELSE @ReblogName END,
+                                Summary = CASE WHEN @Summary IS NULL THEN Summary ELSE @Summary END,
+                                Quote = CASE WHEN @Quote IS NULL THEN Quote ELSE @Quote END,
+                                Body = CASE WHEN @Body IS NULL THEN Body ELSE @Body END,
+                                Tags = CASE WHEN @Tags IS NULL THEN Tags ELSE @Tags END,
+                                Link = CASE WHEN @Link IS NULL THEN Link ELSE @Link END,
+                                PhotoURL = CASE WHEN @PhotoURL IS NULL THEN PhotoURL ELSE @PhotoURL END,
+                                PhotoCaption = CASE WHEN @PhotoCaption IS NULL THEN PhotoCaption ELSE @PhotoCaption END,
+                                DownloadedFiles = CASE WHEN @DownloadedFiles IS NULL THEN DownloadedFiles ELSE @DownloadedFiles END,
+                                AudioCaption = CASE WHEN @AudioCaption IS NULL THEN AudioCaption ELSE @AudioCaption END,
+                                Question = CASE WHEN @Question IS NULL THEN Question ELSE @Question END,
+                                Answer = CASE WHEN @Answer IS NULL THEN Answer ELSE @Answer END,
+                                Title = CASE WHEN @Title IS NULL THEN Title ELSE @Title END,
+                                PostType = CASE WHEN @PostType IS NULL THEN PostType ELSE @PostType END,
                                 HasImage = @HasImage,
                                 DateModified = @DateModified
-                                WHERE BlogName = @BlogName AND PostID = @PostID";
+                                WHERE BlogName = @BlogName AND PostID = @PostID AND (
+                                    (@reblogURL IS NOT NULL AND IFNULL(reblogURL, '') <> @reblogURL) OR
+                                    (@PostDate IS NOT NULL AND IFNULL(PostDate, '') <> @PostDate) OR
+                                    (@PostURL IS NOT NULL AND IFNULL(PostURL, '') <> @PostURL) OR
+                                    (@Slug IS NOT NULL AND IFNULL(Slug, '') <> @Slug) OR
+                                    (@ReblogKey IS NOT NULL AND IFNULL(ReblogKey, '') <> @ReblogKey) OR
+                                    (@ReblogName IS NOT NULL AND IFNULL(ReblogName, '') <> @ReblogName) OR
+                                    (@Summary IS NOT NULL AND IFNULL(Summary, '') <> @Summary) OR
+                                    (@Quote IS NOT NULL AND IFNULL(Quote, '') <> @Quote) OR
+                                    (@Body IS NOT NULL AND IFNULL(Body, '') <> @Body) OR
+                                    (@Tags IS NOT NULL AND IFNULL(Tags, '') <> @Tags) OR
+                                    (@Link IS NOT NULL AND IFNULL(Link, '') <> @Link) OR
+                                    (@PhotoURL IS NOT NULL AND IFNULL(PhotoURL, '') <> @PhotoURL) OR
+                                    (@PhotoCaption IS NOT NULL AND IFNULL(PhotoCaption, '') <> @PhotoCaption) OR
+                                    (@DownloadedFiles IS NOT NULL AND IFNULL(DownloadedFiles, '') <> @DownloadedFiles) OR
+                                    (@AudioCaption IS NOT NULL AND IFNULL(AudioCaption, '') <> @AudioCaption) OR
+                                    (@Question IS NOT NULL AND IFNULL(Question, '') <> @Question) OR
+                                    (@Answer IS NOT NULL AND IFNULL(Answer, '') <> @Answer) OR
+                                    (@Title IS NOT NULL AND IFNULL(Title, '') <> @Title) OR
+                                    (@PostType IS NOT NULL AND IFNULL(PostType, '') <> @PostType) OR
+                                    IFNULL(HasImage, 0) <> @HasImage
+                                )";
 
                     using (var cmd = new SQLiteCommand(updateSql, connection))
                     {
@@ -1984,7 +2174,7 @@ namespace URLNotesGrabberCORE
                                   PhotoURL, PhotoCaption, DownloadedFiles, AudioCaption,
                                   Question, Answer, Title, PostType,
                                   HasImage, DateCreated, DateModified
-                           FROM Posts WHERE BlogName = @BlogName";
+                           FROM Posts WHERE BlogName = @BlogName" + AndIsActive("Posts", "", DBPath);
             using var cmd = new SQLiteCommand(sql, connection);
             cmd.Parameters.AddWithValue("@BlogName", blogName);
             using var reader = cmd.ExecuteReader();
@@ -2033,7 +2223,7 @@ namespace URLNotesGrabberCORE
                                   PhotoURL, PhotoCaption, DownloadedFiles, AudioCaption,
                                   Question, Answer, Title, PostType,
                                   HasImage, DateCreated, DateModified
-                           FROM Posts WHERE BlogName = @BlogName AND PostID = @PostID";
+                           FROM Posts WHERE BlogName = @BlogName AND PostID = @PostID" + AndIsActive("Posts", "", DBPath);
             using var cmd = new SQLiteCommand(sql, connection);
             cmd.Parameters.AddWithValue("@BlogName", blogName);
             cmd.Parameters.AddWithValue("@PostID", postId);
@@ -2083,7 +2273,7 @@ namespace URLNotesGrabberCORE
                                   PhotoURL, PhotoCaption, DownloadedFiles, AudioCaption,
                                   Question, Answer, Title, PostType,
                                   HasImage, DateCreated, DateModified
-                           FROM Posts WHERE PostID = @PostID LIMIT 1";
+                           FROM Posts WHERE PostID = @PostID" + AndIsActive("Posts", "", DBPath) + @" LIMIT 1";
             using var cmd = new SQLiteCommand(sql, connection);
             cmd.Parameters.AddWithValue("@PostID", postId);
             using var reader = cmd.ExecuteReader();
@@ -2118,7 +2308,11 @@ namespace URLNotesGrabberCORE
             };
         }
 
-        public static void SetBlogTTFolderPath(string blogName, string? path, string? DBPath = null)
+        // Returns true only when a row's TTFolderPath actually changed. A false means either
+        // the row already held this value or no row matched the name -- callers must not
+        // report a write they did not get, which is how a --updatepaths run could once print
+        // "Updated <blog>" for every metadata file while leaving the column entirely NULL.
+        public static bool SetBlogTTFolderPath(string blogName, string? path, string? DBPath = null)
         {
             DBPath ??= GetDefaultDbPath();
             try { AddBlog(blogName, false, DBPath); } catch { }
@@ -2126,24 +2320,40 @@ namespace URLNotesGrabberCORE
             using var connection = new SQLiteConnection("Data Source=" + DBPath);
             connection.Open();
             using var cmd = new SQLiteCommand(
-                "UPDATE Blogs SET TTFolderPath = @path, DateModified = @modified WHERE BlogName = @name",
+                "UPDATE Blogs SET TTFolderPath = @path, DateModified = @modified WHERE BlogName = @name AND IFNULL(TTFolderPath, '') <> IFNULL(@path, '')",
                 connection);
             cmd.Parameters.AddWithValue("@path", (object?)path ?? DBNull.Value);
             cmd.Parameters.AddWithValue("@modified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
             cmd.Parameters.AddWithValue("@name", blogName);
-            cmd.ExecuteNonQuery();
+            return cmd.ExecuteNonQuery() > 0;
+        }
+
+        // Whether a Blogs row exists under this exact name. BlogName is a BINARY-collated
+        // primary key, so a metadata filename that differs only in case is a different blog
+        // as far as the UPDATE above is concerned -- worth telling the user about.
+        public static bool BlogExists(string blogName, string? DBPath = null)
+        {
+            DBPath ??= GetDefaultDbPath();
+
+            using var connection = new SQLiteConnection("Data Source=" + DBPath);
+            connection.Open();
+            using var cmd = new SQLiteCommand("SELECT 1 FROM Blogs WHERE BlogName = @name", connection);
+            cmd.Parameters.AddWithValue("@name", blogName);
+            return cmd.ExecuteScalar() != null;
         }
 
         // Partial UPDATE used by the correct-apply path. fieldsToUpdate maps
         // ThreeTxtFileHelper prefix names ("Reblog URL", "Body", etc.) to non-empty
         // values pulled from a BAK file. Only those columns + DateModified are written;
         // other content columns and all engagement columns are left intact.
-        // Returns true if a row was matched (and therefore updated).
+        // Returns true if a row was actually changed. A row whose columns already hold
+        // the incoming values is left alone, DateModified included.
         public static bool UpdatePostContentFields(string blogName, string postId, IDictionary<string, string> fieldsToUpdate, string? DBPath = null)
         {
             DBPath ??= GetDefaultDbPath();
 
             var setClauses = new List<string>();
+            var changedClauses = new List<string>();
             var parameters = new List<(string Name, object Value)>();
 
             foreach (var kvp in fieldsToUpdate)
@@ -2153,6 +2363,7 @@ namespace URLNotesGrabberCORE
                 if (column == null) continue;
                 string paramName = "@p" + parameters.Count;
                 setClauses.Add($"{column} = {paramName}");
+                changedClauses.Add($"IFNULL({column}, '') <> {paramName}");
                 parameters.Add((paramName, kvp.Value));
             }
 
@@ -2164,7 +2375,7 @@ namespace URLNotesGrabberCORE
             using var connection = new SQLiteConnection("Data Source=" + DBPath);
             connection.Open();
 
-            string sql = $"UPDATE Posts SET {string.Join(", ", setClauses)} WHERE BlogName = @BlogName AND PostID = @PostID";
+            string sql = $"UPDATE Posts SET {string.Join(", ", setClauses)} WHERE BlogName = @BlogName AND PostID = @PostID AND ({string.Join(" OR ", changedClauses)})";
             using var cmd = new SQLiteCommand(sql, connection);
             foreach (var (name, value) in parameters)
                 cmd.Parameters.AddWithValue(name, value);
@@ -2202,24 +2413,48 @@ namespace URLNotesGrabberCORE
             };
         }
 
-        public static List<(string BlogName, string? TTFolderPath)> GetAllBlogsWithTTFolderPath(string? DBPath = null)
+        // Export targets only: active blogs that actually carry a TTFolderPath.
+        // Blogs is a 144k-row crawl registry and only the few hundred blogs downloaded
+        // locally have a folder, so returning the unset rows made --output print a skip
+        // line for every blog Tumblr has ever handed us.
+        public static List<(string BlogName, string TTFolderPath)> GetAllBlogsWithTTFolderPath(string? DBPath = null)
         {
             DBPath ??= GetDefaultDbPath();
-            var results = new List<(string, string?)>();
+            var results = new List<(string, string)>();
 
             using var connection = new SQLiteConnection("Data Source=" + DBPath);
             connection.Open();
-            using var cmd = new SQLiteCommand("SELECT BlogName, TTFolderPath FROM Blogs", connection);
+            using var cmd = new SQLiteCommand(
+                "SELECT BlogName, TRIM(TTFolderPath) FROM Blogs WHERE IsActive = 1 AND IFNULL(TRIM(TTFolderPath), '') <> '' ORDER BY BlogName",
+                connection);
             using var reader = cmd.ExecuteReader();
             while (reader.Read())
-            {
-                string name = reader.GetString(0);
-                string? path = reader.IsDBNull(1) ? null : reader.GetString(1);
-                results.Add((name, path));
-            }
+                results.Add((reader.GetString(0), reader.GetString(1)));
             return results;
         }
 
+        // Companion counts for the messages --output and --updatepaths print about coverage.
+        public static int CountActiveBlogs(string? DBPath = null)
+        {
+            DBPath ??= GetDefaultDbPath();
+
+            using var connection = new SQLiteConnection("Data Source=" + DBPath);
+            connection.Open();
+            using var cmd = new SQLiteCommand("SELECT COUNT(*) FROM Blogs WHERE IsActive = 1", connection);
+            return Convert.ToInt32(cmd.ExecuteScalar());
+        }
+
+        public static int CountBlogsWithTTFolderPath(string? DBPath = null)
+        {
+            DBPath ??= GetDefaultDbPath();
+
+            using var connection = new SQLiteConnection("Data Source=" + DBPath);
+            connection.Open();
+            using var cmd = new SQLiteCommand(
+                "SELECT COUNT(*) FROM Blogs WHERE IFNULL(TRIM(TTFolderPath), '') <> ''", connection);
+            return Convert.ToInt32(cmd.ExecuteScalar());
+        }
+
         private static string SafeStr(SQLiteDataReader reader, int ordinal)
         {
             return reader.IsDBNull(ordinal) ? string.Empty : reader.GetValue(ordinal)?.ToString() ?? string.Empty;
@@ -2609,6 +2844,10 @@ namespace URLNotesGrabberCORE
 
         public static void MarkAvailable(ApiKeyConfig key)
         {
+            // Called after every successful call; skip the write and the log line when nothing was flagged.
+            if (GetRetryUntil(key) == 0)
+                return;
+
             using var conn = new System.Data.SQLite.SQLiteConnection("Data Source=" + _dbPath);
             conn.Open();
             using var cmd = new System.Data.SQLite.SQLiteCommand(
@@ -2702,6 +2941,15 @@ namespace URLNotesGrabberCORE
 
         private static string FormatKeyLabel(ApiKeyConfig key) => $"[Key#{key.KeyNumber}]";
 
+        private static string SummarizeBody(string body)
+        {
+            if (string.IsNullOrWhiteSpace(body))
+                return "(empty)";
+
+            var flat = System.Text.RegularExpressions.Regex.Replace(body, @"<[^>]+>|\s+", " ").Trim();
+            return flat.Length <= 80 ? flat : flat.Substring(0, 80) + "...";
+        }
+
         private static int GetRetryDelaySecondsFromHeaders(IEnumerable<HeaderParameter>? headers)
         {
             if (headers == null)
@@ -2780,11 +3028,27 @@ namespace URLNotesGrabberCORE
                 Console.WriteLine($"{FormatKeyLabel(key)} {timestamp}\t{DateTime.Now}\t{DataAccess.UpdateAPICount()}");
                 var myDeserializedClass = new Root();
 
+                // Never reached the API: there is no body to interpret, so the post's state is still unknown.
+                if (response.ResponseStatus != ResponseStatus.Completed)
+                {
+                    myDeserializedClass.statusCode = response.ResponseStatus.ToString();
+                    myDeserializedClass.transientFailure = true;
+                    Console.WriteLine($"[Transient] {FormatKeyLabel(key)} transport {response.ResponseStatus}: {response.ErrorException?.Message}");
+                    return myDeserializedClass;
+                }
+
                 try
                 {
                     var deserializedResult = JsonConvert.DeserializeObject<Root>(myJsonResponse);
-                    if (deserializedResult != null)
+                    if (deserializedResult == null)
                     {
+                        // Empty body behind an HTTP status: an edge/proxy response, not the API.
+                        myDeserializedClass.statusCode = response.StatusCode.ToString();
+                        myDeserializedClass.transientFailure = true;
+                        Console.WriteLine($"[Transient] {FormatKeyLabel(key)} HTTP {(int)response.StatusCode} {response.StatusDescription} — empty body");
+                        return myDeserializedClass;
+                    }
+
                     myDeserializedClass = deserializedResult;
                     myDeserializedClass.rawJson = myJsonResponse;
 
@@ -2796,128 +3060,31 @@ namespace URLNotesGrabberCORE
                     bool metaIndicatesRateLimit = myDeserializedClass.meta != null && myDeserializedClass.meta.status == 429;
                     bool metaMsgIndicatesRateLimit = myDeserializedClass.meta != null && !string.IsNullOrEmpty(myDeserializedClass.meta.msg) && myDeserializedClass.meta.msg.IndexOf("Too Many", StringComparison.OrdinalIgnoreCase) >= 0;
 
-                        if (metaIndicatesRateLimit || metaMsgIndicatesRateLimit || (response != null && (response.StatusDescription?.IndexOf("Too Many", StringComparison.OrdinalIgnoreCase) >= 0 || response.StatusCode == System.Net.HttpStatusCode.TooManyRequests)))
+                    if (metaIndicatesRateLimit || metaMsgIndicatesRateLimit || response.StatusDescription?.IndexOf("Too Many", StringComparison.OrdinalIgnoreCase) >= 0 || response.StatusCode == System.Net.HttpStatusCode.TooManyRequests)
                     {
-                            if (response?.Headers != null)
-                            {
-                                bool checkResetLocal = false;
-                                foreach (var header in response.Headers)
-                                {
-                                    string? headerName = header?.Name;
-                                    string? headerValue = header?.Value?.ToString();
-                                    if (string.IsNullOrEmpty(headerName) || string.IsNullOrEmpty(headerValue))
-                                        continue;
-
-                                    if (string.Equals(headerName, "Retry-After", StringComparison.OrdinalIgnoreCase))
-                                    {
-                                        if (int.TryParse(headerValue, out int retrySecs))
-                                            myDeserializedClass.retryInSeconds = Math.Max(myDeserializedClass.retryInSeconds, retrySecs);
-                                        else if (DateTimeOffset.TryParse(headerValue, out var dto))
-                                            myDeserializedClass.retryInSeconds = Math.Max(myDeserializedClass.retryInSeconds, (int)Math.Max(0, (dto - DateTimeOffset.UtcNow).TotalSeconds));
-                                    }
-
-                                    if (headerName.IndexOf("X-RateLimit-Reset", StringComparison.OrdinalIgnoreCase) >= 0 && long.TryParse(headerValue, out long epoch))
-                                    {
-                                        var secs = (int)Math.Max(0, epoch - DateTimeOffset.UtcNow.ToUnixTimeSeconds());
-                                        myDeserializedClass.retryInSeconds = Math.Max(myDeserializedClass.retryInSeconds, secs);
-                                    }
-
-                                    if (headerName.Contains("Remaining", StringComparison.OrdinalIgnoreCase) && headerValue == "0")
-                                        checkResetLocal = true;
-
-                                    if (checkResetLocal && headerName.IndexOf("Reset", StringComparison.OrdinalIgnoreCase) >= 0)
-                                    {
-                                        if (int.TryParse(headerValue, out int resetValue))
-                                            myDeserializedClass.retryInSeconds = Math.Max(myDeserializedClass.retryInSeconds, resetValue);
-                                        else if (long.TryParse(headerValue, out long epochVal))
-                                            myDeserializedClass.retryInSeconds = Math.Max(myDeserializedClass.retryInSeconds, (int)Math.Max(0, epochVal - DateTimeOffset.UtcNow.ToUnixTimeSeconds()));
-                                    }
-                                }
-                            }
-
+                        myDeserializedClass.retryInSeconds = Math.Max(myDeserializedClass.retryInSeconds, GetRetryDelaySecondsFromHeaders(response.Headers));
                         myDeserializedClass.statusCode = "TooManyRequests";
                     }
                 }
-                }
                 catch (Exception ex)
                 {
-                    Console.WriteLine($"Failed JSON: {myJsonResponse}");
-                    Console.WriteLine(ex.ToString());
+                    // A body that will not parse came from infrastructure (CDN/proxy/WAF), not the Tumblr
+                    // API, so it says nothing about this post. Retryable, not a failure of the post itself.
+                    myDeserializedClass.statusCode = response.StatusCode.ToString();
 
-                    if (!response.IsSuccessful)
-                    {
-                        string? statusStr = null;
-                        try { statusStr = response != null ? response.StatusCode.ToString() : null; } catch { statusStr = null; }
-                        Console.WriteLine($"{statusStr}\t{response?.StatusDescription}");
-                        if (!string.IsNullOrEmpty(statusStr))
-                            myDeserializedClass.statusCode = statusStr;
-
-                        bool checkReset = false;
-
-                        if ((myDeserializedClass.statusCode != "NotFound" || myDeserializedClass.retryInSeconds > 0) && response.Headers != null)
-                        {
-                            bool foundRateLimitHeader = false;
-                            foreach (var header in response.Headers)
-                            {
-                                if (header.Name != null && header.Value != null)
-                                {
-                                    Console.WriteLine($"{header.Name}  -   {header.Value}");
-                                    var headerValue = header.Value?.ToString();
-                                    if (!string.IsNullOrEmpty(headerValue))
-                                    {
-                                        if (string.Equals(header.Name, "Retry-After", StringComparison.OrdinalIgnoreCase))
-                                        {
-                                            if (int.TryParse(headerValue, out int retrySecs))
-                                            {
-                                                if (myDeserializedClass.retryInSeconds < retrySecs)
-                                                    myDeserializedClass.retryInSeconds = retrySecs;
-                                            }
-                                            else if (DateTimeOffset.TryParse(headerValue, out DateTimeOffset dto))
-                                            {
-                                                var secs = (int)Math.Max(0, (dto - DateTimeOffset.UtcNow).TotalSeconds);
-                                                if (myDeserializedClass.retryInSeconds < secs)
-                                                    myDeserializedClass.retryInSeconds = secs;
-                                            }
-                                        }
-
-                                        if (checkReset && header.Name.Contains("Reset", StringComparison.OrdinalIgnoreCase))
-                                        {
-                                            if (int.TryParse(headerValue, out int resetValue))
-                                            {
-                                                if (myDeserializedClass.retryInSeconds < resetValue)
-                                                    myDeserializedClass.retryInSeconds = resetValue;
-                                            }
-                                            else if (long.TryParse(headerValue, out long epochVal))
-                                            {
-                                                var secs = (int)Math.Max(0, epochVal - DateTimeOffset.UtcNow.ToUnixTimeSeconds());
-                                                if (myDeserializedClass.retryInSeconds < secs)
-                                                    myDeserializedClass.retryInSeconds = secs;
-                                            }
-                                        }
-
-                                        if (header.Name.IndexOf("X-RateLimit-Reset", StringComparison.OrdinalIgnoreCase) >= 0)
-                                        {
-                                            if (long.TryParse(headerValue, out long epoch))
-                                            {
-                                                var secs = (int)Math.Max(0, epoch - DateTimeOffset.UtcNow.ToUnixTimeSeconds());
-                                                if (myDeserializedClass.retryInSeconds < secs)
-                                                    myDeserializedClass.retryInSeconds = secs;
-                                            }
-                                            foundRateLimitHeader = true;
-                                        }
-                                    }
-                                    if (header.Name.Contains("Remaining", StringComparison.OrdinalIgnoreCase) && header.Value.ToString() == "0")
-                                        checkReset = true;
-                                    else
-                                        checkReset = false;
-                                }
-
-                                if (foundRateLimitHeader || (response != null && response.StatusCode == System.Net.HttpStatusCode.TooManyRequests))
+                    if (response.StatusCode == System.Net.HttpStatusCode.TooManyRequests)
                     {
+                        myDeserializedClass.retryInSeconds = GetRetryDelaySecondsFromHeaders(response.Headers);
                         myDeserializedClass.statusCode = "TooManyRequests";
                     }
-                            }
-                        }
+                    else
+                    {
+                        myDeserializedClass.transientFailure = true;
+                        Console.WriteLine($"[Transient] {FormatKeyLabel(key)} HTTP {(int)response.StatusCode} {response.StatusDescription} — unparseable body: {SummarizeBody(myJsonResponse)}");
+
+                        // A 2xx that will not parse is a genuine surprise; keep the detail for that case only.
+                        if (response.IsSuccessful)
+                            Console.WriteLine(ex.ToString());
                     }
                 }
 

URLNotesGrabberCORE/LegacyPostsDbImporter.cs

@@ -29,6 +29,8 @@ namespace URLNotesGrabberCORE
             Console.WriteLine($"Reading legacy posts.db: {legacyDbPath}");
 
             int blogsCopied = 0;
+            int blogPathsWritten = 0;
+            int blogsWithoutPath = 0;
             int postsUpserted = 0;
             int errors = 0;
 
@@ -48,7 +50,13 @@ namespace URLNotesGrabberCORE
                         if (string.IsNullOrWhiteSpace(blogName)) continue;
                         try
                         {
-                            DataAccess.SetBlogTTFolderPath(blogName, ttFolderPath);
+                            // A legacy row whose TTFolderPath was already NULL copies nothing.
+                            // Counting it as "copied" is what hid the fact that this import has
+                            // never populated a single path.
+                            if (string.IsNullOrWhiteSpace(ttFolderPath))
+                                blogsWithoutPath++;
+                            else if (DataAccess.SetBlogTTFolderPath(blogName, ttFolderPath.Trim()))
+                                blogPathsWritten++;
                             blogsCopied++;
                         }
                         catch (Exception ex)
@@ -58,7 +66,7 @@ namespace URLNotesGrabberCORE
                         }
                     }
                 }
-                Console.WriteLine($"  Blogs copied: {blogsCopied}");
+                Console.WriteLine($"  Blogs seen: {blogsCopied}, TTFolderPath written: {blogPathsWritten}, legacy rows with no path: {blogsWithoutPath}");
 
                 // 2) Copy Posts
                 try
@@ -138,7 +146,8 @@ namespace URLNotesGrabberCORE
             }
 
             Console.WriteLine($"\n========== Legacy import summary ==========");
-            Console.WriteLine($"Blogs copied:    {blogsCopied}");
+            Console.WriteLine($"Blogs seen:      {blogsCopied}");
+            Console.WriteLine($"Paths written:   {blogPathsWritten} (legacy rows with no path: {blogsWithoutPath})");
             Console.WriteLine($"Posts upserted:  {postsUpserted}");
             Console.WriteLine($"Errors:          {errors}");
             return errors == 0 ? 0 : 2;

URLNotesGrabberCORE/OutputMode.cs

@@ -8,28 +8,51 @@ namespace URLNotesGrabberCORE
     // field order). Reads from TL.db via DataAccess.GetAllPostsForBlog.
     public static class OutputMode
     {
-        public static int Run(IConfiguration config)
+        public static int Run(IConfiguration config, string[]? args = null)
         {
             DataAccess.EnsureTTFileHelperColumnsExist();
 
-            var blogs = DataAccess.GetAllBlogsWithTTFolderPath();
-            Console.WriteLine($"Found {blogs.Count} blog(s) to process.");
+            string dbPath = DataAccess.GetActiveDbPath();
+            Console.WriteLine($"Database: {Path.GetFullPath(dbPath)}");
 
-            foreach (var (blogName, ttFolderPath) in blogs)
+            if (!RefreshPaths(config, args ?? Array.Empty<string>()))
+                return 1;
+
+            var blogs = DataAccess.GetAllBlogsWithTTFolderPath();
+            int activeBlogs = DataAccess.CountActiveBlogs();
+            Console.WriteLine($"{blogs.Count} of {activeBlogs} active blog(s) have a TTFolderPath.");
+
+            if (blogs.Count == 0)
+            {
+                Console.WriteLine($"\nNothing to export: no blog in {Path.GetFullPath(dbPath)} has a TTFolderPath.");
+                Console.WriteLine("Point --output at a TumblThree root so it can populate them: --output <root>,");
+                Console.WriteLine("or set appSettings:PathTTRoot so the refresh runs automatically.");
+                return 1;
+            }
+
+            int missingFolderCount = 0;
+            int writtenCount = 0;
+
+            foreach (var (blogName, folder) in blogs)
             {
                 Console.WriteLine($"\nProcessing blog: {blogName}");
 
-                if (string.IsNullOrWhiteSpace(ttFolderPath) || !Directory.Exists(ttFolderPath))
+                // A stored path that this machine cannot see means the value was written on
+                // another machine -- re-running --updatepaths locally is the fix, so say so
+                // rather than lumping it in with "not set".
+                if (!Directory.Exists(folder))
                 {
-                    Console.WriteLine($"  TTFolderPath does not exist or is not set. Skipping.");
+                    Console.WriteLine($"  TTFolderPath folder not found: {folder}. Skipping.");
+                    missingFolderCount++;
                     continue;
                 }
 
-                Console.WriteLine($"  TTFolderPath: {ttFolderPath}");
+                Console.WriteLine($"  TTFolderPath: {folder}");
+                writtenCount++;
 
                 try
                 {
-                    foreach (var bakFile in Directory.GetFiles(ttFolderPath, "*.bak"))
+                    foreach (var bakFile in Directory.GetFiles(folder, "*.bak"))
                         File.Delete(bakFile);
                 }
                 catch (Exception ex)
@@ -37,7 +60,7 @@ namespace URLNotesGrabberCORE
                     Console.WriteLine($"  Error deleting .bak files: {ex.Message}");
                 }
 
-                RenameExistingTxtFilesToBak(ttFolderPath);
+                RenameExistingTxtFilesToBak(folder);
 
                 var posts = DataAccess.GetAllPostsForBlog(blogName);
                 Console.WriteLine($"  Found {posts.Count} post(s) for this blog.");
@@ -46,7 +69,7 @@ namespace URLNotesGrabberCORE
                 foreach (var typeGroup in grouped)
                 {
                     string postType = typeGroup.Key ?? "Unknown";
-                    string outputFilePath = Path.Combine(ttFolderPath, $"{postType}.txt");
+                    string outputFilePath = Path.Combine(folder, $"{postType}.txt");
                     var ordered = typeGroup.OrderBy(p => p.Date).ToList();
                     Console.WriteLine($"    Writing {ordered.Count} post(s) to {postType}.txt");
 
@@ -65,10 +88,57 @@ namespace URLNotesGrabberCORE
                 }
             }
 
-            Console.WriteLine("\nOutput mode complete.");
+            Console.WriteLine($"\nOutput mode complete. {writtenCount} blog(s) exported, {missingFolderCount} skipped for a missing folder.");
+
+            if (writtenCount == 0)
+                Console.WriteLine("Every TTFolderPath points at a folder this machine cannot see. The paths were most likely written on another machine -- re-run --updatepaths <root> here so they match local drive letters.");
+
             return 0;
         }
 
+        // Re-reads the TumblThree Index metadata into Blogs.TTFolderPath before exporting.
+        // A TL.db synced between machines cannot hold one absolute path that is valid on
+        // both, so the stored paths are only trustworthy on the machine that wrote them --
+        // which makes this refresh part of a normal export rather than a separate chore.
+        // Returns false only when the run should stop.
+        private static bool RefreshPaths(IConfiguration config, string[] args)
+        {
+            var settings = config.GetSection("appSettings");
+
+            if (args.Any(a => string.Equals(a, "--norefresh", StringComparison.OrdinalIgnoreCase)))
+            {
+                Console.WriteLine("Path refresh skipped (--norefresh); exporting to whatever paths TL.db already holds.");
+                return true;
+            }
+
+            string? root = args.FirstOrDefault(a => !a.StartsWith("--", StringComparison.Ordinal))
+                           ?? settings.GetValue<string>("PathTTRoot");
+
+            var result = UpdateBlogPathsRunner.Scan(root, verbose: false);
+
+            switch (result.Outcome)
+            {
+                case UpdateBlogPathsRunner.ScanOutcome.NoRootConfigured:
+                    Console.WriteLine("No TumblThree root configured (appSettings:PathTTRoot is empty and none was passed),");
+                    Console.WriteLine("so TTFolderPath was not refreshed. Pass one as --output <root> to refresh it.");
+                    return true;
+
+                case UpdateBlogPathsRunner.ScanOutcome.IndexFolderMissing:
+                    // Silently exporting stale paths here would defeat the point of folding
+                    // the refresh in, so a bad root is a hard stop.
+                    Console.WriteLine($"Index folder not found at: {result.IndexPath}");
+                    Console.WriteLine("Fix the root (or pass --norefresh to export the paths already in TL.db).");
+                    return false;
+
+                default:
+                    Console.WriteLine($"Refreshed paths from {result.IndexPath}: " +
+                                      $"{result.MetadataFiles} metadata file(s), {result.Written} written, " +
+                                      $"{result.Unchanged} already correct, {result.NoLocation} without a location, " +
+                                      $"{result.NoMatchingRow} without a blog row, {result.Errors} error(s).");
+                    return true;
+            }
+        }
+
         private static void RenameExistingTxtFilesToBak(string folderPath)
         {
             try

URLNotesGrabberCORE/Program.cs

@@ -281,7 +281,7 @@ namespace URLNotesGrabberCORE
                             managedCollectRun = true;
                         }
 
-                        CollectNotes(settings.GetValue<string>("PathOutput"), withoutNotesOnly, beforeDate, managedCollectRun).GetAwaiter().GetResult();
+                        exitCode = CollectNotes(settings.GetValue<string>("PathOutput"), withoutNotesOnly, beforeDate, managedCollectRun).GetAwaiter().GetResult();
                         break;
 
                     case "--blogsR": //collect notes from all posts
@@ -343,7 +343,7 @@ namespace URLNotesGrabberCORE
                         break;
 
                     case "--output":
-                        exitCode = OutputMode.Run(config);
+                        exitCode = OutputMode.Run(config, args.Skip(1).ToArray());
                         break;
 
                     case "--revert":
@@ -439,7 +439,9 @@ namespace URLNotesGrabberCORE
 
             Console.WriteLine("--ingest [blogname]\t Ingest Tumblr .txt exports from appSettings:PathTTRoot into TL.db (all blogs, or single blog if name given)");
 
-            Console.WriteLine("--output\t Export posts from TL.db back to .txt files in each blog's TTFolderPath");
+            Console.WriteLine("--output [rootPath]\t Refresh Blogs.TTFolderPath from <root>\\Index (or appSettings:PathTTRoot), then export posts from TL.db back to .txt files in each blog's folder");
+
+            Console.WriteLine("--output --norefresh\t Export without refreshing TTFolderPath first");
 
             Console.WriteLine("--revert [blogname]\t Recursively scan the PathInput tree and restore *.bak back to *.txt (current .txt saved as next-free .bkN); optional blogname filters by path substring");
 
@@ -452,7 +454,7 @@ namespace URLNotesGrabberCORE
             Console.WriteLine("--importposts [path-to-posts.db]\t One-time migration: copy legacy ThreeTxtFileHelper posts.db rows into TL.db");
 
             Console.WriteLine();
-            Console.WriteLine("Exit status: 0 = success; 1 = unexpected error; 2 = usage error (unknown command or bad/missing arguments)");
+            Console.WriteLine("Exit status: 0 = success; 1 = unexpected error; 2 = usage error (unknown command or bad/missing arguments); 3 = incomplete (--collect paused on a rate limit, or skipped posts after transient API failures) - relaunch to resume");
         }
 
         static void WritePostBlogsToFile(string outPath)
@@ -582,16 +584,11 @@ namespace URLNotesGrabberCORE
 
         protected static string NormalizeBlogFolderName(string folderName)
         {
-            return folderName
-                .Replace("_1", "")
-                .Replace("_2", "")
-                .Replace("_3", "")
-                .Replace("_4", "")
-                .Replace("_5", "")
-                .Replace("_6", "")
-                .Replace("_7", "")
-                .Replace("_8", "")
-                .Replace("_9", "");
+            // Archive tools suffix duplicate blog folders with _1, _2, ... _10 and beyond. Strip only a
+            // trailing numeric suffix: unanchored substring removal ate the "_1" inside "_10" and left the
+            // "0" welded to the name (zomb-eh_10 -> zomb-eh0), and mangled any blog whose real name
+            // contains "_1". A blog name is never a prefix of itself plus "_<digits>", so this is safe.
+            return System.Text.RegularExpressions.Regex.Replace(folderName, @"_\d+$", "");
         }
 
         static async Task FetchAndStoreReplyText(string blogName, long postID, long timestamp)
@@ -847,7 +844,9 @@ namespace URLNotesGrabberCORE
 
                 RateLimiter limiter = new SlidingWindowRateLimiter(new SlidingWindowRateLimiterOptions
                 {
-                    PermitLimit = 300,
+                    // 1/sec average, matching --collect: the CDN reacts to aggregate traffic from the IP,
+                    // not to per-command rates.
+                    PermitLimit = 60,
                     QueueProcessingOrder = QueueProcessingOrder.OldestFirst,
                     QueueLimit = 1,
                     Window = TimeSpan.FromMinutes(1),
@@ -883,7 +882,9 @@ namespace URLNotesGrabberCORE
 
                     while (hasMoreLikes)
                     {
-                        using RateLimitLease lease = limiter.AttemptAcquire(1);
+                        // Wait for a permit rather than giving up on one: the limiter paces the loop, it is
+                        // not a failure condition. Only one acquire is ever pending, so QueueLimit = 1 suffices.
+                        using RateLimitLease lease = await limiter.AcquireAsync(1);
                         if (!lease.IsAcquired)
                         {
                             Console.WriteLine("!@@@@@@@ - Rate Limited Exceeded: No Lease Available");
@@ -1129,6 +1130,43 @@ if (shouldInsert)
             }
         }
 
+        // Backoff between in-place retries of a transient infrastructure failure. Most CDN 403s and edge
+        // 5xxs clear within a few seconds, so retrying here saves the post its single attempt for the pass.
+        static readonly int[] TransientBackoffSeconds = { 1, 4, 10 };
+
+        // Fetches one page, retrying transient failures in place. Rate limits are returned to the caller
+        // untouched — those are handled by pausing the whole run, not by retrying this post.
+        static async Task<Root> FetchNotesPage(Tuple<string, long, long, long> post, string beforeTimestamp)
+        {
+            Root response = null!;
+
+            for (int attempt = 0; ; attempt++)
+            {
+                var key = ApiKeyPool.GetCurrentKey();
+                response = await APIAccess.GrabNotes(key, post.Item1, post.Item2, beforeTimestamp);
+
+                if (response.statusCode == "TooManyRequests")
+                {
+                    ApiKeyPool.MarkRateLimited(key, response.retryInSeconds > 0 ? response.retryInSeconds : 60);
+                    return response;
+                }
+
+                if (!response.transientFailure)
+                {
+                    // Only a response that actually reached the API says anything about the key's standing.
+                    ApiKeyPool.MarkAvailable(key);
+                    return response;
+                }
+
+                if (attempt >= TransientBackoffSeconds.Length)
+                    return response;
+
+                int delay = TransientBackoffSeconds[attempt];
+                Console.WriteLine($"[Transient] retry {attempt + 1}/{TransientBackoffSeconds.Length} in {delay}s");
+                await Task.Delay(delay * 1000);
+            }
+        }
+
         static async Task<string> GrabNotes(Tuple<string, long, long, long> post)
         {
             try
@@ -1146,18 +1184,16 @@ if (shouldInsert)
                 string beforeTimestamp = post.Item3.ToString();
                 bool hasReplies = false;
                 const int maxPages = 500;
-                var key = ApiKeyPool.GetCurrentKey();
-                var response = await APIAccess.GrabNotes(key, post.Item1, post.Item2, beforeTimestamp);
+                var response = await FetchNotesPage(post, beforeTimestamp);
 
                 if (response.statusCode == "TooManyRequests")
-                {
-                    int retry = response.retryInSeconds > 0 ? response.retryInSeconds : 60;
-                    ApiKeyPool.MarkRateLimited(key, retry);
                     return "TooManyRequests";
-                }
 
-                if (response.meta?.status != 429)
-                    ApiKeyPool.MarkAvailable(key);
+                if (response.transientFailure)
+                {
+                    Console.WriteLine($"[Skip] {post.Item1}/{post.Item2} — {response.statusCode} after {TransientBackoffSeconds.Length} retries");
+                    return "Transient";
+                }
 
                 if (IsNotFound(response))
                 {
@@ -1166,19 +1202,6 @@ if (shouldInsert)
                     Thread.Sleep(1000);
                     return "NotFound";
                 }
-                if (response == null)
-                {
-                    Console.WriteLine("##### Response is null - API Failure? ###");
-                    return "FAILURE";
-                }
-                if (response.statusCode == "TooManyRequests")
-                {
-                    int retry = response.retryInSeconds > 0 ? response.retryInSeconds : 60;
-                    ApiKeyPool.MarkRateLimited(key, retry);
-
-                    ApiKeyPool.SleepUntilAnyAvailable(30);
-                    return response.statusCode;
-                }
 
                 // Pagination loop
                 while (true)
@@ -1228,16 +1251,15 @@ if (shouldInsert)
                         break;
                     }
 
-                    key = ApiKeyPool.GetCurrentKey();
-                    response = await APIAccess.GrabNotes(key, post.Item1, post.Item2, beforeTimestamp);
+                    response = await FetchNotesPage(post, beforeTimestamp);
                     if (response.statusCode == "TooManyRequests")
-                    {
-                        int retry = response.retryInSeconds > 0 ? response.retryInSeconds : 60;
-                        ApiKeyPool.MarkRateLimited(key, retry);
                         return "TooManyRequests";
+
+                    if (response.transientFailure)
+                    {
+                        Console.WriteLine($"[Skip] {post.Item1}/{post.Item2} — {response.statusCode} on page {page} after {TransientBackoffSeconds.Length} retries");
+                        return "Transient";
                     }
-                    if (response.meta?.status != 429)
-                        ApiKeyPool.MarkAvailable(key);
 
                     if (IsNotFound(response))
                     {
@@ -1268,7 +1290,11 @@ if (shouldInsert)
             return "UNKNOWN";
         }
 
-        static async Task CollectNotes(string outPath, bool withoutNotesOnly = true, DateTime? beforeDate = null, bool managedRun = false)
+        // Consecutive transient failures that mean the API edge is rejecting traffic wholesale rather than
+        // blipping on one post. Past this, skipping post-by-post would just hammer a closed door.
+        const int MaxConsecutiveTransient = 10;
+
+        static async Task<int> CollectNotes(string outPath, bool withoutNotesOnly = true, DateTime? beforeDate = null, bool managedRun = false)
         {
             List<Tuple<string, long, long, long>> posts = DataAccess.GetPosts(withoutNotesOnly, beforeDate);
 
@@ -1277,9 +1303,14 @@ if (shouldInsert)
             // post that keeps returning FAILURE/UNKNOWN. Successful/NotFound posts drop out via the DB filter anyway.
             HashSet<(string, long)> attempted = new HashSet<(string, long)>();
 
+            int skipped = 0;
+            int consecutiveTransient = 0;
+
             RateLimiter limiter = new SlidingWindowRateLimiter(new SlidingWindowRateLimiterOptions
             {
-                PermitLimit = 300,
+                // 1/sec average. Sustained higher rates draw CDN-level 403s that the API's own rate-limit
+                // headers never warn about, so this sits well under the per-key quota on purpose.
+                PermitLimit = 60,
                 QueueProcessingOrder = QueueProcessingOrder.OldestFirst,
                 QueueLimit = 1,
                 Window = TimeSpan.FromMinutes(1),
@@ -1304,19 +1335,35 @@ if (shouldInsert)
 
                         ApiKeyPool.SleepUntilAnyAvailable(30);
 
-                        string status;
-
-                        using RateLimitLease lease = limiter.AttemptAcquire(1);
-                        if (lease.IsAcquired)
+                        // Wait for a permit rather than giving up on one: the limiter paces the loop, it is
+                        // not a failure condition. Only one acquire is ever pending, so QueueLimit = 1 suffices.
+                        using RateLimitLease lease = await limiter.AcquireAsync(1);
+                        if (!lease.IsAcquired)
                         {
+                            Console.WriteLine("!@@@@@@@ - Rate Limited Exceeded: No Lease Available");
+                            return 3; // abort without completing the run so a later launch resumes
+                        }
+
                         Console.WriteLine("{0,32} - {1,15} - {2}", post.Item1, post.Item2, DateTimeOffset.FromUnixTimeSeconds(post.Item4).ToString());
-                            status = await GrabNotes(post);
+                        string status = await GrabNotes(post);
+
+                        if (status == "Transient")
+                        {
+                            // Retries in GrabNotes are already exhausted. Skip the post so the pass can make
+                            // progress; it stays unmarked in the DB, so the next launch picks it up again.
+                            attempted.Add((post.Item1, post.Item2));
+                            skipped++;
+                            consecutiveTransient++;
+
+                            if (consecutiveTransient >= MaxConsecutiveTransient)
+                            {
+                                Console.WriteLine($"[Abort] {consecutiveTransient} consecutive transient failures - the API edge is rejecting traffic. Pausing run; relaunch to resume. ({skipped} post(s) skipped)");
+                                return 3;
+                            }
                         }
                         else
                         {
-                            Console.WriteLine("!@@@@@@@ - Rate Limited Exceeded: No Lease Available");
-                            return; // throttle: abort without completing the run so a later launch resumes
-                        }
+                            consecutiveTransient = 0;
 
                             if (status == "Success")
                             {
@@ -1333,8 +1380,8 @@ if (shouldInsert)
                             {
                                 // Throttle, not a real per-post failure: don't consume this post's single attempt.
                                 // Abort the pass without completing so a later launch resumes against the same cutoff.
-                            Console.WriteLine("GrabNotes Result: TooManyRequests - pausing run; relaunch to resume.");
-                            return;
+                                Console.WriteLine($"GrabNotes Result: TooManyRequests - pausing run; relaunch to resume. ({skipped} post(s) skipped)");
+                                return 3;
                             }
                             else
                             {
@@ -1342,6 +1389,7 @@ if (shouldInsert)
                                 attempted.Add((post.Item1, post.Item2));
                                 Console.WriteLine("GrabNotes Result: " + status);
                             }
+                        }
 
                         // Re-fetch the updated list after processing the current post
                         posts = DataAccess.GetPosts(withoutNotesOnly, beforeDate);
@@ -1354,14 +1402,42 @@ if (shouldInsert)
                     DataAccess.CompleteCollectRun();
                     Console.WriteLine("Full re-check run complete.");
                 }
+
+                if (skipped > 0)
+                {
+                    Console.WriteLine($"Pass finished with {skipped} post(s) skipped after transient failures; relaunch to retry them.");
+                    return 3;
+                }
+
+                return 0;
             }
             catch (Exception ex)
             {
                 Console.WriteLine(ex.ToString());
+                return 1;
             }
         }
 
 
+        // Field prefixes TraverseDirectory recognizes as the start of a new record field.
+        // Used to know where a multi-line Body/Downloaded files value ends.
+        private static readonly string[] TraverseDirectoryFieldPrefixes = new[]
+        {
+            "Post id:", "Reblog url:", "Reblog name:", "Reblog root url:", "Downloaded files:",
+            "Reblog key:", "Date:", "Body:", "Post url:", "Answer:", "Audio Caption:", "Blog Name:",
+            "Link:", "Photo Caption:", "Photo url:", "Question:", "Quote:", "Slug:", "Summary:",
+            "Tags:", "Title:"
+        };
+
+        private static bool IsTraverseDirectoryFieldLine(string line)
+        {
+            foreach (var prefix in TraverseDirectoryFieldPrefixes)
+            {
+                if (line.StartsWith(prefix, StringComparison.OrdinalIgnoreCase)) return true;
+            }
+            return false;
+        }
+
         static void TraverseDirectory(string path, string outPath, List<string> contains, ref int postsAdded, string blogName = "", string startFromBlogName = "", bool logRecordImports = false)
         {
             DateTime directoryStart = DateTime.Now;
@@ -1398,8 +1474,10 @@ if (shouldInsert)
                                 var urls = new List<string>();
                                 var reblog = new ReblogRecord();
 
-                                foreach (string line in File.ReadLines(file))
+                                string[] fileLines = File.ReadAllLines(file);
+                                for (int lineIndex = 0; lineIndex < fileLines.Length; lineIndex++)
                                 {
+                                    string line = fileLines[lineIndex];
                                     if (line.StartsWith("Post id:", StringComparison.OrdinalIgnoreCase))
                                     {
                                         if (reblog.reblogName != "." && reblog.postID != "." && reblog.date != "." && reblog.reblogURL != "." && reblog.downloadedFiles == ".")
@@ -1420,7 +1498,7 @@ if (shouldInsert)
                                                 DataAccess.AddPost(curDir, long.Parse(reblog.postID), reblog.reblogURL, reblog.date, reblog.postURL, reblog.slug, reblog.reblogKey,
                                                                     reblog.reblogName, reblog.summary, reblog.quote, reblog.body, reblog.tags, reblog.link, reblog.photoURL,
                                                                     reblog.photoCaption, reblog.downloadedFiles, reblog.audioCaption, reblog.question, reblog.answer,
-                                                                    reblog.title, false);
+                                                                    reblog.title, false, rootURL: reblog.rootURL);
                                                 recordImportStopwatch.Stop();
 
                                                 postsAdded++;
@@ -1445,9 +1523,21 @@ if (shouldInsert)
                                     {
                                         reblog.reblogName = line.Substring(13).Trim();
                                     }
+                                    if (line.StartsWith(@"Reblog root url:", StringComparison.OrdinalIgnoreCase))
+                                    {
+                                        reblog.rootURL = line.Substring(16).Trim();
+                                    }
                                     if (line.StartsWith(@"Downloaded files:", StringComparison.OrdinalIgnoreCase))
                                     {
-                                        reblog.downloadedFiles = line.Substring(17).Trim();
+                                        var valueLines = new List<string> { line.Substring(17).Trim() };
+                                        int nextLineIndex = lineIndex + 1;
+                                        while (nextLineIndex < fileLines.Length && !IsTraverseDirectoryFieldLine(fileLines[nextLineIndex]))
+                                        {
+                                            valueLines.Add(fileLines[nextLineIndex]);
+                                            nextLineIndex++;
+                                        }
+                                        reblog.downloadedFiles = string.Join("\n", valueLines).Trim();
+                                        lineIndex = nextLineIndex - 1;
                                     }
                                     if (line.StartsWith(@"Reblog key:", StringComparison.OrdinalIgnoreCase))
                                     {
@@ -1459,7 +1549,15 @@ if (shouldInsert)
                                     }
                                     if (line.StartsWith(@"Body:", StringComparison.OrdinalIgnoreCase))
                                     {
-                                        reblog.body = line.Substring(6).Trim();
+                                        var valueLines = new List<string> { line.Substring(6).Trim() };
+                                        int nextLineIndex = lineIndex + 1;
+                                        while (nextLineIndex < fileLines.Length && !IsTraverseDirectoryFieldLine(fileLines[nextLineIndex]))
+                                        {
+                                            valueLines.Add(fileLines[nextLineIndex]);
+                                            nextLineIndex++;
+                                        }
+                                        reblog.body = string.Join("\n", valueLines).Trim();
+                                        lineIndex = nextLineIndex - 1;
                                     }
                                     if (line.StartsWith(@"Post url:", StringComparison.OrdinalIgnoreCase))
                                     {
@@ -1548,7 +1646,7 @@ if (shouldInsert)
                                             DataAccess.AddPost(curDir, long.Parse(reblog.postID), reblog.reblogURL, reblog.date, reblog.postURL, reblog.slug, reblog.reblogKey,
                                                                     reblog.reblogName, reblog.summary, reblog.quote, reblog.body, reblog.tags, reblog.link, reblog.photoURL,
                                                                     reblog.photoCaption, reblog.downloadedFiles, reblog.audioCaption, reblog.question, reblog.answer,
-                                                                    reblog.title, true);
+                                                                    reblog.title, true, rootURL: reblog.rootURL);
                                             recordImportStopwatch.Stop();
 
                                             postsAdded++;

URLNotesGrabberCORE/ResponseNotes.cs

@@ -85,6 +85,10 @@ namespace URLNotesGrabberCORE
         public int retryInSeconds { get; set; }
 
         public string rawJson { get; set; }
+
+        // The request never reached the Tumblr API (transport error, or an edge/CDN response with a
+        // non-JSON body). Says nothing about the post, so the caller should retry rather than fail it.
+        public bool transientFailure { get; set; }
     }
 
     // Classes for Posts API endpoint (for reply_text)

URLNotesGrabberCORE/TL.db.md

@@ -0,0 +1,333 @@
+# `TL.db` — schema notes
+
+The SQLite database behind **URLNotesGrabberCORE** and its sibling crawlers, and the one
+[Rolodex](https://git.basso.land/jim/Rolodex) reads.
+
+Everything below was read out of the live file, not inferred from code. Counts are as of
+**2026-07-29**; re-run the queries at the bottom to refresh them.
+
+- Journal mode: **WAL** — `TL.db-wal` and `TL.db-shm` live beside the file and are part of
+  the database. Copying `TL.db` alone gives you whatever was last checkpointed, not the
+  current state.
+- Page size: 4096.
+
+---
+
+## The three content tables
+
+| Table | Rows | What it is |
+|---|--:|---|
+| `Blogs` | 144,367 | The crawl registry — one row per known blog, plus crawl-state flags |
+| `Posts` | 14,589 | Stored post content. Only 3,602 blogs actually have any |
+| `Notes` | 1,189,604 | The engagement graph: `NoteBlogName` acted on `(RootBlogName, PostID)` |
+
+The engagement graph is the interesting part. 31,888 distinct blogs appear as engagers —
+far more than the 3,602 that have stored posts — which is what makes this a social graph
+rather than a post archive.
+
+### `Blogs`
+
+```sql
+CREATE TABLE "Blogs" (
+    "BlogName"              TEXT,
+    "HasBeenOutput"         INTEGER DEFAULT 0,
+    "IsActive"              INTEGER DEFAULT 1,
+    "DateAdded"             TEXT NOT NULL DEFAULT '12/24/25',
+    "ByLikes"               INTEGER NOT NULL DEFAULT 0,
+    "LikesPulled"           INTEGER NOT NULL DEFAULT 0,
+    "LikesCursor"           INTEGER DEFAULT 0,
+    "DateModified"          TEXT,
+    "DateCreated"           TEXT,
+    LikesNewestTimestamp    INTEGER DEFAULT 0,
+    LikesLastRefreshed      INTEGER DEFAULT 0,
+    LikesLastNewCount       INTEGER DEFAULT 0,
+    TTFolderPath            TEXT,
+    PRIMARY KEY("BlogName")
+);
+```
+
+`BlogName` is the primary key, so it is the only indexed way in. There is no index on any
+flag or date — filtering or sorting on those scans all 144k rows, which is affordable
+here and is not on `Notes`.
+
+Flag distribution: `IsActive = 1` on 144,366 of 144,367 rows, `HasBeenOutput = 1` on
+5,369, `ByLikes = 1` on 2. `IsActive` carries a second meaning as of Rolodex — see
+[`Blogs.IsActive`](#blogsisactive--now-written-by-two-applications) below.
+
+The columns after `DateCreated` were added later by `ALTER TABLE`, which is why they carry
+no quoting in the stored DDL. That is the normal way this schema grows.
+
+**`DateAdded` is not written consistently.** 126,423 rows hold ISO `yyyy-MM-dd HH:mm:ss`;
+17,944 hold US-format `M/d/yy` from a bulk import. As text those two sort into different
+parts of the table, so anything ordering or range-filtering on this column has to
+normalise first — see `DateSql` in Rolodex.
+
+### `Posts`
+
+```sql
+CREATE TABLE "Posts" (
+    "BlogName"              TEXT,
+    "PostID"                INTEGER,
+    "HasNotesGathered"      INTEGER DEFAULT 0,
+    "reblogURL"             TEXT,
+    "NotFound"              INTEGER DEFAULT 0,
+    "PostDate"              TEXT,
+    "NotesGatheredDateTime" INTEGER NOT NULL DEFAULT 1729746000,
+    "HasImage"              INTEGER NOT NULL DEFAULT 0,
+    "PostURL"               TEXT,
+    "Slug"                  TEXT,
+    "ReblogKey"             TEXT,
+    "ReblogName"            TEXT,
+    "Summary"               TEXT,
+    "Quote"                 TEXT,
+    "Body"                  TEXT,
+    "Tags"                  TEXT,
+    "Link"                  TEXT,
+    "PhotoURL"              TEXT,
+    "PhotoCaption"          TEXT,
+    "DownloadedFiles"       TEXT,
+    "AudioCaption"          TEXT,
+    "Question"              TEXT,
+    "Answer"                TEXT,
+    "Title"                 TEXT,
+    "ByLikes"               INTEGER NOT NULL DEFAULT 0,
+    "RootBlogName"          TEXT,
+    "RootURL"               TEXT,
+    "DateModified"          TEXT,
+    "DateCreated"           TEXT,
+    PostType                TEXT,
+    PRIMARY KEY("BlogName","PostID")
+);
+```
+
+**The key is `(BlogName, PostID)`, not `PostID`.** This matters more than it looks: 325
+post IDs exist under more than one blog, so an ID on its own is both ambiguous *and*
+unindexed. Any lookup should carry the blog name, and a batch lookup should group by blog
+so it stays on the leading column of the key.
+
+Notable:
+
+- **`PostType` is `NULL` on all 14,589 rows.** The column exists but nothing has ever
+  populated it. Treat it as unpopulated rather than as a type discriminator.
+- `HasImage = 1` on 14,268 rows — nearly all of them. It records that the post *had* a
+  picture, not that a usable URL was kept, so it is not a reliable predictor that anything
+  will render.
+- `PhotoURL` is largely unused; in practice the image markup lives inside `Body`.
+- `NotFound = 1` on 4,663 rows — posts that have since been deleted upstream.
+- The content columns (`Body`, `Quote`, `Question`, `Answer`, …) are the heavy ones. List
+  views should not select them.
+
+### `Notes`
+
+```sql
+CREATE TABLE "Notes" (
+    "RootBlogName"    TEXT,
+    "PostID"          INTEGER,
+    "NoteBlogName"    TEXT,
+    "TimeStamp"       INTEGER,
+    "Type"            TEXT,
+    "replyText"       TEXT DEFAULT '.',
+    "DatetimeCrawled" TEXT DEFAULT '2/12/26 12am',
+    "DateModified"    TEXT,
+    "DateCreated"     TEXT,
+    PRIMARY KEY("RootBlogName","PostID","TimeStamp","Type","NoteBlogName")
+);
+
+CREATE INDEX "Notes_idx_06e01ae3" ON "Notes" ("TimeStamp" DESC);
+CREATE INDEX "ix_NoteBlogName01"  ON "Notes" ("NoteBlogName");
+```
+
+One row per engagement event. `TimeStamp` is **unix seconds** — unlike every date column
+elsewhere in the schema, which are text.
+
+| `Type` | Rows | Share |
+|---|--:|--:|
+| `like` | 947,955 | 79.7% |
+| `reblog` | 224,323 | 18.9% |
+| `reply` | 15,201 | 1.3% |
+| `posted` | 2,106 | 0.2% |
+| `post_attribution` | 19 | — |
+
+At 1.19M rows this is the table that dictates how the whole database has to be queried:
+
+- **Nothing should run an unbounded `SELECT` or a bare `COUNT(*)` here.** A count scans
+  the lot on every call.
+- The only fast access paths are the primary key's leading columns (`RootBlogName`, then
+  `PostID`) and `ix_NoteBlogName01` on `NoteBlogName`. "Notes received by a blog" and
+  "notes given by a blog" are both cheap; almost nothing else is.
+- Ordering by anything but `TimeStamp` is a full sort of whatever the filters leave.
+- `replyText` is `'.'` on 1,174,706 rows — only `reply` notes carry real text.
+
+### Referential integrity
+
+There are no foreign keys, and the tables do not perfectly agree:
+
+- 4 `Posts` rows name a blog with no `Blogs` row.
+- 15 of the 31,888 distinct engagers have no `Blogs` row.
+
+So a name appearing in `Notes` or `Posts` is not a guarantee that the registry knows about
+it. Joins from those tables back to `Blogs` should tolerate a miss.
+
+---
+
+## The `'.'` placeholder convention
+
+**The crawler writes a single dot into text columns it has no value for, rather than
+`NULL`.** This is the single most surprising thing about the schema and it affects every
+consumer.
+
+| Column | `'.'` rows |
+|---|--:|
+| `Notes.replyText` | 1,174,706 |
+| `Posts.Title` | 13,144 |
+| `Posts.Body` | 172 |
+
+Any query whose output reaches a human should collapse it:
+
+```sql
+NULLIF(NULLIF(SomeColumn, '.'), '') AS SomeColumn
+```
+
+Empty string turns up too, hence the double `NULLIF`. Not every column is affected —
+`Blogs.TTFolderPath` and `Blogs.DateModified` currently have zero dot rows — but new
+columns tend to acquire them, so treat cleaning as the default for any text column
+rendered to a user.
+
+---
+
+## Supporting tables
+
+Crawler bookkeeping. Rolodex ignores all of these.
+
+| Table | Rows | What it is |
+|---|--:|---|
+| `DailyAPICount` | 133 | `(Date TEXT PK, APICount INTEGER)` — per-day API call tally against the rate limit |
+| `ApiKeyPoolState` | 2 | `(KeyName TEXT PK, RetryUntil INTEGER)` — per-key backoff; `RetryUntil` is unix seconds |
+| `ApiKeyPoolMeta` | 1 | `(Id PK CHECK (Id = 1), LastIndex)` — round-robin cursor. Singleton by check constraint |
+| `CollectRunState` | 1 | `(Id PK CHECK (Id = 1), RunCutoff, RunComplete, RunStarted, RunCompletedAt)` — resume state for an interrupted collection run. Also a singleton |
+
+---
+
+## `Blogs.IsActive` — now written by two applications
+
+`IsActive` has always been the crawler's work-selection flag. `GetBlogs` in
+`DataAccess.cs` joins on it to decide what to collect:
+
+```sql
+SELECT NoteBlogName, count(*) FROM notes
+INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName
+WHERE blogs.IsActive = @isActive AND ...
+```
+
+Nothing inside the crawler *writes* it — it is an input, set from outside.
+
+**Rolodex is now one of the things that sets it.** Removing a blog through the Rolodex UI
+runs exactly this:
+
+```sql
+UPDATE Blogs SET IsActive = 0 WHERE BlogName = ?;
+```
+
+Rolodex adds no column and changes no schema. It reuses this flag because the two meanings
+were judged to be one decision: a blog you do not want in the browsing UI is a blog you do
+not want to keep crawling. Removal therefore stops collection, and the Rolodex
+confirmation screen says so before anyone commits.
+
+- `1` (or absent/NULL) — live. Crawled, and visible in Rolodex.
+- `0` — removed. Not crawled, hidden from the Rolodex registry, dashboard counts and
+  engagement rollups.
+
+Restoring is the same `UPDATE` with a `1`. Nothing is destroyed either way: the blog's
+`Posts` and `Notes` rows are never touched, and Rolodex deliberately keeps showing them
+under its Posts and Notes pages. Removing a blog hides the blog, not what it collected.
+
+### What other tools need to know
+
+1. **Setting `IsActive = 0` now also hides the blog from Rolodex**, and setting it back to
+   `1` makes it reappear. If another tool deactivates blogs in bulk, it is also removing
+   them from the browsing UI — which may be exactly right, but it is no longer a
+   crawler-only decision.
+2. **Re-crawling a removed blog will not bring it back**, since nothing in the crawler
+   writes the flag. An `INSERT OR REPLACE` on the `Blogs` row *would*, by resetting it to
+   the column default of `1`. Prefer an `UPDATE` of the specific columns, or
+   `INSERT … ON CONFLICT DO UPDATE SET` naming only the columns being refreshed.
+3. **NULL is treated as live.** The column is `INTEGER DEFAULT 1` with no `NOT NULL`, so
+   Rolodex reads it through `COALESCE(IsActive, 1)`. A NULL therefore leaves the blog
+   visible rather than stranding it outside both the registry and the removed list, where
+   no screen could reach it. Write `0` or `1`, not NULL.
+4. **Backing the feature out is a configuration change, not a migration.** Because there is
+   no Rolodex-owned column, setting `Rolodex__EnableBlogDeletion=false` is the whole of it;
+   there is nothing to drop. Any blogs already at `IsActive = 0` simply go back to being
+   ordinary inactive blogs.
+
+---
+
+## `Posts.IsActive` and `Notes.IsActive` — optional, and not in this database yet
+
+The same flag is being extended to the two content tables, with the same meaning: `0` is
+removed, anything else — including `NULL` — is live. **Neither column exists in the live
+`TL.db` as of 2026-07-29**; the DDL quoted above for `Posts` and `Notes` is complete. Like
+`Blogs.IsActive`, they are written from outside this crawler.
+
+The crawler therefore treats both as optional, and as nothing it owns:
+
+- **It never writes them.** No `INSERT` column list names `IsActive`, no `UPDATE` sets it,
+  and `MapPrefixToColumn` — the only place a column name is chosen at runtime — cannot map
+  to it. Re-crawling a removed post or note refreshes its content and leaves the flag at
+  `0`. There is no `INSERT OR REPLACE` on `Posts` or `Notes` for a default to be reset by.
+- **It filters on them only when they exist.** `HasIsActiveColumn` in `DataAccess.cs` asks
+  `PRAGMA table_info` once per table per database path and caches the answer; the filter
+  is `COALESCE(IsActive, 1) = 1`, and it is dropped entirely when the column is absent.
+  Naming a missing column is a hard SQLite error, so this is what lets one build run
+  against databases on both sides of the change. The cache lives for the process — adding
+  the columns to a live database takes effect on the next run.
+
+Every read that selects posts or notes carries the filter: `GetPosts`, `GetReplies`,
+`GetRepliesWithMissingText`, `GetRepliesWithFilledText`, `GetAllPostTextColumns`,
+`GetAllPostsForBlog`, `GetPost`, `GetPostByIdAnyBlog`, and the engagement queries that
+count or join `Notes` (`GetBlogs`, `GetBlogsAll`, `GetBlogsForLikes`). The one deliberate
+omission is the `LEFT JOIN Notes` in `GetPosts`: nothing is selected from it and it can
+neither add nor remove a row, so filtering it would buy nothing.
+
+`LegacyPostsDbImporter` is unfiltered too — it reads a foreign legacy database whose
+`Posts` table is not this schema.
+
+Two consequences worth stating plainly, both inherited from how `Blogs.IsActive` is
+handled:
+
+1. **Removal hides a row; it does not freeze it.** The write paths are keyed on a post the
+   caller already selected, so an ingest or a correction run still overwrites the content
+   of a removed post. Only selection is filtered.
+2. **`NULL` is live.** Write `0` or `1`, not `NULL`, but a `NULL` leaves the row visible
+   rather than stranding it.
+
+---
+
+## Reproducing the numbers
+
+```sql
+SELECT 'Blogs', COUNT(*) FROM Blogs
+UNION ALL SELECT 'Posts', COUNT(*) FROM Posts
+UNION ALL SELECT 'Notes', COUNT(*) FROM Notes;
+
+-- note type mix
+SELECT Type, COUNT(*) FROM Notes GROUP BY Type ORDER BY 2 DESC;
+
+-- the two date shapes in Blogs.DateAdded
+SELECT CASE WHEN DateAdded LIKE '____-__-__%' THEN 'ISO' ELSE 'US' END, COUNT(*)
+FROM Blogs GROUP BY 1;
+
+-- post IDs that are ambiguous without a blog name
+SELECT COUNT(*) FROM (
+    SELECT PostID FROM Posts GROUP BY PostID HAVING COUNT(DISTINCT BlogName) > 1);
+
+-- rows that reference a blog the registry does not have
+SELECT COUNT(*) FROM Posts p
+WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = p.BlogName);
+```
+
+Open the file read-only so an inspection can never disturb a running crawl:
+
+```bash
+sqlite3 "file:TL.db?mode=ro" ".schema"
+```

URLNotesGrabberCORE/UpdateBlogPathsRunner.cs

@@ -4,34 +4,64 @@ namespace URLNotesGrabberCORE
 {
     // Port of ThreeTxtFileHelper/UpdateBlogPaths.cs. Reads .tumblr / .tmblrpriv metadata
     // files from a root\Index folder and populates Blogs.TTFolderPath in TL.db.
+    //
+    // Scan() is the reusable engine: --updatepaths wraps it as a standalone command and
+    // --output calls it as a refresh step, because a TL.db synced between machines cannot
+    // hold one absolute path that is correct on both.
     public static class UpdateBlogPathsRunner
     {
-        public static int Run(string rootPath)
+        public enum ScanOutcome
+        {
+            Completed,
+            NoRootConfigured,
+            IndexFolderMissing
+        }
+
+        public sealed class ScanResult
+        {
+            public ScanOutcome Outcome { get; init; }
+            public string RootPath { get; init; } = string.Empty;
+            public string IndexPath { get; init; } = string.Empty;
+            public int MetadataFiles { get; init; }
+            public int Written { get; init; }
+            public int Unchanged { get; init; }
+            public int NoLocation { get; init; }
+            public int NoMatchingRow { get; init; }
+            public int Errors { get; init; }
+        }
+
+        // verbose: log a line per metadata file. --updatepaths wants that detail; --output
+        // only wants the counts, since a few hundred lines before the export would bury it.
+        public static ScanResult Scan(string? rootPath, bool verbose)
         {
             if (string.IsNullOrWhiteSpace(rootPath))
-            {
-                Console.WriteLine("UpdateBlogPaths: rootPath is required.");
-                return 1;
-            }
+                return new ScanResult { Outcome = ScanOutcome.NoRootConfigured };
 
             DataAccess.EnsureTTFileHelperColumnsExist();
 
             string indexPath = Path.Combine(rootPath, "Index");
             if (!Directory.Exists(indexPath))
             {
-                Console.WriteLine($"Index folder not found at: {indexPath}");
-                return 1;
+                return new ScanResult
+                {
+                    Outcome = ScanOutcome.IndexFolderMissing,
+                    RootPath = rootPath,
+                    IndexPath = indexPath
+                };
             }
 
-            Console.WriteLine($"Scanning Index folder: {indexPath}");
-
             var blogFiles = Directory.GetFiles(indexPath, "*.tumblr")
                 .Concat(Directory.GetFiles(indexPath, "*.tmblrpriv"))
                 .ToList();
 
+            if (verbose)
                 Console.WriteLine($"Found {blogFiles.Count} blog metadata files");
 
             int updatedCount = 0;
+            int unchangedCount = 0;
+            int noLocationCount = 0;
+            int noRowCount = 0;
+            int errorCount = 0;
 
             foreach (var blogFile in blogFiles)
             {
@@ -44,27 +74,91 @@ namespace URLNotesGrabberCORE
 
                     if (root.TryGetProperty("FileDownloadLocation", out JsonElement locationElement))
                     {
-                        string? fileDownloadLocation = locationElement.GetString();
+                        string? fileDownloadLocation = locationElement.GetString()?.Trim();
                         if (!string.IsNullOrWhiteSpace(fileDownloadLocation))
                         {
-                            DataAccess.SetBlogTTFolderPath(blogName, fileDownloadLocation);
+                            // Report the database's answer, not the fact that the file parsed.
+                            if (DataAccess.SetBlogTTFolderPath(blogName, fileDownloadLocation))
+                            {
                                 updatedCount++;
+                                if (verbose)
                                     Console.WriteLine($"Updated {blogName}: {fileDownloadLocation}");
                             }
+                            else if (DataAccess.BlogExists(blogName))
+                            {
+                                unchangedCount++;
+                            }
+                            else
+                            {
+                                noRowCount++;
+                                Console.WriteLine($"No Blogs row named '{blogName}' -- path not stored (name may differ in case)");
+                            }
                         }
                         else
                         {
+                            noLocationCount++;
+                            if (verbose)
+                                Console.WriteLine($"Empty FileDownloadLocation in {blogFile}");
+                        }
+                    }
+                    else
+                    {
+                        noLocationCount++;
+                        if (verbose)
                             Console.WriteLine($"No FileDownloadLocation found in {blogFile}");
                     }
                 }
                 catch (Exception ex)
                 {
+                    errorCount++;
                     Console.WriteLine($"Error processing {blogFile}: {ex.Message}");
                 }
             }
 
-            Console.WriteLine($"\nUpdated {updatedCount} blogs with TTFolderPath");
-            return 0;
+            return new ScanResult
+            {
+                Outcome = ScanOutcome.Completed,
+                RootPath = rootPath,
+                IndexPath = indexPath,
+                MetadataFiles = blogFiles.Count,
+                Written = updatedCount,
+                Unchanged = unchangedCount,
+                NoLocation = noLocationCount,
+                NoMatchingRow = noRowCount,
+                Errors = errorCount
+            };
+        }
+
+        public static int Run(string rootPath)
+        {
+            if (string.IsNullOrWhiteSpace(rootPath))
+            {
+                Console.WriteLine("UpdateBlogPaths: rootPath is required.");
+                return 1;
+            }
+
+            string indexPath = Path.Combine(rootPath, "Index");
+            Console.WriteLine($"Scanning Index folder: {indexPath}");
+
+            var result = Scan(rootPath, verbose: true);
+
+            if (result.Outcome == ScanOutcome.IndexFolderMissing)
+            {
+                Console.WriteLine($"Index folder not found at: {result.IndexPath}");
+                return 1;
+            }
+
+            Console.WriteLine($"\n========== UpdateBlogPaths summary ==========");
+            Console.WriteLine($"Metadata files:           {result.MetadataFiles}");
+            Console.WriteLine($"TTFolderPath written:     {result.Written}");
+            Console.WriteLine($"Already correct:          {result.Unchanged}");
+            Console.WriteLine($"No FileDownloadLocation:  {result.NoLocation}");
+            Console.WriteLine($"No matching blog row:     {result.NoMatchingRow}");
+            Console.WriteLine($"Errors:                   {result.Errors}");
+
+            Console.WriteLine($"\nBlogs now holding a TTFolderPath: {DataAccess.CountBlogsWithTTFolderPath()}");
+
+            return result.Errors == 0 ? 0 : 2;
         }
     }
 }

verify-db-schema.sql

@@ -142,6 +142,9 @@ ORDER BY et.tbl;
 -- 1c. EXTRA / UNEXPECTED COLUMNS: present in the DB but not in the expected
 --     list above. Informational only -- e.g. a NEWER backup, or a column this
 --     script's expected-list hasn't been updated for. Not an error by itself.
+--     Posts.IsActive and Notes.IsActive are listed here and NOT in 1a on
+--     purpose: they are written by other tools, the app only reads them when
+--     present, and it must not be told to add them. See TL.db.md.
 WITH expected(tbl, col) AS (
   VALUES
     ('Posts','BlogName'),('Posts','PostID'),('Posts','HasNotesGathered'),('Posts','reblogURL'),
@@ -150,14 +153,14 @@ WITH expected(tbl, col) AS (
     ('Posts','Quote'),('Posts','Body'),('Posts','Tags'),('Posts','Link'),('Posts','PhotoURL'),
     ('Posts','PhotoCaption'),('Posts','DownloadedFiles'),('Posts','AudioCaption'),('Posts','Question'),
     ('Posts','Answer'),('Posts','Title'),('Posts','ByLikes'),('Posts','RootBlogName'),('Posts','RootURL'),
-    ('Posts','DateModified'),('Posts','DateCreated'),('Posts','PostType'),
+    ('Posts','DateModified'),('Posts','DateCreated'),('Posts','PostType'),('Posts','IsActive'),
     ('Blogs','BlogName'),('Blogs','HasBeenOutput'),('Blogs','IsActive'),('Blogs','DateAdded'),
     ('Blogs','ByLikes'),('Blogs','DateModified'),('Blogs','DateCreated'),('Blogs','LikesPulled'),
     ('Blogs','LikesCursor'),('Blogs','LikesNewestTimestamp'),('Blogs','LikesLastRefreshed'),
     ('Blogs','LikesLastNewCount'),('Blogs','TTFolderPath'),
     ('Notes','RootBlogName'),('Notes','PostID'),('Notes','NoteBlogName'),('Notes','TimeStamp'),
     ('Notes','Type'),('Notes','DatetimeCrawled'),('Notes','DateModified'),('Notes','DateCreated'),
-    ('Notes','replyText'),
+    ('Notes','replyText'),('Notes','IsActive'),
     ('DailyAPICount','Date'),('DailyAPICount','APICount'),
     ('ApiKeyPoolState','KeyName'),('ApiKeyPoolState','RetryUntil'),
     ('ApiKeyPoolMeta','Id'),('ApiKeyPoolMeta','LastIndex')