TL.db syncs between machines whose absolute paths differ, so a single
TTFolderPath column cannot be correct on both at once -- the stored paths are
only trustworthy on the machine that wrote them. That made --updatepaths a
mandatory prelude to every --output rather than the one-time setup step it
looks like.
--output now refreshes the column from the TumblThree Index metadata before
exporting. The root comes from the first non-flag argument, else
appSettings:PathTTRoot. With no root available it says so and exports whatever
TL.db already holds; a root whose Index folder is missing is a hard stop, since
silently exporting stale paths is the failure this change exists to prevent.
--norefresh skips the refresh for a pure export.
The scan is extracted from UpdateBlogPathsRunner.Run into a reusable Scan() that
returns counts instead of only printing them, so --updatepaths keeps its
per-file detail while --output prints a single summary line rather than a few
hundred lines ahead of the export.
Verified against a throwaway database: a stale cross-machine path is repaired
and the export lands in the correct local folder; no configured root warns and
continues (exit 0); a missing Index folder stops (exit 1); --norefresh skips the
refresh and exports (exit 0).
Co-Authored-By: Claude Opus 5 <[email protected]>
--output iterated all 156k active Blogs rows and printed a "does not exist or is
not set" skip line for each, which is nearly every blog in the crawl registry --
only the few hundred downloaded locally ever have a folder. The signal was
buried in six figures of noise.
GetAllBlogsWithTTFolderPath now selects only active blogs carrying a non-empty
path, so --output processes export targets and nothing else. When none exist it
says so once, names the database it read, points at --updatepaths, and returns
non-zero instead of reporting success. A stored path this machine cannot see is
now reported separately from an unset one, with the path shown, because the two
are fixed in different places. Paths are trimmed before Directory.Exists, which
stray whitespace in a .tumblr FileDownloadLocation would otherwise defeat.
Both writers counted optimistically. UpdateBlogPathsRunner printed its
per-blog success line and incremented its total from the metadata file parsing,
never checking whether the UPDATE matched a row; LegacyPostsDbImporter counted a
blog as copied even when the legacy TTFolderPath was NULL. Either could report
full success having written nothing -- which is consistent with TL.db holding
zero populated paths across all 156,492 active blogs despite 20,679 posts having
merged. SetBlogTTFolderPath now returns whether a row changed, and both callers
report written / already-correct / no-matching-row separately.
Verified against a throwaway database: no-paths case, export case (stale .txt
rotated to .bak, per-PostType files, date-sorted), missing-folder case,
--updatepaths honest counts, and an idempotent rerun reporting already-correct.
Co-Authored-By: Claude Opus 5 <[email protected]>
UpsertPostFromTextFile (the persistence layer under --ingest) uses NULL as
its "this file's record had no line for that field" sentinel -- the direct
analog of the "." convention just fixed in UpdatePost. IngestMode strips a
trailing "_N" off the folder name before it ever reaches this function, so
a duplicate export folder deliberately collapses onto the same BlogName --
reconciling multiple differently-formatted files for one post is the whole
point of --ingest. Files are walked in raw filesystem enumeration order,
never sorted, so which file's call lands last for a given (BlogName,
PostID) is arbitrary.
The UPDATE branch set every column unconditionally, so whichever file
processed last for a PostID nulled out every field its own record didn't
carry, silently erasing real content another file had. Worse than the "."
case: that one only caused churn (two writes cancelling out); this one
loses data, in an order that depends on filesystem enumeration.
Every content column is now guarded the same way, NULL instead of "." as
the sentinel: `col = CASE WHEN @col IS NULL THEN col ELSE @col END` in the
SET list, `(@col IS NOT NULL AND IFNULL(col,'') <> @col) OR ...` in the
change-detection. Narrow the same way: only a missing line (NULL) is the
sentinel -- G() already distinguishes that from present-but-blank (""), so
an explicit empty field still overwrites.
HasImage is deliberately left unguarded and documented as a known gap:
IngestMode always computes a concrete bool, defaulting false when a file
has no "Has Image:" line, so this function can't currently tell "no image"
from "not reported" without changing the parameter to bool? and threading
that through IngestMode/LegacyPostsDbImporter too.
Verified against a throwaway DB using the exact SQL text and parameter
binding: a full-format record's real Title/Slug/Tags now survive a
same-PostID partial record whose format doesn't carry those fields, in
both file orders, while a genuine content change and an explicit empty
value still write and still move DateModified.
Co-Authored-By: Claude Opus 5 <[email protected]>
ReblogRecord (TraverseDirectory's .txt-export parser) and the --likes API
path both default every content field to the literal "." when their source
has no value for that field, then pass it straight into UpdatePost. A blog
with two export folders in different field formats -- a duplicate "_2"
folder, or an export whose field set changed over time -- sends one record
with real Title/Tags/Slug and another with those fields "." because that
format never had a line for them. Re-importing both on every run flipped
the row back and forth forever: net content never changed, but
DateModified moved on every pass since each write really did change a
column relative to the other write, just not relative to the true value.
Every content column in UpdatePost's SET list is now guarded the same way
RootBlogName/RootURL already were -- a "." parameter leaves the existing
value alone instead of overwriting it -- and the change-detection WHERE
clause carries the same exception, so a "."-only difference no longer
fires the UPDATE at all. Deliberately narrow: only the literal "." is the
sentinel, so an explicit empty string from a real record still overwrites.
Verified against a throwaway DB using the exact SQL text and parameter
binding from UpdatePost, reproducing the an-angry-wolf/adore-blk scenario
found in the live DB: re-importing conflicting "." records now writes zero
rows and leaves DateModified untouched, while a genuine content change
still fires and still moves it.
Co-Authored-By: Claude Opus 5 <[email protected]>
Seven UPDATE statements wrote DateModified unconditionally, so re-crawling
or re-ingesting identical content marked Blogs, Posts and Notes rows as
modified. Each now carries a WHERE guard covering every column in its SET
list, so SQLite matches zero rows on a no-op.
Guarded: AddPost's insert-failure fallback and blog stamp, AddNote's blog
stamp, UpdateBlogLikesNewestTimestamp, UpdateNoteReplyText,
UpsertPostFromTextFile, SetBlogTTFolderPath, UpdatePostContentFields.
Also:
- Blogs.DateAdded is no longer rewritten when a new post arrives for a
known blog. A new post is not a new blog, and rewriting the column both
destroyed the registration date and made every insert look like a change.
- Posts.NotesGatheredDateTime is crawl bookkeeping that moves on every
pass, so it no longer moves DateModified on its own. It is still written
each pass, but the timestamp is wrapped in a CASE on the pre-UPDATE
HasNotesGathered value so only the flag flipping counts.
These statements now return 0 rows for "found but unchanged" as well as
"not found"; CorrectMode's postsUpdated tally consequently counts rows
actually changed, matching what its dry-run diff reports.
Co-Authored-By: Claude Opus 5 <[email protected]>
The INSERT named a DateCreated column that DailyAPICount (Date, APICount)
never had, so every call threw "no such column: DateCreated" into an
empty catch block. Today's row was never created and the tally sat idle
since 2026-04-13. Drop the column from the INSERT, and report the three
silent failure points (insert error, missing row after insert, update
matching zero rows) instead of swallowing them.
Both columns carry the meaning Blogs.IsActive has: 0 = removed by another
tool, anything else (including NULL) = live. Neither exists in the live
TL.db yet, and both are added from outside this crawler, so the code has
to work on databases either side of the change - naming a missing column
is a hard SQLite error.
HasIsActiveColumn asks PRAGMA table_info once per table per database path
and caches it; AndIsActive/WhereIsActive return "COALESCE(IsActive, 1) = 1"
or an empty string. Every read that selects posts or notes now carries the
filter: GetPosts (both branches, including the per-blog count subquery),
GetReplies, GetRepliesWithMissingText, GetRepliesWithFilledText,
GetAllPostTextColumns, GetAllPostsForBlog, GetPost, GetPostByIdAnyBlog,
and the engagement queries that count or join Notes - GetBlogs,
GetBlogsAll and both note-joining variants of GetBlogsForLikes.
The LEFT JOIN Notes in GetPosts is left alone on purpose: nothing is
selected from it and it can neither add nor remove a row.
LegacyPostsDbImporter is left alone too - it reads a foreign legacy
schema.
Writes were already safe and are documented rather than changed: no
INSERT column list names IsActive, no UPDATE sets it, MapPrefixToColumn
cannot map to it, and there is no INSERT OR REPLACE on Posts or Notes for
a column default to be reset by. Re-crawling a removed row refreshes its
content and leaves the flag at 0. As with Blogs, exclusion belongs at
selection, so the update paths stay keyed on rows the caller already
chose.
Verified against three synthetic databases - no IsActive columns, columns
present with a removed post and its notes, and columns present but NULL -
by running every affected reader: the queries are valid in all three, the
removed rows drop out only where the columns exist, NULL reads as live,
and AddPost/AddNote/UpsertPostFromTextFile/UpdatePostContentFields leave
an IsActive = 0 row at 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
Replaces the Blogs.IsDeleted section. Rolodex adds no column of its own;
it reuses the crawler's existing IsActive flag, so removing a blog in the
UI also stops it being collected.
Co-Authored-By: Claude Opus 5 <[email protected]>
TL.db.md documents the live schema: the three content tables and their
row counts, the '.' placeholder convention the crawler writes instead of
NULL, the two incompatible date formats in Blogs.DateAdded, and the
access paths that matter on the 1.19M-row Notes table.
It also covers Blogs.IsDeleted, which Rolodex adds by ALTER TABLE and
this crawler must not write.
The file was sitting untracked next to the database it describes.
Co-Authored-By: Claude Opus 5 <[email protected]>
Blogs.IsActive was honored only by GetBlogs and GetBlogsAll, so a blog
with IsActive = 0 was still selected for likes crawling and for output
mode. Add the filter to every remaining query that selects blog records:
- GetBlogsForLikes, all three variants (specific blog, ignoreCooldown,
cooldown) - this is the selector that spends API quota
- GetAllBlogsWithTTFolderPath
Writes are deliberately untouched. The UPDATE statements are keyed on a
blog the caller already selected; filtering them would let the crawler
fetch a blog, pay the API cost, then fail to persist its cursor and
re-fetch the same pages on every run. Exclusion belongs at selection.
LegacyPostsDbImporter is also untouched: it reads a foreign legacy
schema that may not have the column.
Co-Authored-By: Claude Opus 5 <[email protected]>
NormalizeBlogFolderName removed "_1".."_9" as unanchored substrings, so a
folder suffixed past a single digit lost the wrong characters: "_10" hit
the "_1" rule and left the trailing "0" welded to the name, importing
zomb-eh_10 as blog "zomb-eh0". That name does not exist on Tumblr, so
every post imported under it 404s on --collect forever.
Anchor the strip to a trailing _<digits> instead. This also fixes blogs
whose real name contains "_1" (some_1blog no longer becomes someblog) and
folders suffixed "_0", which were not stripped at all.
Verified against the live folder tree: zomb-eh_10 is the only existing
folder whose normalized name changes.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A non-JSON response body (CDN 403/5xx HTML, empty body, transport error)
never reached the Tumblr API, so it says nothing about the post being
fetched. These were recorded as FAILURE, which consumed the post's single
attempt for the pass and cleared the API key's rate-limit flag on the way
through.
Classify them as Root.transientFailure and retry in place (1s/4s/10s)
before skipping. Skipped posts stay unmarked in the DB so a later launch
retries them. Ten consecutive transient failures now aborts the pass
rather than skipping post-by-post against an edge refusing all traffic.
Also:
- MarkAvailable() only on a response that reached the API, and it is now
a no-op when the key was not flagged (was writing to the DB and logging
on every single call)
- Only a real 429 counts as a rate limit; stop inferring one from
X-RateLimit-* headers, which Tumblr sends on every response
- Limiters pace with AcquireAsync instead of AttemptAcquire, which did
not wait and aborted the run once a window was saturated
- Throttle --collect and --likes from 300/min to 60/min
- Log one line per transient failure instead of the HTML body and stack
trace; keep full detail only for a 2xx that fails to parse
- --collect returns exit 3 when a pass ends incomplete
Co-Authored-By: Claude Opus 4.8 <[email protected]>
TraverseDirectory only ever read the single line immediately after
"Body:"/"Downloaded files:", silently dropping every continuation
line (multi-paragraph HTML bodies, multiple downloaded filenames).
Switch to an indexed line scan so those two fields collect lines
until the next recognized field prefix, matching how IngestMode.cs
already handles multi-line values.
TraverseDirectory (the no-args ingest path) never read the "Reblog
root url:" line, so RootURL stayed unset even though AddPost/UpdatePost
already support it via the API-based --likes flow. New scraper output
now includes this field; wire it through both AddPost call sites.
IsAllRateLimited() short-circuited true for any pool with 0 or 1
keys, with minRetrySeconds left at 0 regardless of whether that key
was actually rate-limited. SleepUntilAnyAvailable() checks
"minRetry <= 0" to decide whether to skip sleeping, so with exactly
one key it always skipped the wait and let callers hammer the API
again immediately after a 429, even mid-cooldown.
The per-key loop already computes this correctly for any key count;
the special case only needs to cover the true no-keys edge case,
where there's nothing to wait on.
Replace the try/finally { connection.Close(); } pattern used across
most of DataAccess.cs with using declarations, so disposal happens
automatically and can't be skipped by a future edit that adds an
early return before the finally. Left the shared-connection
(ownsConnection) call sites alone since those intentionally outlive
a single method call.
Also drop a stray unused `using static ... JSType` import, and make
a missing ContainsList config setting fail with a clear
InvalidOperationException instead of a NullReferenceException from
Split(',') on null.
Posts.hasImage/DateModified fallback update built its WHERE clause via
raw string concatenation of blogName/postID, unlike every other query
in this method — a blog name containing a single quote would break or
inject into the query. Switch it to parameters.
--parse, --blogsO, and --bop indexed args[1..3] before checking
args.Length, so a missing argument threw IndexOutOfRangeException
instead of hitting the intended usage message.
Rename every multi-character option/command from single-dash to double-dash (--likes, --collect, --force, etc.) to follow the POSIX long-option convention. Single-character short options (-h, -V, -?) keep their single dash, as POSIX prescribes.
Breaking: existing invocations/scripts using single-dash forms now report Unknown Command and must be updated. Run profile (launchSettings.json) and CLI docs (copilot-instructions.md) updated to match.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
These two commands were handled by the switch but never listed in help.
--help now covers every command the program accepts.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The new 0/1/2 exit codes had no footprint in --help; add an Exit status
line so the documented behavior matches what the program now returns.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Keep the existing single-dash switch style and case-insensitive matching,
but add the cheap, non-breaking POSIX wins:
- `--` end-of-options: tokens after a bare `--` are treated as literal operands
- `--help`/`-h` (alongside `-?`) and `-V`/`--version`
- Main returns a real exit code: 2 for usage errors, propagates handler
return codes, and a top-level catch yields a quiet 1 on unhandled errors
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Mode 0 (full re-check) previously reset its cutoff to now on every
launch, so an interrupted run restarted from scratch, and a post that
kept returning a non-Success/non-NotFound status could loop forever.
- Add single-row CollectRunState table + accessors (EnsureCollectRunStateTableExists,
GetCollectRunState, BeginCollectRun, CompleteCollectRun) mirroring the
ApiKeyPoolMeta pattern, to persist a frozen run cutoff and completion flag.
- -collect 0 with no explicit date is now a managed run: resume against the
stored cutoff if a run is in progress, else start a new run; mark complete
when the pass finishes so the next launch starts fresh. Explicit-date and
mode 1 behavior unchanged.
- CollectNotes makes a single attempt pass via an in-process attempted set;
FAILURE/UNKNOWN are logged once, TooManyRequests/no-lease aborts without
completing so a later launch resumes.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
-revert was DB-driven, searching each blog's Blogs.TTFolderPath
non-recursively for *.bak. That tree differs from the no-parameter run,
which recursively walks PathInput. Rewrite RevertMode to recursively walk
PathInput (filesystem-only, no DB), with the optional [blogname] argument
now filtering by path substring. Restore mechanics unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Inverse of -output. For each blog with a TTFolderPath, restores every
*.bak over its *.txt, first preserving the current *.txt as the
next-free *.bkN, then consuming the *.bak. Confirms before running and
supports an optional single-blog filter.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The every-50-file progress line wasn't enough to know which blog was
currently being processed on a long run. Now -ingest prints a
"entering folder: <name>" line whenever the source folder changes, and
the every-50 progress line also prefixes the folder name.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
`-ingest <blogname>` now restricts the run to one blog's folder, mirroring
the existing -parse <blogname> ergonomics. `-ingest` with no arg still
processes every blog under appSettings:PathTTRoot (or PathInput fallback).
Breaking vs 3aff849: the first positional arg is interpreted as a blog
name, not a path. Configure the root via appSettings:PathTTRoot.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Folds the standalone ThreeTxtFileHelper tool into URLNotesGrabberCORE so
text-file ingest/output/correct lives alongside the API scraper. Adds
new flags -ingest, -output, -correct (with -apply), -updatepaths, and a
one-time -importposts <posts.db> migration.
Schema: Blogs.TTFolderPath and Posts.PostType are added by an idempotent
migration. On (BlogName, PostID) collisions, content columns are
overwritten while engagement columns (ByLikes, RootBlogName, RootURL,
HasNotesGathered, NotFound, NotesGatheredDateTime, Likes*) are preserved.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
After backfill completes for a blog, -likes can now pick up only newer
likes instead of being a one-shot pull. Tracks a per-blog
liked_timestamp high-water mark and stops the refresh walk once it
crosses the stored mark. A configurable cooldown (LikesRefreshCooldownDays,
default 7) gates which blogs are re-checked on each run. -force bypasses
the cooldown.
The migration adds three columns to Blogs (LikesNewestTimestamp,
LikesLastRefreshed, LikesLastNewCount) and does a one-time reset of all
likes tracking state so the new high-water mark starts from a clean
baseline. Existing ByLikes posts remain; the UNIQUE constraint absorbs
re-inserts during the first re-backfill.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
AddBlog/AddPost/UpdatePost/UpdatePostSetDate now reuse a single SQLiteConnection
when an import session is active, instead of opening and closing one per call.
AddBlog also short-circuits on an in-memory HashSet of blog names already
attempted this run. Other entry points are unaffected since they never call
BeginImportSession.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Per-iteration call in CollectNotes was dumping ~30 lines of newline-padded
SQL to the console. Replace with a whitespace-collapsed single-line print.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
- GetPosts: match NotesGatheredDateTime = 0 instead of IS NULL for the
beforeDate cutoff.
- GetBlogsForLikes: drop the EXISTS-Posts predicate so blogs with notes
but no posts rows are still eligible for likes collection.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Pre-flight check on the CollectNotes loop now sleeps until at least one
key recovers instead of issuing a wasted 429-bound request per iteration.
Extracts the countdown into ApiKeyPool.SleepUntilAnyAvailable (30s refresh)
and reuses it in CollectLikes and GrabNotes.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Changed CollectNotes from async void to async Task and await it with
.GetAwaiter().GetResult() to match the pattern used by -replies and -likes.
Previously the method would return immediately after the first await,
causing Main to exit before the loop could process more than one post.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Tumblr API 404s were leaving Posts.NotFound=0 and Notes.replyText='.',
so GetRepliesWithFilledText kept re-selecting the same dead post and
the loop spun forever. Mark NotFound=1 and replies '?' on 404.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Apply the URL format to the remaining 6 console log lines that still
referenced posts in <blog>/<id> form (404 logs, max-page-limit log,
DumpReplies console output).
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Console-friendly URL format for log lines that reference a specific
post — easier to copy/paste into a browser when investigating output.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Previous fix removed '.' from the GetRepliesWithFilledText SELECT,
which broke processing of legacy null-substitute rows that need to be
re-fetched.
Restored '.' in the SELECT. To avoid the infinite loop when an actual
API reply is the literal string ".", normalize it to ". " (dot +
trailing space) inside UpdateNoteReplyText so the stored value no
longer matches the sentinel.
Also extended the fan-out UPDATE guard to match the SELECT criteria
(NULL / '' / '.') so legacy '.' rows in other reblog copies can be
filled in too. The guard still refuses to overwrite '?' or
already-fetched text.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
jimandClaude Opus 5