Both columns carry the meaning Blogs.IsActive has: 0 = removed by another
tool, anything else (including NULL) = live. Neither exists in the live
TL.db yet, and both are added from outside this crawler, so the code has
to work on databases either side of the change - naming a missing column
is a hard SQLite error.
HasIsActiveColumn asks PRAGMA table_info once per table per database path
and caches it; AndIsActive/WhereIsActive return "COALESCE(IsActive, 1) = 1"
or an empty string. Every read that selects posts or notes now carries the
filter: GetPosts (both branches, including the per-blog count subquery),
GetReplies, GetRepliesWithMissingText, GetRepliesWithFilledText,
GetAllPostTextColumns, GetAllPostsForBlog, GetPost, GetPostByIdAnyBlog,
and the engagement queries that count or join Notes - GetBlogs,
GetBlogsAll and both note-joining variants of GetBlogsForLikes.
The LEFT JOIN Notes in GetPosts is left alone on purpose: nothing is
selected from it and it can neither add nor remove a row.
LegacyPostsDbImporter is left alone too - it reads a foreign legacy
schema.
Writes were already safe and are documented rather than changed: no
INSERT column list names IsActive, no UPDATE sets it, MapPrefixToColumn
cannot map to it, and there is no INSERT OR REPLACE on Posts or Notes for
a column default to be reset by. Re-crawling a removed row refreshes its
content and leaves the flag at 0. As with Blogs, exclusion belongs at
selection, so the update paths stay keyed on rows the caller already
chose.
Verified against three synthetic databases - no IsActive columns, columns
present with a removed post and its notes, and columns present but NULL -
by running every affected reader: the queries are valid in all three, the
removed rows drop out only where the columns exist, NULL reads as live,
and AddPost/AddNote/UpsertPostFromTextFile/UpdatePostContentFields leave
an IsActive = 0 row at 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
Replaces the Blogs.IsDeleted section. Rolodex adds no column of its own;
it reuses the crawler's existing IsActive flag, so removing a blog in the
UI also stops it being collected.
Co-Authored-By: Claude Opus 5 <[email protected]>
TL.db.md documents the live schema: the three content tables and their
row counts, the '.' placeholder convention the crawler writes instead of
NULL, the two incompatible date formats in Blogs.DateAdded, and the
access paths that matter on the 1.19M-row Notes table.
It also covers Blogs.IsDeleted, which Rolodex adds by ALTER TABLE and
this crawler must not write.
The file was sitting untracked next to the database it describes.
Co-Authored-By: Claude Opus 5 <[email protected]>
Blogs.IsActive was honored only by GetBlogs and GetBlogsAll, so a blog
with IsActive = 0 was still selected for likes crawling and for output
mode. Add the filter to every remaining query that selects blog records:
- GetBlogsForLikes, all three variants (specific blog, ignoreCooldown,
cooldown) - this is the selector that spends API quota
- GetAllBlogsWithTTFolderPath
Writes are deliberately untouched. The UPDATE statements are keyed on a
blog the caller already selected; filtering them would let the crawler
fetch a blog, pay the API cost, then fail to persist its cursor and
re-fetch the same pages on every run. Exclusion belongs at selection.
LegacyPostsDbImporter is also untouched: it reads a foreign legacy
schema that may not have the column.
Co-Authored-By: Claude Opus 5 <[email protected]>
NormalizeBlogFolderName removed "_1".."_9" as unanchored substrings, so a
folder suffixed past a single digit lost the wrong characters: "_10" hit
the "_1" rule and left the trailing "0" welded to the name, importing
zomb-eh_10 as blog "zomb-eh0". That name does not exist on Tumblr, so
every post imported under it 404s on --collect forever.
Anchor the strip to a trailing _<digits> instead. This also fixes blogs
whose real name contains "_1" (some_1blog no longer becomes someblog) and
folders suffixed "_0", which were not stripped at all.
Verified against the live folder tree: zomb-eh_10 is the only existing
folder whose normalized name changes.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A non-JSON response body (CDN 403/5xx HTML, empty body, transport error)
never reached the Tumblr API, so it says nothing about the post being
fetched. These were recorded as FAILURE, which consumed the post's single
attempt for the pass and cleared the API key's rate-limit flag on the way
through.
Classify them as Root.transientFailure and retry in place (1s/4s/10s)
before skipping. Skipped posts stay unmarked in the DB so a later launch
retries them. Ten consecutive transient failures now aborts the pass
rather than skipping post-by-post against an edge refusing all traffic.
Also:
- MarkAvailable() only on a response that reached the API, and it is now
a no-op when the key was not flagged (was writing to the DB and logging
on every single call)
- Only a real 429 counts as a rate limit; stop inferring one from
X-RateLimit-* headers, which Tumblr sends on every response
- Limiters pace with AcquireAsync instead of AttemptAcquire, which did
not wait and aborted the run once a window was saturated
- Throttle --collect and --likes from 300/min to 60/min
- Log one line per transient failure instead of the HTML body and stack
trace; keep full detail only for a 2xx that fails to parse
- --collect returns exit 3 when a pass ends incomplete
Co-Authored-By: Claude Opus 4.8 <[email protected]>
TraverseDirectory only ever read the single line immediately after
"Body:"/"Downloaded files:", silently dropping every continuation
line (multi-paragraph HTML bodies, multiple downloaded filenames).
Switch to an indexed line scan so those two fields collect lines
until the next recognized field prefix, matching how IngestMode.cs
already handles multi-line values.
TraverseDirectory (the no-args ingest path) never read the "Reblog
root url:" line, so RootURL stayed unset even though AddPost/UpdatePost
already support it via the API-based --likes flow. New scraper output
now includes this field; wire it through both AddPost call sites.
IsAllRateLimited() short-circuited true for any pool with 0 or 1
keys, with minRetrySeconds left at 0 regardless of whether that key
was actually rate-limited. SleepUntilAnyAvailable() checks
"minRetry <= 0" to decide whether to skip sleeping, so with exactly
one key it always skipped the wait and let callers hammer the API
again immediately after a 429, even mid-cooldown.
The per-key loop already computes this correctly for any key count;
the special case only needs to cover the true no-keys edge case,
where there's nothing to wait on.
Replace the try/finally { connection.Close(); } pattern used across
most of DataAccess.cs with using declarations, so disposal happens
automatically and can't be skipped by a future edit that adds an
early return before the finally. Left the shared-connection
(ownsConnection) call sites alone since those intentionally outlive
a single method call.
Also drop a stray unused `using static ... JSType` import, and make
a missing ContainsList config setting fail with a clear
InvalidOperationException instead of a NullReferenceException from
Split(',') on null.
Posts.hasImage/DateModified fallback update built its WHERE clause via
raw string concatenation of blogName/postID, unlike every other query
in this method — a blog name containing a single quote would break or
inject into the query. Switch it to parameters.
--parse, --blogsO, and --bop indexed args[1..3] before checking
args.Length, so a missing argument threw IndexOutOfRangeException
instead of hitting the intended usage message.
Rename every multi-character option/command from single-dash to double-dash (--likes, --collect, --force, etc.) to follow the POSIX long-option convention. Single-character short options (-h, -V, -?) keep their single dash, as POSIX prescribes.
Breaking: existing invocations/scripts using single-dash forms now report Unknown Command and must be updated. Run profile (launchSettings.json) and CLI docs (copilot-instructions.md) updated to match.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
These two commands were handled by the switch but never listed in help.
--help now covers every command the program accepts.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The new 0/1/2 exit codes had no footprint in --help; add an Exit status
line so the documented behavior matches what the program now returns.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Keep the existing single-dash switch style and case-insensitive matching,
but add the cheap, non-breaking POSIX wins:
- `--` end-of-options: tokens after a bare `--` are treated as literal operands
- `--help`/`-h` (alongside `-?`) and `-V`/`--version`
- Main returns a real exit code: 2 for usage errors, propagates handler
return codes, and a top-level catch yields a quiet 1 on unhandled errors
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Mode 0 (full re-check) previously reset its cutoff to now on every
launch, so an interrupted run restarted from scratch, and a post that
kept returning a non-Success/non-NotFound status could loop forever.
- Add single-row CollectRunState table + accessors (EnsureCollectRunStateTableExists,
GetCollectRunState, BeginCollectRun, CompleteCollectRun) mirroring the
ApiKeyPoolMeta pattern, to persist a frozen run cutoff and completion flag.
- -collect 0 with no explicit date is now a managed run: resume against the
stored cutoff if a run is in progress, else start a new run; mark complete
when the pass finishes so the next launch starts fresh. Explicit-date and
mode 1 behavior unchanged.
- CollectNotes makes a single attempt pass via an in-process attempted set;
FAILURE/UNKNOWN are logged once, TooManyRequests/no-lease aborts without
completing so a later launch resumes.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
-revert was DB-driven, searching each blog's Blogs.TTFolderPath
non-recursively for *.bak. That tree differs from the no-parameter run,
which recursively walks PathInput. Rewrite RevertMode to recursively walk
PathInput (filesystem-only, no DB), with the optional [blogname] argument
now filtering by path substring. Restore mechanics unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Inverse of -output. For each blog with a TTFolderPath, restores every
*.bak over its *.txt, first preserving the current *.txt as the
next-free *.bkN, then consuming the *.bak. Confirms before running and
supports an optional single-blog filter.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The every-50-file progress line wasn't enough to know which blog was
currently being processed on a long run. Now -ingest prints a
"entering folder: <name>" line whenever the source folder changes, and
the every-50 progress line also prefixes the folder name.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
`-ingest <blogname>` now restricts the run to one blog's folder, mirroring
the existing -parse <blogname> ergonomics. `-ingest` with no arg still
processes every blog under appSettings:PathTTRoot (or PathInput fallback).
Breaking vs 3aff849: the first positional arg is interpreted as a blog
name, not a path. Configure the root via appSettings:PathTTRoot.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Folds the standalone ThreeTxtFileHelper tool into URLNotesGrabberCORE so
text-file ingest/output/correct lives alongside the API scraper. Adds
new flags -ingest, -output, -correct (with -apply), -updatepaths, and a
one-time -importposts <posts.db> migration.
Schema: Blogs.TTFolderPath and Posts.PostType are added by an idempotent
migration. On (BlogName, PostID) collisions, content columns are
overwritten while engagement columns (ByLikes, RootBlogName, RootURL,
HasNotesGathered, NotFound, NotesGatheredDateTime, Likes*) are preserved.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
After backfill completes for a blog, -likes can now pick up only newer
likes instead of being a one-shot pull. Tracks a per-blog
liked_timestamp high-water mark and stops the refresh walk once it
crosses the stored mark. A configurable cooldown (LikesRefreshCooldownDays,
default 7) gates which blogs are re-checked on each run. -force bypasses
the cooldown.
The migration adds three columns to Blogs (LikesNewestTimestamp,
LikesLastRefreshed, LikesLastNewCount) and does a one-time reset of all
likes tracking state so the new high-water mark starts from a clean
baseline. Existing ByLikes posts remain; the UNIQUE constraint absorbs
re-inserts during the first re-backfill.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
AddBlog/AddPost/UpdatePost/UpdatePostSetDate now reuse a single SQLiteConnection
when an import session is active, instead of opening and closing one per call.
AddBlog also short-circuits on an in-memory HashSet of blog names already
attempted this run. Other entry points are unaffected since they never call
BeginImportSession.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Per-iteration call in CollectNotes was dumping ~30 lines of newline-padded
SQL to the console. Replace with a whitespace-collapsed single-line print.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
- GetPosts: match NotesGatheredDateTime = 0 instead of IS NULL for the
beforeDate cutoff.
- GetBlogsForLikes: drop the EXISTS-Posts predicate so blogs with notes
but no posts rows are still eligible for likes collection.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Pre-flight check on the CollectNotes loop now sleeps until at least one
key recovers instead of issuing a wasted 429-bound request per iteration.
Extracts the countdown into ApiKeyPool.SleepUntilAnyAvailable (30s refresh)
and reuses it in CollectLikes and GrabNotes.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Changed CollectNotes from async void to async Task and await it with
.GetAwaiter().GetResult() to match the pattern used by -replies and -likes.
Previously the method would return immediately after the first await,
causing Main to exit before the loop could process more than one post.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Tumblr API 404s were leaving Posts.NotFound=0 and Notes.replyText='.',
so GetRepliesWithFilledText kept re-selecting the same dead post and
the loop spun forever. Mark NotFound=1 and replies '?' on 404.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Apply the URL format to the remaining 6 console log lines that still
referenced posts in <blog>/<id> form (404 logs, max-page-limit log,
DumpReplies console output).
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Console-friendly URL format for log lines that reference a specific
post — easier to copy/paste into a browser when investigating output.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Previous fix removed '.' from the GetRepliesWithFilledText SELECT,
which broke processing of legacy null-substitute rows that need to be
re-fetched.
Restored '.' in the SELECT. To avoid the infinite loop when an actual
API reply is the literal string ".", normalize it to ". " (dot +
trailing space) inside UpdateNoteReplyText so the stored value no
longer matches the sentinel.
Also extended the fan-out UPDATE guard to match the SELECT criteria
(NULL / '' / '.') so legacy '.' rows in other reblog copies can be
filled in too. The guard still refuses to overwrite '?' or
already-fetched text.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
'.' was used as a sentinel for 'not yet fetched' in the SELECT query,
but it is also valid reply text. This caused any post where a reply
text was literally '.' to stay in the work queue forever.
Also fixes the fan-out UPDATE guard: previously it used
IFNULL(replyText, '.') <> @newValue, which would overwrite '?'
(confirmed-empty) rows with '.' when processing a dot reply elsewhere
in the reblog chain, pulling completed posts back into the queue and
causing the remaining counter to increase.
Changes:
- Remove OR replyText = '.' from GetRepliesWithFilledText SELECT
- Restrict UpdateNoteReplyText fan-out to NULL/'' rows only
- Use '?' not '.' as null-coalesce fallback in UpdateNoteReplyText
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Previously, the '?' marker only fired when page 1 returned empty notes.
Posts whose page 1 contained only likes/reblogs and whose page 2 came
back empty were never marked, so GetRepliesWithFilledText kept reselecting
them every iteration. Now mark after the pagination loop whenever
rowsUpdated==0 and at least one page returned 200 OK.
UpdateAllNoteReplyTextForPost now returns int so the caller can fold
the count into emptyReplyCount/rowsUpdated for accurate Done logging.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
When the API responds 200 OK with notes:[] on the first page,
the post has no conversational notes per Tumblr. Stamp all the
post's reply rows '?' so the per-iteration requery stops handing
this post back forever. Only fires for first-page 200 OK so a
later-page empty (end of pagination) doesn't poison good rows.
A reply by a given blog at a given timestamp is the same reply
across the original post and every reblog of it. Drop PostID from
the UpdateNoteReplyText WHERE clause so a single API hit fills in
the replyText on every matching row at once.
Pair that with a per-iteration requery (limit 1) of the work list
so posts whose replies were already filled in as a side-effect of
a previous post's update are skipped without burning an API call.
The API returns reply timestamps that are ~1s ahead of what -collect
originally stored, so an exact TimeStamp match in the UPDATE was
hitting zero rows for every note - the API call worked, the reply
text came back, but nothing landed in the DB. Match within +-5s
instead. Also return rowsAffected from UpdateNoteReplyText and
report it separately from notes-seen in the summary so misses are
visible.
The DB-derived MAX(N.timestamp)+1 for a post is unreliable - many notes
were stored with the same timestamp (likely crawl time, not the actual
note timestamp), so passing it as before_timestamp excluded all real
replies and returned an empty notes array. Diagnostics showed two
unrelated posts coming back with identical totals (1625/1021/603) and
zero notes.
Now we start with no before_timestamp (newest page) and let pagination
walk backward via each page's last-note timestamp.
Wrap FetchAndStoreReplyText in a per-post pagination loop (up to 10
pages, advancing before_timestamp via the last note's timestamp) so
posts with >50 notes are fully walked. Log raw response (meta + first
500 chars of JSON) when a page returns no notes so empty results are
diagnosable. Stop blanket-marking every reply on a post with '?' on
the first empty response - rows stay '.' and remain retry-eligible;
only individual replies that come back with empty reply_text are
marked '?'.
Call ApiKeyPool.MarkAvailable(key) after successful reply fetch in FetchAndStoreReplyText, matching behavior in CollectLikes. This produces the colored '[Pool] Key#N (...) rate-limit cleared' line for each API call, allowing operators to visually track which API key is being used per request.
Co-Authored-By: Claude Haiku 4.5 <[email protected]>
jimandClaude Opus 5