IsAllRateLimited() short-circuited true for any pool with 0 or 1
keys, with minRetrySeconds left at 0 regardless of whether that key
was actually rate-limited. SleepUntilAnyAvailable() checks
"minRetry <= 0" to decide whether to skip sleeping, so with exactly
one key it always skipped the wait and let callers hammer the API
again immediately after a 429, even mid-cooldown.
The per-key loop already computes this correctly for any key count;
the special case only needs to cover the true no-keys edge case,
where there's nothing to wait on.
Replace the try/finally { connection.Close(); } pattern used across
most of DataAccess.cs with using declarations, so disposal happens
automatically and can't be skipped by a future edit that adds an
early return before the finally. Left the shared-connection
(ownsConnection) call sites alone since those intentionally outlive
a single method call.
Also drop a stray unused `using static ... JSType` import, and make
a missing ContainsList config setting fail with a clear
InvalidOperationException instead of a NullReferenceException from
Split(',') on null.
Posts.hasImage/DateModified fallback update built its WHERE clause via
raw string concatenation of blogName/postID, unlike every other query
in this method — a blog name containing a single quote would break or
inject into the query. Switch it to parameters.
--parse, --blogsO, and --bop indexed args[1..3] before checking
args.Length, so a missing argument threw IndexOutOfRangeException
instead of hitting the intended usage message.
Rename every multi-character option/command from single-dash to double-dash (--likes, --collect, --force, etc.) to follow the POSIX long-option convention. Single-character short options (-h, -V, -?) keep their single dash, as POSIX prescribes.
Breaking: existing invocations/scripts using single-dash forms now report Unknown Command and must be updated. Run profile (launchSettings.json) and CLI docs (copilot-instructions.md) updated to match.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
These two commands were handled by the switch but never listed in help.
--help now covers every command the program accepts.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The new 0/1/2 exit codes had no footprint in --help; add an Exit status
line so the documented behavior matches what the program now returns.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Keep the existing single-dash switch style and case-insensitive matching,
but add the cheap, non-breaking POSIX wins:
- `--` end-of-options: tokens after a bare `--` are treated as literal operands
- `--help`/`-h` (alongside `-?`) and `-V`/`--version`
- Main returns a real exit code: 2 for usage errors, propagates handler
return codes, and a top-level catch yields a quiet 1 on unhandled errors
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Mode 0 (full re-check) previously reset its cutoff to now on every
launch, so an interrupted run restarted from scratch, and a post that
kept returning a non-Success/non-NotFound status could loop forever.
- Add single-row CollectRunState table + accessors (EnsureCollectRunStateTableExists,
GetCollectRunState, BeginCollectRun, CompleteCollectRun) mirroring the
ApiKeyPoolMeta pattern, to persist a frozen run cutoff and completion flag.
- -collect 0 with no explicit date is now a managed run: resume against the
stored cutoff if a run is in progress, else start a new run; mark complete
when the pass finishes so the next launch starts fresh. Explicit-date and
mode 1 behavior unchanged.
- CollectNotes makes a single attempt pass via an in-process attempted set;
FAILURE/UNKNOWN are logged once, TooManyRequests/no-lease aborts without
completing so a later launch resumes.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
-revert was DB-driven, searching each blog's Blogs.TTFolderPath
non-recursively for *.bak. That tree differs from the no-parameter run,
which recursively walks PathInput. Rewrite RevertMode to recursively walk
PathInput (filesystem-only, no DB), with the optional [blogname] argument
now filtering by path substring. Restore mechanics unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Inverse of -output. For each blog with a TTFolderPath, restores every
*.bak over its *.txt, first preserving the current *.txt as the
next-free *.bkN, then consuming the *.bak. Confirms before running and
supports an optional single-blog filter.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The every-50-file progress line wasn't enough to know which blog was
currently being processed on a long run. Now -ingest prints a
"entering folder: <name>" line whenever the source folder changes, and
the every-50 progress line also prefixes the folder name.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
`-ingest <blogname>` now restricts the run to one blog's folder, mirroring
the existing -parse <blogname> ergonomics. `-ingest` with no arg still
processes every blog under appSettings:PathTTRoot (or PathInput fallback).
Breaking vs 3aff849: the first positional arg is interpreted as a blog
name, not a path. Configure the root via appSettings:PathTTRoot.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Folds the standalone ThreeTxtFileHelper tool into URLNotesGrabberCORE so
text-file ingest/output/correct lives alongside the API scraper. Adds
new flags -ingest, -output, -correct (with -apply), -updatepaths, and a
one-time -importposts <posts.db> migration.
Schema: Blogs.TTFolderPath and Posts.PostType are added by an idempotent
migration. On (BlogName, PostID) collisions, content columns are
overwritten while engagement columns (ByLikes, RootBlogName, RootURL,
HasNotesGathered, NotFound, NotesGatheredDateTime, Likes*) are preserved.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
After backfill completes for a blog, -likes can now pick up only newer
likes instead of being a one-shot pull. Tracks a per-blog
liked_timestamp high-water mark and stops the refresh walk once it
crosses the stored mark. A configurable cooldown (LikesRefreshCooldownDays,
default 7) gates which blogs are re-checked on each run. -force bypasses
the cooldown.
The migration adds three columns to Blogs (LikesNewestTimestamp,
LikesLastRefreshed, LikesLastNewCount) and does a one-time reset of all
likes tracking state so the new high-water mark starts from a clean
baseline. Existing ByLikes posts remain; the UNIQUE constraint absorbs
re-inserts during the first re-backfill.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
AddBlog/AddPost/UpdatePost/UpdatePostSetDate now reuse a single SQLiteConnection
when an import session is active, instead of opening and closing one per call.
AddBlog also short-circuits on an in-memory HashSet of blog names already
attempted this run. Other entry points are unaffected since they never call
BeginImportSession.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Per-iteration call in CollectNotes was dumping ~30 lines of newline-padded
SQL to the console. Replace with a whitespace-collapsed single-line print.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
- GetPosts: match NotesGatheredDateTime = 0 instead of IS NULL for the
beforeDate cutoff.
- GetBlogsForLikes: drop the EXISTS-Posts predicate so blogs with notes
but no posts rows are still eligible for likes collection.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Pre-flight check on the CollectNotes loop now sleeps until at least one
key recovers instead of issuing a wasted 429-bound request per iteration.
Extracts the countdown into ApiKeyPool.SleepUntilAnyAvailable (30s refresh)
and reuses it in CollectLikes and GrabNotes.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Changed CollectNotes from async void to async Task and await it with
.GetAwaiter().GetResult() to match the pattern used by -replies and -likes.
Previously the method would return immediately after the first await,
causing Main to exit before the loop could process more than one post.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Tumblr API 404s were leaving Posts.NotFound=0 and Notes.replyText='.',
so GetRepliesWithFilledText kept re-selecting the same dead post and
the loop spun forever. Mark NotFound=1 and replies '?' on 404.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Apply the URL format to the remaining 6 console log lines that still
referenced posts in <blog>/<id> form (404 logs, max-page-limit log,
DumpReplies console output).
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Console-friendly URL format for log lines that reference a specific
post — easier to copy/paste into a browser when investigating output.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Previous fix removed '.' from the GetRepliesWithFilledText SELECT,
which broke processing of legacy null-substitute rows that need to be
re-fetched.
Restored '.' in the SELECT. To avoid the infinite loop when an actual
API reply is the literal string ".", normalize it to ". " (dot +
trailing space) inside UpdateNoteReplyText so the stored value no
longer matches the sentinel.
Also extended the fan-out UPDATE guard to match the SELECT criteria
(NULL / '' / '.') so legacy '.' rows in other reblog copies can be
filled in too. The guard still refuses to overwrite '?' or
already-fetched text.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
'.' was used as a sentinel for 'not yet fetched' in the SELECT query,
but it is also valid reply text. This caused any post where a reply
text was literally '.' to stay in the work queue forever.
Also fixes the fan-out UPDATE guard: previously it used
IFNULL(replyText, '.') <> @newValue, which would overwrite '?'
(confirmed-empty) rows with '.' when processing a dot reply elsewhere
in the reblog chain, pulling completed posts back into the queue and
causing the remaining counter to increase.
Changes:
- Remove OR replyText = '.' from GetRepliesWithFilledText SELECT
- Restrict UpdateNoteReplyText fan-out to NULL/'' rows only
- Use '?' not '.' as null-coalesce fallback in UpdateNoteReplyText
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Previously, the '?' marker only fired when page 1 returned empty notes.
Posts whose page 1 contained only likes/reblogs and whose page 2 came
back empty were never marked, so GetRepliesWithFilledText kept reselecting
them every iteration. Now mark after the pagination loop whenever
rowsUpdated==0 and at least one page returned 200 OK.
UpdateAllNoteReplyTextForPost now returns int so the caller can fold
the count into emptyReplyCount/rowsUpdated for accurate Done logging.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
When the API responds 200 OK with notes:[] on the first page,
the post has no conversational notes per Tumblr. Stamp all the
post's reply rows '?' so the per-iteration requery stops handing
this post back forever. Only fires for first-page 200 OK so a
later-page empty (end of pagination) doesn't poison good rows.
A reply by a given blog at a given timestamp is the same reply
across the original post and every reblog of it. Drop PostID from
the UpdateNoteReplyText WHERE clause so a single API hit fills in
the replyText on every matching row at once.
Pair that with a per-iteration requery (limit 1) of the work list
so posts whose replies were already filled in as a side-effect of
a previous post's update are skipped without burning an API call.
The API returns reply timestamps that are ~1s ahead of what -collect
originally stored, so an exact TimeStamp match in the UPDATE was
hitting zero rows for every note - the API call worked, the reply
text came back, but nothing landed in the DB. Match within +-5s
instead. Also return rowsAffected from UpdateNoteReplyText and
report it separately from notes-seen in the summary so misses are
visible.
The DB-derived MAX(N.timestamp)+1 for a post is unreliable - many notes
were stored with the same timestamp (likely crawl time, not the actual
note timestamp), so passing it as before_timestamp excluded all real
replies and returned an empty notes array. Diagnostics showed two
unrelated posts coming back with identical totals (1625/1021/603) and
zero notes.
Now we start with no before_timestamp (newest page) and let pagination
walk backward via each page's last-note timestamp.
Wrap FetchAndStoreReplyText in a per-post pagination loop (up to 10
pages, advancing before_timestamp via the last note's timestamp) so
posts with >50 notes are fully walked. Log raw response (meta + first
500 chars of JSON) when a page returns no notes so empty results are
diagnosable. Stop blanket-marking every reply on a post with '?' on
the first empty response - rows stay '.' and remain retry-eligible;
only individual replies that come back with empty reply_text are
marked '?'.
Call ApiKeyPool.MarkAvailable(key) after successful reply fetch in FetchAndStoreReplyText, matching behavior in CollectLikes. This produces the colored '[Pool] Key#N (...) rate-limit cleared' line for each API call, allowing operators to visually track which API key is being used per request.
Co-Authored-By: Claude Haiku 4.5 <[email protected]>
✅ Captures replies and reblogs with comment
✅ Ignores rollup_notes field (as requested)
✅ Maintains rate limiting and error handling
✅ Console output shows both reply and reblog comment
appsettings.json — Added PoolEnabled to each API section:
- TumblrApi → true
- TumblrApi3 → false
- TumblrApi4 → true
DataAccess.cs — Added:
- ApiKeyConfig class — holds credentials + metadata per key
- ApiKeyPool class — manages pool with round-robin rotation, SQLite-backed state (ApiKeyPoolState, ApiKeyPoolMeta tables)
- GetCurrentKey() — returns next key, skipping rate-limited ones, falls back to earliest-recovery if all are throttled
- MarkRateLimited(key, retryUntil) / MarkAvailable(key) — persists state
- Initialize() — discovers pool-enabled keys, detects single-key override mode
- Refactored APIAccess.GrabNotes(), GrabPostWithReplies(), GrabLikes() to accept ApiKeyConfig param and log [Key#N]
Program.cs — Updated:
- Tracks apiExplicitlySet flag from -api/-api3/-api4
- Initializes ApiKeyPool at startup (pool mode or single-key override)
- All 3 API callers updated to use pool rotation + 429 handling
Startup Output
- Pool mode: [Pool] Active keys: Key#1=TumblrApi, Key#2=TumblrApi4
- Single-key: [Pool] Single-key mode: TumblrApi3
jim
