Note TraverseDirectory's recognized field prefixes, multi-line
Body/Downloaded files continuation, and how RootURL now gets
populated from both the .txt Reblog root url line and the API-based
--likes flow.
TraverseDirectory only ever read the single line immediately after
"Body:"/"Downloaded files:", silently dropping every continuation
line (multi-paragraph HTML bodies, multiple downloaded filenames).
Switch to an indexed line scan so those two fields collect lines
until the next recognized field prefix, matching how IngestMode.cs
already handles multi-line values.
TraverseDirectory (the no-args ingest path) never read the "Reblog
root url:" line, so RootURL stayed unset even though AddPost/UpdatePost
already support it via the API-based --likes flow. New scraper output
now includes this field; wire it through both AddPost call sites.
IsAllRateLimited() short-circuited true for any pool with 0 or 1
keys, with minRetrySeconds left at 0 regardless of whether that key
was actually rate-limited. SleepUntilAnyAvailable() checks
"minRetry <= 0" to decide whether to skip sleeping, so with exactly
one key it always skipped the wait and let callers hammer the API
again immediately after a 429, even mid-cooldown.
The per-key loop already computes this correctly for any key count;
the special case only needs to cover the true no-keys edge case,
where there's nothing to wait on.
Replace the try/finally { connection.Close(); } pattern used across
most of DataAccess.cs with using declarations, so disposal happens
automatically and can't be skipped by a future edit that adds an
early return before the finally. Left the shared-connection
(ownsConnection) call sites alone since those intentionally outlive
a single method call.
Also drop a stray unused `using static ... JSType` import, and make
a missing ContainsList config setting fail with a clear
InvalidOperationException instead of a NullReferenceException from
Split(',') on null.
Posts.hasImage/DateModified fallback update built its WHERE clause via
raw string concatenation of blogName/postID, unlike every other query
in this method — a blog name containing a single quote would break or
inject into the query. Switch it to parameters.
--parse, --blogsO, and --bop indexed args[1..3] before checking
args.Length, so a missing argument threw IndexOutOfRangeException
instead of hitting the intended usage message.
Rename every multi-character option/command from single-dash to double-dash (--likes, --collect, --force, etc.) to follow the POSIX long-option convention. Single-character short options (-h, -V, -?) keep their single dash, as POSIX prescribes.
Breaking: existing invocations/scripts using single-dash forms now report Unknown Command and must be updated. Run profile (launchSettings.json) and CLI docs (copilot-instructions.md) updated to match.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
These two commands were handled by the switch but never listed in help.
--help now covers every command the program accepts.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The new 0/1/2 exit codes had no footprint in --help; add an Exit status
line so the documented behavior matches what the program now returns.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Keep the existing single-dash switch style and case-insensitive matching,
but add the cheap, non-breaking POSIX wins:
- `--` end-of-options: tokens after a bare `--` are treated as literal operands
- `--help`/`-h` (alongside `-?`) and `-V`/`--version`
- Main returns a real exit code: 2 for usage errors, propagates handler
return codes, and a top-level catch yields a quiet 1 on unhandled errors
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Removed outdated binary files and the `run_500_times.bat` script, which automated repetitive runs of `URLNotesGrabberCORE`. The batch script is no longer needed or has been replaced.
Added `verify-db-schema.sql`, a new script to verify and align the SQLite database schema (`TL.db`) with the application's expected schema. The script includes:
- A verification section to identify missing/extra columns or tables.
- An optional fix section with `ALTER TABLE` statements to add missing columns.
The SQL script ensures database compatibility while preserving data integrity and avoiding destructive operations.
Mode 0 (full re-check) previously reset its cutoff to now on every
launch, so an interrupted run restarted from scratch, and a post that
kept returning a non-Success/non-NotFound status could loop forever.
- Add single-row CollectRunState table + accessors (EnsureCollectRunStateTableExists,
GetCollectRunState, BeginCollectRun, CompleteCollectRun) mirroring the
ApiKeyPoolMeta pattern, to persist a frozen run cutoff and completion flag.
- -collect 0 with no explicit date is now a managed run: resume against the
stored cutoff if a run is in progress, else start a new run; mark complete
when the pass finishes so the next launch starts fresh. Explicit-date and
mode 1 behavior unchanged.
- CollectNotes makes a single attempt pass via an in-process attempted set;
FAILURE/UNKNOWN are logged once, TooManyRequests/no-lease aborts without
completing so a later launch resumes.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
-revert was DB-driven, searching each blog's Blogs.TTFolderPath
non-recursively for *.bak. That tree differs from the no-parameter run,
which recursively walks PathInput. Rewrite RevertMode to recursively walk
PathInput (filesystem-only, no DB), with the optional [blogname] argument
now filtering by path substring. Restore mechanics unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Inverse of -output. For each blog with a TTFolderPath, restores every
*.bak over its *.txt, first preserving the current *.txt as the
next-free *.bkN, then consuming the *.bak. Confirms before running and
supports an optional single-blog filter.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The every-50-file progress line wasn't enough to know which blog was
currently being processed on a long run. Now -ingest prints a
"entering folder: <name>" line whenever the source folder changes, and
the every-50 progress line also prefixes the folder name.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
`-ingest <blogname>` now restricts the run to one blog's folder, mirroring
the existing -parse <blogname> ergonomics. `-ingest` with no arg still
processes every blog under appSettings:PathTTRoot (or PathInput fallback).
Breaking vs 3aff849: the first positional arg is interpreted as a blog
name, not a path. Configure the root via appSettings:PathTTRoot.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Folds the standalone ThreeTxtFileHelper tool into URLNotesGrabberCORE so
text-file ingest/output/correct lives alongside the API scraper. Adds
new flags -ingest, -output, -correct (with -apply), -updatepaths, and a
one-time -importposts <posts.db> migration.
Schema: Blogs.TTFolderPath and Posts.PostType are added by an idempotent
migration. On (BlogName, PostID) collisions, content columns are
overwritten while engagement columns (ByLikes, RootBlogName, RootURL,
HasNotesGathered, NotFound, NotesGatheredDateTime, Likes*) are preserved.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
After backfill completes for a blog, -likes can now pick up only newer
likes instead of being a one-shot pull. Tracks a per-blog
liked_timestamp high-water mark and stops the refresh walk once it
crosses the stored mark. A configurable cooldown (LikesRefreshCooldownDays,
default 7) gates which blogs are re-checked on each run. -force bypasses
the cooldown.
The migration adds three columns to Blogs (LikesNewestTimestamp,
LikesLastRefreshed, LikesLastNewCount) and does a one-time reset of all
likes tracking state so the new high-water mark starts from a clean
baseline. Existing ByLikes posts remain; the UNIQUE constraint absorbs
re-inserts during the first re-backfill.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
AddBlog/AddPost/UpdatePost/UpdatePostSetDate now reuse a single SQLiteConnection
when an import session is active, instead of opening and closing one per call.
AddBlog also short-circuits on an in-memory HashSet of blog names already
attempted this run. Other entry points are unaffected since they never call
BeginImportSession.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Per-iteration call in CollectNotes was dumping ~30 lines of newline-padded
SQL to the console. Replace with a whitespace-collapsed single-line print.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
- GetPosts: match NotesGatheredDateTime = 0 instead of IS NULL for the
beforeDate cutoff.
- GetBlogsForLikes: drop the EXISTS-Posts predicate so blogs with notes
but no posts rows are still eligible for likes collection.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Pre-flight check on the CollectNotes loop now sleeps until at least one
key recovers instead of issuing a wasted 429-bound request per iteration.
Extracts the countdown into ApiKeyPool.SleepUntilAnyAvailable (30s refresh)
and reuses it in CollectLikes and GrabNotes.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Changed CollectNotes from async void to async Task and await it with
.GetAwaiter().GetResult() to match the pattern used by -replies and -likes.
Previously the method would return immediately after the first await,
causing Main to exit before the loop could process more than one post.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Tumblr API 404s were leaving Posts.NotFound=0 and Notes.replyText='.',
so GetRepliesWithFilledText kept re-selecting the same dead post and
the loop spun forever. Mark NotFound=1 and replies '?' on 404.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Apply the URL format to the remaining 6 console log lines that still
referenced posts in <blog>/<id> form (404 logs, max-page-limit log,
DumpReplies console output).
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Console-friendly URL format for log lines that reference a specific
post — easier to copy/paste into a browser when investigating output.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Previous fix removed '.' from the GetRepliesWithFilledText SELECT,
which broke processing of legacy null-substitute rows that need to be
re-fetched.
Restored '.' in the SELECT. To avoid the infinite loop when an actual
API reply is the literal string ".", normalize it to ". " (dot +
trailing space) inside UpdateNoteReplyText so the stored value no
longer matches the sentinel.
Also extended the fan-out UPDATE guard to match the SELECT criteria
(NULL / '' / '.') so legacy '.' rows in other reblog copies can be
filled in too. The guard still refuses to overwrite '?' or
already-fetched text.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
'.' was used as a sentinel for 'not yet fetched' in the SELECT query,
but it is also valid reply text. This caused any post where a reply
text was literally '.' to stay in the work queue forever.
Also fixes the fan-out UPDATE guard: previously it used
IFNULL(replyText, '.') <> @newValue, which would overwrite '?'
(confirmed-empty) rows with '.' when processing a dot reply elsewhere
in the reblog chain, pulling completed posts back into the queue and
causing the remaining counter to increase.
Changes:
- Remove OR replyText = '.' from GetRepliesWithFilledText SELECT
- Restrict UpdateNoteReplyText fan-out to NULL/'' rows only
- Use '?' not '.' as null-coalesce fallback in UpdateNoteReplyText
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Previously, the '?' marker only fired when page 1 returned empty notes.
Posts whose page 1 contained only likes/reblogs and whose page 2 came
back empty were never marked, so GetRepliesWithFilledText kept reselecting
them every iteration. Now mark after the pagination loop whenever
rowsUpdated==0 and at least one page returned 200 OK.
UpdateAllNoteReplyTextForPost now returns int so the caller can fold
the count into emptyReplyCount/rowsUpdated for accurate Done logging.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
When the API responds 200 OK with notes:[] on the first page,
the post has no conversational notes per Tumblr. Stamp all the
post's reply rows '?' so the per-iteration requery stops handing
this post back forever. Only fires for first-page 200 OK so a
later-page empty (end of pagination) doesn't poison good rows.
A reply by a given blog at a given timestamp is the same reply
across the original post and every reblog of it. Drop PostID from
the UpdateNoteReplyText WHERE clause so a single API hit fills in
the replyText on every matching row at once.
Pair that with a per-iteration requery (limit 1) of the work list
so posts whose replies were already filled in as a side-effect of
a previous post's update are skipped without burning an API call.
The API returns reply timestamps that are ~1s ahead of what -collect
originally stored, so an exact TimeStamp match in the UPDATE was
hitting zero rows for every note - the API call worked, the reply
text came back, but nothing landed in the DB. Match within +-5s
instead. Also return rowsAffected from UpdateNoteReplyText and
report it separately from notes-seen in the summary so misses are
visible.
jim
