Both columns carry the meaning Blogs.IsActive has: 0 = removed by another
tool, anything else (including NULL) = live. Neither exists in the live
TL.db yet, and both are added from outside this crawler, so the code has
to work on databases either side of the change - naming a missing column
is a hard SQLite error.
HasIsActiveColumn asks PRAGMA table_info once per table per database path
and caches it; AndIsActive/WhereIsActive return "COALESCE(IsActive, 1) = 1"
or an empty string. Every read that selects posts or notes now carries the
filter: GetPosts (both branches, including the per-blog count subquery),
GetReplies, GetRepliesWithMissingText, GetRepliesWithFilledText,
GetAllPostTextColumns, GetAllPostsForBlog, GetPost, GetPostByIdAnyBlog,
and the engagement queries that count or join Notes - GetBlogs,
GetBlogsAll and both note-joining variants of GetBlogsForLikes.
The LEFT JOIN Notes in GetPosts is left alone on purpose: nothing is
selected from it and it can neither add nor remove a row.
LegacyPostsDbImporter is left alone too - it reads a foreign legacy
schema.
Writes were already safe and are documented rather than changed: no
INSERT column list names IsActive, no UPDATE sets it, MapPrefixToColumn
cannot map to it, and there is no INSERT OR REPLACE on Posts or Notes for
a column default to be reset by. Re-crawling a removed row refreshes its
content and leaves the flag at 0. As with Blogs, exclusion belongs at
selection, so the update paths stay keyed on rows the caller already
chose.
Verified against three synthetic databases - no IsActive columns, columns
present with a removed post and its notes, and columns present but NULL -
by running every affected reader: the queries are valid in all three, the
removed rows drop out only where the columns exist, NULL reads as live,
and AddPost/AddNote/UpsertPostFromTextFile/UpdatePostContentFields leave
an IsActive = 0 row at 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
Replaces the Blogs.IsDeleted section. Rolodex adds no column of its own;
it reuses the crawler's existing IsActive flag, so removing a blog in the
UI also stops it being collected.
Co-Authored-By: Claude Opus 5 <[email protected]>
TL.db.md documents the live schema: the three content tables and their
row counts, the '.' placeholder convention the crawler writes instead of
NULL, the two incompatible date formats in Blogs.DateAdded, and the
access paths that matter on the 1.19M-row Notes table.
It also covers Blogs.IsDeleted, which Rolodex adds by ALTER TABLE and
this crawler must not write.
The file was sitting untracked next to the database it describes.
Co-Authored-By: Claude Opus 5 <[email protected]>
Blogs.IsActive was honored only by GetBlogs and GetBlogsAll, so a blog
with IsActive = 0 was still selected for likes crawling and for output
mode. Add the filter to every remaining query that selects blog records:
- GetBlogsForLikes, all three variants (specific blog, ignoreCooldown,
cooldown) - this is the selector that spends API quota
- GetAllBlogsWithTTFolderPath
Writes are deliberately untouched. The UPDATE statements are keyed on a
blog the caller already selected; filtering them would let the crawler
fetch a blog, pay the API cost, then fail to persist its cursor and
re-fetch the same pages on every run. Exclusion belongs at selection.
LegacyPostsDbImporter is also untouched: it reads a foreign legacy
schema that may not have the column.
Co-Authored-By: Claude Opus 5 <[email protected]>
Strip only a trailing numeric suffix from blog folder names, fixing
zomb-eh_10 importing as blog 'zomb-eh0'.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
NormalizeBlogFolderName removed "_1".."_9" as unanchored substrings, so a
folder suffixed past a single digit lost the wrong characters: "_10" hit
the "_1" rule and left the trailing "0" welded to the name, importing
zomb-eh_10 as blog "zomb-eh0". That name does not exist on Tumblr, so
every post imported under it 404s on --collect forever.
Anchor the strip to a trailing _<digits> instead. This also fixes blogs
whose real name contains "_1" (some_1blog no longer becomes someblog) and
folders suffixed "_0", which were not stripped at all.
Verified against the live folder tree: zomb-eh_10 is the only existing
folder whose normalized name changes.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Retry transient CDN failures instead of failing the post; throttle
--collect and --likes to 60/min.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A non-JSON response body (CDN 403/5xx HTML, empty body, transport error)
never reached the Tumblr API, so it says nothing about the post being
fetched. These were recorded as FAILURE, which consumed the post's single
attempt for the pass and cleared the API key's rate-limit flag on the way
through.
Classify them as Root.transientFailure and retry in place (1s/4s/10s)
before skipping. Skipped posts stay unmarked in the DB so a later launch
retries them. Ten consecutive transient failures now aborts the pass
rather than skipping post-by-post against an edge refusing all traffic.
Also:
- MarkAvailable() only on a response that reached the API, and it is now
a no-op when the key was not flagged (was writing to the DB and logging
on every single call)
- Only a real 429 counts as a rate limit; stop inferring one from
X-RateLimit-* headers, which Tumblr sends on every response
- Limiters pace with AcquireAsync instead of AttemptAcquire, which did
not wait and aborted the run once a window was saturated
- Throttle --collect and --likes from 300/min to 60/min
- Log one line per transient failure instead of the HTML body and stack
trace; keep full detail only for a 2xx that fails to parse
- --collect returns exit 3 when a pass ends incomplete
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Note TraverseDirectory's recognized field prefixes, multi-line
Body/Downloaded files continuation, and how RootURL now gets
populated from both the .txt Reblog root url line and the API-based
--likes flow.
TraverseDirectory only ever read the single line immediately after
"Body:"/"Downloaded files:", silently dropping every continuation
line (multi-paragraph HTML bodies, multiple downloaded filenames).
Switch to an indexed line scan so those two fields collect lines
until the next recognized field prefix, matching how IngestMode.cs
already handles multi-line values.
TraverseDirectory (the no-args ingest path) never read the "Reblog
root url:" line, so RootURL stayed unset even though AddPost/UpdatePost
already support it via the API-based --likes flow. New scraper output
now includes this field; wire it through both AddPost call sites.
IsAllRateLimited() short-circuited true for any pool with 0 or 1
keys, with minRetrySeconds left at 0 regardless of whether that key
was actually rate-limited. SleepUntilAnyAvailable() checks
"minRetry <= 0" to decide whether to skip sleeping, so with exactly
one key it always skipped the wait and let callers hammer the API
again immediately after a 429, even mid-cooldown.
The per-key loop already computes this correctly for any key count;
the special case only needs to cover the true no-keys edge case,
where there's nothing to wait on.
Replace the try/finally { connection.Close(); } pattern used across
most of DataAccess.cs with using declarations, so disposal happens
automatically and can't be skipped by a future edit that adds an
early return before the finally. Left the shared-connection
(ownsConnection) call sites alone since those intentionally outlive
a single method call.
Also drop a stray unused `using static ... JSType` import, and make
a missing ContainsList config setting fail with a clear
InvalidOperationException instead of a NullReferenceException from
Split(',') on null.
Posts.hasImage/DateModified fallback update built its WHERE clause via
raw string concatenation of blogName/postID, unlike every other query
in this method — a blog name containing a single quote would break or
inject into the query. Switch it to parameters.
--parse, --blogsO, and --bop indexed args[1..3] before checking
args.Length, so a missing argument threw IndexOutOfRangeException
instead of hitting the intended usage message.
Rename every multi-character option/command from single-dash to double-dash (--likes, --collect, --force, etc.) to follow the POSIX long-option convention. Single-character short options (-h, -V, -?) keep their single dash, as POSIX prescribes.
Breaking: existing invocations/scripts using single-dash forms now report Unknown Command and must be updated. Run profile (launchSettings.json) and CLI docs (copilot-instructions.md) updated to match.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
These two commands were handled by the switch but never listed in help.
--help now covers every command the program accepts.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The new 0/1/2 exit codes had no footprint in --help; add an Exit status
line so the documented behavior matches what the program now returns.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Keep the existing single-dash switch style and case-insensitive matching,
but add the cheap, non-breaking POSIX wins:
- `--` end-of-options: tokens after a bare `--` are treated as literal operands
- `--help`/`-h` (alongside `-?`) and `-V`/`--version`
- Main returns a real exit code: 2 for usage errors, propagates handler
return codes, and a top-level catch yields a quiet 1 on unhandled errors
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Removed outdated binary files and the `run_500_times.bat` script, which automated repetitive runs of `URLNotesGrabberCORE`. The batch script is no longer needed or has been replaced.
Added `verify-db-schema.sql`, a new script to verify and align the SQLite database schema (`TL.db`) with the application's expected schema. The script includes:
- A verification section to identify missing/extra columns or tables.
- An optional fix section with `ALTER TABLE` statements to add missing columns.
The SQL script ensures database compatibility while preserving data integrity and avoiding destructive operations.
Mode 0 (full re-check) previously reset its cutoff to now on every
launch, so an interrupted run restarted from scratch, and a post that
kept returning a non-Success/non-NotFound status could loop forever.
- Add single-row CollectRunState table + accessors (EnsureCollectRunStateTableExists,
GetCollectRunState, BeginCollectRun, CompleteCollectRun) mirroring the
ApiKeyPoolMeta pattern, to persist a frozen run cutoff and completion flag.
- -collect 0 with no explicit date is now a managed run: resume against the
stored cutoff if a run is in progress, else start a new run; mark complete
when the pass finishes so the next launch starts fresh. Explicit-date and
mode 1 behavior unchanged.
- CollectNotes makes a single attempt pass via an in-process attempted set;
FAILURE/UNKNOWN are logged once, TooManyRequests/no-lease aborts without
completing so a later launch resumes.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
-revert was DB-driven, searching each blog's Blogs.TTFolderPath
non-recursively for *.bak. That tree differs from the no-parameter run,
which recursively walks PathInput. Rewrite RevertMode to recursively walk
PathInput (filesystem-only, no DB), with the optional [blogname] argument
now filtering by path substring. Restore mechanics unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Inverse of -output. For each blog with a TTFolderPath, restores every
*.bak over its *.txt, first preserving the current *.txt as the
next-free *.bkN, then consuming the *.bak. Confirms before running and
supports an optional single-blog filter.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The every-50-file progress line wasn't enough to know which blog was
currently being processed on a long run. Now -ingest prints a
"entering folder: <name>" line whenever the source folder changes, and
the every-50 progress line also prefixes the folder name.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
`-ingest <blogname>` now restricts the run to one blog's folder, mirroring
the existing -parse <blogname> ergonomics. `-ingest` with no arg still
processes every blog under appSettings:PathTTRoot (or PathInput fallback).
Breaking vs 3aff849: the first positional arg is interpreted as a blog
name, not a path. Configure the root via appSettings:PathTTRoot.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Folds the standalone ThreeTxtFileHelper tool into URLNotesGrabberCORE so
text-file ingest/output/correct lives alongside the API scraper. Adds
new flags -ingest, -output, -correct (with -apply), -updatepaths, and a
one-time -importposts <posts.db> migration.
Schema: Blogs.TTFolderPath and Posts.PostType are added by an idempotent
migration. On (BlogName, PostID) collisions, content columns are
overwritten while engagement columns (ByLikes, RootBlogName, RootURL,
HasNotesGathered, NotFound, NotesGatheredDateTime, Likes*) are preserved.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
After backfill completes for a blog, -likes can now pick up only newer
likes instead of being a one-shot pull. Tracks a per-blog
liked_timestamp high-water mark and stops the refresh walk once it
crosses the stored mark. A configurable cooldown (LikesRefreshCooldownDays,
default 7) gates which blogs are re-checked on each run. -force bypasses
the cooldown.
The migration adds three columns to Blogs (LikesNewestTimestamp,
LikesLastRefreshed, LikesLastNewCount) and does a one-time reset of all
likes tracking state so the new high-water mark starts from a clean
baseline. Existing ByLikes posts remain; the UNIQUE constraint absorbs
re-inserts during the first re-backfill.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
AddBlog/AddPost/UpdatePost/UpdatePostSetDate now reuse a single SQLiteConnection
when an import session is active, instead of opening and closing one per call.
AddBlog also short-circuits on an in-memory HashSet of blog names already
attempted this run. Other entry points are unaffected since they never call
BeginImportSession.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Per-iteration call in CollectNotes was dumping ~30 lines of newline-padded
SQL to the console. Replace with a whitespace-collapsed single-line print.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
- GetPosts: match NotesGatheredDateTime = 0 instead of IS NULL for the
beforeDate cutoff.
- GetBlogsForLikes: drop the EXISTS-Posts predicate so blogs with notes
but no posts rows are still eligible for likes collection.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Pre-flight check on the CollectNotes loop now sleeps until at least one
key recovers instead of issuing a wasted 429-bound request per iteration.
Extracts the countdown into ApiKeyPool.SleepUntilAnyAvailable (30s refresh)
and reuses it in CollectLikes and GrabNotes.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Changed CollectNotes from async void to async Task and await it with
.GetAwaiter().GetResult() to match the pattern used by -replies and -likes.
Previously the method would return immediately after the first await,
causing Main to exit before the loop could process more than one post.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Tumblr API 404s were leaving Posts.NotFound=0 and Notes.replyText='.',
so GetRepliesWithFilledText kept re-selecting the same dead post and
the loop spun forever. Mark NotFound=1 and replies '?' on 404.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Apply the URL format to the remaining 6 console log lines that still
referenced posts in <blog>/<id> form (404 logs, max-page-limit log,
DumpReplies console output).
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Console-friendly URL format for log lines that reference a specific
post — easier to copy/paste into a browser when investigating output.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
jim
