Notes.RootBlogName/NoteBlogName/Type became RootBlogId/NoteBlogId/TypeId
on 2026-08-07, resolved through the new BlogNames and NoteTypes tables.
There is no compatibility view, so every affected statement is a hard cut.
All 14 call sites in DataAccess.cs are ported:
- Notes->Blogs joins go through Blogs.BlogId in one integer hop; the
Notes->Posts join in GetRepliesWithFilledText is the only one that must
route through BlogNames, since Posts carries no BlogId
- AddNote registers both blog names and the note type with INSERT OR
IGNORE before inserting, in one transaction committed before the console
sleep. Registering the type matters: an unseen type would resolve to
NULL and fail NOT NULL on TypeId, silently losing the note
- The LEFT JOIN Notes in GetPosts is dropped rather than translated. It
selected nothing, could not remove a row, and its duplicates were
collapsed by the query's own GROUP BY
- Duplicate-key detection moves to IsNotesDuplicateKey, matching the
constraint and table instead of an exact column list. The old literal
string is what broke on this rename
- EnsureReplyTextColumnExists drops DEFAULT '.', matching the migrated
schema: new rows get NULL, not a placeholder nobody wrote
verify-db-schema.sql gains BlogNames, NoteTypes, Blogs.BlogId and the new
Notes columns, plus query 1d naming a pre-migration file and pointing at
normalize-notes.sql. Blogs.BlogId is deliberately not auto-fixable -- an
added-but-empty column makes engagement joins return zero rows silently.
Verified against the live 148 MB file: query plans hit the intended
indexes, and the BlogId join matches an independent name-resolved
formulation exactly on all 4,267 GetBlogs and 2,637 GetBlogsForLikes rows.
RolodexRepository.cs (16 sites) lives in the Rolodex repo and is not
covered here.
Co-Authored-By: Claude Opus 5 <[email protected]>
--output iterated all 156k active Blogs rows and printed a "does not exist or is
not set" skip line for each, which is nearly every blog in the crawl registry --
only the few hundred downloaded locally ever have a folder. The signal was
buried in six figures of noise.
GetAllBlogsWithTTFolderPath now selects only active blogs carrying a non-empty
path, so --output processes export targets and nothing else. When none exist it
says so once, names the database it read, points at --updatepaths, and returns
non-zero instead of reporting success. A stored path this machine cannot see is
now reported separately from an unset one, with the path shown, because the two
are fixed in different places. Paths are trimmed before Directory.Exists, which
stray whitespace in a .tumblr FileDownloadLocation would otherwise defeat.
Both writers counted optimistically. UpdateBlogPathsRunner printed its
per-blog success line and incremented its total from the metadata file parsing,
never checking whether the UPDATE matched a row; LegacyPostsDbImporter counted a
blog as copied even when the legacy TTFolderPath was NULL. Either could report
full success having written nothing -- which is consistent with TL.db holding
zero populated paths across all 156,492 active blogs despite 20,679 posts having
merged. SetBlogTTFolderPath now returns whether a row changed, and both callers
report written / already-correct / no-matching-row separately.
Verified against a throwaway database: no-paths case, export case (stale .txt
rotated to .bak, per-PostType files, date-sorted), missing-folder case,
--updatepaths honest counts, and an idempotent rerun reporting already-correct.
Co-Authored-By: Claude Opus 5 <[email protected]>
UpsertPostFromTextFile (the persistence layer under --ingest) uses NULL as
its "this file's record had no line for that field" sentinel -- the direct
analog of the "." convention just fixed in UpdatePost. IngestMode strips a
trailing "_N" off the folder name before it ever reaches this function, so
a duplicate export folder deliberately collapses onto the same BlogName --
reconciling multiple differently-formatted files for one post is the whole
point of --ingest. Files are walked in raw filesystem enumeration order,
never sorted, so which file's call lands last for a given (BlogName,
PostID) is arbitrary.
The UPDATE branch set every column unconditionally, so whichever file
processed last for a PostID nulled out every field its own record didn't
carry, silently erasing real content another file had. Worse than the "."
case: that one only caused churn (two writes cancelling out); this one
loses data, in an order that depends on filesystem enumeration.
Every content column is now guarded the same way, NULL instead of "." as
the sentinel: `col = CASE WHEN @col IS NULL THEN col ELSE @col END` in the
SET list, `(@col IS NOT NULL AND IFNULL(col,'') <> @col) OR ...` in the
change-detection. Narrow the same way: only a missing line (NULL) is the
sentinel -- G() already distinguishes that from present-but-blank (""), so
an explicit empty field still overwrites.
HasImage is deliberately left unguarded and documented as a known gap:
IngestMode always computes a concrete bool, defaulting false when a file
has no "Has Image:" line, so this function can't currently tell "no image"
from "not reported" without changing the parameter to bool? and threading
that through IngestMode/LegacyPostsDbImporter too.
Verified against a throwaway DB using the exact SQL text and parameter
binding: a full-format record's real Title/Slug/Tags now survive a
same-PostID partial record whose format doesn't carry those fields, in
both file orders, while a genuine content change and an explicit empty
value still write and still move DateModified.
Co-Authored-By: Claude Opus 5 <[email protected]>
ReblogRecord (TraverseDirectory's .txt-export parser) and the --likes API
path both default every content field to the literal "." when their source
has no value for that field, then pass it straight into UpdatePost. A blog
with two export folders in different field formats -- a duplicate "_2"
folder, or an export whose field set changed over time -- sends one record
with real Title/Tags/Slug and another with those fields "." because that
format never had a line for them. Re-importing both on every run flipped
the row back and forth forever: net content never changed, but
DateModified moved on every pass since each write really did change a
column relative to the other write, just not relative to the true value.
Every content column in UpdatePost's SET list is now guarded the same way
RootBlogName/RootURL already were -- a "." parameter leaves the existing
value alone instead of overwriting it -- and the change-detection WHERE
clause carries the same exception, so a "."-only difference no longer
fires the UPDATE at all. Deliberately narrow: only the literal "." is the
sentinel, so an explicit empty string from a real record still overwrites.
Verified against a throwaway DB using the exact SQL text and parameter
binding from UpdatePost, reproducing the an-angry-wolf/adore-blk scenario
found in the live DB: re-importing conflicting "." records now writes zero
rows and leaves DateModified untouched, while a genuine content change
still fires and still moves it.
Co-Authored-By: Claude Opus 5 <[email protected]>
Seven UPDATE statements wrote DateModified unconditionally, so re-crawling
or re-ingesting identical content marked Blogs, Posts and Notes rows as
modified. Each now carries a WHERE guard covering every column in its SET
list, so SQLite matches zero rows on a no-op.
Guarded: AddPost's insert-failure fallback and blog stamp, AddNote's blog
stamp, UpdateBlogLikesNewestTimestamp, UpdateNoteReplyText,
UpsertPostFromTextFile, SetBlogTTFolderPath, UpdatePostContentFields.
Also:
- Blogs.DateAdded is no longer rewritten when a new post arrives for a
known blog. A new post is not a new blog, and rewriting the column both
destroyed the registration date and made every insert look like a change.
- Posts.NotesGatheredDateTime is crawl bookkeeping that moves on every
pass, so it no longer moves DateModified on its own. It is still written
each pass, but the timestamp is wrapped in a CASE on the pre-UPDATE
HasNotesGathered value so only the flag flipping counts.
These statements now return 0 rows for "found but unchanged" as well as
"not found"; CorrectMode's postsUpdated tally consequently counts rows
actually changed, matching what its dry-run diff reports.
Co-Authored-By: Claude Opus 5 <[email protected]>
The INSERT named a DateCreated column that DailyAPICount (Date, APICount)
never had, so every call threw "no such column: DateCreated" into an
empty catch block. Today's row was never created and the tally sat idle
since 2026-04-13. Drop the column from the INSERT, and report the three
silent failure points (insert error, missing row after insert, update
matching zero rows) instead of swallowing them.
Both columns carry the meaning Blogs.IsActive has: 0 = removed by another
tool, anything else (including NULL) = live. Neither exists in the live
TL.db yet, and both are added from outside this crawler, so the code has
to work on databases either side of the change - naming a missing column
is a hard SQLite error.
HasIsActiveColumn asks PRAGMA table_info once per table per database path
and caches it; AndIsActive/WhereIsActive return "COALESCE(IsActive, 1) = 1"
or an empty string. Every read that selects posts or notes now carries the
filter: GetPosts (both branches, including the per-blog count subquery),
GetReplies, GetRepliesWithMissingText, GetRepliesWithFilledText,
GetAllPostTextColumns, GetAllPostsForBlog, GetPost, GetPostByIdAnyBlog,
and the engagement queries that count or join Notes - GetBlogs,
GetBlogsAll and both note-joining variants of GetBlogsForLikes.
The LEFT JOIN Notes in GetPosts is left alone on purpose: nothing is
selected from it and it can neither add nor remove a row.
LegacyPostsDbImporter is left alone too - it reads a foreign legacy
schema.
Writes were already safe and are documented rather than changed: no
INSERT column list names IsActive, no UPDATE sets it, MapPrefixToColumn
cannot map to it, and there is no INSERT OR REPLACE on Posts or Notes for
a column default to be reset by. Re-crawling a removed row refreshes its
content and leaves the flag at 0. As with Blogs, exclusion belongs at
selection, so the update paths stay keyed on rows the caller already
chose.
Verified against three synthetic databases - no IsActive columns, columns
present with a removed post and its notes, and columns present but NULL -
by running every affected reader: the queries are valid in all three, the
removed rows drop out only where the columns exist, NULL reads as live,
and AddPost/AddNote/UpsertPostFromTextFile/UpdatePostContentFields leave
an IsActive = 0 row at 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
Blogs.IsActive was honored only by GetBlogs and GetBlogsAll, so a blog
with IsActive = 0 was still selected for likes crawling and for output
mode. Add the filter to every remaining query that selects blog records:
- GetBlogsForLikes, all three variants (specific blog, ignoreCooldown,
cooldown) - this is the selector that spends API quota
- GetAllBlogsWithTTFolderPath
Writes are deliberately untouched. The UPDATE statements are keyed on a
blog the caller already selected; filtering them would let the crawler
fetch a blog, pay the API cost, then fail to persist its cursor and
re-fetch the same pages on every run. Exclusion belongs at selection.
LegacyPostsDbImporter is also untouched: it reads a foreign legacy
schema that may not have the column.
Co-Authored-By: Claude Opus 5 <[email protected]>
A non-JSON response body (CDN 403/5xx HTML, empty body, transport error)
never reached the Tumblr API, so it says nothing about the post being
fetched. These were recorded as FAILURE, which consumed the post's single
attempt for the pass and cleared the API key's rate-limit flag on the way
through.
Classify them as Root.transientFailure and retry in place (1s/4s/10s)
before skipping. Skipped posts stay unmarked in the DB so a later launch
retries them. Ten consecutive transient failures now aborts the pass
rather than skipping post-by-post against an edge refusing all traffic.
Also:
- MarkAvailable() only on a response that reached the API, and it is now
a no-op when the key was not flagged (was writing to the DB and logging
on every single call)
- Only a real 429 counts as a rate limit; stop inferring one from
X-RateLimit-* headers, which Tumblr sends on every response
- Limiters pace with AcquireAsync instead of AttemptAcquire, which did
not wait and aborted the run once a window was saturated
- Throttle --collect and --likes from 300/min to 60/min
- Log one line per transient failure instead of the HTML body and stack
trace; keep full detail only for a 2xx that fails to parse
- --collect returns exit 3 when a pass ends incomplete
Co-Authored-By: Claude Opus 4.8 <[email protected]>
TraverseDirectory (the no-args ingest path) never read the "Reblog
root url:" line, so RootURL stayed unset even though AddPost/UpdatePost
already support it via the API-based --likes flow. New scraper output
now includes this field; wire it through both AddPost call sites.
IsAllRateLimited() short-circuited true for any pool with 0 or 1
keys, with minRetrySeconds left at 0 regardless of whether that key
was actually rate-limited. SleepUntilAnyAvailable() checks
"minRetry <= 0" to decide whether to skip sleeping, so with exactly
one key it always skipped the wait and let callers hammer the API
again immediately after a 429, even mid-cooldown.
The per-key loop already computes this correctly for any key count;
the special case only needs to cover the true no-keys edge case,
where there's nothing to wait on.
Replace the try/finally { connection.Close(); } pattern used across
most of DataAccess.cs with using declarations, so disposal happens
automatically and can't be skipped by a future edit that adds an
early return before the finally. Left the shared-connection
(ownsConnection) call sites alone since those intentionally outlive
a single method call.
Also drop a stray unused `using static ... JSType` import, and make
a missing ContainsList config setting fail with a clear
InvalidOperationException instead of a NullReferenceException from
Split(',') on null.
Posts.hasImage/DateModified fallback update built its WHERE clause via
raw string concatenation of blogName/postID, unlike every other query
in this method — a blog name containing a single quote would break or
inject into the query. Switch it to parameters.
--parse, --blogsO, and --bop indexed args[1..3] before checking
args.Length, so a missing argument threw IndexOutOfRangeException
instead of hitting the intended usage message.
Mode 0 (full re-check) previously reset its cutoff to now on every
launch, so an interrupted run restarted from scratch, and a post that
kept returning a non-Success/non-NotFound status could loop forever.
- Add single-row CollectRunState table + accessors (EnsureCollectRunStateTableExists,
GetCollectRunState, BeginCollectRun, CompleteCollectRun) mirroring the
ApiKeyPoolMeta pattern, to persist a frozen run cutoff and completion flag.
- -collect 0 with no explicit date is now a managed run: resume against the
stored cutoff if a run is in progress, else start a new run; mark complete
when the pass finishes so the next launch starts fresh. Explicit-date and
mode 1 behavior unchanged.
- CollectNotes makes a single attempt pass via an in-process attempted set;
FAILURE/UNKNOWN are logged once, TooManyRequests/no-lease aborts without
completing so a later launch resumes.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Folds the standalone ThreeTxtFileHelper tool into URLNotesGrabberCORE so
text-file ingest/output/correct lives alongside the API scraper. Adds
new flags -ingest, -output, -correct (with -apply), -updatepaths, and a
one-time -importposts <posts.db> migration.
Schema: Blogs.TTFolderPath and Posts.PostType are added by an idempotent
migration. On (BlogName, PostID) collisions, content columns are
overwritten while engagement columns (ByLikes, RootBlogName, RootURL,
HasNotesGathered, NotFound, NotesGatheredDateTime, Likes*) are preserved.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
After backfill completes for a blog, -likes can now pick up only newer
likes instead of being a one-shot pull. Tracks a per-blog
liked_timestamp high-water mark and stops the refresh walk once it
crosses the stored mark. A configurable cooldown (LikesRefreshCooldownDays,
default 7) gates which blogs are re-checked on each run. -force bypasses
the cooldown.
The migration adds three columns to Blogs (LikesNewestTimestamp,
LikesLastRefreshed, LikesLastNewCount) and does a one-time reset of all
likes tracking state so the new high-water mark starts from a clean
baseline. Existing ByLikes posts remain; the UNIQUE constraint absorbs
re-inserts during the first re-backfill.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
AddBlog/AddPost/UpdatePost/UpdatePostSetDate now reuse a single SQLiteConnection
when an import session is active, instead of opening and closing one per call.
AddBlog also short-circuits on an in-memory HashSet of blog names already
attempted this run. Other entry points are unaffected since they never call
BeginImportSession.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Per-iteration call in CollectNotes was dumping ~30 lines of newline-padded
SQL to the console. Replace with a whitespace-collapsed single-line print.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
- GetPosts: match NotesGatheredDateTime = 0 instead of IS NULL for the
beforeDate cutoff.
- GetBlogsForLikes: drop the EXISTS-Posts predicate so blogs with notes
but no posts rows are still eligible for likes collection.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Pre-flight check on the CollectNotes loop now sleeps until at least one
key recovers instead of issuing a wasted 429-bound request per iteration.
Extracts the countdown into ApiKeyPool.SleepUntilAnyAvailable (30s refresh)
and reuses it in CollectLikes and GrabNotes.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Console-friendly URL format for log lines that reference a specific
post — easier to copy/paste into a browser when investigating output.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Previous fix removed '.' from the GetRepliesWithFilledText SELECT,
which broke processing of legacy null-substitute rows that need to be
re-fetched.
Restored '.' in the SELECT. To avoid the infinite loop when an actual
API reply is the literal string ".", normalize it to ". " (dot +
trailing space) inside UpdateNoteReplyText so the stored value no
longer matches the sentinel.
Also extended the fan-out UPDATE guard to match the SELECT criteria
(NULL / '' / '.') so legacy '.' rows in other reblog copies can be
filled in too. The guard still refuses to overwrite '?' or
already-fetched text.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
'.' was used as a sentinel for 'not yet fetched' in the SELECT query,
but it is also valid reply text. This caused any post where a reply
text was literally '.' to stay in the work queue forever.
Also fixes the fan-out UPDATE guard: previously it used
IFNULL(replyText, '.') <> @newValue, which would overwrite '?'
(confirmed-empty) rows with '.' when processing a dot reply elsewhere
in the reblog chain, pulling completed posts back into the queue and
causing the remaining counter to increase.
Changes:
- Remove OR replyText = '.' from GetRepliesWithFilledText SELECT
- Restrict UpdateNoteReplyText fan-out to NULL/'' rows only
- Use '?' not '.' as null-coalesce fallback in UpdateNoteReplyText
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Previously, the '?' marker only fired when page 1 returned empty notes.
Posts whose page 1 contained only likes/reblogs and whose page 2 came
back empty were never marked, so GetRepliesWithFilledText kept reselecting
them every iteration. Now mark after the pagination loop whenever
rowsUpdated==0 and at least one page returned 200 OK.
UpdateAllNoteReplyTextForPost now returns int so the caller can fold
the count into emptyReplyCount/rowsUpdated for accurate Done logging.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
A reply by a given blog at a given timestamp is the same reply
across the original post and every reblog of it. Drop PostID from
the UpdateNoteReplyText WHERE clause so a single API hit fills in
the replyText on every matching row at once.
Pair that with a per-iteration requery (limit 1) of the work list
so posts whose replies were already filled in as a side-effect of
a previous post's update are skipped without burning an API call.
The API returns reply timestamps that are ~1s ahead of what -collect
originally stored, so an exact TimeStamp match in the UPDATE was
hitting zero rows for every note - the API call worked, the reply
text came back, but nothing landed in the DB. Match within +-5s
instead. Also return rowsAffected from UpdateNoteReplyText and
report it separately from notes-seen in the summary so misses are
visible.
✅ Captures replies and reblogs with comment
✅ Ignores rollup_notes field (as requested)
✅ Maintains rate limiting and error handling
✅ Console output shows both reply and reblog comment
appsettings.json — Added PoolEnabled to each API section:
- TumblrApi → true
- TumblrApi3 → false
- TumblrApi4 → true
DataAccess.cs — Added:
- ApiKeyConfig class — holds credentials + metadata per key
- ApiKeyPool class — manages pool with round-robin rotation, SQLite-backed state (ApiKeyPoolState, ApiKeyPoolMeta tables)
- GetCurrentKey() — returns next key, skipping rate-limited ones, falls back to earliest-recovery if all are throttled
- MarkRateLimited(key, retryUntil) / MarkAvailable(key) — persists state
- Initialize() — discovers pool-enabled keys, detects single-key override mode
- Refactored APIAccess.GrabNotes(), GrabPostWithReplies(), GrabLikes() to accept ApiKeyConfig param and log [Key#N]
Program.cs — Updated:
- Tracks apiExplicitlySet flag from -api/-api3/-api4
- Initializes ApiKeyPool at startup (pool mode or single-key override)
- All 3 API callers updated to use pool rotation + 429 handling
Startup Output
- Pool mode: [Pool] Active keys: Key#1=TumblrApi, Key#2=TumblrApi4
- Single-key: [Pool] Single-key mode: TumblrApi3
- Add SQLite import mode pragmas for faster bulk inserts
- Implement -urldump command to extract all URLs from posts
- Add -start [blogname] CLI option for partial traversal
- Support toggling file logging and record import logging via config
- Only update DB rows if values change to reduce writes
- Add DateCreated fields to relevant tables
- Enhance logging and output formatting
- Refactor directory traversal for better control and reporting
Allows switching between multiple Tumblr API credential sets at runtime using new command-line arguments (-api3, -api4, -api [section]). API keys are now read from the selected section in appsettings.json, with validation for required keys. Also updates the SQL query for selecting blogs needing likes to use more precise filtering and ordering. Usage/help output is updated to reflect new options.
Introduce byLikes parameter to AddBlog, AddPost, and UpdatePost methods, updating SQL logic and schema to store this flag. Ensure all relevant calls and SQL statements handle the new ByLikes column, allowing tracking of whether entries were added "by likes." Also standardize hasImage boolean handling in SQL.
Add Tumblr likes fetching and DB tracking support
Implemented -likes command to fetch/process Tumblr blog likes.
Added DB columns and logic to track likes progress (LikesPulled, LikesCursor).
Integrated API call, pagination, and rate-limit handling for likes.
Extended AddBlog/AddPost/UpdatePost for likes-related fields.
Added DateModified tracking to DB operations.
Improved error handling and updated launch/app settings.
Added RERUN.sqbpro with SQL queries and DB settings.
Enhanced DataAccess.cs filtering for 'zomb-eh' and enabled SQL debug output.
Commented out Tumblr URL output in Program.cs for cleaner logs.
Deleted TLDB.7z binary archive.
- Add database migration and update logic for replyText column in Notes
- Implement batch collection of missing reply text via Tumblr API
- Add new API integration to fetch reply text for replies
- Enhance DataAccess with methods to query/update replyText
- Update CLI: -replies now collects and stores reply text
- Improve logging (archive logs/), error handling, and output
- Remove TL.db from source control
jimandClaude Opus 5