The API returns reply timestamps that are ~1s ahead of what -collect
originally stored, so an exact TimeStamp match in the UPDATE was
hitting zero rows for every note - the API call worked, the reply
text came back, but nothing landed in the DB. Match within +-5s
instead. Also return rowsAffected from UpdateNoteReplyText and
report it separately from notes-seen in the summary so misses are
visible.
The DB-derived MAX(N.timestamp)+1 for a post is unreliable - many notes
were stored with the same timestamp (likely crawl time, not the actual
note timestamp), so passing it as before_timestamp excluded all real
replies and returned an empty notes array. Diagnostics showed two
unrelated posts coming back with identical totals (1625/1021/603) and
zero notes.
Now we start with no before_timestamp (newest page) and let pagination
walk backward via each page's last-note timestamp.
Wrap FetchAndStoreReplyText in a per-post pagination loop (up to 10
pages, advancing before_timestamp via the last note's timestamp) so
posts with >50 notes are fully walked. Log raw response (meta + first
500 chars of JSON) when a page returns no notes so empty results are
diagnosable. Stop blanket-marking every reply on a post with '?' on
the first empty response - rows stay '.' and remain retry-eligible;
only individual replies that come back with empty reply_text are
marked '?'.
Call ApiKeyPool.MarkAvailable(key) after successful reply fetch in FetchAndStoreReplyText, matching behavior in CollectLikes. This produces the colored '[Pool] Key#N (...) rate-limit cleared' line for each API call, allowing operators to visually track which API key is being used per request.
Co-Authored-By: Claude Haiku 4.5 <[email protected]>
✅ Captures replies and reblogs with comment
✅ Ignores rollup_notes field (as requested)
✅ Maintains rate limiting and error handling
✅ Console output shows both reply and reblog comment
appsettings.json — Added PoolEnabled to each API section:
- TumblrApi → true
- TumblrApi3 → false
- TumblrApi4 → true
DataAccess.cs — Added:
- ApiKeyConfig class — holds credentials + metadata per key
- ApiKeyPool class — manages pool with round-robin rotation, SQLite-backed state (ApiKeyPoolState, ApiKeyPoolMeta tables)
- GetCurrentKey() — returns next key, skipping rate-limited ones, falls back to earliest-recovery if all are throttled
- MarkRateLimited(key, retryUntil) / MarkAvailable(key) — persists state
- Initialize() — discovers pool-enabled keys, detects single-key override mode
- Refactored APIAccess.GrabNotes(), GrabPostWithReplies(), GrabLikes() to accept ApiKeyConfig param and log [Key#N]
Program.cs — Updated:
- Tracks apiExplicitlySet flag from -api/-api3/-api4
- Initializes ApiKeyPool at startup (pool mode or single-key override)
- All 3 API callers updated to use pool rotation + 429 handling
Startup Output
- Pool mode: [Pool] Active keys: Key#1=TumblrApi, Key#2=TumblrApi4
- Single-key: [Pool] Single-key mode: TumblrApi3
- Add SQLite import mode pragmas for faster bulk inserts
- Implement -urldump command to extract all URLs from posts
- Add -start [blogname] CLI option for partial traversal
- Support toggling file logging and record import logging via config
- Only update DB rows if values change to reduce writes
- Add DateCreated fields to relevant tables
- Enhance logging and output formatting
- Refactor directory traversal for better control and reporting
Allows switching between multiple Tumblr API credential sets at runtime using new command-line arguments (-api3, -api4, -api [section]). API keys are now read from the selected section in appsettings.json, with validation for required keys. Also updates the SQL query for selecting blogs needing likes to use more precise filtering and ordering. Usage/help output is updated to reflect new options.
Introduce byLikes parameter to AddBlog, AddPost, and UpdatePost methods, updating SQL logic and schema to store this flag. Ensure all relevant calls and SQL statements handle the new ByLikes column, allowing tracking of whether entries were added "by likes." Also standardize hasImage boolean handling in SQL.
Add Tumblr likes fetching and DB tracking support
Implemented -likes command to fetch/process Tumblr blog likes.
Added DB columns and logic to track likes progress (LikesPulled, LikesCursor).
Integrated API call, pagination, and rate-limit handling for likes.
Extended AddBlog/AddPost/UpdatePost for likes-related fields.
Added DateModified tracking to DB operations.
Improved error handling and updated launch/app settings.
Added RERUN.sqbpro with SQL queries and DB settings.
Enhanced DataAccess.cs filtering for 'zomb-eh' and enabled SQL debug output.
Commented out Tumblr URL output in Program.cs for cleaner logs.
Deleted TLDB.7z binary archive.
- Add database migration and update logic for replyText column in Notes
- Implement batch collection of missing reply text via Tumblr API
- Add new API integration to fetch reply text for replies
- Enhance DataAccess with methods to query/update replyText
- Update CLI: -replies now collects and stores reply text
- Improve logging (archive logs/), error handling, and output
- Remove TL.db from source control
- Added `beforeDate` parameter to filter posts by `NotesGatheredDateTime`.
- Improved SQL queries with additional joins and conditions.
- Refactored exception handling for better resource cleanup.
- Enhanced string comparisons to support case-insensitivity.
- Added `GetReplies` method for fetching replies.
- Improved handling of `ReblogRecord` data and filtering logic.
- Updated `-collect` command to support optional date filtering.
- Adjusted `launchSettings.json` for testing with specific parameters.
- Improved logging and error reporting in API and database operations.
jim
