8 Commits
Author SHA1 Message Date
jimandClaude Opus 4.8 003a504d5e fix: retry transient CDN failures instead of failing the post
A non-JSON response body (CDN 403/5xx HTML, empty body, transport error)
never reached the Tumblr API, so it says nothing about the post being
fetched. These were recorded as FAILURE, which consumed the post's single
attempt for the pass and cleared the API key's rate-limit flag on the way
through.

Classify them as Root.transientFailure and retry in place (1s/4s/10s)
before skipping. Skipped posts stay unmarked in the DB so a later launch
retries them. Ten consecutive transient failures now aborts the pass
rather than skipping post-by-post against an edge refusing all traffic.

Also:
- MarkAvailable() only on a response that reached the API, and it is now
  a no-op when the key was not flagged (was writing to the DB and logging
  on every single call)
- Only a real 429 counts as a rate limit; stop inferring one from
  X-RateLimit-* headers, which Tumblr sends on every response
- Limiters pace with AcquireAsync instead of AttemptAcquire, which did
  not wait and aborted the run once a window was saturated
- Throttle --collect and --likes from 300/min to 60/min
- Log one line per transient failure instead of the HTML body and stack
  trace; keep full detail only for a 2xx that fails to parse
- --collect returns exit 3 when a pass ends incomplete

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-07-22 12:10:49 -05:00
jim abdfe30203 Uses mode=conversation to fetch only notes with text
 Captures replies and reblogs with comment
 Ignores rollup_notes field (as requested)
 Maintains rate limiting and error handling
 Console output shows both reply and reblog comment
2026-05-05 11:49:10 -05:00
jim eb86bc8f66 Add byLikes tracking to blogs and posts in DataAccess
Introduce byLikes parameter to AddBlog, AddPost, and UpdatePost methods, updating SQL logic and schema to store this flag. Ensure all relevant calls and SQL statements handle the new ByLikes column, allowing tracking of whether entries were added "by likes." Also standardize hasImage boolean handling in SQL.

Add Tumblr likes fetching and DB tracking support

Implemented -likes command to fetch/process Tumblr blog likes.
Added DB columns and logic to track likes progress (LikesPulled, LikesCursor).
Integrated API call, pagination, and rate-limit handling for likes.
Extended AddBlog/AddPost/UpdatePost for likes-related fields.
Added DateModified tracking to DB operations.
Improved error handling and updated launch/app settings.
2026-04-02 10:57:13 -05:00
jim bc9985a6f0 Add support for fetching and storing reply text
- Add database migration and update logic for replyText column in Notes
- Implement batch collection of missing reply text via Tumblr API
- Add new API integration to fetch reply text for replies
- Enhance DataAccess with methods to query/update replyText
- Update CLI: -replies now collects and stores reply text
- Improve logging (archive logs/), error handling, and output
- Remove TL.db from source control
2026-02-12 23:12:54 -06:00
jim 2640854f79 feat: Refactor database access to use parameterized queries and improve error handling; update launch settings and add VSCode configuration files 2025-11-21 16:26:33 -06:00
jim 64ef0e9079 feat: Initialize URLNotesGrabberCORE project with core application structure, data access, and configuration. 2025-11-19 15:51:52 -06:00
jim 5cddf2a427 Improved 429 retries 2024-11-01 08:01:15 -05:00
jim 225934d3b9 Add project files. 2024-10-30 09:14:53 -05:00