Files

T

jimandClaude Opus 4.8 003a504d5e fix: retry transient CDN failures instead of failing the post

A non-JSON response body (CDN 403/5xx HTML, empty body, transport error)
never reached the Tumblr API, so it says nothing about the post being
fetched. These were recorded as FAILURE, which consumed the post's single
attempt for the pass and cleared the API key's rate-limit flag on the way
through.

Classify them as Root.transientFailure and retry in place (1s/4s/10s)
before skipping. Skipped posts stay unmarked in the DB so a later launch
retries them. Ten consecutive transient failures now aborts the pass
rather than skipping post-by-post against an edge refusing all traffic.

Also:
- MarkAvailable() only on a response that reached the API, and it is now
  a no-op when the key was not flagged (was writing to the DB and logging
  on every single call)
- Only a real 429 counts as a rate limit; stop inferring one from
  X-RateLimit-* headers, which Tumblr sends on every response
- Limiters pace with AcquireAsync instead of AttemptAcquire, which did
  not wait and aborted the run once a window was saturated
- Throttle --collect and --likes from 300/min to 60/min
- Log one line per transient failure instead of the HTML body and stack
  trace; keep full detail only for a 2xx that fails to parse
- --collect returns exit 3 when a pass ends incomplete

Co-Authored-By: Claude Opus 4.8 <[email protected]>

2026-07-22 12:10:49 -05:00

3.3 KiB

Raw Blame History

AGENTS.md

Project Overview

Primary Language: C# (.NET 8 Console Application)
Key Libraries: RestSharp, Newtonsoft.Json, System.Data.SQLite, Microsoft.Extensions.Configuration
Purpose: Tumblr API data harvester for collecting notes, posts, likes, and replies, storing results in SQLite.

Architectural Patterns

CLI entry point in Program.cs with workflow orchestration
DataAccess.cs: Database operations, ApiKeyPool (API key management), APIAccess (Tumblr client)
ResponseNotes.cs: Tumblr API response models
Round-robin API key rotation with rate-limit tracking
Automatic console color assignment per API key for output differentiation

Developer Guidelines

Code Formatting

4-space indentation, no tabs, match existing C# style
PascalCase for public members, camelCase for locals
Minimize code comments unless explicitly requested
Use only existing project libraries; no new dependencies without confirmation
Match accessibility modifiers (public for models, internal for helpers)

Error Handling

Wrap file/network operations in try-catch
Log non-critical errors (e.g., config write failures) with [Warning] prefix
Preserve console color state: use save/restore pattern for temporary color changes
API rate limits must use ApiKeyPool.MarkRateLimited()/MarkAvailable()

API Failure Classification

Tumblr sits behind a CDN that returns HTML error pages (403, 5xx) which never reach the API. These say nothing about the item being fetched, so they must not be recorded as per-item failures.

A response body that will not parse as JSON did not come from the API. Flag it with Root.transientFailure, never as FAILURE
Transient failures retry in place (TransientBackoffSeconds) before the item is skipped; a skipped item stays unmarked in the DB so a later launch retries it
MaxConsecutiveTransient consecutive transient failures aborts the pass rather than skipping item-by-item against an edge that is refusing all traffic
Only call ApiKeyPool.MarkAvailable() on a response that actually reached the API. A transport or CDN failure says nothing about the key's standing and must not clear its flag
Only a real HTTP 429 (or meta.status == 429) counts as a rate limit. Do not infer one from the presence of X-RateLimit-* headers, which Tumblr sends on every response
Rate limiters must pace with await AcquireAsync(). AttemptAcquire() does not wait, so a saturated window aborts the run instead of throttling it
Long-running commands return exit 3 when a pass ends incomplete (rate-limit pause, breaker trip, or skipped items), so a caller can distinguish that from a clean run

Testing

No existing test suite; use xUnit if adding tests
Test critical logic: ApiKeyPool init, color parsing, config persistence
Avoid testing one-off CLI workflows

Git Commit Messages

Imperative mood ("Add feature" not "Added feature")
Prefix with type: feat:, fix:, chore:, docs:
Keep messages under 72 characters
Never commit sensitive data (API keys/tokens)

API Key Color Rules

Unconfigured keys auto-assign colors from a preset palette
Auto-assigned colors persist to appsettings.json
All output for an active key uses its assigned color
Temporary color changes (e.g., errors) must restore the key's color afterward

3.3 KiB Raw Blame History