fix: retry transient CDN failures instead of failing the post
A non-JSON response body (CDN 403/5xx HTML, empty body, transport error) never reached the Tumblr API, so it says nothing about the post being fetched. These were recorded as FAILURE, which consumed the post's single attempt for the pass and cleared the API key's rate-limit flag on the way through. Classify them as Root.transientFailure and retry in place (1s/4s/10s) before skipping. Skipped posts stay unmarked in the DB so a later launch retries them. Ten consecutive transient failures now aborts the pass rather than skipping post-by-post against an edge refusing all traffic. Also: - MarkAvailable() only on a response that reached the API, and it is now a no-op when the key was not flagged (was writing to the DB and logging on every single call) - Only a real 429 counts as a rate limit; stop inferring one from X-RateLimit-* headers, which Tumblr sends on every response - Limiters pace with AcquireAsync instead of AttemptAcquire, which did not wait and aborted the run once a window was saturated - Throttle --collect and --likes from 300/min to 60/min - Log one line per transient failure instead of the HTML body and stack trace; keep full detail only for a 2xx that fails to parse - --collect returns exit 3 when a pass ends incomplete Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
@@ -27,6 +27,25 @@
|
||||
- Preserve console color state: use save/restore pattern for temporary color changes
|
||||
- API rate limits must use `ApiKeyPool.MarkRateLimited()`/`MarkAvailable()`
|
||||
|
||||
### API Failure Classification
|
||||
Tumblr sits behind a CDN that returns HTML error pages (403, 5xx) which never reach the API. These
|
||||
say nothing about the item being fetched, so they must not be recorded as per-item failures.
|
||||
|
||||
- A response body that will not parse as JSON did not come from the API. Flag it with
|
||||
`Root.transientFailure`, never as `FAILURE`
|
||||
- Transient failures retry in place (`TransientBackoffSeconds`) before the item is skipped; a skipped
|
||||
item stays unmarked in the DB so a later launch retries it
|
||||
- `MaxConsecutiveTransient` consecutive transient failures aborts the pass rather than skipping
|
||||
item-by-item against an edge that is refusing all traffic
|
||||
- Only call `ApiKeyPool.MarkAvailable()` on a response that actually reached the API. A transport or
|
||||
CDN failure says nothing about the key's standing and must not clear its flag
|
||||
- Only a real HTTP 429 (or `meta.status == 429`) counts as a rate limit. Do not infer one from the
|
||||
presence of `X-RateLimit-*` headers, which Tumblr sends on every response
|
||||
- Rate limiters must pace with `await AcquireAsync()`. `AttemptAcquire()` does not wait, so a
|
||||
saturated window aborts the run instead of throttling it
|
||||
- Long-running commands return exit 3 when a pass ends incomplete (rate-limit pause, breaker trip, or
|
||||
skipped items), so a caller can distinguish that from a clean run
|
||||
|
||||
### Testing
|
||||
- No existing test suite; use xUnit if adding tests
|
||||
- Test critical logic: `ApiKeyPool` init, color parsing, config persistence
|
||||
|
||||
Reference in New Issue
Block a user