URLNotesGrabberCore/AGENTS.md

AGENTS.md

Project Overview

• Primary Language: C# (.NET 8 Console Application)
• Key Libraries: RestSharp, Newtonsoft.Json, System.Data.SQLite, Microsoft.Extensions.Configuration
• Purpose: Tumblr API data harvester for collecting notes, posts, likes, and replies, storing results in SQLite.

Architectural Patterns

• CLI entry point in Program.cs with workflow orchestration
• DataAccess.cs: Database operations, ApiKeyPool (API key management), APIAccess (Tumblr client)
• ResponseNotes.cs: Tumblr API response models
• Round-robin API key rotation with rate-limit tracking
• Automatic console color assignment per API key for output differentiation
• No-argument mode (Program.TraverseDirectory) ingests .txt blog export files into Posts via DataAccess.AddPost. Recognized field prefixes live in TraverseDirectoryFieldPrefixes; Body: and Downloaded files: collect every following line up to the next recognized prefix (multi-line values). RootURL is populated from a Reblog root url: line the same way it's populated from the API-based --likes flow — both paths converge on DataAccess.AddPost's rootURL parameter, which UpdatePost only overwrites when the incoming value is non-empty (existing RootURL is preserved otherwise)

Developer Guidelines

Code Formatting

• 4-space indentation, no tabs, match existing C# style
• PascalCase for public members, camelCase for locals
• Minimize code comments unless explicitly requested
• Use only existing project libraries; no new dependencies without confirmation
• Match accessibility modifiers (public for models, internal for helpers)

Error Handling

• Wrap file/network operations in try-catch
• Log non-critical errors (e.g., config write failures) with [Warning] prefix
• Preserve console color state: use save/restore pattern for temporary color changes
• API rate limits must use ApiKeyPool.MarkRateLimited()/MarkAvailable()

API Failure Classification

Tumblr sits behind a CDN that returns HTML error pages (403, 5xx) which never reach the API. These say nothing about the item being fetched, so they must not be recorded as per-item failures.

• A response body that will not parse as JSON did not come from the API. Flag it with Root.transientFailure, never as FAILURE
• Transient failures retry in place (TransientBackoffSeconds) before the item is skipped; a skipped item stays unmarked in the DB so a later launch retries it
• MaxConsecutiveTransient consecutive transient failures aborts the pass rather than skipping item-by-item against an edge that is refusing all traffic
• Only call ApiKeyPool.MarkAvailable() on a response that actually reached the API. A transport or CDN failure says nothing about the key's standing and must not clear its flag
• Only a real HTTP 429 (or meta.status == 429) counts as a rate limit. Do not infer one from the presence of X-RateLimit-* headers, which Tumblr sends on every response
• Rate limiters must pace with await AcquireAsync(). AttemptAcquire() does not wait, so a saturated window aborts the run instead of throttling it
• Long-running commands return exit 3 when a pass ends incomplete (rate-limit pause, breaker trip, or skipped items), so a caller can distinguish that from a clean run

IsActive Is Not Ours To Write

Blogs.IsActive, Posts.IsActive and Notes.IsActive are removal flags set by other tools (Rolodex). 0 means removed; anything else, including NULL, means live. Full detail in URLNotesGrabberCORE/TL.db.md.

• Never write any IsActive column. Not in an INSERT column list, not in an UPDATE, and never via INSERT OR REPLACE on these tables — that resets the column default and un-removes the row. Re-crawling a removed row must refresh its content and leave the flag where it was
• Filter at selection, not at write. Every query that selects posts, notes or blogs excludes removed rows. Update statements stay keyed on a row the caller already selected; filtering them would spend API quota and then fail to persist the result
• Posts.IsActive and Notes.IsActive are optional — they are absent from databases that predate them, and naming a missing column is a hard SQLite error. Compose the filter with AndIsActive/WhereIsActive in DataAccess.cs, which return COALESCE(IsActive, 1) = 1 only when HasIsActiveColumn finds the column. Blogs.IsActive is not optional and is filtered directly
• Do not add these columns from this app, and do not add them to the missing-column list in verify-db-schema.sql

Testing

• No existing test suite; use xUnit if adding tests
• Test critical logic: ApiKeyPool init, color parsing, config persistence
• Avoid testing one-off CLI workflows

Git Commit Messages

• Imperative mood ("Add feature" not "Added feature")
• Prefix with type: feat:, fix:, chore:, docs:
• Keep messages under 72 characters
• Never commit sensitive data (API keys/tokens)

API Key Color Rules

• Unconfigured keys auto-assign colors from a preset palette
• Auto-assigned colors persist to appsettings.json
• All output for an active key uses its assigned color
• Temporary color changes (e.g., errors) must restore the key's color afterward