URLNotesGrabberCore/AGENTS.md

AGENTS.md

Project Overview

• Primary Language: C# (.NET 8 Console Application)
• Key Libraries: RestSharp, Newtonsoft.Json, System.Data.SQLite, Microsoft.Extensions.Configuration
• Purpose: Tumblr API data harvester for collecting notes, posts, likes, and replies, storing results in SQLite.

Architectural Patterns

• CLI entry point in Program.cs with workflow orchestration
• DataAccess.cs: Database operations, ApiKeyPool (API key management), APIAccess (Tumblr client)
• ResponseNotes.cs: Tumblr API response models
• Round-robin API key rotation with rate-limit tracking
• Automatic console color assignment per API key for output differentiation
• No-argument mode (Program.TraverseDirectory) ingests .txt blog export files into Posts via DataAccess.AddPost. Recognized field prefixes live in TraverseDirectoryFieldPrefixes; Body: and Downloaded files: collect every following line up to the next recognized prefix (multi-line values). RootURL is populated from a Reblog root url: line the same way it's populated from the API-based --likes flow — both paths converge on DataAccess.AddPost's rootURL parameter, which UpdatePost only overwrites when the incoming value is non-empty (existing RootURL is preserved otherwise)

Developer Guidelines

Code Formatting

• 4-space indentation, no tabs, match existing C# style
• PascalCase for public members, camelCase for locals
• Minimize code comments unless explicitly requested
• Use only existing project libraries; no new dependencies without confirmation
• Match accessibility modifiers (public for models, internal for helpers)

Error Handling

• Wrap file/network operations in try-catch
• Log non-critical errors (e.g., config write failures) with [Warning] prefix
• Preserve console color state: use save/restore pattern for temporary color changes
• API rate limits must use ApiKeyPool.MarkRateLimited()/MarkAvailable()

API Failure Classification

Tumblr sits behind a CDN that returns HTML error pages (403, 5xx) which never reach the API. These say nothing about the item being fetched, so they must not be recorded as per-item failures.

• A response body that will not parse as JSON did not come from the API. Flag it with Root.transientFailure, never as FAILURE
• Transient failures retry in place (TransientBackoffSeconds) before the item is skipped; a skipped item stays unmarked in the DB so a later launch retries it
• MaxConsecutiveTransient consecutive transient failures aborts the pass rather than skipping item-by-item against an edge that is refusing all traffic
• Only call ApiKeyPool.MarkAvailable() on a response that actually reached the API. A transport or CDN failure says nothing about the key's standing and must not clear its flag
• Only a real HTTP 429 (or meta.status == 429) counts as a rate limit. Do not infer one from the presence of X-RateLimit-* headers, which Tumblr sends on every response
• Rate limiters must pace with await AcquireAsync(). AttemptAcquire() does not wait, so a saturated window aborts the run instead of throttling it
• Long-running commands return exit 3 when a pass ends incomplete (rate-limit pause, breaker trip, or skipped items), so a caller can distinguish that from a clean run

IsActive Is Not Ours To Write

Blogs.IsActive, Posts.IsActive and Notes.IsActive are removal flags set by other tools (Rolodex). 0 means removed; anything else, including NULL, means live. Full detail in URLNotesGrabberCORE/TL.db.md.

• Never write any IsActive column. Not in an INSERT column list, not in an UPDATE, and never via INSERT OR REPLACE on these tables — that resets the column default and un-removes the row. Re-crawling a removed row must refresh its content and leave the flag where it was
• Filter at selection, not at write. Every query that selects posts, notes or blogs excludes removed rows. Update statements stay keyed on a row the caller already selected; filtering them would spend API quota and then fail to persist the result
• Posts.IsActive and Notes.IsActive are optional — they are absent from databases that predate them, and naming a missing column is a hard SQLite error. Compose the filter with AndIsActive/WhereIsActive in DataAccess.cs, which return COALESCE(IsActive, 1) = 1 only when HasIsActiveColumn finds the column. Blogs.IsActive is not optional and is filtered directly
• Do not add these columns from this app, and do not add them to the missing-column list in verify-db-schema.sql

DateModified Tracks Real Changes Only

Blogs.DateModified, Posts.DateModified and Notes.DateModified must move only when a column beside DateModified itself actually changed. Re-crawling or re-ingesting identical content has to leave the row — and its timestamp — untouched, or downstream consumers cannot tell a refreshed row from a rewritten one.

• Enforce it in the WHERE clause, not in C#. Every UPDATE that sets DateModified ends with an AND (<col> <> @param OR ...) term covering every column in its SET list, so SQLite matches zero rows on a no-op and never writes
• Compare NULL-safely: IFNULL(col, '') <> IFNULL(@param, '') for text, IFNULL(col, 0) <> @param for integer flags. A bare col <> @param is NULL on a NULL column and silently skips the row that most needs writing
• Where NULL is not equivalent to the default, spell it out. The HasBeenOutput = 0 stamps use (HasBeenOutput IS NULL OR HasBeenOutput <> 0) because the selection queries test HasBeenOutput = 0, which a NULL would never match
• Dynamic SET lists (UpdatePostContentFields) build the guard alongside the assignments so the two lists cannot drift apart
• These statements now return 0 rows for "found but unchanged" as well as "not found". Callers that read ExecuteNonQuery() must not treat 0 as "row missing"

Posts.NotesGatheredDateTime is crawl bookkeeping, not content. It moves on every -collect pass and says nothing about the post, so it must never move DateModified on its own. UpdatePostMarkNotesCollected still writes it every pass but wraps the timestamp in DateModified = CASE WHEN IFNULL(HasNotesGathered, 0) <> 1 THEN @dateModified ELSE DateModified END — SQLite evaluates SET expressions against the pre-UPDATE row, so only the flag flipping counts as a modification. Use this shape for any column that has to be refreshed unconditionally without being a change. Blogs.LikesLastRefreshed is the deliberate exception: a refresh pass is treated as a real event on the blog row.

Blogs.DateAdded is write-once. AddBlog's INSERT is the only place that sets it. A new post arriving for a known blog reopens HasBeenOutput but must leave DateAdded alone — a new post is not a new blog, and rewriting the column both destroys the registration date and makes every insert look like a change.

Testing

• No existing test suite; use xUnit if adding tests
• Test critical logic: ApiKeyPool init, color parsing, config persistence
• Avoid testing one-off CLI workflows

Git Commit Messages

• Imperative mood ("Add feature" not "Added feature")
• Prefix with type: feat:, fix:, chore:, docs:
• Keep messages under 72 characters
• Never commit sensitive data (API keys/tokens)

API Key Color Rules

• Unconfigured keys auto-assign colors from a preset palette
• Auto-assigned colors persist to appsettings.json
• All output for an active key uses its assigned color
• Temporary color changes (e.g., errors) must restore the key's color afterward