Author SHA1 Message Date
jim f541ec4260 fix: correctly detect rate-limit state for single-key API pools
IsAllRateLimited() short-circuited true for any pool with 0 or 1
keys, with minRetrySeconds left at 0 regardless of whether that key
was actually rate-limited. SleepUntilAnyAvailable() checks
"minRetry <= 0" to decide whether to skip sleeping, so with exactly
one key it always skipped the wait and let callers hammer the API
again immediately after a 429, even mid-cooldown.

The per-key loop already computes this correctly for any key count;
the special case only needs to cover the true no-keys edge case,
where there's nothing to wait on.
2026-06-30 21:22:12 -05:00
jim f549f020e1 refactor: standardize SQLiteConnection disposal via using; guard config
Replace the try/finally { connection.Close(); } pattern used across
most of DataAccess.cs with using declarations, so disposal happens
automatically and can't be skipped by a future edit that adds an
early return before the finally. Left the shared-connection
(ownsConnection) call sites alone since those intentionally outlive
a single method call.

Also drop a stray unused `using static ... JSType` import, and make
a missing ContainsList config setting fail with a clear
InvalidOperationException instead of a NullReferenceException from
Split(',') on null.
2026-06-30 20:54:18 -05:00
jim 0ff80a0fd3 fix: parameterize AddPost fallback UPDATE, guard args indexing
Posts.hasImage/DateModified fallback update built its WHERE clause via
raw string concatenation of blogName/postID, unlike every other query
in this method — a blog name containing a single quote would break or
inject into the query. Switch it to parameters.

--parse, --blogsO, and --bop indexed args[1..3] before checking
args.Length, so a missing argument threw IndexOutOfRangeException
instead of hitting the intended usage message.
2026-06-30 20:41:51 -05:00
jimandClaude Opus 4.8 4df73367fb BREAKING: switch all multi-char commands to POSIX --double-dash
Rename every multi-character option/command from single-dash to double-dash (--likes, --collect, --force, etc.) to follow the POSIX long-option convention. Single-character short options (-h, -V, -?) keep their single dash, as POSIX prescribes.

Breaking: existing invocations/scripts using single-dash forms now report Unknown Command and must be updated. Run profile (launchSettings.json) and CLI docs (copilot-instructions.md) updated to match.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-09 15:26:38 -05:00
jimandClaude Opus 4.8 a437fa87d3 Document -post and -bop commands in --help
These two commands were handled by the switch but never listed in help.
--help now covers every command the program accepts.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-09 15:19:51 -05:00
jimandClaude Opus 4.8 32a1583efd Document exit-status codes in --help output
The new 0/1/2 exit codes had no footprint in --help; add an Exit status
line so the documented behavior matches what the program now returns.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-09 15:18:25 -05:00
jimandClaude Opus 4.8 03676432bd Add POSIX-friendly CLI handling: --, --help/--version, exit codes
Keep the existing single-dash switch style and case-insensitive matching,
but add the cheap, non-breaking POSIX wins:

- `--` end-of-options: tokens after a bare `--` are treated as literal operands
- `--help`/`-h` (alongside `-?`) and `-V`/`--version`
- Main returns a real exit code: 2 for usage errors, propagates handler
  return codes, and a top-level catch yields a quiet 1 on unhandled errors

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-09 15:12:10 -05:00
jim 8d6b9212c1 Remove binaries, batch script; add DB schema verifier
Removed outdated binary files and the `run_500_times.bat` script, which automated repetitive runs of `URLNotesGrabberCORE`. The batch script is no longer needed or has been replaced.

Added `verify-db-schema.sql`, a new script to verify and align the SQLite database schema (`TL.db`) with the application's expected schema. The script includes:
- A verification section to identify missing/extra columns or tables.
- An optional fix section with `ALTER TABLE` statements to add missing columns.

The SQL script ensures database compatibility while preserving data integrity and avoiding destructive operations.
2026-06-03 15:54:22 -05:00
jimandClaude Opus 4.8 18f172fe96 Make -collect 0 a resumable, single-pass full re-check
Mode 0 (full re-check) previously reset its cutoff to now on every
launch, so an interrupted run restarted from scratch, and a post that
kept returning a non-Success/non-NotFound status could loop forever.

- Add single-row CollectRunState table + accessors (EnsureCollectRunStateTableExists,
  GetCollectRunState, BeginCollectRun, CompleteCollectRun) mirroring the
  ApiKeyPoolMeta pattern, to persist a frozen run cutoff and completion flag.
- -collect 0 with no explicit date is now a managed run: resume against the
  stored cutoff if a run is in progress, else start a new run; mark complete
  when the pass finishes so the next launch starts fresh. Explicit-date and
  mode 1 behavior unchanged.
- CollectNotes makes a single attempt pass via an in-process attempted set;
  FAILURE/UNKNOWN are logged once, TooManyRequests/no-lease aborts without
  completing so a later launch resumes.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-03 15:48:26 -05:00
jimandClaude Opus 4.8 b576a9cdf3 fix: make -revert scan the PathInput tree like the no-parameter run
-revert was DB-driven, searching each blog's Blogs.TTFolderPath
non-recursively for *.bak. That tree differs from the no-parameter run,
which recursively walks PathInput. Rewrite RevertMode to recursively walk
PathInput (filesystem-only, no DB), with the optional [blogname] argument
now filtering by path substring. Restore mechanics unchanged.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-05-28 16:30:26 -05:00
jimandClaude Opus 4.8 5973920894 feat: add -revert mode to restore *.bak back to *.txt
Inverse of -output. For each blog with a TTFolderPath, restores every
*.bak over its *.txt, first preserving the current *.txt as the
next-free *.bkN, then consuming the *.bak. Confirms before running and
supports an optional single-blog filter.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-05-28 16:11:35 -05:00
jim 494d6aa2d4 Shortened sleep 2026-05-19 16:48:44 -05:00
jim 18c5ac5401 fix: correct foreach syntax for .NET 8 compatibility 2026-05-19 16:27:38 -05:00
jim 21e848efb7 feat: add [X remaining] progress counter to likes mode output 2026-05-19 15:48:18 -05:00
jim 24c5449e0c No longer copies db to output directory 2026-05-18 21:32:01 -05:00
jimandClaude Opus 4.7 c55569eadf chore: log folder transitions during -ingest
The every-50-file progress line wasn't enough to know which blog was
currently being processed on a long run. Now -ingest prints a
"entering folder: <name>" line whenever the source folder changes, and
the every-50 progress line also prefixes the folder name.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-18 15:40:27 -05:00
jimandClaude Opus 4.7 cf3f97ddc4 feat: add single-blog filter to -ingest
`-ingest <blogname>` now restricts the run to one blog's folder, mirroring
the existing -parse <blogname> ergonomics. `-ingest` with no arg still
processes every blog under appSettings:PathTTRoot (or PathInput fallback).

Breaking vs 3aff849: the first positional arg is interpreted as a blog
name, not a path. Configure the root via appSettings:PathTTRoot.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-18 15:36:42 -05:00
jimandClaude Opus 4.7 3aff849216 feat: merge ThreeTxtFileHelper into URLNotesGrabberCORE
Folds the standalone ThreeTxtFileHelper tool into URLNotesGrabberCORE so
text-file ingest/output/correct lives alongside the API scraper. Adds
new flags -ingest, -output, -correct (with -apply), -updatepaths, and a
one-time -importposts <posts.db> migration.

Schema: Blogs.TTFolderPath and Posts.PostType are added by an idempotent
migration. On (BlogName, PostID) collisions, content columns are
overwritten while engagement columns (ByLikes, RootBlogName, RootURL,
HasNotesGathered, NotFound, NotesGatheredDateTime, Likes*) are preserved.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-18 12:43:17 -05:00
jim 5781e121d2 Merge branch 'claude/gifted-dhawan-7e5fc7' 2026-05-16 14:02:07 -05:00
jimandClaude Sonnet 4.6 e27190e4f9 add incremental refresh + cooldown to -likes mode
After backfill completes for a blog, -likes can now pick up only newer
likes instead of being a one-shot pull. Tracks a per-blog
liked_timestamp high-water mark and stops the refresh walk once it
crosses the stored mark. A configurable cooldown (LikesRefreshCooldownDays,
default 7) gates which blogs are re-checked on each run. -force bypasses
the cooldown.

The migration adds three columns to Blogs (LikesNewestTimestamp,
LikesLastRefreshed, LikesLastNewCount) and does a one-time reset of all
likes tracking state so the new high-water mark starts from a clean
baseline. Existing ByLikes posts remain; the UNIQUE constraint absorbs
re-inserts during the first re-backfill.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-16 14:00:45 -05:00
jim 7c667bd579 Merge branch 'claude/eager-hugle-2f3520' 2026-05-15 11:53:34 -05:00
jimandClaude Opus 4.7 2634ff8967 speed up no-args directory import with shared connection + lazy blog cache
AddBlog/AddPost/UpdatePost/UpdatePostSetDate now reuse a single SQLiteConnection
when an import session is active, instead of opening and closing one per call.
AddBlog also short-circuits on an in-memory HashSet of blog names already
attempted this run. Other entry points are unaffected since they never call
BeginImportSession.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-15 11:51:57 -05:00
jim 69ce36a1f5 sleep call on finding note 2026-05-15 11:49:54 -05:00
jim 52293f2021 Merge branch 'claude/goofy-antonelli-73ea26' 2026-05-11 09:09:25 -05:00
jimandClaude Opus 4.7 2b86ce9119 collapse GetPosts SQL log to a single line
Per-iteration call in CollectNotes was dumping ~30 lines of newline-padded
SQL to the console. Replace with a whitespace-collapsed single-line print.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-11 09:09:24 -05:00
jim a5c118ce3b Merge branch 'claude/goofy-antonelli-73ea26' 2026-05-11 09:03:12 -05:00
jimandClaude Opus 4.7 47497d02bf tweak GetPosts beforeDate filter and broaden GetBlogsForLikes
- GetPosts: match NotesGatheredDateTime = 0 instead of IS NULL for the
  beforeDate cutoff.
- GetBlogsForLikes: drop the EXISTS-Posts predicate so blogs with notes
  but no posts rows are still eligible for likes collection.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-11 09:03:01 -05:00
jimandClaude Opus 4.7 5ae1d4feb8 fix: skip API calls when all keys are rate-limited in -collect mode
Pre-flight check on the CollectNotes loop now sleeps until at least one
key recovers instead of issuing a wasted 429-bound request per iteration.
Extracts the countdown into ApiKeyPool.SleepUntilAnyAvailable (30s refresh)
and reuses it in CollectLikes and GrabNotes.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-11 08:37:47 -05:00
jim 2034b43e83 Merge branch 'claude/wizardly-antonelli-9f2795' 2026-05-09 20:24:53 -05:00
jimandClaude Opus 4.7 4f00adcc05 fix: -collect mode now processes all posts instead of exiting after first
Changed CollectNotes from async void to async Task and await it with
.GetAwaiter().GetResult() to match the pattern used by -replies and -likes.
Previously the method would return immediately after the first await,
causing Main to exit before the loop could process more than one post.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-09 20:24:50 -05:00
jim bcbaea8c48 Merge branch 'claude/wizardly-antonelli-9f2795' 2026-05-08 21:42:39 -05:00
jimandClaude Opus 4.7 349b465f8a fix: release 404 posts from -replies queue
Tumblr API 404s were leaving Posts.NotFound=0 and Notes.replyText='.',
so GetRepliesWithFilledText kept re-selecting the same dead post and
the loop spun forever. Mark NotFound=1 and replies '?' on 404.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-08 21:42:30 -05:00
jim acc5e885a8 feat: require PathDB config in appsettings.json 2026-05-08 16:24:43 -05:00
jimandClaude Sonnet 4.6 b68a9833d8 chore: format remaining blog/post log output as <blog>.tumblr.com/post/<id>
Apply the URL format to the remaining 6 console log lines that still
referenced posts in <blog>/<id> form (404 logs, max-page-limit log,
DumpReplies console output).

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-07 16:16:29 -05:00
jim 89f2addf01 Merge branch 'claude/awesome-edison-d12b61' 2026-05-07 16:13:30 -05:00
jimandClaude Sonnet 4.6 b963489741 chore: format blog/post log output as <blog>.tumblr.com/post/<id>
Console-friendly URL format for log lines that reference a specific
post — easier to copy/paste into a browser when investigating output.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-07 16:12:00 -05:00
jim d18ab92019 Merge branch 'claude/awesome-edison-d12b61' 2026-05-07 15:58:13 -05:00
jimandClaude Sonnet 4.6 4737baa288 fix: keep '.' as needs-processing sentinel, normalize real-dot replies
Previous fix removed '.' from the GetRepliesWithFilledText SELECT,
which broke processing of legacy null-substitute rows that need to be
re-fetched.

Restored '.' in the SELECT. To avoid the infinite loop when an actual
API reply is the literal string ".", normalize it to ". " (dot +
trailing space) inside UpdateNoteReplyText so the stored value no
longer matches the sentinel.

Also extended the fan-out UPDATE guard to match the SELECT criteria
(NULL / '' / '.') so legacy '.' rows in other reblog copies can be
filled in too. The guard still refuses to overwrite '?' or
already-fetched text.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-07 15:56:53 -05:00
jim 51c40fe2bd Merge branch 'claude/awesome-edison-d12b61' 2026-05-07 15:49:40 -05:00
jimandClaude Sonnet 4.6 51e2a8a1e1 fix: stop infinite loop when reply text is a literal dot
'.' was used as a sentinel for 'not yet fetched' in the SELECT query,
but it is also valid reply text. This caused any post where a reply
text was literally '.' to stay in the work queue forever.

Also fixes the fan-out UPDATE guard: previously it used
IFNULL(replyText, '.') <> @newValue, which would overwrite '?'
(confirmed-empty) rows with '.' when processing a dot reply elsewhere
in the reblog chain, pulling completed posts back into the queue and
causing the remaining counter to increase.

Changes:
- Remove OR replyText = '.' from GetRepliesWithFilledText SELECT
- Restrict UpdateNoteReplyText fan-out to NULL/'' rows only
- Use '?' not '.' as null-coalesce fallback in UpdateNoteReplyText

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-07 15:49:35 -05:00
jim 0213478a40 Merge branch 'claude/epic-bardeen-cbbb7e' into master 2026-05-07 15:02:48 -05:00
jimandClaude Opus 4.7 081528a81a fix: mark replies '?' after pagination yields no reply notes
Previously, the '?' marker only fired when page 1 returned empty notes.
Posts whose page 1 contained only likes/reblogs and whose page 2 came
back empty were never marked, so GetRepliesWithFilledText kept reselecting
them every iteration. Now mark after the pagination loop whenever
rowsUpdated==0 and at least one page returned 200 OK.

UpdateAllNoteReplyTextForPost now returns int so the caller can fold
the count into emptyReplyCount/rowsUpdated for accurate Done logging.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-07 15:01:04 -05:00
jim f2539dc9d0 fix: mark post's replies '?' on confirmed empty notes response
When the API responds 200 OK with notes:[] on the first page,
the post has no conversational notes per Tumblr. Stamp all the
post's reply rows '?' so the per-iteration requery stops handing
this post back forever. Only fires for first-page 200 OK so a
later-page empty (end of pagination) doesn't poison good rows.
2026-05-07 14:25:00 -05:00
jim 2438997c7c chore: untrack and gitignore .claude local settings + worktrees
Local Claude Code config; should not be in the repo.
2026-05-07 14:20:41 -05:00
jim ac79727040 feat: fan reply updates across reblog chains, requery per post
A reply by a given blog at a given timestamp is the same reply
across the original post and every reblog of it. Drop PostID from
the UpdateNoteReplyText WHERE clause so a single API hit fills in
the replyText on every matching row at once.

Pair that with a per-iteration requery (limit 1) of the work list
so posts whose replies were already filled in as a side-effect of
a previous post's update are skipped without burning an API call.
2026-05-07 14:19:54 -05:00
jim 64a914c14d fix: tolerate ~5s timestamp drift in UpdateNoteReplyText
The API returns reply timestamps that are ~1s ahead of what -collect
originally stored, so an exact TimeStamp match in the UPDATE was
hitting zero rows for every note - the API call worked, the reply
text came back, but nothing landed in the DB. Match within +-5s
instead. Also return rowsAffected from UpdateNoteReplyText and
report it separately from notes-seen in the summary so misses are
visible.
2026-05-07 14:10:37 -05:00
jim 1c7b6a1e86 fix: start replies fetch from newest, ignore stale DB timestamp
The DB-derived MAX(N.timestamp)+1 for a post is unreliable - many notes
were stored with the same timestamp (likely crawl time, not the actual
note timestamp), so passing it as before_timestamp excluded all real
replies and returned an empty notes array. Diagnostics showed two
unrelated posts coming back with identical totals (1625/1021/603) and
zero notes.

Now we start with no before_timestamp (newest page) and let pagination
walk backward via each page's last-note timestamp.
2026-05-07 14:00:48 -05:00
jim 1da8a702ba fix: pull reply_text from Tumblr in -replies mode
Wrap FetchAndStoreReplyText in a per-post pagination loop (up to 10
pages, advancing before_timestamp via the last note's timestamp) so
posts with >50 notes are fully walked. Log raw response (meta + first
500 chars of JSON) when a page returns no notes so empty results are
diagnosable. Stop blanket-marking every reply on a post with '?' on
the first empty response - rows stay '.' and remain retry-eligible;
only individual replies that come back with empty reply_text are
marked '?'.
2026-05-07 13:54:54 -05:00
jim b70f356d24 SQLite DB Project File 2026-05-06 16:51:56 -05:00
jimandClaude Haiku 4.5 539994b742 fix: emit per-key colored output in -replies mode on successful API calls
Call ApiKeyPool.MarkAvailable(key) after successful reply fetch in FetchAndStoreReplyText, matching behavior in CollectLikes. This produces the colored '[Pool] Key#N (...) rate-limit cleared' line for each API call, allowing operators to visually track which API key is being used per request.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
2026-05-06 15:27:23 -05:00
jim c39d555d6e 2026-05-06 14:50:30 -05:00
jim a7d175a5a8 feat: enhance replies command progress output 2026-05-06 10:32:49 -05:00
jim 0adaef62b1 fix: update GetRepliesWithFilledText to join Posts table and filter NotFound 2026-05-06 09:50:36 -05:00
jim 1507086269 fix: Correct CollectMissingReplyText to use API key rate limiting 2026-05-06 09:16:36 -05:00
jim 9c77ecce4b fix: color no-replies messages yellow for visual distinction 2026-05-05 18:07:19 -05:00
jim 78e3a070b2 fix: make reply text collection process all remaining posts without limit 2026-05-05 17:57:23 -05:00
jim 9589a9c090 feat: add green color for replies and cyan for reblog comments 2026-05-05 17:43:48 -05:00
jim abdfe30203 Uses mode=conversation to fetch only notes with text
 Captures replies and reblogs with comment
 Ignores rollup_notes field (as requested)
 Maintains rate limiting and error handling
 Console output shows both reply and reblog comment
2026-05-05 11:49:10 -05:00
jim 354bb9cc8e Additional fixes 2026-05-05 00:04:40 -05:00
jim 5e62b58261 feat: add API key color assignment, auto-persist, AGENTS.md 2026-05-04 10:33:46 -05:00
jim 34e5745e99 Fixed bugs on API key pool 2026-05-03 13:59:51 -05:00
jim 6cec7afb53 Changes Summary
appsettings.json — Added PoolEnabled to each API section:
- TumblrApi → true
- TumblrApi3 → false
- TumblrApi4 → true
DataAccess.cs — Added:
- ApiKeyConfig class — holds credentials + metadata per key
- ApiKeyPool class — manages pool with round-robin rotation, SQLite-backed state (ApiKeyPoolState, ApiKeyPoolMeta tables)
  - GetCurrentKey() — returns next key, skipping rate-limited ones, falls back to earliest-recovery if all are throttled
  - MarkRateLimited(key, retryUntil) / MarkAvailable(key) — persists state
  - Initialize() — discovers pool-enabled keys, detects single-key override mode
- Refactored APIAccess.GrabNotes(), GrabPostWithReplies(), GrabLikes() to accept ApiKeyConfig param and log [Key#N]
Program.cs — Updated:
- Tracks apiExplicitlySet flag from -api/-api3/-api4
- Initializes ApiKeyPool at startup (pool mode or single-key override)
- All 3 API callers updated to use pool rotation + 429 handling
Startup Output
- Pool mode: [Pool] Active keys: Key#1=TumblrApi, Key#2=TumblrApi4
- Single-key: [Pool] Single-key mode: TumblrApi3
2026-05-03 13:31:28 -05:00
jim de96c7c1c5 Updated db location logic 2026-05-03 08:41:21 -05:00
jim 059c9cd527 Improve import speed, add URL dump, and CLI options
- Add SQLite import mode pragmas for faster bulk inserts
- Implement -urldump command to extract all URLs from posts
- Add -start [blogname] CLI option for partial traversal
- Support toggling file logging and record import logging via config
- Only update DB rows if values change to reduce writes
- Add DateCreated fields to relevant tables
- Enhance logging and output formatting
- Refactor directory traversal for better control and reporting
2026-04-15 15:14:43 -05:00
jim 390e1284cb Add support for selecting Tumblr API credentials via CLI
Allows switching between multiple Tumblr API credential sets at runtime using new command-line arguments (-api3, -api4, -api [section]). API keys are now read from the selected section in appsettings.json, with validation for required keys. Also updates the SQL query for selecting blogs needing likes to use more precise filtering and ordering. Usage/help output is updated to reflect new options.
2026-04-03 13:23:38 -05:00
25 changed files with 8207 additions and 958 deletions
+9 -9
View File
@@ -22,18 +22,18 @@ This document provides essential context for AI agents working with URLNotesGrab
```powershell ```powershell
dotnet build dotnet build
dotnet run # Process all files in input directory dotnet run # Process all files in input directory
dotnet run -- -parse [blogname] # Process specific blog dotnet run -- --parse [blogname] # Process specific blog
dotnet run -- -test [blogname] [postID] # Test API for specific post dotnet run -- --test [blogname] [postID] # Test API for specific post
``` ```
### Command-Line Interface ### Command-Line Interface
- `-parse [blogname]`: Parse text files for specific blog - `--parse [blogname]`: Parse text files for specific blog
- `-test [blogname] [postID]`: Test API note collection - `--test [blogname] [postID]`: Test API note collection
- `-posts`: Export post blogs to file - `--posts`: Export post blogs to file
- `-blogs`: Export blog list to file - `--blogs`: Export blog list to file
- `-collect`: Collect notes for all posts in DB - `--collect`: Collect notes for all posts in DB
- `-blogsR`: Export reply blogs to file - `--blogsR`: Export reply blogs to file
- `-blogsO [start] [stop]`: Export blogs within range - `--blogsO [start] [stop]`: Export blogs within range
## Project Conventions ## Project Conventions
+10
View File
@@ -361,3 +361,13 @@ MigrationBackup/
# Fody - auto-generated XML schema # Fody - auto-generated XML schema
FodyWeavers.xsd FodyWeavers.xsd
/URLNotesGrabberCORE/TL.db
# Claude Code local settings + worktrees
.claude/settings.local.json
.claude/worktrees/
/URLNotesGrabberCORE/TL.db
/URLNotesGrabberCORE/tl.db-shm
/URLNotesGrabberCORE/tl.db-wal
/U/jim/documents/web copies/blogs
BIN
View File
Binary file not shown.
+45
View File
@@ -0,0 +1,45 @@
# AGENTS.md
## Project Overview
- **Primary Language**: C# (.NET 8 Console Application)
- **Key Libraries**: RestSharp, Newtonsoft.Json, System.Data.SQLite, Microsoft.Extensions.Configuration
- **Purpose**: Tumblr API data harvester for collecting notes, posts, likes, and replies, storing results in SQLite.
## Architectural Patterns
- CLI entry point in `Program.cs` with workflow orchestration
- `DataAccess.cs`: Database operations, `ApiKeyPool` (API key management), `APIAccess` (Tumblr client)
- `ResponseNotes.cs`: Tumblr API response models
- Round-robin API key rotation with rate-limit tracking
- Automatic console color assignment per API key for output differentiation
## Developer Guidelines
### Code Formatting
- 4-space indentation, no tabs, match existing C# style
- PascalCase for public members, camelCase for locals
- Minimize code comments unless explicitly requested
- Use only existing project libraries; no new dependencies without confirmation
- Match accessibility modifiers (`public` for models, `internal` for helpers)
### Error Handling
- Wrap file/network operations in `try-catch`
- Log non-critical errors (e.g., config write failures) with `[Warning]` prefix
- Preserve console color state: use save/restore pattern for temporary color changes
- API rate limits must use `ApiKeyPool.MarkRateLimited()`/`MarkAvailable()`
### Testing
- No existing test suite; use xUnit if adding tests
- Test critical logic: `ApiKeyPool` init, color parsing, config persistence
- Avoid testing one-off CLI workflows
### Git Commit Messages
- Imperative mood ("Add feature" not "Added feature")
- Prefix with type: `feat:`, `fix:`, `chore:`, `docs:`
- Keep messages under 72 characters
- Never commit sensitive data (API keys/tokens)
### API Key Color Rules
- Unconfigured keys auto-assign colors from a preset palette
- Auto-assigned colors persist to `appsettings.json`
- All output for an active key uses its assigned color
- Temporary color changes (e.g., errors) must restore the key's color afterward
+26 -7
View File
@@ -1,4 +1,4 @@
<?xml version="1.0" encoding="UTF-8"?><sqlb_project><db path="D:/NextCloud/C#/URLNotesGrabberCORE/URLNotesGrabberCORE/bin/Debug/net8.0/TL.db" readonly="0" foreign_keys="1" case_sensitive_like="0" temp_store="0" wal_autocheckpoint="1000" synchronous="2"/><attached/><window><main_tabs open="structure browser pragmas query" current="3"/></window><tab_structure><column_width id="0" width="300"/><column_width id="1" width="0"/><column_width id="2" width="100"/><column_width id="3" width="3571"/><column_width id="4" width="0"/><expanded_item id="0" parent="1"/><expanded_item id="1" parent="1"/><expanded_item id="2" parent="1"/><expanded_item id="3" parent="1"/></tab_structure><tab_browse><table title="Posts" custom_title="0" dock_id="4" table="4,5:mainPosts"/><dock_state state="000000ff00000000fd00000001000000020000077400000365fc0100000005fb000000160064006f0063006b00420072006f00770073006500310100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500320100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500330100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500350100000000000005f40000000000000000fc0000000000000774000001eb00fffffffa000000000100000002fb000000160064006f0063006b00420072006f00770073006500340100000000ffffffff000001eb00fffffffb000000160064006f0063006b00420072006f00770073006500340000000000ffffffff0000000000000000000007740000000000000004000000040000000800000008fc00000000"/><default_encoding codec=""/><browse_table_settings><table schema="main" name="Blogs" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="3" mode="1"/></sort><column_widths><column index="1" value="257"/><column index="2" value="95"/><column index="3" value="54"/><column index="4" value="156"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Notes" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="4" mode="1"/></sort><column_widths><column index="1" value="207"/><column index="2" value="151"/><column index="3" value="263"/><column index="4" value="127"/><column index="5" value="59"/><column index="6" value="300"/><column index="7" value="191"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Posts" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="5" mode="1"/></sort><column_widths><column index="1" value="247"/><column index="2" value="151"/><column index="3" value="206"/><column index="4" value="300"/><column index="5" value="110"/><column index="6" value="191"/><column index="7" value="267"/><column index="8" value="116"/><column index="9" value="300"/><column index="10" value="300"/><column index="11" value="119"/><column index="12" value="255"/><column index="13" value="300"/><column index="14" value="71"/><column index="15" value="300"/><column index="16" value="300"/><column index="17" value="59"/><column index="18" value="113"/><column index="19" value="151"/><column index="20" value="300"/><column index="21" value="148"/><column index="22" value="300"/><column index="23" value="300"/><column index="24" value="59"/></column_widths><filter_values><column index="7" value="=1"/><column index="4" value="&lt;&gt;1"/></filter_values><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table></browse_table_settings></tab_browse><tab_sql><sql name="SQL 1">UPDATE Posts <?xml version="1.0" encoding="UTF-8"?><sqlb_project><db path="C:/Users/jim/Nextcloud/C#/URLNotesGrabberCORE/URLNotesGrabberCORE/TL.db" readonly="0" foreign_keys="1" case_sensitive_like="0" temp_store="0" wal_autocheckpoint="1000" synchronous="2"/><attached/><window><main_tabs open="structure browser pragmas query" current="3"/></window><tab_structure><column_width id="0" width="300"/><column_width id="1" width="0"/><column_width id="2" width="100"/><column_width id="3" width="4305"/><column_width id="4" width="0"/><expanded_item id="0" parent="1"/><expanded_item id="1" parent="1"/><expanded_item id="2" parent="1"/><expanded_item id="3" parent="1"/></tab_structure><tab_browse><table title="Posts" custom_title="0" dock_id="4" table="4,5:mainPosts"/><dock_state state="000000ff00000000fd00000001000000020000077200000379fc0100000006fb000000160064006f0063006b00420072006f00770073006500310100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500320100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500330100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500350100000000000005f40000000000000000fb000000160064006f0063006b00420072006f00770073006500340100000000000007720000011700fffffffb000000160064006f0063006b00420072006f00770073006500340100000000000005f40000000000000000000007720000000000000004000000040000000800000008fc00000000"/><default_encoding codec=""/><browse_table_settings><table schema="main" name="ApiKeyPoolMeta" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="29"/><column index="2" value="64"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Blogs" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="257"/><column index="2" value="95"/><column index="3" value="54"/><column index="4" value="156"/><column index="5" value="51"/><column index="6" value="71"/><column index="7" value="85"/><column index="8" value="156"/><column index="9" value="156"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Posts" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="28" mode="1"/></sort><column_widths><column index="1" value="241"/><column index="2" value="148"/><column index="3" value="126"/><column index="4" value="300"/><column index="5" value="75"/><column index="6" value="187"/><column index="7" value="159"/><column index="8" value="75"/><column index="9" value="300"/><column index="10" value="300"/><column index="11" value="78"/><column index="12" value="249"/><column index="13" value="300"/><column index="14" value="53"/><column index="15" value="300"/><column index="16" value="300"/><column index="17" value="41"/><column index="18" value="75"/><column index="19" value="96"/><column index="20" value="300"/><column index="21" value="96"/><column index="22" value="300"/><column index="23" value="300"/><column index="24" value="42"/><column index="25" value="60"/><column index="26" value="218"/><column index="27" value="920"/><column index="28" value="156"/><column index="29" value="156"/></column_widths><filter_values><column index="24" value="=1"/><column index="28" value="&gt;2026-05-06 20:00:01"/></filter_values><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table></browse_table_settings></tab_browse><tab_sql><sql name="SQL 1">UPDATE Posts
SET HasNotesGathered = 0 SET HasNotesGathered = 0
WHERE (BlogName, PostID) IN ( WHERE (BlogName, PostID) IN (
SELECT p.BlogName, p.PostID SELECT p.BlogName, p.PostID
@@ -14,7 +14,7 @@ WHERE (BlogName, PostID) IN (
) )
ORDER BY P.PostDate ASC ORDER BY P.PostDate ASC
--LIMIT 500 --LIMIT 500
);</sql><sql name="SQL 2">select * );</sql><sql name="Mark Blogs">select *
from Blogs from Blogs
--update blogs set HasBeenOutput = 1 --update blogs set HasBeenOutput = 1
where HasBeenOutput = 0 where HasBeenOutput = 0
@@ -29,11 +29,12 @@ blogname in
'nudenymph', 'nudenymph',
'caylachief' 'caylachief'
)</sql><sql name="SQL 3*">select RootBlogName, PostID, NoteBlogName || '.tumblr.com' as NoteBlogName, DatetimeCrawled, TimeStamp, type, RootBlogName || '.tumblr.com/post/' || postid, datetime(timestamp, 'unixepoch') )</sql><sql name="New Notes">select RootBlogName, PostID, NoteBlogName || '.tumblr.com' as NoteBlogName, DatetimeCrawled, TimeStamp, type, RootBlogName || '.tumblr.com/post/' || postid, datetime(timestamp, 'unixepoch')
from Notes from Notes
where --type like 'r%' and where
DatetimeCrawled &lt;&gt; '2/12/26 12am' DatetimeCrawled &gt; '2026-05-14 02:50:05' --and type like 'r%'
order by DatetimeCrawled desc, TimeStamp desc</sql><sql name="SQL 5">SELECT distinct order by DatetimeCrawled desc</sql><sql name="Pull Blogs*">SELECT distinct
'''' || blogname || ''',',
blogs.* blogs.*
, blogname || '.tumblr.com' , blogname || '.tumblr.com'
FROM FROM
@@ -45,4 +46,22 @@ WHERE
order by order by
Notes.Type desc, Notes.Type desc,
DateAdded desc DateAdded desc
LIMIT 500;</sql><current_tab id="2"/></tab_sql></sqlb_project> LIMIT 100;</sql><sql name="SQL 7">WITH ReplyCounts AS (
SELECT
NoteBlogName,
COUNT(DISTINCT replyText) AS DistinctReplyCount
FROM Notes
where replyText &lt;&gt; '.'
GROUP BY NoteBlogName
)
SELECT
n.RootBlogName || '.tumblr.com/post/' || n.PostID AS PostURL, postid,
n.NoteBlogName,
n.replyText,
c.DistinctReplyCount
FROM Notes n
JOIN ReplyCounts c ON n.NoteBlogName = c.NoteBlogName
where replyText &lt;&gt; '.' and type &lt;&gt; 'reply'
--AND N.NoteBlogName NOT IN ( 'roadblocker21', 'thesaddemon666', 'edwardabbeyhoffman', 'tattedsoldier20', 'zomb-eh', 'animalistic13', 'indken', 'maccloud1592',
--'moss-wizard', 'supertrucker12682', 'exploringthrupics', 'padeyepete' )
order by c.DistinctReplyCount desc, n.NoteBlogName, n.DateModified desc, replyText, RootBlogName, PostID</sql><current_tab id="3"/></tab_sql></sqlb_project>
BIN
View File
Binary file not shown.
BIN
View File
Binary file not shown.
BIN
View File
Binary file not shown.
+277
View File
@@ -0,0 +1,277 @@
using Microsoft.Extensions.Configuration;
namespace URLNotesGrabberCORE
{
// Port of ThreeTxtFileHelper RunCorrectionMode (dry-run) and RunFullCorrectionMode (apply).
// Scans a BAK directory of .txt files, parses posts with multi-line field support,
// and either reports or applies content-column corrections to TL.db.Posts.
// Apply path only writes non-empty values (mirrors original ThreeTxtFileHelper semantics)
// and never touches engagement columns.
public static class CorrectMode
{
public static int Run(IConfiguration config, string[] args, bool applyChanges)
{
DataAccess.EnsureTTFileHelperColumnsExist();
// Resolve BAK path: explicit arg > appSettings:PathTTBackup > derive from PathTTRoot/PathInput
string? bakRootPath = args.Length > 0 ? args[0] : config["appSettings:PathTTBackup"];
if (string.IsNullOrWhiteSpace(bakRootPath))
{
string? root = config["appSettings:PathTTRoot"];
if (string.IsNullOrWhiteSpace(root)) root = config["appSettings:PathInput"];
if (!string.IsNullOrWhiteSpace(root))
bakRootPath = root.TrimEnd('\\', '/') + "_BAK\\";
}
if (string.IsNullOrWhiteSpace(bakRootPath) || !Directory.Exists(bakRootPath))
{
Console.WriteLine($"BAK directory not found: {bakRootPath}");
return 1;
}
Console.WriteLine($"BAK source path: {bakRootPath}");
Console.WriteLine(applyChanges
? "Correction mode: APPLY - non-empty fields from BAK overwrite DB columns\n"
: "Correction mode: Dry-run - reports multi-line field updates available\n");
string prefixesPath = config["appSettings:PathPrefixes"] ?? "prefixes.txt";
if (!Path.IsPathRooted(prefixesPath))
prefixesPath = Path.Combine(AppContext.BaseDirectory, prefixesPath);
if (!File.Exists(prefixesPath))
{
Console.WriteLine($"Prefixes file not found: {prefixesPath}");
return 1;
}
var allowedPrefixes = new HashSet<string>(File.ReadLines(prefixesPath), StringComparer.OrdinalIgnoreCase);
if (applyChanges)
{
Console.Write("WARNING: This will overwrite non-empty fields in matching posts from BAK files. Continue? (yes/no): ");
string? response = Console.ReadLine();
if (string.IsNullOrWhiteSpace(response) || !response.Equals("yes", StringComparison.OrdinalIgnoreCase))
{
Console.WriteLine("Operation cancelled.");
return 0;
}
}
var bakTxtFiles = new List<string>();
try
{
foreach (var dir in Directory.GetDirectories(bakRootPath, "*", SearchOption.AllDirectories))
bakTxtFiles.AddRange(Directory.GetFiles(dir, "*.txt"));
bakTxtFiles.AddRange(Directory.GetFiles(bakRootPath, "*.txt"));
}
catch (Exception ex)
{
Console.WriteLine($"Error scanning BAK directory: {ex.Message}");
return 1;
}
Console.WriteLine($"Found {bakTxtFiles.Count} file(s) in BAK directory\n");
int totalPostsFound = 0;
int postsWithUpdates = 0;
int postsUpdated = 0;
int postsNotFound = 0;
var correctionLog = new List<string>();
var updateLog = new List<string>();
foreach (string bakFile in bakTxtFiles)
{
Console.WriteLine($"Processing BAK file: {Path.GetFileName(bakFile)}");
try
{
var bakPosts = ParsePostsFromFile(bakFile, allowedPrefixes);
Console.WriteLine($" Found {bakPosts.Count} post(s) in this file");
foreach (var (postId, bakData) in bakPosts)
{
totalPostsFound++;
var dbPost = DataAccess.GetPostByIdAnyBlog(postId);
if (dbPost == null)
{
postsNotFound++;
continue;
}
if (applyChanges)
{
bool updated = DataAccess.UpdatePostContentFields(dbPost.BlogName, dbPost.PostId, bakData);
if (updated)
{
postsUpdated++;
updateLog.Add($"Post ID: {postId} - Updated from {Path.GetFileName(bakFile)}");
}
}
else
{
var updateList = BuildDryRunDiff(bakData, dbPost);
if (updateList.Count > 0)
{
postsWithUpdates++;
correctionLog.Add($"\nPost ID: {postId}");
correctionLog.Add($" File: {Path.GetFileName(bakFile)}");
correctionLog.Add($" Fields to update:");
correctionLog.AddRange(updateList);
}
}
}
}
catch (Exception ex)
{
Console.WriteLine($" ERROR processing file: {ex.Message}");
}
}
if (applyChanges)
{
Console.WriteLine($"\n========== CORRECTION COMPLETE ==========");
Console.WriteLine($"Total posts found in BAK files: {totalPostsFound}");
Console.WriteLine($"Posts updated in database: {postsUpdated}");
Console.WriteLine($"Posts not found in database: {postsNotFound}");
string logPath = config["appSettings:PathCorrectionApplied"] ?? "correction_applied.txt";
try
{
var logLines = new List<string>
{
$"Correction Applied: {DateTime.Now:yyyy-MM-dd HH:mm:ss}",
$"Total posts found in BAK files: {totalPostsFound}",
$"Posts updated in database: {postsUpdated}",
$"Posts not found in database: {postsNotFound}",
"",
"Updated Posts:"
};
logLines.AddRange(updateLog);
File.WriteAllLines(logPath, logLines);
Console.WriteLine($"Update log saved to: {logPath}");
}
catch (Exception ex)
{
Console.WriteLine($"Error writing log file: {ex.Message}");
}
}
else
{
Console.WriteLine($"\n========== CORRECTION REPORT (DRY RUN) ==========");
Console.WriteLine($"Total posts found in BAK files: {totalPostsFound}");
Console.WriteLine($"Posts with multi-line field updates available: {postsWithUpdates}");
if (correctionLog.Count > 0)
{
string logPath = config["appSettings:PathCorrectionReport"] ?? "correction_report.txt";
try
{
File.WriteAllLines(logPath, correctionLog);
Console.WriteLine($"\nDetailed report saved to: {logPath}");
}
catch (Exception ex)
{
Console.WriteLine($"Error writing report file: {ex.Message}");
}
}
else
{
Console.WriteLine("\nNo multi-line field updates found.");
}
Console.WriteLine("\nDry-run complete. No database changes were made.");
Console.WriteLine("If updates look correct, re-run with `-correct -apply` to apply changes.");
}
return 0;
}
private static List<string> BuildDryRunDiff(Dictionary<string, string> bakData, TTPostRecord dbPost)
{
var updates = new List<string>();
foreach (var (fieldName, bakValue) in bakData)
{
if (string.IsNullOrWhiteSpace(bakValue)) continue;
string? currentValue = fieldName.ToLowerInvariant() switch
{
"reblog url" => dbPost.ReblogUrl,
"date" => dbPost.Date,
"has image" => dbPost.HasImage,
"post url" => dbPost.PostUrl,
"slug" => dbPost.Slug,
"reblog key" => dbPost.ReblogKey,
"reblog name" => dbPost.ReblogName,
"summary" => dbPost.Summary,
"quote" => dbPost.Quote,
"body" => dbPost.Body,
"tags" => dbPost.Tags,
"link" => dbPost.Link,
"photo url" => dbPost.PhotoUrl,
"photo caption" => dbPost.PhotoCaption,
"downloaded files" => dbPost.DownloadedFiles,
"audio caption" => dbPost.AudioCaption,
"question" => dbPost.Question,
"answer" => dbPost.Answer,
"title" => dbPost.Title,
_ => null
};
if (bakValue != currentValue && bakValue.Contains('\n'))
updates.Add($" {fieldName}: [MULTILINE]");
}
return updates;
}
private static Dictionary<string, Dictionary<string, string>> ParsePostsFromFile(string filePath, HashSet<string> allowedPrefixes)
{
var posts = new Dictionary<string, Dictionary<string, string>>(StringComparer.OrdinalIgnoreCase);
var lines = File.ReadAllLines(filePath);
int lineIndex = 0;
string currentPostId = "";
var currentPostData = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
while (lineIndex < lines.Length)
{
string line = lines[lineIndex];
string searchText = line.Length > 25 ? line.Substring(0, 25) : line;
int colonIndex = searchText.IndexOf(": ");
if (colonIndex > 0)
{
string prefix = line.Substring(0, colonIndex).Trim();
if (!string.IsNullOrWhiteSpace(prefix) && allowedPrefixes.Contains(prefix))
{
if (string.Equals(prefix, "Post ID", StringComparison.OrdinalIgnoreCase))
{
if (!string.IsNullOrWhiteSpace(currentPostId) && currentPostData.Count > 0)
posts[currentPostId] = new Dictionary<string, string>(currentPostData, StringComparer.OrdinalIgnoreCase);
currentPostId = line.Substring(colonIndex + 2).Trim();
currentPostData.Clear();
lineIndex++;
continue;
}
var valueLines = new List<string> { line.Substring(colonIndex + 2).Trim() };
int nextLineIndex = lineIndex + 1;
while (nextLineIndex < lines.Length)
{
string nextLine = lines[nextLineIndex];
string nextSearch = nextLine.Length > 25 ? nextLine.Substring(0, 25) : nextLine;
int nextColon = nextSearch.IndexOf(": ");
if (nextColon > 0)
{
string nextPrefix = nextLine.Substring(0, nextColon).Trim();
if (!string.IsNullOrWhiteSpace(nextPrefix) && allowedPrefixes.Contains(nextPrefix))
break;
}
valueLines.Add(nextLine);
nextLineIndex++;
}
currentPostData[prefix] = string.Join("\n", valueLines);
lineIndex = nextLineIndex;
continue;
}
}
lineIndex++;
}
if (!string.IsNullOrWhiteSpace(currentPostId) && currentPostData.Count > 0)
posts[currentPostId] = new Dictionary<string, string>(currentPostData, StringComparer.OrdinalIgnoreCase);
return posts;
}
}
}
File diff suppressed because it is too large Load Diff
+219
View File
@@ -0,0 +1,219 @@
using System.Text.RegularExpressions;
using Microsoft.Extensions.Configuration;
namespace URLNotesGrabberCORE
{
// Port of ThreeTxtFileHelper RunIngestMode. Scans a root folder for .txt files,
// parses Tumblr-export fields (multi-line aware, prefix-driven), upserts each
// post into TL.db.Posts via DataAccess.UpsertPostFromTextFile.
public static class IngestMode
{
public static int Run(IConfiguration config, string[] args)
{
string? targetBlog = args.Length > 0 ? args[0]?.Trim() : null;
if (string.IsNullOrWhiteSpace(targetBlog)) targetBlog = null;
string? rootPath = config["appSettings:PathTTRoot"];
if (string.IsNullOrWhiteSpace(rootPath))
rootPath = config["appSettings:PathInput"];
if (string.IsNullOrWhiteSpace(rootPath))
{
Console.WriteLine("Ingest: no root path configured. Set appSettings:PathTTRoot or appSettings:PathInput.");
return 1;
}
if (!Directory.Exists(rootPath))
{
Console.WriteLine($"Directory not found: {rootPath}");
return 1;
}
DataAccess.EnsureTTFileHelperColumnsExist();
string prefixesPath = config["appSettings:PathPrefixes"] ?? "prefixes.txt";
if (!Path.IsPathRooted(prefixesPath))
prefixesPath = Path.Combine(AppContext.BaseDirectory, prefixesPath);
if (!File.Exists(prefixesPath))
{
Console.WriteLine($"Prefixes file not found: {prefixesPath}");
return 1;
}
var allowedPrefixes = new HashSet<string>(File.ReadLines(prefixesPath), StringComparer.OrdinalIgnoreCase);
Console.WriteLine($"Loaded {allowedPrefixes.Count} prefixes from {prefixesPath}");
Console.WriteLine($"========== Ingest Settings ==========");
Console.WriteLine($"Root path: {rootPath}");
Console.WriteLine($"Blog filter: {(targetBlog == null ? "(all blogs)" : targetBlog)}");
Console.WriteLine($"=====================================");
var txtFiles = new List<string>();
try
{
var dirs = Directory.GetDirectories(rootPath, "*", SearchOption.AllDirectories);
Console.WriteLine($"Found {dirs.Length} directories under root.");
foreach (var dir in dirs)
{
try { txtFiles.AddRange(Directory.GetFiles(dir, "*.txt")); }
catch (Exception ex) { Console.WriteLine($" Skipping {dir}: {ex.Message}"); }
}
txtFiles.AddRange(Directory.GetFiles(rootPath, "*.txt"));
}
catch (Exception ex)
{
Console.WriteLine($"Error scanning root: {ex.Message}");
return 1;
}
Console.WriteLine($"Processing {txtFiles.Count} .txt file(s)...");
int filesProcessed = 0;
int filesSkipped = 0;
int postsTouched = 0;
string? lastBlogFolder = null;
try
{
DataAccess.EnableImportModePragmas();
DataAccess.BeginImportSession();
foreach (string file in txtFiles)
{
try
{
string rawBlogName = Path.GetFileName(Path.GetDirectoryName(file) ?? "unknown");
string blogName = Regex.Replace(rawBlogName, @"_\d+$", "");
string postType = Path.GetFileNameWithoutExtension(file);
if (targetBlog != null && !string.Equals(blogName, targetBlog, StringComparison.OrdinalIgnoreCase))
{
filesSkipped++;
continue;
}
filesProcessed++;
if (lastBlogFolder != rawBlogName)
{
Console.WriteLine($"[{filesProcessed}/{txtFiles.Count}] >> entering folder: {rawBlogName}");
lastBlogFolder = rawBlogName;
}
else if (filesProcessed % 50 == 0)
{
Console.WriteLine($"[{filesProcessed}/{txtFiles.Count}] {rawBlogName}/{Path.GetFileName(file)}");
}
string currentPostId = "";
var currentPostData = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
void Flush()
{
if (!string.IsNullOrWhiteSpace(currentPostId) && currentPostData.Count > 0)
{
UpsertPostFromParsedData(blogName, currentPostId, postType, currentPostData);
postsTouched++;
}
}
var lines = File.ReadAllLines(file);
int lineIndex = 0;
while (lineIndex < lines.Length)
{
string line = lines[lineIndex];
string searchText = line.Length > 25 ? line.Substring(0, 25) : line;
int colonIndex = searchText.IndexOf(": ");
if (colonIndex > 0)
{
string prefix = line.Substring(0, colonIndex).Trim();
if (!string.IsNullOrWhiteSpace(prefix) && allowedPrefixes.Contains(prefix))
{
if (string.Equals(prefix, "Post ID", StringComparison.OrdinalIgnoreCase))
{
Flush();
currentPostId = line.Substring(colonIndex + 2).Trim();
currentPostData.Clear();
lineIndex++;
continue;
}
var valueLines = new List<string> { line.Substring(colonIndex + 2).Trim() };
int nextLineIndex = lineIndex + 1;
while (nextLineIndex < lines.Length)
{
string nextLine = lines[nextLineIndex];
string nextSearch = nextLine.Length > 25 ? nextLine.Substring(0, 25) : nextLine;
int nextColon = nextSearch.IndexOf(": ");
if (nextColon > 0)
{
string nextPrefix = nextLine.Substring(0, nextColon).Trim();
if (!string.IsNullOrWhiteSpace(nextPrefix) && allowedPrefixes.Contains(nextPrefix))
break;
}
valueLines.Add(nextLine);
nextLineIndex++;
}
currentPostData[prefix] = string.Join("\n", valueLines);
lineIndex = nextLineIndex;
continue;
}
}
lineIndex++;
}
Flush();
}
catch (Exception ex)
{
Console.WriteLine($" ERROR processing file {file}: {ex.Message}");
}
}
}
finally
{
DataAccess.EndImportSession();
DataAccess.RestoreImportModePragmas();
}
Console.WriteLine($"\nIngest complete. Files processed: {filesProcessed}. Files skipped (blog filter): {filesSkipped}. Posts touched: {postsTouched}.");
return 0;
}
private static void UpsertPostFromParsedData(string blogName, string postId, string postType, Dictionary<string, string> data)
{
string? G(string key) => data.TryGetValue(key, out var v) ? v : null;
string? hasImageStr = G("Has Image");
bool hasImage = !string.IsNullOrWhiteSpace(hasImageStr)
&& (hasImageStr.Equals("true", StringComparison.OrdinalIgnoreCase)
|| hasImageStr == "1"
|| hasImageStr.Equals("yes", StringComparison.OrdinalIgnoreCase));
DataAccess.UpsertPostFromTextFile(
blogName: blogName,
postID: postId,
reblogURL: G("reblog URL"),
postDate: G("Date"),
postURL: G("Post URL"),
slug: G("Slug"),
reblogKey: G("Reblog Key"),
reblogName: G("Reblog Name"),
summary: G("Summary"),
quote: G("Quote"),
body: G("Body"),
tags: G("Tags"),
link: G("Link"),
photoURL: G("Photo URL"),
photoCaption: G("Photo Caption"),
downloadedFiles: G("Downloaded Files"),
audioCaption: G("Audio Caption"),
question: G("Question"),
answer: G("Answer"),
title: G("Title"),
postType: postType,
hasImage: hasImage);
}
}
}
@@ -0,0 +1,147 @@
using System.Data.SQLite;
namespace URLNotesGrabberCORE
{
// One-time migration: opens a legacy ThreeTxtFileHelper posts.db, copies its
// Blog + PostData rows into the merged TL.db via DataAccess.
// Conflict rule on (BlogName, PostId): ThreeTxtFileHelper wins on the 22 content
// columns + PostType + DateModified (handled inside UpsertPostFromTextFile).
// Engagement columns in TL.db (ByLikes, RootBlogName, RootURL, HasNotesGathered,
// NotFound, NotesGatheredDateTime) are preserved.
public static class LegacyPostsDbImporter
{
public static int Run(string legacyDbPath)
{
if (string.IsNullOrWhiteSpace(legacyDbPath))
{
Console.WriteLine("LegacyPostsDbImporter: path to legacy posts.db is required.");
return 1;
}
if (!File.Exists(legacyDbPath))
{
Console.WriteLine($"Legacy posts.db not found at: {legacyDbPath}");
return 1;
}
DataAccess.EnsureTTFileHelperColumnsExist();
Console.WriteLine($"Reading legacy posts.db: {legacyDbPath}");
int blogsCopied = 0;
int postsUpserted = 0;
int errors = 0;
try
{
using var src = new SQLiteConnection("Data Source=" + legacyDbPath + ";Read Only=True;");
src.Open();
// 1) Copy Blogs (BlogName + TTFolderPath)
using (var cmd = new SQLiteCommand("SELECT BlogName, TTFolderPath FROM Blogs", src))
using (var reader = cmd.ExecuteReader())
{
while (reader.Read())
{
string blogName = reader.IsDBNull(0) ? string.Empty : reader.GetString(0);
string? ttFolderPath = reader.IsDBNull(1) ? null : reader.GetString(1);
if (string.IsNullOrWhiteSpace(blogName)) continue;
try
{
DataAccess.SetBlogTTFolderPath(blogName, ttFolderPath);
blogsCopied++;
}
catch (Exception ex)
{
errors++;
Console.WriteLine($" Blog copy failed for '{blogName}': {ex.Message}");
}
}
}
Console.WriteLine($" Blogs copied: {blogsCopied}");
// 2) Copy Posts
try
{
DataAccess.EnableImportModePragmas();
DataAccess.BeginImportSession();
string sql = @"SELECT BlogName, PostId, ReblogUrl, Date, HasImage, PostUrl, Slug,
ReblogKey, ReblogName, Summary, Quote, Body, Tags, Link,
PhotoUrl, PhotoCaption, DownloadedFiles, AudioCaption,
Question, Answer, Title, PostType
FROM Posts";
using var cmd = new SQLiteCommand(sql, src);
using var reader = cmd.ExecuteReader();
while (reader.Read())
{
try
{
string blogName = reader.IsDBNull(0) ? string.Empty : reader.GetString(0);
string postId = reader.IsDBNull(1) ? string.Empty : reader.GetString(1);
if (string.IsNullOrWhiteSpace(blogName) || string.IsNullOrWhiteSpace(postId)) continue;
string? hasImageRaw = reader.IsDBNull(4) ? null : reader.GetValue(4)?.ToString();
bool hasImage = !string.IsNullOrWhiteSpace(hasImageRaw)
&& (hasImageRaw.Equals("true", StringComparison.OrdinalIgnoreCase)
|| hasImageRaw == "1"
|| hasImageRaw.Equals("yes", StringComparison.OrdinalIgnoreCase));
DataAccess.UpsertPostFromTextFile(
blogName: blogName,
postID: postId,
reblogURL: reader.IsDBNull(2) ? null : reader.GetString(2),
postDate: reader.IsDBNull(3) ? null : reader.GetString(3),
postURL: reader.IsDBNull(5) ? null : reader.GetString(5),
slug: reader.IsDBNull(6) ? null : reader.GetString(6),
reblogKey: reader.IsDBNull(7) ? null : reader.GetString(7),
reblogName: reader.IsDBNull(8) ? null : reader.GetString(8),
summary: reader.IsDBNull(9) ? null : reader.GetString(9),
quote: reader.IsDBNull(10) ? null : reader.GetString(10),
body: reader.IsDBNull(11) ? null : reader.GetString(11),
tags: reader.IsDBNull(12) ? null : reader.GetString(12),
link: reader.IsDBNull(13) ? null : reader.GetString(13),
photoURL: reader.IsDBNull(14) ? null : reader.GetString(14),
photoCaption: reader.IsDBNull(15) ? null : reader.GetString(15),
downloadedFiles: reader.IsDBNull(16) ? null : reader.GetString(16),
audioCaption: reader.IsDBNull(17) ? null : reader.GetString(17),
question: reader.IsDBNull(18) ? null : reader.GetString(18),
answer: reader.IsDBNull(19) ? null : reader.GetString(19),
title: reader.IsDBNull(20) ? null : reader.GetString(20),
postType: reader.IsDBNull(21) ? null : reader.GetString(21),
hasImage: hasImage);
postsUpserted++;
if (postsUpserted % 500 == 0)
Console.WriteLine($" ... {postsUpserted} posts upserted");
}
catch (Exception ex)
{
errors++;
if (errors < 20)
Console.WriteLine($" Post upsert error: {ex.Message}");
}
}
}
finally
{
DataAccess.EndImportSession();
DataAccess.RestoreImportModePragmas();
}
Console.WriteLine($" Posts upserted: {postsUpserted}");
}
catch (Exception ex)
{
Console.WriteLine($"Fatal error reading legacy posts.db: {ex.Message}");
return 1;
}
Console.WriteLine($"\n========== Legacy import summary ==========");
Console.WriteLine($"Blogs copied: {blogsCopied}");
Console.WriteLine($"Posts upserted: {postsUpserted}");
Console.WriteLine($"Errors: {errors}");
return errors == 0 ? 0 : 2;
}
}
}
+136
View File
@@ -0,0 +1,136 @@
using Microsoft.Extensions.Configuration;
namespace URLNotesGrabberCORE
{
// Port of ThreeTxtFileHelper RunOutputMode + WritePostToFile + RenameExistingTxtFilesToBak.
// For each Blog with a TTFolderPath, renames any existing .txt files in that folder to .bak,
// then writes one .txt per PostType containing all posts of that type (date-sorted, fixed
// field order). Reads from TL.db via DataAccess.GetAllPostsForBlog.
public static class OutputMode
{
public static int Run(IConfiguration config)
{
DataAccess.EnsureTTFileHelperColumnsExist();
var blogs = DataAccess.GetAllBlogsWithTTFolderPath();
Console.WriteLine($"Found {blogs.Count} blog(s) to process.");
foreach (var (blogName, ttFolderPath) in blogs)
{
Console.WriteLine($"\nProcessing blog: {blogName}");
if (string.IsNullOrWhiteSpace(ttFolderPath) || !Directory.Exists(ttFolderPath))
{
Console.WriteLine($" TTFolderPath does not exist or is not set. Skipping.");
continue;
}
Console.WriteLine($" TTFolderPath: {ttFolderPath}");
try
{
foreach (var bakFile in Directory.GetFiles(ttFolderPath, "*.bak"))
File.Delete(bakFile);
}
catch (Exception ex)
{
Console.WriteLine($" Error deleting .bak files: {ex.Message}");
}
RenameExistingTxtFilesToBak(ttFolderPath);
var posts = DataAccess.GetAllPostsForBlog(blogName);
Console.WriteLine($" Found {posts.Count} post(s) for this blog.");
var grouped = posts.GroupBy(p => p.PostType ?? "Unknown");
foreach (var typeGroup in grouped)
{
string postType = typeGroup.Key ?? "Unknown";
string outputFilePath = Path.Combine(ttFolderPath, $"{postType}.txt");
var ordered = typeGroup.OrderBy(p => p.Date).ToList();
Console.WriteLine($" Writing {ordered.Count} post(s) to {postType}.txt");
using var writer = new StreamWriter(outputFilePath, false, System.Text.Encoding.UTF8);
bool isFirst = true;
foreach (var post in ordered)
{
if (!isFirst)
{
writer.WriteLine();
writer.WriteLine();
}
WritePostToFile(writer, post);
isFirst = false;
}
}
}
Console.WriteLine("\nOutput mode complete.");
return 0;
}
private static void RenameExistingTxtFilesToBak(string folderPath)
{
try
{
foreach (var txtFile in Directory.GetFiles(folderPath, "*.txt"))
{
string bakPath = Path.ChangeExtension(txtFile, ".bak");
if (File.Exists(bakPath)) File.Delete(bakPath);
File.Move(txtFile, bakPath, overwrite: true);
}
}
catch (Exception ex)
{
Console.WriteLine($" Error renaming txt files to .bak: {ex.Message}");
}
}
private static void WritePostToFile(StreamWriter writer, TTPostRecord post)
{
var startColumns = new[] { "Post ID", "Date", "Post URL", "Slug", "Reblog Key", "Reblog URL", "Reblog Name", "Title", "Body" };
var endColumns = new[] { "Tags", "Downloaded Files" };
var columns = new Dictionary<string, string>();
if (!string.IsNullOrWhiteSpace(post.PostId)) columns["Post ID"] = post.PostId;
if (!string.IsNullOrWhiteSpace(post.Date)) columns["Date"] = post.Date!;
if (!string.IsNullOrWhiteSpace(post.PostUrl)) columns["Post URL"] = post.PostUrl!;
if (!string.IsNullOrWhiteSpace(post.Slug)) columns["Slug"] = post.Slug!;
if (!string.IsNullOrWhiteSpace(post.ReblogKey)) columns["Reblog Key"] = post.ReblogKey!;
if (!string.IsNullOrWhiteSpace(post.ReblogUrl)) columns["Reblog URL"] = post.ReblogUrl!;
if (!string.IsNullOrWhiteSpace(post.ReblogName)) columns["Reblog Name"] = post.ReblogName!;
if (!string.IsNullOrWhiteSpace(post.Title)) columns["Title"] = post.Title!;
if (!string.IsNullOrWhiteSpace(post.Body)) columns["Body"] = post.Body!;
if (!string.IsNullOrWhiteSpace(post.HasImage)) columns["Has Image"] = post.HasImage!;
if (!string.IsNullOrWhiteSpace(post.Summary)) columns["Summary"] = post.Summary!;
if (!string.IsNullOrWhiteSpace(post.Quote)) columns["Quote"] = post.Quote!;
if (!string.IsNullOrWhiteSpace(post.Link)) columns["Link"] = post.Link!;
if (!string.IsNullOrWhiteSpace(post.PhotoUrl)) columns["Photo URL"] = post.PhotoUrl!;
if (!string.IsNullOrWhiteSpace(post.PhotoCaption)) columns["Photo Caption"] = post.PhotoCaption!;
if (!string.IsNullOrWhiteSpace(post.AudioCaption)) columns["Audio Caption"] = post.AudioCaption!;
if (!string.IsNullOrWhiteSpace(post.Question)) columns["Question"] = post.Question!;
if (!string.IsNullOrWhiteSpace(post.Answer)) columns["Answer"] = post.Answer!;
if (!string.IsNullOrWhiteSpace(post.Tags)) columns["Tags"] = post.Tags!;
if (!string.IsNullOrWhiteSpace(post.DownloadedFiles)) columns["Downloaded Files"] = post.DownloadedFiles!;
foreach (var col in startColumns)
{
if (columns.ContainsKey(col))
{
writer.WriteLine($"{col}: {columns[col]}");
columns.Remove(col);
}
}
var remaining = columns.Keys.Where(k => !endColumns.Contains(k)).OrderBy(k => k).ToList();
foreach (var col in remaining)
writer.WriteLine($"{col}: {columns[col]}");
foreach (var col in endColumns)
{
if (columns.ContainsKey(col))
writer.WriteLine($"{col}: {columns[col]}");
}
}
}
}
File diff suppressed because it is too large Load Diff
@@ -2,7 +2,7 @@
"profiles": { "profiles": {
"URLNotesGrabberCORE": { "URLNotesGrabberCORE": {
"commandName": "Project", "commandName": "Project",
"commandLineArgs": "-likes timothywrite" "commandLineArgs": "--collect 1 --api4"
} }
} }
} }
+3 -2
View File
@@ -5,12 +5,12 @@ using System.Text;
using System.Threading.Tasks; using System.Threading.Tasks;
using Newtonsoft.Json; using Newtonsoft.Json;
namespace URLNotesGrabberCORE namespace URLNotesGrabberCORE
{ {
internal class ResponseNotes internal class ResponseNotes
{ {
} }
// Root myDeserializedClass = JsonConvert.DeserializeObject<Root>(myJsonResponse); // Root myDeserializedClass = JsonConvert.DeserializeObject<Root>(myJsonResponse);
public class AvatarUrl public class AvatarUrl
{ {
@@ -42,7 +42,7 @@ namespace URLNotesGrabberCORE
public class Note public class Note
{ {
public string type { get; set; } public string type { get; set; }
public int timestamp { get; set; } public long timestamp { get; set; }
public string blog_name { get; set; } public string blog_name { get; set; }
public string blog_uuid { get; set; } public string blog_uuid { get; set; }
public string blog_url { get; set; } public string blog_url { get; set; }
@@ -51,6 +51,7 @@ namespace URLNotesGrabberCORE
public AvatarUrl avatar_url { get; set; } public AvatarUrl avatar_url { get; set; }
public string post_id { get; set; } public string post_id { get; set; }
public string reblog_parent_blog_name { get; set; } public string reblog_parent_blog_name { get; set; }
public string reply_text { get; set; }
} }
public class QueryParams public class QueryParams
+110
View File
@@ -0,0 +1,110 @@
using Microsoft.Extensions.Configuration;
namespace URLNotesGrabberCORE
{
// Inverse of OutputMode. Recursively walks the PathInput tree (the same directory tree the
// no-parameter run uses) and restores every *.bak back to its *.txt, first preserving the
// current *.txt as the next-free *.bkN. Consumes the *.bak (File.Move). Filesystem-only;
// does not read the DB. An optional blogname argument filters by path substring.
public static class RevertMode
{
public static int Run(IConfiguration config, string? blogFilter = null)
{
string? root = config["appSettings:PathInput"];
if (string.IsNullOrWhiteSpace(root) || !Directory.Exists(root))
{
Console.WriteLine($"PathInput is not set or does not exist: '{root}'. Nothing to revert.");
return 0;
}
Console.WriteLine($"Searching for .bak files under: {root}");
// Recursively collect every *.bak, optionally filtered by path substring (blogname).
var bakFiles = EnumerateBakFiles(root)
.Where(f => string.IsNullOrWhiteSpace(blogFilter)
|| f.IndexOf(blogFilter, StringComparison.OrdinalIgnoreCase) >= 0)
.ToList();
if (bakFiles.Count == 0)
{
Console.WriteLine("No .bak files found. Nothing to revert.");
return 0;
}
Console.Write($"WARNING: This will restore {bakFiles.Count} .bak file(s) over their .txt files. " +
$"Current .txt files are preserved as the next-free .bkN. Continue? (yes/no): ");
string? response = Console.ReadLine();
if (string.IsNullOrWhiteSpace(response) || !response.Equals("yes", StringComparison.OrdinalIgnoreCase))
{
Console.WriteLine("Operation cancelled.");
return 0;
}
int restored = 0, backedUp = 0;
foreach (var bakFile in bakFiles)
{
try
{
string txtPath = Path.ChangeExtension(bakFile, ".txt");
if (File.Exists(txtPath))
{
string bkPath = NextFreeBkPath(txtPath);
File.Move(txtPath, bkPath);
backedUp++;
Console.WriteLine($" Backed up {Path.GetFileName(txtPath)} -> {Path.GetFileName(bkPath)}");
}
File.Move(bakFile, txtPath);
restored++;
Console.WriteLine($" Restored {bakFile} -> {Path.GetFileName(txtPath)}");
}
catch (Exception ex)
{
Console.WriteLine($" Error reverting {bakFile}: {ex.Message}");
}
}
Console.WriteLine($"\nRevert mode complete. Restored {restored} file(s); backed up {backedUp} current .txt file(s).");
return 0;
}
// Recursively yields every *.bak path under root. Per-directory try/catch so an
// inaccessible folder doesn't abort the whole walk (mirrors TraverseDirectory).
private static IEnumerable<string> EnumerateBakFiles(string path)
{
string[] subDirs;
try { subDirs = Directory.GetDirectories(path); }
catch (Exception ex)
{
Console.WriteLine($" Skipping '{path}': {ex.Message}");
yield break;
}
foreach (var dir in subDirs)
foreach (var bak in EnumerateBakFiles(dir))
yield return bak;
string[] bakFiles;
try { bakFiles = Directory.GetFiles(path, "*.bak"); }
catch (Exception ex)
{
Console.WriteLine($" Skipping files in '{path}': {ex.Message}");
yield break;
}
foreach (var bak in bakFiles)
yield return bak;
}
// Returns the lowest unused .bkN path for a given .txt file (.bk1, .bk2, ...).
private static string NextFreeBkPath(string txtFile)
{
for (int n = 1; ; n++)
{
string candidate = Path.ChangeExtension(txtFile, $".bk{n}");
if (!File.Exists(candidate)) return candidate;
}
}
}
}
@@ -29,6 +29,9 @@
<CopyToOutputDirectory>Always</CopyToOutputDirectory> <CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None> </None>
<None Update="TL.db"> <None Update="TL.db">
<CopyToOutputDirectory>Never</CopyToOutputDirectory>
</None>
<None Update="prefixes.txt">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory> <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None> </None>
</ItemGroup> </ItemGroup>
@@ -0,0 +1,70 @@
using System.Text.Json;
namespace URLNotesGrabberCORE
{
// Port of ThreeTxtFileHelper/UpdateBlogPaths.cs. Reads .tumblr / .tmblrpriv metadata
// files from a root\Index folder and populates Blogs.TTFolderPath in TL.db.
public static class UpdateBlogPathsRunner
{
public static int Run(string rootPath)
{
if (string.IsNullOrWhiteSpace(rootPath))
{
Console.WriteLine("UpdateBlogPaths: rootPath is required.");
return 1;
}
DataAccess.EnsureTTFileHelperColumnsExist();
string indexPath = Path.Combine(rootPath, "Index");
if (!Directory.Exists(indexPath))
{
Console.WriteLine($"Index folder not found at: {indexPath}");
return 1;
}
Console.WriteLine($"Scanning Index folder: {indexPath}");
var blogFiles = Directory.GetFiles(indexPath, "*.tumblr")
.Concat(Directory.GetFiles(indexPath, "*.tmblrpriv"))
.ToList();
Console.WriteLine($"Found {blogFiles.Count} blog metadata files");
int updatedCount = 0;
foreach (var blogFile in blogFiles)
{
try
{
string blogName = Path.GetFileNameWithoutExtension(blogFile);
string jsonContent = File.ReadAllText(blogFile);
using JsonDocument doc = JsonDocument.Parse(jsonContent);
JsonElement root = doc.RootElement;
if (root.TryGetProperty("FileDownloadLocation", out JsonElement locationElement))
{
string? fileDownloadLocation = locationElement.GetString();
if (!string.IsNullOrWhiteSpace(fileDownloadLocation))
{
DataAccess.SetBlogTTFolderPath(blogName, fileDownloadLocation);
updatedCount++;
Console.WriteLine($"Updated {blogName}: {fileDownloadLocation}");
}
}
else
{
Console.WriteLine($"No FileDownloadLocation found in {blogFile}");
}
}
catch (Exception ex)
{
Console.WriteLine($"Error processing {blogFile}: {ex.Message}");
}
}
Console.WriteLine($"\nUpdated {updatedCount} blogs with TTFolderPath");
return 0;
}
}
}
+30 -4
View File
@@ -1,4 +1,4 @@
{ {
"appSettings": { "appSettings": {
"PathInput": "u:\\jim\\Documents\\Web Copies\\blogs\\", "PathInput": "u:\\jim\\Documents\\Web Copies\\blogs\\",
"PathOutput": "u:\\jim\\Documents\\Web Copies\\blogs\\GetNotes.txt", "PathOutput": "u:\\jim\\Documents\\Web Copies\\blogs\\GetNotes.txt",
@@ -6,14 +6,40 @@
"PathOutputPosts": "u:\\jim\\Documents\\Web Copies\\blogs\\GetPosts.txt", "PathOutputPosts": "u:\\jim\\Documents\\Web Copies\\blogs\\GetPosts.txt",
"PathOutputBlogs": "u:\\jim\\Documents\\Web Copies\\blogs\\GetBlogs.txt", "PathOutputBlogs": "u:\\jim\\Documents\\Web Copies\\blogs\\GetBlogs.txt",
"PathOutputReplies": "u:\\jim\\Documents\\Web Copies\\blogs\\GetReplies.txt", "PathOutputReplies": "u:\\jim\\Documents\\Web Copies\\blogs\\GetReplies.txt",
"PathDB": "u:\\jim\\Documents\\Web Copies\\blogs\\TL.db", "PathOutputUrls": "u:\\jim\\Documents\\Web Copies\\blogs\\GetUrls.txt",
"PathDB": "..\\..\\..\\tl.db",
"ContainsList": "zombaee,zomb-eh,ahzombae,thebugandme,lovingbabybug,h4rdspot", "ContainsList": "zombaee,zomb-eh,ahzombae,thebugandme,lovingbabybug,h4rdspot",
"PostIDToExclude": "730084470076686336,726309940037287936,639974102120235008,184045126218" "PostIDToExclude": "730084470076686336,726309940037287936,639974102120235008,184045126218",
"EnableFileLogging": false,
"LogTraversalRecordImports": false,
"LikesRefreshCooldownDays": 7,
"PathTTRoot": "",
"PathTTBackup": "",
"PathPrefixes": "prefixes.txt",
"PathCorrectionReport": "correction_report.txt",
"PathCorrectionApplied": "correction_applied.txt"
}, },
"TumblrApi": { "TumblrApi": {
"ConsumerKey": "PtsBCGumcsgihyynxUh8b47Jfmi7uXhEIOU46bmhdsUSJ5mEP3", "ConsumerKey": "PtsBCGumcsgihyynxUh8b47Jfmi7uXhEIOU46bmhdsUSJ5mEP3",
"ConsumerSecret": "sA8BwNVTVKqBRRJmHbAD6NuyKPJ3bb9cei2bYhMgqT8cLX8tSG", "ConsumerSecret": "sA8BwNVTVKqBRRJmHbAD6NuyKPJ3bb9cei2bYhMgqT8cLX8tSG",
"OAuthToken": "HPJI6IijHoKN6WzBumG7KjS7g01iCu07jQsulpueKbWN1ZJ35J", "OAuthToken": "HPJI6IijHoKN6WzBumG7KjS7g01iCu07jQsulpueKbWN1ZJ35J",
"OAuthTokenSecret": "ajkN0Z1kKrsJaIXZEDI8zLsjRSQxFkfgO1i5k78FuQYCQjSY7w" "OAuthTokenSecret": "ajkN0Z1kKrsJaIXZEDI8zLsjRSQxFkfgO1i5k78FuQYCQjSY7w",
"PoolEnabled": true,
"Color": "Cyan"
},
"TumblrApi3": {
"ConsumerKey": "Jmoh13AS9hKBYsf939uENQsiWUuZJLFT3do4YK9P0bpspVuxkH",
"ConsumerSecret": "8LcHjuqpOS9gZZPaML1joT330w5NqyGdreSqSBazdAWCADzcms",
"OAuthToken": "Gm2esFTeb5LKsb4A2YJGrF2Udycfc0AF8afof2qNGyzNSq1qWM",
"OAuthTokenSecret": "Kfokor80s5fVue91OmNHidCdILZ9AnHtRplrct2P6Q75jgNTLa",
"PoolEnabled": false
},
"TumblrApi4": {
"ConsumerKey": "gOuydIEENkRmEvvuf57R69yPRb39FC0Egb9p6ntxWuCuFzlsV3",
"ConsumerSecret": "0my913slXAHgra4mEEYV98stJ3wXGjPTWJeviiZXwJ2AqzYtIF ",
"OAuthToken": "JHk08YufSrsetTR1Ekuh10cPOzC6rjqU5WwVu3uV4vvLzp0w8w",
"OAuthTokenSecret": "1oryr4WMBxuDrt4vL4sJQs5qNagkH3KxwIf61M8SfvklTrhHSb",
"PoolEnabled": true,
"Color": "Yellow"
} }
} }
+20
View File
@@ -0,0 +1,20 @@
Post ID
reblog URL
Date
Has Image
Post URL
Slug
Reblog Key
Reblog Name
Summary
Quote
Body
Tags
Link
Photo URL
Photo Caption
Downloaded Files
Audio Caption
Question
Answer
Title
-52
View File
@@ -1,52 +0,0 @@
@echo off
REM Batch file to run URLNotesGrabberCORE 500 times in a loop
setlocal enabledelayedexpansion
REM Set the path to the executable
REM Update this path if your executable is in a different location
set APP_PATH=URLNotesGrabberCORE.exe
REM Check if the executable exists
if not exist "%APP_PATH%" (
echo Error: %APP_PATH% not found in the current directory.
echo Please ensure the executable is in the same directory as this batch file,
echo or update the APP_PATH variable with the correct path.
pause
exit /b 1
)
REM Loop counter
set ITERATIONS=500
set COUNTER=0
echo Starting to run %APP_PATH% %ITERATIONS% times...
echo.
:LOOP
set /a COUNTER+=1
echo [%COUNTER%/%ITERATIONS%] Running iteration %COUNTER%...
echo Started at: %date% %time%
REM Run the application with -replies option
call "%APP_PATH%" -replies
REM Check if the application ran successfully
if errorlevel 1 (
echo Warning: Application exited with error code !ERRORLEVEL! on iteration %COUNTER%
) else (
echo Iteration %COUNTER% completed successfully.
)
echo Completed at: %date% %time%
echo.
REM Check if we've reached 500 iterations
if %COUNTER% lss %ITERATIONS% (
goto LOOP
)
echo.
echo Completed all %ITERATIONS% iterations!
echo.
pause
+4005
View File
File diff suppressed because one or more lines are too long
+199
View File
@@ -0,0 +1,199 @@
-- ============================================================================
-- verify-db-schema.sql
--
-- Purpose: Verify that a TL.db (e.g. a restored backup) has every column the
-- current URLNotesGrabberCORE code expects. The app has NO startup
-- migration: missing columns only get added when specific modes run,
-- and a referenced-but-missing column causes a "no such column" crash.
--
-- How to use (DB Browser for SQLite):
-- 1. File > Open Database -> pick the restored backup.
-- 2. Execute SQL tab. Run SECTION 1 (it is read-only).
-- * Zero rows from every query = schema is fully aligned, you're done.
-- * Rows in "MISSING COLUMNS" = copy the run_this_to_fix text.
-- 3. If columns are missing: KEEP A COPY OF THE BACKUP FIRST, then go to
-- SECTION 2, uncomment ONLY the ALTER lines that match the report, and run.
-- 4. Re-run SECTION 1 to confirm zero rows.
--
-- This script never UPDATEs/DELETEs/DROPs. In particular it deliberately does
-- NOT replicate the likes-reset that the app's -likes migration performs
-- (DataAccess.cs:375), so existing likes high-water marks are preserved.
-- ============================================================================
-- ============================================================================
-- SECTION 1 -- VERIFICATION (read-only)
-- ============================================================================
-- Expected schema for the current code version.
-- alter_stmt is a runnable ALTER for additively-fixable columns; for base
-- columns it is a 'MANUAL REVIEW' note (a missing base column means the backup
-- predates the table's creation or is damaged -- do not blindly auto-add).
WITH expected(tbl, col, alter_stmt) AS (
VALUES
-- Posts (base columns: manual review if missing)
('Posts','BlogName', 'MANUAL REVIEW - base/PK column missing'),
('Posts','PostID', 'MANUAL REVIEW - base/PK column missing'),
('Posts','HasNotesGathered', 'MANUAL REVIEW - base column missing'),
('Posts','reblogURL', 'MANUAL REVIEW - base column missing'),
('Posts','NotFound', 'MANUAL REVIEW - base column missing'),
('Posts','PostDate', 'MANUAL REVIEW - base column missing'),
('Posts','NotesGatheredDateTime', 'MANUAL REVIEW - base column missing'),
('Posts','HasImage', 'MANUAL REVIEW - base column missing'),
('Posts','PostURL', 'MANUAL REVIEW - base column missing'),
('Posts','Slug', 'MANUAL REVIEW - base column missing'),
('Posts','ReblogKey', 'MANUAL REVIEW - base column missing'),
('Posts','ReblogName', 'MANUAL REVIEW - base column missing'),
('Posts','Summary', 'MANUAL REVIEW - base column missing'),
('Posts','Quote', 'MANUAL REVIEW - base column missing'),
('Posts','Body', 'MANUAL REVIEW - base column missing'),
('Posts','Tags', 'MANUAL REVIEW - base column missing'),
('Posts','Link', 'MANUAL REVIEW - base column missing'),
('Posts','PhotoURL', 'MANUAL REVIEW - base column missing'),
('Posts','PhotoCaption', 'MANUAL REVIEW - base column missing'),
('Posts','DownloadedFiles', 'MANUAL REVIEW - base column missing'),
('Posts','AudioCaption', 'MANUAL REVIEW - base column missing'),
('Posts','Question', 'MANUAL REVIEW - base column missing'),
('Posts','Answer', 'MANUAL REVIEW - base column missing'),
('Posts','Title', 'MANUAL REVIEW - base column missing'),
('Posts','ByLikes', 'MANUAL REVIEW - base column missing'),
('Posts','RootBlogName', 'MANUAL REVIEW - base column missing'),
('Posts','RootURL', 'MANUAL REVIEW - base column missing'),
('Posts','DateModified', 'MANUAL REVIEW - base column missing'),
('Posts','DateCreated', 'MANUAL REVIEW - base column missing'),
-- Posts (additive migration column, auto-fixable)
('Posts','PostType', 'ALTER TABLE Posts ADD COLUMN PostType TEXT;'),
-- Blogs (base columns: manual review if missing)
('Blogs','BlogName', 'MANUAL REVIEW - base/PK column missing'),
('Blogs','HasBeenOutput', 'MANUAL REVIEW - base column missing'),
('Blogs','IsActive', 'MANUAL REVIEW - base column missing'),
('Blogs','DateAdded', 'MANUAL REVIEW - base column missing'),
('Blogs','ByLikes', 'MANUAL REVIEW - base column missing'),
('Blogs','DateModified', 'MANUAL REVIEW - base column missing'),
('Blogs','DateCreated', 'MANUAL REVIEW - base column missing'),
-- Blogs (additive migration columns, auto-fixable)
('Blogs','LikesPulled', 'ALTER TABLE Blogs ADD COLUMN LikesPulled INTEGER DEFAULT 0;'),
('Blogs','LikesCursor', 'ALTER TABLE Blogs ADD COLUMN LikesCursor INTEGER DEFAULT 0;'),
('Blogs','LikesNewestTimestamp', 'ALTER TABLE Blogs ADD COLUMN LikesNewestTimestamp INTEGER DEFAULT 0;'),
('Blogs','LikesLastRefreshed', 'ALTER TABLE Blogs ADD COLUMN LikesLastRefreshed INTEGER DEFAULT 0;'),
('Blogs','LikesLastNewCount', 'ALTER TABLE Blogs ADD COLUMN LikesLastNewCount INTEGER DEFAULT 0;'),
('Blogs','TTFolderPath', 'ALTER TABLE Blogs ADD COLUMN TTFolderPath TEXT;'),
-- Notes (base columns: manual review if missing)
('Notes','RootBlogName', 'MANUAL REVIEW - base/PK column missing'),
('Notes','PostID', 'MANUAL REVIEW - base/PK column missing'),
('Notes','NoteBlogName', 'MANUAL REVIEW - base/PK column missing'),
('Notes','TimeStamp', 'MANUAL REVIEW - base/PK column missing'),
('Notes','Type', 'MANUAL REVIEW - base/PK column missing'),
('Notes','DatetimeCrawled', 'MANUAL REVIEW - base column missing'),
('Notes','DateModified', 'MANUAL REVIEW - base column missing'),
('Notes','DateCreated', 'MANUAL REVIEW - base column missing'),
-- Notes (additive migration column, auto-fixable)
('Notes','replyText', 'ALTER TABLE Notes ADD COLUMN replyText TEXT DEFAULT ''.'';'),
-- DailyAPICount (base columns)
('DailyAPICount','Date', 'MANUAL REVIEW - base/PK column missing'),
('DailyAPICount','APICount', 'MANUAL REVIEW - base column missing'),
-- ApiKeyPoolState (created at runtime by EnsureApiKeyPoolTables; auto-fixable by re-running app, but safe to add)
('ApiKeyPoolState','KeyName', 'MANUAL REVIEW - run app once to auto-create ApiKeyPool tables'),
('ApiKeyPoolState','RetryUntil', 'MANUAL REVIEW - run app once to auto-create ApiKeyPool tables'),
('ApiKeyPoolMeta','Id', 'MANUAL REVIEW - run app once to auto-create ApiKeyPool tables'),
('ApiKeyPoolMeta','LastIndex', 'MANUAL REVIEW - run app once to auto-create ApiKeyPool tables')
),
actual(tbl, col) AS (
SELECT 'Posts', name FROM pragma_table_info('Posts')
UNION ALL SELECT 'Blogs', name FROM pragma_table_info('Blogs')
UNION ALL SELECT 'Notes', name FROM pragma_table_info('Notes')
UNION ALL SELECT 'DailyAPICount', name FROM pragma_table_info('DailyAPICount')
UNION ALL SELECT 'ApiKeyPoolState', name FROM pragma_table_info('ApiKeyPoolState')
UNION ALL SELECT 'ApiKeyPoolMeta', name FROM pragma_table_info('ApiKeyPoolMeta')
)
-- 1a. MISSING COLUMNS: columns the code needs that the DB does not have.
-- Zero rows = good. Otherwise copy run_this_to_fix into SECTION 2.
SELECT
e.tbl AS table_name,
e.col AS missing_column,
e.alter_stmt AS run_this_to_fix
FROM expected e
LEFT JOIN actual a
ON a.tbl = e.tbl AND lower(a.col) = lower(e.col)
WHERE a.col IS NULL
ORDER BY (e.alter_stmt LIKE 'ALTER%') DESC, e.tbl, e.col;
-- 1b. MISSING TABLES: expected tables that don't exist at all in this DB.
-- Zero rows = good.
WITH expected_tables(tbl) AS (
VALUES ('Posts'),('Blogs'),('Notes'),('DailyAPICount'),
('ApiKeyPoolState'),('ApiKeyPoolMeta')
)
SELECT et.tbl AS missing_table
FROM expected_tables et
WHERE NOT EXISTS (
SELECT 1 FROM sqlite_master
WHERE type = 'table' AND lower(name) = lower(et.tbl)
)
ORDER BY et.tbl;
-- 1c. EXTRA / UNEXPECTED COLUMNS: present in the DB but not in the expected
-- list above. Informational only -- e.g. a NEWER backup, or a column this
-- script's expected-list hasn't been updated for. Not an error by itself.
WITH expected(tbl, col) AS (
VALUES
('Posts','BlogName'),('Posts','PostID'),('Posts','HasNotesGathered'),('Posts','reblogURL'),
('Posts','NotFound'),('Posts','PostDate'),('Posts','NotesGatheredDateTime'),('Posts','HasImage'),
('Posts','PostURL'),('Posts','Slug'),('Posts','ReblogKey'),('Posts','ReblogName'),('Posts','Summary'),
('Posts','Quote'),('Posts','Body'),('Posts','Tags'),('Posts','Link'),('Posts','PhotoURL'),
('Posts','PhotoCaption'),('Posts','DownloadedFiles'),('Posts','AudioCaption'),('Posts','Question'),
('Posts','Answer'),('Posts','Title'),('Posts','ByLikes'),('Posts','RootBlogName'),('Posts','RootURL'),
('Posts','DateModified'),('Posts','DateCreated'),('Posts','PostType'),
('Blogs','BlogName'),('Blogs','HasBeenOutput'),('Blogs','IsActive'),('Blogs','DateAdded'),
('Blogs','ByLikes'),('Blogs','DateModified'),('Blogs','DateCreated'),('Blogs','LikesPulled'),
('Blogs','LikesCursor'),('Blogs','LikesNewestTimestamp'),('Blogs','LikesLastRefreshed'),
('Blogs','LikesLastNewCount'),('Blogs','TTFolderPath'),
('Notes','RootBlogName'),('Notes','PostID'),('Notes','NoteBlogName'),('Notes','TimeStamp'),
('Notes','Type'),('Notes','DatetimeCrawled'),('Notes','DateModified'),('Notes','DateCreated'),
('Notes','replyText'),
('DailyAPICount','Date'),('DailyAPICount','APICount'),
('ApiKeyPoolState','KeyName'),('ApiKeyPoolState','RetryUntil'),
('ApiKeyPoolMeta','Id'),('ApiKeyPoolMeta','LastIndex')
),
actual(tbl, col) AS (
SELECT 'Posts', name FROM pragma_table_info('Posts')
UNION ALL SELECT 'Blogs', name FROM pragma_table_info('Blogs')
UNION ALL SELECT 'Notes', name FROM pragma_table_info('Notes')
UNION ALL SELECT 'DailyAPICount', name FROM pragma_table_info('DailyAPICount')
UNION ALL SELECT 'ApiKeyPoolState', name FROM pragma_table_info('ApiKeyPoolState')
UNION ALL SELECT 'ApiKeyPoolMeta', name FROM pragma_table_info('ApiKeyPoolMeta')
)
SELECT a.tbl AS table_name, a.col AS unexpected_column
FROM actual a
LEFT JOIN expected e
ON e.tbl = a.tbl AND lower(e.col) = lower(a.col)
WHERE e.col IS NULL
ORDER BY a.tbl, a.col;
-- ============================================================================
-- SECTION 2 -- FIX (opt-in, additive only)
--
-- Run ONLY the lines that query 1a flagged with an ALTER statement.
-- KEEP A COPY OF THE BACKUP FIRST. SQLite has no "ADD COLUMN IF NOT EXISTS",
-- so running an ALTER for a column that already exists throws a harmless
-- "duplicate column name" error and changes nothing -- just run the flagged
-- subset. These are the 8 additive migration columns and nothing else; the
-- likes high-water-mark reset is intentionally NOT included.
-- ============================================================================
-- ALTER TABLE Posts ADD COLUMN PostType TEXT;
-- ALTER TABLE Blogs ADD COLUMN LikesPulled INTEGER DEFAULT 0;
-- ALTER TABLE Blogs ADD COLUMN LikesCursor INTEGER DEFAULT 0;
-- ALTER TABLE Blogs ADD COLUMN LikesNewestTimestamp INTEGER DEFAULT 0;
-- ALTER TABLE Blogs ADD COLUMN LikesLastRefreshed INTEGER DEFAULT 0;
-- ALTER TABLE Blogs ADD COLUMN LikesLastNewCount INTEGER DEFAULT 0;
-- ALTER TABLE Blogs ADD COLUMN TTFolderPath TEXT;
-- ALTER TABLE Notes ADD COLUMN replyText TEXT DEFAULT '.';
+1
View File
@@ -0,0 +1 @@
ollama launch opencode --model qwen-coder-32k