Commit Graph
146 Commits
Author SHA1 Message Date
jim c5bdd93708 Merge branch 'claude/blogs-likes-criteria-5dca9a' into master 2026-10-21 13:38:34 -05:00
jimandClaude Opus 5.5 d16ab14691 feat(likes): show blog progress count and percentage
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-21 13:38:33 -05:00
jimandClaude Opus 5.5 3448d75030 fix: trim ContainsList entries and drop empty ones
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-21 13:37:14 -05:00
jim 445b52f0c8 Merge branch 'claude/blogs-likes-criteria-5dca9a' into master 2026-10-21 13:31:38 -05:00
jimandClaude Opus 5.5 c5676eb084 fix(likes): select all active blogs, not only self-rebloggers
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-21 13:31:28 -05:00
jimandClaude 7557c9a168 feat(likes): accept multiple blog names with --likes
Change --likes from accepting a single blog name to accepting a list.
Passes all non-flag args after --likes as blog names. Database stores
bare blog names, so users pass just the name without .tumblr.com.

Co-Authored-By: Claude <[email protected]>
2026-10-19 13:41:05 -05:00
jim ca99586176 Merge branch 'claude/collect-query-count-output-c430c8' into master 2026-10-06 15:29:26 -05:00
jimandClaude Opus 5.5 f40d824a8c feat(collect): print candidate row count after the query
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-06 15:29:18 -05:00
jim bde6506e01 Merge branch 'claude/friendly-bhaskara-b06186' into master 2026-10-02 09:26:26 -05:00
jimandClaude Opus 5.5 06a3f8c142 fix(traverse): import the last no-download record in each file
TraverseDirectory imported a record without downloaded files only when
the next "Post id:" line appeared, so the final record of every export
file was never evaluated and a matching post at the end of a file never
reached Posts.

Move the check into a local function and call it again after the line
loop, so the last record is flushed under exactly the same conditions as
every other one. HasImage values at both call sites are unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 09:26:21 -05:00
jim 49e581c136 Merge branch 'claude/post-modifieddate-rescan-f25914' into master 2026-10-02 09:15:53 -05:00
jimandClaude Opus 5.5 e18fb5ca10 fix(posts): make HasImage raise-only to stop rescan churn
TraverseDirectory's two AddPost call sites pass opposite hasImage values
for one post (duplicate _N folders, /blog/private bodies on own-blog
reblogs), and --ingest does the same across duplicate files. The
overwrite flipped the flag 0->1->0 and bumped DateModified every pass.
UpdatePost, AddPost's fallback and UpsertPostFromTextFile now only raise
HasImage, matching ByLikes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 09:15:42 -05:00
jim 5af284a49f Merge branch 'claude/blognames-table-design-0573fb' into master 2026-09-28 11:33:29 -05:00
jimandClaude Opus 5.5 8fe2ffeb96 fix(db): make Blogs.BlogId the only blog ID and drop BlogNames
Blogs.BlogId was a one-time copy of BlogNames and nothing kept it
current: 12,238 blogs first seen after 2026-08-07 had a BlogNames ID
but a NULL Blogs.BlogId, so GetBlogs' join on BlogId silently skipped
them and their 23,148 notes.

- retire-blognames.sql: stub Blogs rows for the 17 unregistered note
  participants, backfill IDs (none renumbered), make ix_Blogs_BlogId
  UNIQUE, drop BlogNames, and add triggers that stop a Blogs row with
  a BlogId from being deleted, renamed or renumbered
- AddNote registers both blogs via RegisterBlog (Blogs row + MAX+1 ID)
  and every query resolves names through Blogs instead of BlogNames
- verify-db-schema.sql reports a DB that still has BlogNames (1e)
- Update TL.db.md, AGENTS.md and the DB Browser saved queries

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-28 11:33:04 -05:00
jim c8c43c4918 Merge branch 'claude/datetime-format-consistency-b68faa' into master 2026-09-21 14:07:29 -05:00
jimandClaude Opus 5 2948a4aff0 fix(posts): normalize PostDate to yyyy-MM-dd HH:mm:ss GMT
Text-file exports can carry RFC 1123 dates ("Fri, 14 Feb 2025 15:20:09 GMT"),
which sort on the weekday name and fall outside --fromDate/--toDate
comparisons. PostDates.Normalize converts them on every PostDate write path
(AddPost, UpsertPostFromTextFile, UpdatePostContentFields).
normalize-postdate.sql fixes the 4 existing rows.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-21 14:07:05 -05:00
jimandClaude Sonnet 5 b8231d6a4c Merge branch 'claude/session-fa480e' into master
fix(collect): exclude posts from IsActive=0 blogs in --collect 1

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-09-03 14:35:57 -05:00
jimandClaude Sonnet 5 bbf05b3863 fix(collect): exclude posts from IsActive=0 blogs in --collect 1
Blogs.IsActive is the crawler's work-selection flag (Rolodex removal
sets it to 0) and is independent of Posts.IsActive/Notes.IsActive --
deactivating a blog never touches its posts' own IsActive column, so
--collect 1 kept re-queuing posts for blogs that had been deactivated.

Join Blogs into the PostsWithCount CTE's source filter and require
COALESCE(BL.IsActive, 1) = 1. Since the hardcoded zomb-eh re-queue
branch reads from PostsWithCount rather than Posts directly, it now
inherits this filter automatically -- if zomb-eh is ever deactivated,
its rows disappear from PostsWithCount and the union branch
contributes nothing, with no special-case code needed.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-09-03 14:34:57 -05:00
jim ea2afc9d35 feat(collect): add --toDate upper bound for zomb-eh re-queue branch
Mirrors --fromDate: bounds the zomb-eh periodic re-queue branch by
PostDate <= the given date. Stacks alongside --fromDate and the
cooldown clause rather than replacing either, so --fromDate/--toDate
and --force can all be combined, and each applies with or without
--force.

Named --toDate rather than --end to pair with --fromDate -- --start
already exists as an unrelated flag (resume folder traversal at a
blog name).
2026-08-25 12:51:32 -05:00
jim 7bf270e47d feat(collect): add --fromDate lower bound for zomb-eh re-queue branch
The zomb-eh periodic re-queue branch in GetPosts had no lower bound on
the post's original PostDate -- it re-queued every already-collected
zomb-eh post past the 3-day cooldown, regardless of age.

Add --fromDate <datetime> to bound that branch by PostDate >= the given
date. It stacks with the existing cooldown clause rather than replacing
it, so it applies the same way whether or not --force also drops the
cooldown.
2026-08-24 14:42:43 -05:00
jim 58d7b1d05e Merge branch 'claude/collect-command-issue-09fcb8' into master 2026-08-23 04:34:50 -05:00
jimandClaude Opus 5 f41957fd2f feat(collect): let --force ignore the re-collect cooldown
--collect 1 <blog> returned an empty worklist whenever the periodic
re-queue branch was still inside its 3-day window, with no way to ask
for the re-collect early. GetPosts now composes that age predicate
conditionally, and the existing global --force flag - already "ignore
cooldown" for --likes - drives it.

Only the age gate drops: NotFound = 0, the IsActive filter and the blog
scoping still apply. The flag is a no-op for mode 0, which re-collects
every post regardless, and says so rather than pretending to act.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-23 04:34:45 -05:00
jimandClaude Sonnet 5 a28c5cc9ec fix(dbbrowser): rewrite saved queries for the Notes integer schema
Notes.RootBlogName/NoteBlogName/Type were replaced by RootBlogId/NoteBlogId/
TypeId (resolved via BlogNames and NoteTypes) back on 2026-08-07. The saved
DB Browser for SQLite queries in RERUN.sqbpro and TL.sqbpro still referenced
the old text columns and failed against the migrated TL.db.

Rewrote the 5 affected queries per TL.db.md's porting guide: Notes<->Blogs
joins go through Blogs.BlogId in one hop, Notes<->Posts joins route through
BlogNames (Posts has no BlogId), and type filters resolve through NoteTypes.
Verified read-only against the live TL.db -- all 5 execute without error.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-20 09:07:08 -05:00
jim d6a96f7885 Merge branch 'claude/posttype-noargs-backfill' into master 2026-08-20 08:15:54 -05:00
jimandClaude Opus 5 864b468d96 fix(posttype): run the migration and backfill in no-args mode
The backfill hung off --ingest, --output, --correct, --importposts and
--updatepaths, because those were where EnsureTTFileHelperColumnsExist was
already being called. But the no-argument traversal is the mode that
actually gets run day to day, and it called none of them -- so the command
used most often was the one command that never repaired an untyped row.

TraverseDirectory already types the posts it inserts, from the filename it
is reading. This closes the other half: the rows already sitting untyped
now get fixed by an ordinary run, with no separate maintenance command.

Called before BeginImportSession so it uses its own connection rather than
contending with the import session's, and before the traversal so existing
rows are typed first and newly inserted ones arrive already typed.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-20 08:15:54 -05:00
jim e6a3efba5b Merge branch 'claude/posttype-unknown-fix' into master 2026-08-20 08:01:01 -05:00
jimandClaude Opus 5 34da632e6a fix(posttype): stop Unknown.txt and type posts at their source
PostType becomes an output filename, so an unset or unvalidated value does
not stay a data problem -- it creates a file. OutputMode wrote untyped rows
to `PostType ?? "Unknown"`, IngestMode read that file back and derived the
literal type "Unknown" from its name, and the two would have regenerated
each other indefinitely.

Nothing was setting the type in the first place. AddPost -- the path every
notes/likes harvest goes through -- omitted PostType from its INSERT column
list entirely, so 1790 rows across 469 blogs had none. Ingest could never
repair them: it types a post only when it meets it inside a real export
file, and these posts appear in none.

Type at the source, from what each path actually knows:

- TraverseDirectory takes it from the filename it is already reading
  ("texts.txt" -> "texts"), the same rule ingest uses.
- CollectLikes has no file, so it reads the legacy-format `type` field that
  GrabLikes already requests with npf=false, mapped singular -> plural.
- AddPost/UpdatePost gained the plumbing to carry it. UpdatePost fills a
  missing type but never overwrites one, and its change-detection clause
  had to learn about PostType or the SET would be unreachable for a row
  whose content was already current.

PostTypes is the single source of truth: eight canonical names, and
anything else normalizes to null. Null is safe -- OutputMode skips those
rows -- while a stray value would have become a stray file. IngestMode and
TraverseDirectory now skip non-export .txt files outright, and the legacy
importer no longer passes a pre-column NULL straight back in.

For the rows already stored untyped, content is the only signal left, so
the migration infers from which columns they carry. Verified against a copy
of the live database: 1780 of 1790 typed, 10 left untyped for want of any
content at all, idempotent on a second pass. HasImage is deliberately not
consulted -- it is set on 12,420 of 19,828 known text posts.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-20 08:00:56 -05:00
jim 954ec353a5 Merge branch 'claude/collect-pause-time-bc9610' into master 2026-08-08 22:40:30 -05:00
jimandClaude Opus 5 c9530a3718 perf(collect): halve the new-note console pause
Cut the brief pause after printing a newly inserted note in green from
250ms to 125ms so --collect runs move faster while the highlight is
still noticeable.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-08 22:40:24 -05:00
jim 3d61cbb6ea Merge branch 'claude/collect-1-query-sorting-e12f6c' into master 2026-08-08 21:47:45 -05:00
jimandClaude Opus 5 0d9641db08 feat(collect): add an optional blog filter to --collect
"--collect 1 zomb-eh" now restricts the run to a single blog. The name is
bound as a SQLite parameter and matched exactly against Posts.BlogName,
which leads the primary key, so the predicate uses that index.

Trailing arguments are scanned rather than positionally fixed: a token that
parses as a date is the cutoff, anything else is the blog name, in either
order. --blog=name forces the blog reading for a name that would otherwise
parse as a date.

The filter is a pure filter -- the zomb-eh 3-day refresh branch reads from
the already-scoped PostsWithCount CTE, so a filtered worklist is a strict
subset of the unfiltered one. Verified against TL.db: --collect 1 returns
651 posts, of which --collect 1 zomb-eh returns exactly its 164 and
--collect 1 cs1d3blog exactly its 18.

Blog-scoped runs are excluded from managed-run state, so collecting one
blog cannot mark a full re-check complete.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-08 21:47:38 -05:00
jimandClaude Opus 5 b31d5842cc feat(db)!: port DataAccess to the Notes integer schema
Notes.RootBlogName/NoteBlogName/Type became RootBlogId/NoteBlogId/TypeId
on 2026-08-07, resolved through the new BlogNames and NoteTypes tables.
There is no compatibility view, so every affected statement is a hard cut.

All 14 call sites in DataAccess.cs are ported:

- Notes->Blogs joins go through Blogs.BlogId in one integer hop; the
  Notes->Posts join in GetRepliesWithFilledText is the only one that must
  route through BlogNames, since Posts carries no BlogId
- AddNote registers both blog names and the note type with INSERT OR
  IGNORE before inserting, in one transaction committed before the console
  sleep. Registering the type matters: an unseen type would resolve to
  NULL and fail NOT NULL on TypeId, silently losing the note
- The LEFT JOIN Notes in GetPosts is dropped rather than translated. It
  selected nothing, could not remove a row, and its duplicates were
  collapsed by the query's own GROUP BY
- Duplicate-key detection moves to IsNotesDuplicateKey, matching the
  constraint and table instead of an exact column list. The old literal
  string is what broke on this rename
- EnsureReplyTextColumnExists drops DEFAULT '.', matching the migrated
  schema: new rows get NULL, not a placeholder nobody wrote

verify-db-schema.sql gains BlogNames, NoteTypes, Blogs.BlogId and the new
Notes columns, plus query 1d naming a pre-migration file and pointing at
normalize-notes.sql. Blogs.BlogId is deliberately not auto-fixable -- an
added-but-empty column makes engagement joins return zero rows silently.

Verified against the live 148 MB file: query plans hit the intended
indexes, and the BlogId join matches an independent name-resolved
formulation exactly on all 4,267 GetBlogs and 2,637 GetBlogsForLikes rows.

RolodexRepository.cs (16 sites) lives in the Rolodex repo and is not
covered here.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-07 21:23:13 -05:00
jim c3cf89c3f1 Merge branch 'claude/normalize-notes' into master 2026-08-07 20:54:24 -05:00
jimandClaude Opus 5 ab36085ba8 feat(db)!: replace blog names and note types in Notes with integer IDs
BREAKING CHANGE: Notes.RootBlogName, Notes.NoteBlogName and Notes.Type no
longer exist. They are RootBlogId, NoteBlogId and TypeId, resolved through two
new lookup tables. Every Notes query in this repo and in Rolodex fails against
a migrated database until rewritten. Neither application is ported yet.
TumblThree is unaffected -- it touches only Blogs.

Takes TL.db from 207 MB to 148 MB (-29%); cumulative with this morning's
WITHOUT ROWID change, 267 MB to 148 MB (-45%). The names were text repeated
across 1.18M rows, in the table and again in every index over it.

  BlogNames(BlogId, BlogName)   20,430 rows, the ID authority
  NoteTypes(TypeId, Type)       5 rows, a table rather than a CHECK so a new
                                type is an INSERT not a migration
  Blogs.BlogId                  new, additive, NULL on the 168,202 blogs with
                                no notes

Blogs.BlogId exists so Notes reaches Blogs in one integer hop instead of going
through BlogNames and ending in the text comparison this change was meant to
remove. It costs 2 MB and is purely additive, which is what leaves TumblThree
untouched.

BlogNames is built from Notes rather than from Blogs, deliberately: 12 engagers
have no registry row, and sourcing it from Blogs would have dropped their notes
through the migration's inner joins.

Proven lossless before and after applying to the live file: the old text shape
was reconstructed from the new schema and diffed against the pre-migration
database in both directions. Zero rows differed either way across all 1,182,333
rows and all ten columns. integrity_check ok, journal_mode still wal.

A view-plus-INSTEAD-OF-triggers compatibility shim was built and measured
first. It worked completely -- reads, INSERT OR IGNORE dedup, both apps' update
paths, cross-table transactions -- but cost 194 ms to 321 ms on Rolodex's
unfiltered Notes page, and a clean break was chosen over carrying it.

TL.db.md gains a "Porting to the integer schema" section: column mapping and
the old-to-new form of every query shape the two applications use, including
the INSERT-OR-IGNORE-into-BlogNames-first pattern for notes naming a blog that
has no ID yet. Roughly 14 call sites in DataAccess.cs, 16 in
RolodexRepository.cs. Every documented snippet was executed against the live
file. Also flags that the duplicate-key error string DataAccess.cs matches on
at two sites now names the new columns and will no longer match.

Unrelated corrections found while refreshing the counts, all of which had
drifted on their own: Posts.PostType is no longer NULL on every row but
populated on 20,679 of 22,468, which invalidates the stated reason both this
document and Rolodex derive post type from content instead of reading it; the
Posts.IsActive and Notes.IsActive columns described as "not in this database
yet" both exist; and the registry is 188,620 blogs, not 144,367.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-07 20:54:24 -05:00
jimandClaude Opus 5 4d37999f8e chore: rotate the TL.db backup archive and refresh local state
Replaces TL 20251212.7z with TL 20260807.7z, taken after today's shrink of
TL.db from 267 MB to 207 MB. Also picks up the RERUN.sqbpro working state and
.claude/tl.db.

Note that .claude/tl.db is not covered by .gitignore, which currently excludes
only .claude/settings.local.json and .claude/worktrees/. Rolodex ignores the
whole .claude/ directory; this repo may want the same.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-07 20:27:23 -05:00
jim 387c023900 Merge branch 'claude/shrink-tl-db' into master 2026-08-07 20:20:29 -05:00
jimandClaude Opus 5 a9bd5a4c37 perf(db): shrink TL.db from 267 MB to 207 MB
The file was already tight -- freelist 0 pages, and a plain VACUUM reclaimed
nothing -- so the saving had to come from schema rather than compaction.
Profiled with dbstat and measured every step on copies of the live file.

Three changes to Notes, applied 2026-08-07:

- Rebuild as WITHOUT ROWID (-32 MB). The 5-column composite primary key was
  stored twice: once in the table, once in a 62 MB autoindex existing only to
  map key -> rowid. Keying the table b-tree on the primary key itself drops the
  second copy. ix_NoteBlogName01 grows 25 -> 58 MB in exchange, since a
  secondary index on such a table carries the whole primary key instead of a
  rowid; net -32 MB.

- Drop Notes_idx_06e01ae3 on TimeStamp DESC (-14 MB). Barely earned its keep as
  a rowid index and would have cost 58 MB after the conversion, cancelling the
  entire exercise. The crawler's only TimeStamp filter (>= 1535778000) excludes
  786 of 1,182,333 rows; Rolodex's default Notes sort carries a three-column
  tiebreaker forcing a full sort regardless; the reply-matching UPDATE uses
  ABS(TimeStamp - ?) <= 5, which no index on the column can serve. Cost is one
  path: Rolodex's Notes page with a date-range filter, 60 ms -> 164 ms.

- Null the DatetimeCrawled placeholder (-13 MB). 1,148,077 rows stored the
  literal DDL default '2/12/26 12am', a backfill marker rather than a crawl
  time. UI-neutral: Rolodex reads the column through DateSql.Sortable, whose
  CASE matches neither format, so those rows already rendered as an em dash.

No application code changed. The schema keeps the same tables, columns, types
and constraints; WITHOUT ROWID is a storage-layout change behind the same SQL
surface, and no consumer referenced rowid on Notes.

Verified against all three consumers on the live file: integrity_check ok, row
counts unchanged (1182333 / 22468 / 188620), journal_mode still wal, crawler
INSERT OR IGNORE still dedupes, Rolodex's NoteBlogName filter still uses
ix_NoteBlogName01, and exactly as many rows read as null through Sortable after
the change as before it. TumblThree touches only Blogs, which is untouched.

Deliberately not done: nulling Notes.DateCreated (a further -12 MB). Unlike
DatetimeCrawled its value parses as a real date, so Rolodex displays and sorts
by it; nulling would turn visible dates into em dashes.

Note that DEFAULT '2/12/26 12am' remains on the column, so any writer inserting
a note without naming it reintroduces the placeholder. Consumer-side date
normalisation must stay.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-07 19:47:15 -05:00
jim 70b32dfc89 Merge branch 'claude/ttfolderpath-not-set-8c9b45' into master 2026-08-05 12:09:55 -05:00
jimandClaude Opus 5 8f4177a0c9 feat: fold the TTFolderPath refresh into --output
TL.db syncs between machines whose absolute paths differ, so a single
TTFolderPath column cannot be correct on both at once -- the stored paths are
only trustworthy on the machine that wrote them. That made --updatepaths a
mandatory prelude to every --output rather than the one-time setup step it
looks like.

--output now refreshes the column from the TumblThree Index metadata before
exporting. The root comes from the first non-flag argument, else
appSettings:PathTTRoot. With no root available it says so and exports whatever
TL.db already holds; a root whose Index folder is missing is a hard stop, since
silently exporting stale paths is the failure this change exists to prevent.
--norefresh skips the refresh for a pure export.

The scan is extracted from UpdateBlogPathsRunner.Run into a reusable Scan() that
returns counts instead of only printing them, so --updatepaths keeps its
per-file detail while --output prints a single summary line rather than a few
hundred lines ahead of the export.

Verified against a throwaway database: a stale cross-machine path is repaired
and the export lands in the correct local folder; no configured root warns and
continues (exit 0); a missing Index folder stops (exit 1); --norefresh skips the
refresh and exports (exit 0).

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-05 12:09:55 -05:00
jim a2763d0026 Merge branch 'claude/ttfolderpath-not-set-8c9b45' into master 2026-08-05 12:02:58 -05:00
jimandClaude Opus 5 721224bc13 fix: make --output and --updatepaths tell the truth about TTFolderPath
--output iterated all 156k active Blogs rows and printed a "does not exist or is
not set" skip line for each, which is nearly every blog in the crawl registry --
only the few hundred downloaded locally ever have a folder. The signal was
buried in six figures of noise.

GetAllBlogsWithTTFolderPath now selects only active blogs carrying a non-empty
path, so --output processes export targets and nothing else. When none exist it
says so once, names the database it read, points at --updatepaths, and returns
non-zero instead of reporting success. A stored path this machine cannot see is
now reported separately from an unset one, with the path shown, because the two
are fixed in different places. Paths are trimmed before Directory.Exists, which
stray whitespace in a .tumblr FileDownloadLocation would otherwise defeat.

Both writers counted optimistically. UpdateBlogPathsRunner printed its
per-blog success line and incremented its total from the metadata file parsing,
never checking whether the UPDATE matched a row; LegacyPostsDbImporter counted a
blog as copied even when the legacy TTFolderPath was NULL. Either could report
full success having written nothing -- which is consistent with TL.db holding
zero populated paths across all 156,492 active blogs despite 20,679 posts having
merged. SetBlogTTFolderPath now returns whether a row changed, and both callers
report written / already-correct / no-matching-row separately.

Verified against a throwaway database: no-paths case, export case (stale .txt
rotated to .bak, per-PostType files, date-sorted), missing-folder case,
--updatepaths honest counts, and an idempotent rerun reporting already-correct.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-05 12:02:49 -05:00
jim ef6629d86a Merge branch 'claude/sqlite-modified-date-logic-3d9bee' into master 2026-08-05 09:36:38 -05:00
jimandClaude Opus 5 6320e2c0c9 fix: stop --ingest's NULL sentinel from clobbering post content
UpsertPostFromTextFile (the persistence layer under --ingest) uses NULL as
its "this file's record had no line for that field" sentinel -- the direct
analog of the "." convention just fixed in UpdatePost. IngestMode strips a
trailing "_N" off the folder name before it ever reaches this function, so
a duplicate export folder deliberately collapses onto the same BlogName --
reconciling multiple differently-formatted files for one post is the whole
point of --ingest. Files are walked in raw filesystem enumeration order,
never sorted, so which file's call lands last for a given (BlogName,
PostID) is arbitrary.

The UPDATE branch set every column unconditionally, so whichever file
processed last for a PostID nulled out every field its own record didn't
carry, silently erasing real content another file had. Worse than the "."
case: that one only caused churn (two writes cancelling out); this one
loses data, in an order that depends on filesystem enumeration.

Every content column is now guarded the same way, NULL instead of "." as
the sentinel: `col = CASE WHEN @col IS NULL THEN col ELSE @col END` in the
SET list, `(@col IS NOT NULL AND IFNULL(col,'') <> @col) OR ...` in the
change-detection. Narrow the same way: only a missing line (NULL) is the
sentinel -- G() already distinguishes that from present-but-blank (""), so
an explicit empty field still overwrites.

HasImage is deliberately left unguarded and documented as a known gap:
IngestMode always computes a concrete bool, defaulting false when a file
has no "Has Image:" line, so this function can't currently tell "no image"
from "not reported" without changing the parameter to bool? and threading
that through IngestMode/LegacyPostsDbImporter too.

Verified against a throwaway DB using the exact SQL text and parameter
binding: a full-format record's real Title/Slug/Tags now survive a
same-PostID partial record whose format doesn't carry those fields, in
both file orders, while a genuine content change and an explicit empty
value still write and still move DateModified.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-05 09:36:17 -05:00
jim 6136901cc7 Merge branch 'claude/sqlite-modified-date-logic-3d9bee' into master 2026-08-05 09:17:48 -05:00
jimandClaude Opus 5 83e35a2323 fix: stop "." export sentinel from clobbering real post content
ReblogRecord (TraverseDirectory's .txt-export parser) and the --likes API
path both default every content field to the literal "." when their source
has no value for that field, then pass it straight into UpdatePost. A blog
with two export folders in different field formats -- a duplicate "_2"
folder, or an export whose field set changed over time -- sends one record
with real Title/Tags/Slug and another with those fields "." because that
format never had a line for them. Re-importing both on every run flipped
the row back and forth forever: net content never changed, but
DateModified moved on every pass since each write really did change a
column relative to the other write, just not relative to the true value.

Every content column in UpdatePost's SET list is now guarded the same way
RootBlogName/RootURL already were -- a "." parameter leaves the existing
value alone instead of overwriting it -- and the change-detection WHERE
clause carries the same exception, so a "."-only difference no longer
fires the UPDATE at all. Deliberately narrow: only the literal "." is the
sentinel, so an explicit empty string from a real record still overwrites.

Verified against a throwaway DB using the exact SQL text and parameter
binding from UpdatePost, reproducing the an-angry-wolf/adore-blk scenario
found in the live DB: re-importing conflicting "." records now writes zero
rows and leaves DateModified untouched, while a genuine content change
still fires and still moves it.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-05 09:17:20 -05:00
jim f0ccac6503 Merge GitTea/master into master 2026-08-03 08:58:07 -05:00
jimandClaude Opus 5 e1d2eb48c2 fix: only bump DateModified when a value actually changed
Seven UPDATE statements wrote DateModified unconditionally, so re-crawling
or re-ingesting identical content marked Blogs, Posts and Notes rows as
modified. Each now carries a WHERE guard covering every column in its SET
list, so SQLite matches zero rows on a no-op.

Guarded: AddPost's insert-failure fallback and blog stamp, AddNote's blog
stamp, UpdateBlogLikesNewestTimestamp, UpdateNoteReplyText,
UpsertPostFromTextFile, SetBlogTTFolderPath, UpdatePostContentFields.

Also:
- Blogs.DateAdded is no longer rewritten when a new post arrives for a
  known blog. A new post is not a new blog, and rewriting the column both
  destroyed the registration date and made every insert look like a change.
- Posts.NotesGatheredDateTime is crawl bookkeeping that moves on every
  pass, so it no longer moves DateModified on its own. It is still written
  each pass, but the timestamp is wrapped in a CASE on the pre-UPDATE
  HasNotesGathered value so only the flag flipping counts.

These statements now return 0 rows for "found but unchanged" as well as
"not found"; CorrectMode's postsUpdated tally consequently counts rows
actually changed, matching what its dry-run diff reports.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-03 08:53:10 -05:00
jim 3c85a05afc Merge branch 'master' into claude/jovial-mayer-77c2b5 2026-07-29 16:26:50 -05:00
jim 60912c882d fix: stop AddAPICount from throwing on missing DateCreated column
The INSERT named a DateCreated column that DailyAPICount (Date, APICount)
never had, so every call threw "no such column: DateCreated" into an
empty catch block. Today's row was never created and the tally sat idle
since 2026-04-13. Drop the column from the INSERT, and report the three
silent failure points (insert error, missing row after insert, update
matching zero rows) instead of swallowing them.
2026-07-29 16:26:08 -05:00
jim 8a4ab2402d docs: record the IsActive no-write rule in AGENTS.md 2026-07-29 16:04:08 -05:00