16 Commits
Author SHA1 Message Date
jimandClaude Opus 5 b31d5842cc feat(db)!: port DataAccess to the Notes integer schema
Notes.RootBlogName/NoteBlogName/Type became RootBlogId/NoteBlogId/TypeId
on 2026-08-07, resolved through the new BlogNames and NoteTypes tables.
There is no compatibility view, so every affected statement is a hard cut.

All 14 call sites in DataAccess.cs are ported:

- Notes->Blogs joins go through Blogs.BlogId in one integer hop; the
  Notes->Posts join in GetRepliesWithFilledText is the only one that must
  route through BlogNames, since Posts carries no BlogId
- AddNote registers both blog names and the note type with INSERT OR
  IGNORE before inserting, in one transaction committed before the console
  sleep. Registering the type matters: an unseen type would resolve to
  NULL and fail NOT NULL on TypeId, silently losing the note
- The LEFT JOIN Notes in GetPosts is dropped rather than translated. It
  selected nothing, could not remove a row, and its duplicates were
  collapsed by the query's own GROUP BY
- Duplicate-key detection moves to IsNotesDuplicateKey, matching the
  constraint and table instead of an exact column list. The old literal
  string is what broke on this rename
- EnsureReplyTextColumnExists drops DEFAULT '.', matching the migrated
  schema: new rows get NULL, not a placeholder nobody wrote

verify-db-schema.sql gains BlogNames, NoteTypes, Blogs.BlogId and the new
Notes columns, plus query 1d naming a pre-migration file and pointing at
normalize-notes.sql. Blogs.BlogId is deliberately not auto-fixable -- an
added-but-empty column makes engagement joins return zero rows silently.

Verified against the live 148 MB file: query plans hit the intended
indexes, and the BlogId join matches an independent name-resolved
formulation exactly on all 4,267 GetBlogs and 2,637 GetBlogsForLikes rows.

RolodexRepository.cs (16 sites) lives in the Rolodex repo and is not
covered here.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-07 21:23:13 -05:00
jim c3cf89c3f1 Merge branch 'claude/normalize-notes' into master 2026-08-07 20:54:24 -05:00
jimandClaude Opus 5 ab36085ba8 feat(db)!: replace blog names and note types in Notes with integer IDs
BREAKING CHANGE: Notes.RootBlogName, Notes.NoteBlogName and Notes.Type no
longer exist. They are RootBlogId, NoteBlogId and TypeId, resolved through two
new lookup tables. Every Notes query in this repo and in Rolodex fails against
a migrated database until rewritten. Neither application is ported yet.
TumblThree is unaffected -- it touches only Blogs.

Takes TL.db from 207 MB to 148 MB (-29%); cumulative with this morning's
WITHOUT ROWID change, 267 MB to 148 MB (-45%). The names were text repeated
across 1.18M rows, in the table and again in every index over it.

  BlogNames(BlogId, BlogName)   20,430 rows, the ID authority
  NoteTypes(TypeId, Type)       5 rows, a table rather than a CHECK so a new
                                type is an INSERT not a migration
  Blogs.BlogId                  new, additive, NULL on the 168,202 blogs with
                                no notes

Blogs.BlogId exists so Notes reaches Blogs in one integer hop instead of going
through BlogNames and ending in the text comparison this change was meant to
remove. It costs 2 MB and is purely additive, which is what leaves TumblThree
untouched.

BlogNames is built from Notes rather than from Blogs, deliberately: 12 engagers
have no registry row, and sourcing it from Blogs would have dropped their notes
through the migration's inner joins.

Proven lossless before and after applying to the live file: the old text shape
was reconstructed from the new schema and diffed against the pre-migration
database in both directions. Zero rows differed either way across all 1,182,333
rows and all ten columns. integrity_check ok, journal_mode still wal.

A view-plus-INSTEAD-OF-triggers compatibility shim was built and measured
first. It worked completely -- reads, INSERT OR IGNORE dedup, both apps' update
paths, cross-table transactions -- but cost 194 ms to 321 ms on Rolodex's
unfiltered Notes page, and a clean break was chosen over carrying it.

TL.db.md gains a "Porting to the integer schema" section: column mapping and
the old-to-new form of every query shape the two applications use, including
the INSERT-OR-IGNORE-into-BlogNames-first pattern for notes naming a blog that
has no ID yet. Roughly 14 call sites in DataAccess.cs, 16 in
RolodexRepository.cs. Every documented snippet was executed against the live
file. Also flags that the duplicate-key error string DataAccess.cs matches on
at two sites now names the new columns and will no longer match.

Unrelated corrections found while refreshing the counts, all of which had
drifted on their own: Posts.PostType is no longer NULL on every row but
populated on 20,679 of 22,468, which invalidates the stated reason both this
document and Rolodex derive post type from content instead of reading it; the
Posts.IsActive and Notes.IsActive columns described as "not in this database
yet" both exist; and the registry is 188,620 blogs, not 144,367.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-07 20:54:24 -05:00
jimandClaude Opus 5 4d37999f8e chore: rotate the TL.db backup archive and refresh local state
Replaces TL 20251212.7z with TL 20260807.7z, taken after today's shrink of
TL.db from 267 MB to 207 MB. Also picks up the RERUN.sqbpro working state and
.claude/tl.db.

Note that .claude/tl.db is not covered by .gitignore, which currently excludes
only .claude/settings.local.json and .claude/worktrees/. Rolodex ignores the
whole .claude/ directory; this repo may want the same.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-07 20:27:23 -05:00
jim 387c023900 Merge branch 'claude/shrink-tl-db' into master 2026-08-07 20:20:29 -05:00
jimandClaude Opus 5 a9bd5a4c37 perf(db): shrink TL.db from 267 MB to 207 MB
The file was already tight -- freelist 0 pages, and a plain VACUUM reclaimed
nothing -- so the saving had to come from schema rather than compaction.
Profiled with dbstat and measured every step on copies of the live file.

Three changes to Notes, applied 2026-08-07:

- Rebuild as WITHOUT ROWID (-32 MB). The 5-column composite primary key was
  stored twice: once in the table, once in a 62 MB autoindex existing only to
  map key -> rowid. Keying the table b-tree on the primary key itself drops the
  second copy. ix_NoteBlogName01 grows 25 -> 58 MB in exchange, since a
  secondary index on such a table carries the whole primary key instead of a
  rowid; net -32 MB.

- Drop Notes_idx_06e01ae3 on TimeStamp DESC (-14 MB). Barely earned its keep as
  a rowid index and would have cost 58 MB after the conversion, cancelling the
  entire exercise. The crawler's only TimeStamp filter (>= 1535778000) excludes
  786 of 1,182,333 rows; Rolodex's default Notes sort carries a three-column
  tiebreaker forcing a full sort regardless; the reply-matching UPDATE uses
  ABS(TimeStamp - ?) <= 5, which no index on the column can serve. Cost is one
  path: Rolodex's Notes page with a date-range filter, 60 ms -> 164 ms.

- Null the DatetimeCrawled placeholder (-13 MB). 1,148,077 rows stored the
  literal DDL default '2/12/26 12am', a backfill marker rather than a crawl
  time. UI-neutral: Rolodex reads the column through DateSql.Sortable, whose
  CASE matches neither format, so those rows already rendered as an em dash.

No application code changed. The schema keeps the same tables, columns, types
and constraints; WITHOUT ROWID is a storage-layout change behind the same SQL
surface, and no consumer referenced rowid on Notes.

Verified against all three consumers on the live file: integrity_check ok, row
counts unchanged (1182333 / 22468 / 188620), journal_mode still wal, crawler
INSERT OR IGNORE still dedupes, Rolodex's NoteBlogName filter still uses
ix_NoteBlogName01, and exactly as many rows read as null through Sortable after
the change as before it. TumblThree touches only Blogs, which is untouched.

Deliberately not done: nulling Notes.DateCreated (a further -12 MB). Unlike
DatetimeCrawled its value parses as a real date, so Rolodex displays and sorts
by it; nulling would turn visible dates into em dashes.

Note that DEFAULT '2/12/26 12am' remains on the column, so any writer inserting
a note without naming it reintroduces the placeholder. Consumer-side date
normalisation must stay.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-07 19:47:15 -05:00
jim 70b32dfc89 Merge branch 'claude/ttfolderpath-not-set-8c9b45' into master 2026-08-05 12:09:55 -05:00
jimandClaude Opus 5 8f4177a0c9 feat: fold the TTFolderPath refresh into --output
TL.db syncs between machines whose absolute paths differ, so a single
TTFolderPath column cannot be correct on both at once -- the stored paths are
only trustworthy on the machine that wrote them. That made --updatepaths a
mandatory prelude to every --output rather than the one-time setup step it
looks like.

--output now refreshes the column from the TumblThree Index metadata before
exporting. The root comes from the first non-flag argument, else
appSettings:PathTTRoot. With no root available it says so and exports whatever
TL.db already holds; a root whose Index folder is missing is a hard stop, since
silently exporting stale paths is the failure this change exists to prevent.
--norefresh skips the refresh for a pure export.

The scan is extracted from UpdateBlogPathsRunner.Run into a reusable Scan() that
returns counts instead of only printing them, so --updatepaths keeps its
per-file detail while --output prints a single summary line rather than a few
hundred lines ahead of the export.

Verified against a throwaway database: a stale cross-machine path is repaired
and the export lands in the correct local folder; no configured root warns and
continues (exit 0); a missing Index folder stops (exit 1); --norefresh skips the
refresh and exports (exit 0).

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-05 12:09:55 -05:00
jim a2763d0026 Merge branch 'claude/ttfolderpath-not-set-8c9b45' into master 2026-08-05 12:02:58 -05:00
jimandClaude Opus 5 721224bc13 fix: make --output and --updatepaths tell the truth about TTFolderPath
--output iterated all 156k active Blogs rows and printed a "does not exist or is
not set" skip line for each, which is nearly every blog in the crawl registry --
only the few hundred downloaded locally ever have a folder. The signal was
buried in six figures of noise.

GetAllBlogsWithTTFolderPath now selects only active blogs carrying a non-empty
path, so --output processes export targets and nothing else. When none exist it
says so once, names the database it read, points at --updatepaths, and returns
non-zero instead of reporting success. A stored path this machine cannot see is
now reported separately from an unset one, with the path shown, because the two
are fixed in different places. Paths are trimmed before Directory.Exists, which
stray whitespace in a .tumblr FileDownloadLocation would otherwise defeat.

Both writers counted optimistically. UpdateBlogPathsRunner printed its
per-blog success line and incremented its total from the metadata file parsing,
never checking whether the UPDATE matched a row; LegacyPostsDbImporter counted a
blog as copied even when the legacy TTFolderPath was NULL. Either could report
full success having written nothing -- which is consistent with TL.db holding
zero populated paths across all 156,492 active blogs despite 20,679 posts having
merged. SetBlogTTFolderPath now returns whether a row changed, and both callers
report written / already-correct / no-matching-row separately.

Verified against a throwaway database: no-paths case, export case (stale .txt
rotated to .bak, per-PostType files, date-sorted), missing-folder case,
--updatepaths honest counts, and an idempotent rerun reporting already-correct.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-05 12:02:49 -05:00
jim ef6629d86a Merge branch 'claude/sqlite-modified-date-logic-3d9bee' into master 2026-08-05 09:36:38 -05:00
jimandClaude Opus 5 6320e2c0c9 fix: stop --ingest's NULL sentinel from clobbering post content
UpsertPostFromTextFile (the persistence layer under --ingest) uses NULL as
its "this file's record had no line for that field" sentinel -- the direct
analog of the "." convention just fixed in UpdatePost. IngestMode strips a
trailing "_N" off the folder name before it ever reaches this function, so
a duplicate export folder deliberately collapses onto the same BlogName --
reconciling multiple differently-formatted files for one post is the whole
point of --ingest. Files are walked in raw filesystem enumeration order,
never sorted, so which file's call lands last for a given (BlogName,
PostID) is arbitrary.

The UPDATE branch set every column unconditionally, so whichever file
processed last for a PostID nulled out every field its own record didn't
carry, silently erasing real content another file had. Worse than the "."
case: that one only caused churn (two writes cancelling out); this one
loses data, in an order that depends on filesystem enumeration.

Every content column is now guarded the same way, NULL instead of "." as
the sentinel: `col = CASE WHEN @col IS NULL THEN col ELSE @col END` in the
SET list, `(@col IS NOT NULL AND IFNULL(col,'') <> @col) OR ...` in the
change-detection. Narrow the same way: only a missing line (NULL) is the
sentinel -- G() already distinguishes that from present-but-blank (""), so
an explicit empty field still overwrites.

HasImage is deliberately left unguarded and documented as a known gap:
IngestMode always computes a concrete bool, defaulting false when a file
has no "Has Image:" line, so this function can't currently tell "no image"
from "not reported" without changing the parameter to bool? and threading
that through IngestMode/LegacyPostsDbImporter too.

Verified against a throwaway DB using the exact SQL text and parameter
binding: a full-format record's real Title/Slug/Tags now survive a
same-PostID partial record whose format doesn't carry those fields, in
both file orders, while a genuine content change and an explicit empty
value still write and still move DateModified.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-05 09:36:17 -05:00
jim 6136901cc7 Merge branch 'claude/sqlite-modified-date-logic-3d9bee' into master 2026-08-05 09:17:48 -05:00
jimandClaude Opus 5 83e35a2323 fix: stop "." export sentinel from clobbering real post content
ReblogRecord (TraverseDirectory's .txt-export parser) and the --likes API
path both default every content field to the literal "." when their source
has no value for that field, then pass it straight into UpdatePost. A blog
with two export folders in different field formats -- a duplicate "_2"
folder, or an export whose field set changed over time -- sends one record
with real Title/Tags/Slug and another with those fields "." because that
format never had a line for them. Re-importing both on every run flipped
the row back and forth forever: net content never changed, but
DateModified moved on every pass since each write really did change a
column relative to the other write, just not relative to the true value.

Every content column in UpdatePost's SET list is now guarded the same way
RootBlogName/RootURL already were -- a "." parameter leaves the existing
value alone instead of overwriting it -- and the change-detection WHERE
clause carries the same exception, so a "."-only difference no longer
fires the UPDATE at all. Deliberately narrow: only the literal "." is the
sentinel, so an explicit empty string from a real record still overwrites.

Verified against a throwaway DB using the exact SQL text and parameter
binding from UpdatePost, reproducing the an-angry-wolf/adore-blk scenario
found in the live DB: re-importing conflicting "." records now writes zero
rows and leaves DateModified untouched, while a genuine content change
still fires and still moves it.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-05 09:17:20 -05:00
jim f0ccac6503 Merge GitTea/master into master 2026-08-03 08:58:07 -05:00
jimandClaude Opus 5 e1d2eb48c2 fix: only bump DateModified when a value actually changed
Seven UPDATE statements wrote DateModified unconditionally, so re-crawling
or re-ingesting identical content marked Blogs, Posts and Notes rows as
modified. Each now carries a WHERE guard covering every column in its SET
list, so SQLite matches zero rows on a no-op.

Guarded: AddPost's insert-failure fallback and blog stamp, AddNote's blog
stamp, UpdateBlogLikesNewestTimestamp, UpdateNoteReplyText,
UpsertPostFromTextFile, SetBlogTTFolderPath, UpdatePostContentFields.

Also:
- Blogs.DateAdded is no longer rewritten when a new post arrives for a
  known blog. A new post is not a new blog, and rewriting the column both
  destroyed the registration date and made every insert look like a change.
- Posts.NotesGatheredDateTime is crawl bookkeeping that moves on every
  pass, so it no longer moves DateModified on its own. It is still written
  each pass, but the timestamp is wrapped in a CASE on the pre-UPDATE
  HasNotesGathered value so only the flag flipping counts.

These statements now return 0 rows for "found but unchanged" as well as
"not found"; CorrectMode's postsUpdated tally consequently counts rows
actually changed, matching what its dry-run diff reports.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-03 08:53:10 -05:00
14 changed files with 1483 additions and 251 deletions
BIN
View File
Binary file not shown.
+117
View File
@@ -47,6 +47,36 @@ say nothing about the item being fetched, so they must not be recorded as per-it
- Long-running commands return exit 3 when a pass ends incomplete (rate-limit pause, breaker trip, or - Long-running commands return exit 3 when a pass ends incomplete (rate-limit pause, breaker trip, or
skipped items), so a caller can distinguish that from a clean run skipped items), so a caller can distinguish that from a clean run
### `Notes` Stores Integer IDs, Not Names
As of 2026-08-07 `Notes.RootBlogName`, `NoteBlogName` and `Type` are gone, replaced by
`RootBlogId`, `NoteBlogId` and `TypeId` resolving through the `BlogNames` and `NoteTypes`
lookup tables. There is no compatibility view — naming an old column is a hard SQLite
error, so unlike `IsActive` this is a hard cut with no runtime probe. Full detail in
`URLNotesGrabberCORE/TL.db.md`.
- **Joining `Notes` to `Blogs` goes through `Blogs.BlogId`**, not `BlogNames`:
`FROM Blogs B INNER JOIN Notes N ON N.NoteBlogId = B.BlogId`. Routing it through
`BlogNames` adds a hop and ends in the text comparison the migration removed
- **Joining `Notes` to `Posts` is the opposite** — `Posts` has only `BlogName`, so it must
go through `BlogNames` (`GetRepliesWithFilledText`). This is the only such join
- **Resolve a name by filtering the lookup, never by scanning `Notes`**:
`WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @name)`. The subquery
is a unique-index probe on 20k rows and does not show against the 1.18M-row table
- **`AddNote` registers both blog names *and* the note type** with `INSERT OR IGNORE`
before inserting, all in one transaction. `NoteTypes` is a table rather than a `CHECK`
constraint precisely so an unseen type is an `INSERT`; without that registration it
would resolve to `NULL` and fail the `NOT NULL` on `TypeId`, losing the note
- **`Blogs.BlogId` is NULL on 168,202 of 188,620 rows** — every blog that has never
appeared in a note. An inner join on it silently drops them. Correct for engagement
queries, wrong for anything listing the registry
- **IDs are stable and must never be renumbered.** They are stored in 1.18M `Notes` rows.
A blog renamed upstream gets a new `BlogNames` row, not an edited one
- Prefer `TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')` over a hardcoded
ID. A negated `TypeId NOT IN (SELECT …)` is only correct because `TypeId` is `NOT NULL`
- Duplicate-key detection uses `IsNotesDuplicateKey`, which matches the constraint and the
table rather than an exact column list. The old literal string comparison broke silently
on this rename — do not reintroduce one
### `IsActive` Is Not Ours To Write ### `IsActive` Is Not Ours To Write
`Blogs.IsActive`, `Posts.IsActive` and `Notes.IsActive` are removal flags set by other tools `Blogs.IsActive`, `Posts.IsActive` and `Notes.IsActive` are removal flags set by other tools
(Rolodex). `0` means removed; anything else, including `NULL`, means live. Full detail in (Rolodex). `0` means removed; anything else, including `NULL`, means live. Full detail in
@@ -67,6 +97,93 @@ say nothing about the item being fetched, so they must not be recorded as per-it
- Do not add these columns from this app, and do not add them to the missing-column list in - Do not add these columns from this app, and do not add them to the missing-column list in
`verify-db-schema.sql` `verify-db-schema.sql`
### `DateModified` Tracks Real Changes Only
`Blogs.DateModified`, `Posts.DateModified` and `Notes.DateModified` must move only when a
column beside `DateModified` itself actually changed. Re-crawling or re-ingesting identical
content has to leave the row — and its timestamp — untouched, or downstream consumers cannot
tell a refreshed row from a rewritten one.
- Enforce it in the `WHERE` clause, not in C#. Every `UPDATE` that sets `DateModified` ends
with an `AND (<col> <> @param OR ...)` term covering every column in its `SET` list, so
SQLite matches zero rows on a no-op and never writes
- Compare NULL-safely: `IFNULL(col, '') <> IFNULL(@param, '')` for text,
`IFNULL(col, 0) <> @param` for integer flags. A bare `col <> @param` is NULL on a NULL
column and silently skips the row that most needs writing
- Where NULL is not equivalent to the default, spell it out. The `HasBeenOutput = 0` stamps
use `(HasBeenOutput IS NULL OR HasBeenOutput <> 0)` because the selection queries test
`HasBeenOutput = 0`, which a NULL would never match
- Dynamic `SET` lists (`UpdatePostContentFields`) build the guard alongside the assignments
so the two lists cannot drift apart
- These statements now return 0 rows for "found but unchanged" as well as "not found".
Callers that read `ExecuteNonQuery()` must not treat 0 as "row missing"
**`Posts.NotesGatheredDateTime` is crawl bookkeeping, not content.** It moves on every
`-collect` pass and says nothing about the post, so it must never move `DateModified` on its
own. `UpdatePostMarkNotesCollected` still writes it every pass but wraps the timestamp in
`DateModified = CASE WHEN IFNULL(HasNotesGathered, 0) <> 1 THEN @dateModified ELSE
DateModified END` — SQLite evaluates `SET` expressions against the pre-`UPDATE` row, so only
the flag flipping counts as a modification. Use this shape for any column that has to be
refreshed unconditionally without being a change. `Blogs.LikesLastRefreshed` is the
deliberate exception: a refresh pass is treated as a real event on the blog row.
**`Blogs.DateAdded` is write-once.** `AddBlog`'s `INSERT` is the only place that sets it. A
new post arriving for a known blog reopens `HasBeenOutput` but must leave `DateAdded` alone —
a new post is not a new blog, and rewriting the column both destroys the registration date
and makes every insert look like a change.
**`"."` in a `Posts` content field means "not supplied", not "empty".** `ReblogRecord`
(`TraverseDirectory`'s parser for the local `.txt` export tree) and the `--likes` API path
both default every content field to the literal string `"."` when their source has no value
for it, then pass that straight to `UpdatePost`. A blog with two export folders in different
field formats (a duplicate `_2` folder, or a Tumblr export whose field set changed over time)
sends one record with a real `Title`/`Tags`/`Slug` and another with those fields `"."`
because that format never had a line for them — and without a guard, re-importing both on
every run flips the row back and forth forever, bumping `DateModified` on every pass even
though the true content never changes.
- Every content column in `UpdatePost`'s `SET` list is guarded the same way `RootBlogName`/
`RootURL` already were: `col = CASE WHEN @col = '.' THEN col ELSE @col END`. A `"."`
parameter leaves the existing value alone instead of overwriting it
- The change-detection `WHERE` clause carries the same exception —
`(@col <> '.' AND IFNULL(col, '') <> @col) OR ...` — so a `"."`-only difference does not
make the statement fire at all, and `DateModified` stays put
- Deliberately narrow: only the literal `"."` is the sentinel. An explicit empty string from
a real record still overwrites, same as before this fix. `postID`, `BlogName`, `hasImage`,
`ByLikes` are not part of this convention and are unaffected
- If a new content field is added to `Posts`/`UpdatePost`, decide explicitly whether its
source can legitimately supply `"."` as "field absent" before deciding whether it needs
the same `CASE` treatment — don't assume every column needs it
**`--ingest` (`UpsertPostFromTextFile`) uses `NULL`, not `"."`, for the same "field absent"
convention, and reconciling exactly this kind of duplicate IS the feature's job.**
`IngestMode` strips a trailing `_N` from the folder name before it ever reaches
`UpsertPostFromTextFile`, so a duplicate export folder collapses onto the same `BlogName` on
purpose — the whole point is to merge multiple differently-formatted files for the same post
into one row. `IngestMode.G(key)` returns `null` (not `"."`) when a field's line is absent
from a given file, `LegacyPostsDbImporter` passes `null` straight from a `NULL` source column,
and files are walked in raw filesystem enumeration order — never sorted — so which file's call
lands last for a given `(BlogName, PostID)` is arbitrary.
- Before the fix, the `UPDATE` branch set every column unconditionally, so whichever file
processed last for a `PostID` would null out every field its own record didn't carry —
silently erasing real `Title`/`Slug`/`Tags`/… another file had, the opposite of what
`--ingest` exists to do. This is worse than the `"."` case above: that one only caused
churn (the two writes canceled out); this one loses data, and which posts lose which
fields depends on filesystem enumeration order
- Same shape of fix, `NULL` instead of `"."` as the sentinel: `col = CASE WHEN @col IS NULL
THEN col ELSE @col END` in the `SET` list, `(@col IS NOT NULL AND IFNULL(col, '') <> @col)
OR ...` in the change-detection
- Same narrow rule: only `NULL` (the field's line was never present in this file) is the
sentinel. `G()` already distinguishes this from "present but blank" — a dictionary miss is
`null`, an empty value after the prefix is `""` — so an explicitly blank field still
overwrites
- `HasImage` is **not** guarded and remains a known gap: `IngestMode` always computes a
concrete `bool` (defaulting `false` when a file has no `Has Image:` line), so there is no
way for this function to tell "this format says no image" from "this format doesn't report
it at all" without changing the parameter to `bool?` and threading that through
`IngestMode`/`LegacyPostsDbImporter`. Fix this the same way if `--ingest` is observed
downgrading a post's `HasImage` from `1` to `0`
### Testing ### Testing
- No existing test suite; use xUnit if adding tests - No existing test suite; use xUnit if adding tests
- Test critical logic: `ApiKeyPool` init, color parsing, config persistence - Test critical logic: `ApiKeyPool` init, color parsing, config persistence
+39 -6
View File
@@ -1,4 +1,4 @@
<?xml version="1.0" encoding="UTF-8"?><sqlb_project><db path="C:/Users/jim/Nextcloud/C#/URLNotesGrabberCORE/URLNotesGrabberCORE/TL.db" readonly="0" foreign_keys="1" case_sensitive_like="0" temp_store="0" wal_autocheckpoint="1000" synchronous="2"/><attached/><window><main_tabs open="structure browser pragmas query" current="3"/></window><tab_structure><column_width id="0" width="300"/><column_width id="1" width="0"/><column_width id="2" width="100"/><column_width id="3" width="4305"/><column_width id="4" width="0"/><expanded_item id="0" parent="1"/><expanded_item id="1" parent="1"/><expanded_item id="2" parent="1"/><expanded_item id="3" parent="1"/></tab_structure><tab_browse><table title="Posts" custom_title="0" dock_id="4" table="4,5:mainPosts"/><dock_state state="000000ff00000000fd00000001000000020000077200000379fc0100000006fb000000160064006f0063006b00420072006f00770073006500310100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500320100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500330100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500350100000000000005f40000000000000000fb000000160064006f0063006b00420072006f00770073006500340100000000000007720000011700fffffffb000000160064006f0063006b00420072006f00770073006500340100000000000005f40000000000000000000007720000000000000004000000040000000800000008fc00000000"/><default_encoding codec=""/><browse_table_settings><table schema="main" name="ApiKeyPoolMeta" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="29"/><column index="2" value="64"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Blogs" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="257"/><column index="2" value="95"/><column index="3" value="54"/><column index="4" value="156"/><column index="5" value="51"/><column index="6" value="71"/><column index="7" value="85"/><column index="8" value="156"/><column index="9" value="156"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Posts" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="28" mode="1"/></sort><column_widths><column index="1" value="241"/><column index="2" value="148"/><column index="3" value="126"/><column index="4" value="300"/><column index="5" value="75"/><column index="6" value="187"/><column index="7" value="159"/><column index="8" value="75"/><column index="9" value="300"/><column index="10" value="300"/><column index="11" value="78"/><column index="12" value="249"/><column index="13" value="300"/><column index="14" value="53"/><column index="15" value="300"/><column index="16" value="300"/><column index="17" value="41"/><column index="18" value="75"/><column index="19" value="96"/><column index="20" value="300"/><column index="21" value="96"/><column index="22" value="300"/><column index="23" value="300"/><column index="24" value="42"/><column index="25" value="60"/><column index="26" value="218"/><column index="27" value="920"/><column index="28" value="156"/><column index="29" value="156"/></column_widths><filter_values><column index="24" value="=1"/><column index="28" value="&gt;2026-05-06 20:00:01"/></filter_values><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table></browse_table_settings></tab_browse><tab_sql><sql name="SQL 1">UPDATE Posts <?xml version="1.0" encoding="UTF-8"?><sqlb_project><db path="D:/NextCloud/C#/URLNotesGrabberCORE/URLNotesGrabberCORE/TL.db" readonly="0" foreign_keys="1" case_sensitive_like="0" temp_store="0" wal_autocheckpoint="1000" synchronous="2"/><attached/><window><main_tabs open="structure browser pragmas query" current="3"/></window><tab_structure><column_width id="0" width="300"/><column_width id="1" width="0"/><column_width id="2" width="100"/><column_width id="3" width="4486"/><column_width id="4" width="0"/><expanded_item id="0" parent="1"/><expanded_item id="1" parent="1"/><expanded_item id="2" parent="1"/><expanded_item id="3" parent="1"/></tab_structure><tab_browse><table title="Posts" custom_title="0" dock_id="4" table="4,5:mainPosts"/><dock_state state="000000ff00000000fd0000000100000002000005470000029afc0100000006fb000000160064006f0063006b00420072006f00770073006500310100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500320100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500330100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500350100000000000005f40000000000000000fb000000160064006f0063006b00420072006f00770073006500340100000000000005470000011100fffffffb000000160064006f0063006b00420072006f00770073006500340100000000000005f40000000000000000000005470000000000000004000000040000000800000008fc00000000"/><default_encoding codec=""/><browse_table_settings><table schema="main" name="ApiKeyPoolMeta" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="29"/><column index="2" value="64"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Blogs" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="7" mode="1"/></sort><column_widths><column index="1" value="257"/><column index="2" value="108"/><column index="3" value="63"/><column index="4" value="156"/><column index="5" value="60"/><column index="6" value="81"/><column index="7" value="85"/><column index="8" value="156"/><column index="9" value="156"/><column index="10" value="151"/><column index="11" value="125"/><column index="12" value="129"/></column_widths><filter_values><column index="4" value="1"/><column index="7" value="&gt;2026-05-27 17:22:36"/></filter_values><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Notes" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="198"/><column index="2" value="144"/><column index="3" value="251"/><column index="4" value="84"/><column index="5" value="53"/><column index="6" value="300"/><column index="7" value="116"/><column index="8" value="152"/><column index="9" value="89"/><column index="10" value="63"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Posts" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="14" mode="1"/></sort><column_widths><column index="1" value="236"/><column index="2" value="144"/><column index="3" value="126"/><column index="4" value="32"/><column index="5" value="32"/><column index="6" value="32"/><column index="7" value="32"/><column index="8" value="32"/><column index="9" value="0"/><column index="10" value="0"/><column index="11" value="0"/><column index="12" value="243"/><column index="13" value="300"/><column index="14" value="53"/><column index="15" value="37351"/><column index="16" value="300"/><column index="17" value="41"/><column index="18" value="75"/><column index="19" value="96"/><column index="20" value="300"/><column index="21" value="96"/><column index="22" value="300"/><column index="23" value="300"/><column index="24" value="548"/><column index="25" value="60"/><column index="26" value="213"/><column index="27" value="532"/><column index="28" value="152"/><column index="29" value="152"/><column index="30" value="69"/><column index="31" value="63"/></column_widths><filter_values><column index="2" value="0"/><column index="1" value="734568371821084672"/></filter_values><conditional_formats/><row_id_formats/><display_formats/><hidden_columns><column index="9" value="1"/><column index="10" value="1"/><column index="11" value="1"/></hidden_columns><plot_y_axes/><global_filter/></table></browse_table_settings></tab_browse><tab_sql><sql name="SQL 1">UPDATE Posts
SET HasNotesGathered = 0 SET HasNotesGathered = 0
WHERE (BlogName, PostID) IN ( WHERE (BlogName, PostID) IN (
SELECT p.BlogName, p.PostID SELECT p.BlogName, p.PostID
@@ -29,11 +29,12 @@ blogname in
'nudenymph', 'nudenymph',
'caylachief' 'caylachief'
)</sql><sql name="New Notes">select RootBlogName, PostID, NoteBlogName || '.tumblr.com' as NoteBlogName, DatetimeCrawled, TimeStamp, type, RootBlogName || '.tumblr.com/post/' || postid, datetime(timestamp, 'unixepoch') )</sql><sql name="New Notes">select P.slug, N.replyText, n.RootBlogName, n.PostID, NoteBlogName || '.tumblr.com' as NoteBlogName, DatetimeCrawled, TimeStamp, type, n.RootBlogName || '.tumblr.com/post/' || n.postid, datetime(timestamp, 'unixepoch')
from Notes from Notes N inner join Posts P on p.PostID = n.PostID
where where
DatetimeCrawled &gt; '2026-05-14 02:50:05' --and type like 'r%' DatetimeCrawled &gt; '2026-08-07 11:47:22' and type like 'r%'
order by DatetimeCrawled desc</sql><sql name="Pull Blogs*">SELECT distinct␍ and P.IsActive = 1
order by n.DatetimeCrawled</sql><sql name="Pull Blogs">SELECT distinct
'''' || blogname || ''',', '''' || blogname || ''',',
blogs.* blogs.*
, blogname || '.tumblr.com' , blogname || '.tumblr.com'
@@ -64,4 +65,36 @@ JOIN ReplyCounts c ON n.NoteBlogName = c.NoteBlogName
where replyText &lt;&gt; '.' and type &lt;&gt; 'reply' where replyText &lt;&gt; '.' and type &lt;&gt; 'reply'
--AND N.NoteBlogName NOT IN ( 'roadblocker21', 'thesaddemon666', 'edwardabbeyhoffman', 'tattedsoldier20', 'zomb-eh', 'animalistic13', 'indken', 'maccloud1592', --AND N.NoteBlogName NOT IN ( 'roadblocker21', 'thesaddemon666', 'edwardabbeyhoffman', 'tattedsoldier20', 'zomb-eh', 'animalistic13', 'indken', 'maccloud1592',
--'moss-wizard', 'supertrucker12682', 'exploringthrupics', 'padeyepete' ) --'moss-wizard', 'supertrucker12682', 'exploringthrupics', 'padeyepete' )
order by c.DistinctReplyCount desc, n.NoteBlogName, n.DateModified desc, replyText, RootBlogName, PostID</sql><current_tab id="3"/></tab_sql></sqlb_project> order by c.DistinctReplyCount desc, n.NoteBlogName, n.DateModified desc, replyText, RootBlogName, PostID</sql><sql name="Collect">WITH PostsWithCount AS ( SELECT P.BlogName, P.PostID, 1925013599 AS LatestNoteTimestamp, P.NotesGatheredDateTime, COUNT(P.PostID) OVER(PARTITION BY P.BlogName) AS CNT, P.HasNotesGathered, P.NotFound, P.PostDate FROM Posts P WHERE COALESCE(P.IsActive, 1) = 1 ), Unioned AS ( SELECT BlogName, PostID, LatestNoteTimestamp, NotesGatheredDateTime, CNT, PostDate FROM PostsWithCount WHERE NotFound = 0 AND HasNotesGathered = 0 UNION SELECT BlogName, PostID, LatestNoteTimestamp, NotesGatheredDateTime, CNT, PostDate FROM PostsWithCount WHERE BlogName = 'zomb-eh' AND NotFound = 0 AND NotesGatheredDateTime &lt; unixepoch('now', 'localtime', '-3 days') ) SELECT U.BlogName, U.PostID, U.LatestNoteTimestamp, U.NotesGatheredDateTime, U.CNT FROM Unioned U WHERE (U.NotesGatheredDateTime &lt; 1786134037 OR U.NotesGatheredDateTime IS NULL) ORDER BY U.NotesGatheredDateTime, U.PostDate DESC, U.BlogName, U.PostID;</sql><sql name="Del Posts">delete from posts where postid in
(
'741662499571728384',
178892849664,
178264721139,
177012868749,
169950081964,
755440787056099328
)</sql><sql name="notes NO post*">select *
-- delete
from notes
where postid not in (select distinct postid from posts where IsActive = 1)</sql><sql name="SQL 9">SELECT
*
FROM
POSTS P
WHERE
P.ByLikes = 1
AND
P.DateCreated &gt; '2026-05-26 17:47:32'
ORDER BY
P.DateCreated desc</sql><sql name="SQL 13">update posts set IsActive = 0 where blogname IN ( 'shoebiedoo', 'redheaded-girlygirl', 'xlittle-ghost' )</sql><sql name="SQL 14*">update Posts␍
set IsActive = 0␍
where postid in␍
(␍
'731937314675310592'␍
)␍
</sql><current_tab id="7"/></tab_sql></sqlb_project>
BIN
View File
Binary file not shown.
+352 -138
View File
@@ -106,6 +106,10 @@ namespace URLNotesGrabberCORE
} }
} }
// The database every DataAccess call defaults to, exposed so modes can report
// which file they actually read when their results are surprising.
public static string GetActiveDbPath() => GetDefaultDbPath();
private static string GetDefaultDbPath() private static string GetDefaultDbPath()
{ {
if (_cachedDbPath != null) if (_cachedDbPath != null)
@@ -209,6 +213,59 @@ namespace URLNotesGrabberCORE
#endregion IsActive #endregion IsActive
#region Notes integer schema
// Notes stopped storing names on 2026-08-07: RootBlogName/NoteBlogName/Type became
// RootBlogId/NoteBlogId/TypeId, resolved through BlogNames and NoteTypes. There is no
// compatibility view -- a query naming an old column fails outright, so this is a hard
// cut rather than an optional column like IsActive. See TL.db.md.
//
// Two shapes recur below and are spelled out inline rather than hidden behind a helper,
// so that every statement reads as the SQL it actually runs:
// (SELECT BlogId FROM BlogNames WHERE BlogName = @name) -- unique-index probe, 20k rows
// (SELECT TypeId FROM NoteTypes WHERE Type = 'reply') -- 5 rows, effectively free
// Joining Notes to Blogs is the one case that must NOT route through BlogNames: Blogs
// carries its own BlogId, so N.NoteBlogId = B.BlogId is a single integer hop. Joining
// Notes to Posts is the opposite case -- Posts has only BlogName, so it has to go
// through BlogNames.
/// <summary>
/// True when the exception is a duplicate-key collision on Notes. The message embeds the
/// primary key's column names, which the integer migration renamed, so this matches on the
/// constraint and the table instead of on an exact column list -- a literal comparison
/// silently inverts into "log every error" the next time a column is renamed.
/// </summary>
private static bool IsNotesDuplicateKey(Exception ex)
{
return ex.Message.Contains("UNIQUE constraint failed", StringComparison.OrdinalIgnoreCase)
&& ex.Message.Contains("Notes.", StringComparison.OrdinalIgnoreCase);
}
/// <summary>
/// Gives a blog name an ID if it does not have one. No read-back and no round trip -- a
/// name that is already registered keeps the ID that 1.18M Notes rows point at.
/// </summary>
private static void RegisterBlogName(SQLiteConnection connection, SQLiteTransaction? transaction, string blogName)
{
using SQLiteCommand command = new SQLiteCommand("INSERT OR IGNORE INTO BlogNames (BlogName) VALUES (@BlogName)", connection, transaction);
command.Parameters.AddWithValue("@BlogName", blogName);
command.ExecuteNonQuery();
}
/// <summary>
/// Same, for a note type. NoteTypes is a table rather than a CHECK constraint precisely so
/// that a type this crawler has not seen before is an INSERT and not a schema migration --
/// without this the type would resolve to NULL and fail the NOT NULL on Notes.TypeId.
/// </summary>
private static void RegisterNoteType(SQLiteConnection connection, SQLiteTransaction? transaction, string type)
{
using SQLiteCommand command = new SQLiteCommand("INSERT OR IGNORE INTO NoteTypes (Type) VALUES (@Type)", connection, transaction);
command.Parameters.AddWithValue("@Type", type);
command.ExecuteNonQuery();
}
#endregion Notes integer schema
public static string Q(string input) public static string Q(string input)
{ {
return "'" + input.Replace("'", "''") + "'"; return "'" + input.Replace("'", "''") + "'";
@@ -367,7 +424,10 @@ namespace URLNotesGrabberCORE
connection.Close(); connection.Close();
connection.Open(); connection.Open();
string addColumnSql = "ALTER TABLE Notes ADD COLUMN replyText TEXT DEFAULT '.';"; // No column default: the migrated schema dropped the DEFAULT '.' that
// is how 1.1M rows acquired a placeholder nobody wrote. New rows get
// NULL, which every reader here already treats as "no reply text".
string addColumnSql = "ALTER TABLE Notes ADD COLUMN replyText TEXT;";
using (SQLiteCommand addCommand = new SQLiteCommand(addColumnSql, connection)) using (SQLiteCommand addCommand = new SQLiteCommand(addColumnSql, connection))
{ {
addCommand.ExecuteNonQuery(); addCommand.ExecuteNonQuery();
@@ -598,7 +658,7 @@ namespace URLNotesGrabberCORE
{ {
if (ownsConnection) connection.Open(); if (ownsConnection) connection.Open();
string updateSql = "UPDATE Posts SET hasImage = @hasImage, DateModified = @DateModified WHERE blogName = @blogName AND postID = @postID"; string updateSql = "UPDATE Posts SET hasImage = @hasImage, DateModified = @DateModified WHERE blogName = @blogName AND postID = @postID AND IFNULL(hasImage, 0) <> @hasImage";
using SQLiteCommand updateCommand = new SQLiteCommand(updateSql, connection); using SQLiteCommand updateCommand = new SQLiteCommand(updateSql, connection);
updateCommand.Parameters.AddWithValue("@hasImage", hasImage ? 1 : 0); updateCommand.Parameters.AddWithValue("@hasImage", hasImage ? 1 : 0);
updateCommand.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss")); updateCommand.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
@@ -623,16 +683,17 @@ namespace URLNotesGrabberCORE
} }
} }
// Only update HasBeenOutput and DateAdded if a new post was inserted // Only reopen the blog for output if a new post was inserted. DateAdded records
// when the blog first entered the registry and is never rewritten here -- a new
// post is not a new blog.
if (rowsInserted == 1) if (rowsInserted == 1)
{ {
try try
{ {
string updateBlogSql = "UPDATE Blogs SET HasBeenOutput = 0, DateAdded = @DateAdded, DateModified = @DateModified WHERE BlogName = @BlogName"; string updateBlogSql = "UPDATE Blogs SET HasBeenOutput = 0, DateModified = @DateModified WHERE BlogName = @BlogName AND (HasBeenOutput IS NULL OR HasBeenOutput <> 0)";
using (var updateBlogCommand = new SQLiteCommand(updateBlogSql, connection)) using (var updateBlogCommand = new SQLiteCommand(updateBlogSql, connection))
{ {
updateBlogCommand.Parameters.AddWithValue("@BlogName", blogName); updateBlogCommand.Parameters.AddWithValue("@BlogName", blogName);
updateBlogCommand.Parameters.AddWithValue("@DateAdded", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
updateBlogCommand.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss")); updateBlogCommand.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
updateBlogCommand.ExecuteNonQuery(); updateBlogCommand.ExecuteNonQuery();
} }
@@ -698,61 +759,85 @@ namespace URLNotesGrabberCORE
try { AddBlog(noteBlogName, false, DBPath); } catch { } try { AddBlog(noteBlogName, false, DBPath); } catch { }
using SQLiteConnection connection2 = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection2 = new SQLiteConnection("Data Source=" + DBPath);
int rowsInserted = 0;
try try
{ {
connection2.Open(); connection2.Open();
// INSERT OR IGNORE, and no IsActive in the column list: re-crawling a note // Notes stores integer IDs, so both participants and the type have to exist in
// that was removed elsewhere leaves the existing row -- and its flag -- alone. // their lookup table before the note can point at them.
string sql = "INSERT OR IGNORE INTO Notes (rootBlogName, noteBlogName, PostID, TimeStamp, Type, DatetimeCrawled, DateModified, DateCreated) values(@rootBlogName, @noteBlogName, @PostID, @TimeStamp, @Type, @DatetimeCrawled, @DateModified, @DateCreated)"; //
using (SQLiteCommand command = new SQLiteCommand(sql, connection2)) // All four statements run in one transaction so a crash cannot leave a name or a
// type registered with no note. The transaction is committed before the console
// output below, which sleeps -- a write lock must not be held across that.
using (SQLiteTransaction transaction = connection2.BeginTransaction())
{ {
command.Parameters.AddWithValue("@rootBlogName", rootBlogName); RegisterBlogName(connection2, transaction, rootBlogName);
command.Parameters.AddWithValue("@noteBlogName", noteBlogName); RegisterBlogName(connection2, transaction, noteBlogName);
command.Parameters.AddWithValue("@PostID", postID); RegisterNoteType(connection2, transaction, type ?? string.Empty);
command.Parameters.AddWithValue("@TimeStamp", timestamp);
command.Parameters.AddWithValue("@Type", type ?? string.Empty);
command.Parameters.AddWithValue("@DatetimeCrawled", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
command.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
command.Parameters.AddWithValue("@DateCreated", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
int rowsInserted = command.ExecuteNonQuery(); // INSERT OR IGNORE, and no IsActive in the column list: re-crawling a note
// that was removed elsewhere leaves the existing row -- and its flag -- alone.
string sql = "INSERT OR IGNORE INTO Notes (RootBlogId, NoteBlogId, PostID, TimeStamp, TypeId, DatetimeCrawled, DateModified, DateCreated) " +
"SELECT (SELECT BlogId FROM BlogNames WHERE BlogName = @rootBlogName), " +
" (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName), " +
" @PostID, @TimeStamp, " +
" (SELECT TypeId FROM NoteTypes WHERE Type = @Type), " +
" @DatetimeCrawled, @DateModified, @DateCreated";
if (rowsInserted == 1) using (SQLiteCommand command = new SQLiteCommand(sql, connection2, transaction))
{ {
ConsoleColor previousColor = Console.ForegroundColor; command.Parameters.AddWithValue("@rootBlogName", rootBlogName);
Console.ForegroundColor = ConsoleColor.Green; command.Parameters.AddWithValue("@noteBlogName", noteBlogName);
Console.WriteLine("{2}\t{0}\t{1}", UnixTimeStampToDateTime(timestamp), noteBlogName, type); command.Parameters.AddWithValue("@PostID", postID);
Console.ForegroundColor = previousColor; command.Parameters.AddWithValue("@TimeStamp", timestamp);
Thread.Sleep(250); // Brief pause to make new notes more noticeable in the console output command.Parameters.AddWithValue("@Type", type ?? string.Empty);
} command.Parameters.AddWithValue("@DatetimeCrawled", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
else command.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
{ command.Parameters.AddWithValue("@DateCreated", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
Console.WriteLine("{2}\t{0}\t{1}", UnixTimeStampToDateTime(timestamp), noteBlogName, type);
rowsInserted = command.ExecuteNonQuery();
} }
// Only update HasBeenOutput if a new note was inserted transaction.Commit();
if (rowsInserted == 1) }
if (rowsInserted == 1)
{
ConsoleColor previousColor = Console.ForegroundColor;
Console.ForegroundColor = ConsoleColor.Green;
Console.WriteLine("{2}\t{0}\t{1}", UnixTimeStampToDateTime(timestamp), noteBlogName, type);
Console.ForegroundColor = previousColor;
Thread.Sleep(250); // Brief pause to make new notes more noticeable in the console output
}
else
{
Console.WriteLine("{2}\t{0}\t{1}", UnixTimeStampToDateTime(timestamp), noteBlogName, type);
}
// Only update HasBeenOutput if a new note was inserted
if (rowsInserted == 1)
{
try
{ {
try // HasBeenOutput IS NULL still counts as a change: the selection queries
// test HasBeenOutput = 0, which a NULL would never match.
string updateSql = "UPDATE Blogs SET HasBeenOutput = 0, DateModified = @DateModified WHERE BlogName = @BlogName AND (HasBeenOutput IS NULL OR HasBeenOutput <> 0)";
using (var updateCommand = new SQLiteCommand(updateSql, connection2))
{ {
string updateSql = "UPDATE Blogs SET HasBeenOutput = 0, DateModified = @DateModified WHERE BlogName = @BlogName"; updateCommand.Parameters.AddWithValue("@BlogName", noteBlogName);
using (var updateCommand = new SQLiteCommand(updateSql, connection2)) updateCommand.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
{ updateCommand.ExecuteNonQuery();
updateCommand.Parameters.AddWithValue("@BlogName", noteBlogName);
updateCommand.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
updateCommand.ExecuteNonQuery();
}
} }
catch { }
} }
catch { }
} }
} }
catch (Exception ex) catch (Exception ex)
{ {
// Breakpoint here // Breakpoint here
if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Notes.RootBlogName, Notes.PostID, Notes.TimeStamp, Notes.Type, Notes.NoteBlogName") if (!IsNotesDuplicateKey(ex))
{ {
Console.WriteLine(ex.Message); Console.WriteLine(ex.Message);
Console.WriteLine("^^^^^ - SHORTCUT"); Console.WriteLine("^^^^^ - SHORTCUT");
@@ -843,6 +928,10 @@ namespace URLNotesGrabberCORE
} }
else else
{ {
// The LEFT OUTER JOIN to Notes that used to sit here has been dropped rather
// than ported. Nothing was selected from it, a LEFT JOIN cannot remove a row,
// and the GROUP BY below collapsed the rows it duplicated -- so it could not
// affect the result, and it cost a join against 1.18M rows on every pass.
sql = "SELECT " + sql = "SELECT " +
" MAX(Posts.BlogName) as BlogName, " + Environment.NewLine + " MAX(Posts.BlogName) as BlogName, " + Environment.NewLine +
" Posts.PostID, " + Environment.NewLine + " Posts.PostID, " + Environment.NewLine +
@@ -852,8 +941,6 @@ namespace URLNotesGrabberCORE
"FROM " + Environment.NewLine + "FROM " + Environment.NewLine +
" Posts " + Environment.NewLine + " Posts " + Environment.NewLine +
" LEFT OUTER JOIN " + Environment.NewLine + " LEFT OUTER JOIN " + Environment.NewLine +
" Notes ON Notes.RootBlogName = Posts.BlogName AND Notes.PostID = Posts.PostID " + Environment.NewLine +
" LEFT OUTER JOIN " + Environment.NewLine +
" ( select BlogName, count(PostID) as CNT from Posts" + WhereIsActive("Posts", "", DBPath) + " group by BlogName) CNT on CNT.blogName = Posts.BlogName " + " ( select BlogName, count(PostID) as CNT from Posts" + WhereIsActive("Posts", "", DBPath) + " group by BlogName) CNT on CNT.blogName = Posts.BlogName " +
"WHERE NotFound = 0 " + AndIsActive("Posts", "Posts", DBPath) + Environment.NewLine; "WHERE NotFound = 0 " + AndIsActive("Posts", "Posts", DBPath) + Environment.NewLine;
@@ -964,7 +1051,11 @@ namespace URLNotesGrabberCORE
{ {
connection.Open(); connection.Open();
string sql = "SELECT distinct RootBlogName as blogName, postID FROM Notes WHERE Notes.type = 'reply'" + AndIsActive("Notes", "Notes", DBPath) + " order by RootBlogName, PostID"; string sql = "SELECT DISTINCT BN.BlogName as blogName, N.PostID" +
" FROM Notes N" +
" INNER JOIN BlogNames BN ON BN.BlogId = N.RootBlogId" +
" WHERE N.TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')" + AndIsActive("Notes", "N", DBPath) +
" ORDER BY BN.BlogName, N.PostID";
using (SQLiteCommand command = new SQLiteCommand(sql, connection)) using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{ {
@@ -1004,12 +1095,15 @@ namespace URLNotesGrabberCORE
{ {
connection.Open(); connection.Open();
string sql = @"SELECT DISTINCT Notes.RootBlogName as blogName, Notes.PostID, // Grouped on the integer rather than the name: the group key is what gets sorted,
MAX(Notes.timestamp) as LatestTimestamp // and BN.BlogName comes along for free off the join.
FROM Notes string sql = @"SELECT BN.BlogName as blogName, N.PostID,
WHERE Notes.type = 'reply' MAX(N.TimeStamp) as LatestTimestamp
AND (Notes.replyText IS NULL OR Notes.replyText = '' OR Notes.replyText = '.')" + AndIsActive("Notes", "Notes", DBPath) + @" FROM Notes N
GROUP BY Notes.RootBlogName, Notes.PostID INNER JOIN BlogNames BN ON BN.BlogId = N.RootBlogId
WHERE N.TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')
AND (N.replyText IS NULL OR N.replyText = '' OR N.replyText = '.')" + AndIsActive("Notes", "N", DBPath) + @"
GROUP BY N.RootBlogId, N.PostID
ORDER BY LatestTimestamp ASC ORDER BY LatestTimestamp ASC
LIMIT @limit"; LIMIT @limit";
@@ -1051,11 +1145,15 @@ namespace URLNotesGrabberCORE
{ {
connection.Open(); connection.Open();
string sql = @"SELECT DISTINCT P.BlogName, P.PostID, MAX(N.timestamp) as LatestTimestamp // Posts carries only BlogName, so this is the one join to Notes that has to go
// through BlogNames -- there is no Posts.BlogId to hop on. The name predicate is
// pushed into the 20k-row lookup, which then feeds integers to the Notes key.
string sql = @"SELECT DISTINCT P.BlogName, P.PostID, MAX(N.TimeStamp) as LatestTimestamp
FROM Posts P FROM Posts P
INNER JOIN Notes N ON N.PostID = P.PostID AND N.RootBlogName = P.BlogName INNER JOIN BlogNames RBN ON RBN.BlogName = P.BlogName
INNER JOIN Notes N ON N.RootBlogId = RBN.BlogId AND N.PostID = P.PostID
WHERE P.NotFound = 0 WHERE P.NotFound = 0
AND N.type = 'reply' AND N.TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')
AND (N.replyText IS NULL OR N.replyText = '' OR N.replyText = '.')" + AndIsActive("Posts", "P", DBPath) + AndIsActive("Notes", "N", DBPath) + @" AND (N.replyText IS NULL OR N.replyText = '' OR N.replyText = '.')" + AndIsActive("Posts", "P", DBPath) + AndIsActive("Notes", "N", DBPath) + @"
GROUP BY P.BlogName, P.PostID GROUP BY P.BlogName, P.PostID
ORDER BY LatestTimestamp ASC"; ORDER BY LatestTimestamp ASC";
@@ -1153,6 +1251,11 @@ namespace URLNotesGrabberCORE
{ {
connection.Open(); connection.Open();
string sql; string sql;
// The two Notes branches below join on Blogs.BlogId, which is NULL for the 168k
// registry rows that have never appeared in a note. The inner join drops them,
// which is correct here -- both branches already require a note to exist -- but
// it is the wrong shape for anything that lists the registry.
if (!string.IsNullOrEmpty(specificBlog)) if (!string.IsNullOrEmpty(specificBlog))
{ {
// Specific blog: always process, bypass cooldown // Specific blog: always process, bypass cooldown
@@ -1172,12 +1275,12 @@ namespace URLNotesGrabberCORE
COALESCE(B.LikesCursor, 0), COALESCE(B.LikesCursor, 0),
COALESCE(B.LikesNewestTimestamp, 0) COALESCE(B.LikesNewestTimestamp, 0)
FROM Blogs B FROM Blogs B
INNER JOIN Notes N ON N.NoteBlogName = B.BlogName INNER JOIN Notes N ON N.NoteBlogId = B.BlogId
WHERE N.TimeStamp >= 1535778000 WHERE N.TimeStamp >= 1535778000
AND N.rootBlogName = B.BlogName AND N.RootBlogId = B.BlogId
AND B.IsActive = 1" + AndIsActive("Notes", "N", DBPath) + @" AND B.IsActive = 1" + AndIsActive("Notes", "N", DBPath) + @"
GROUP BY B.BlogName GROUP BY B.BlogName
ORDER BY MIN(N.Timestamp);"; ORDER BY MIN(N.TimeStamp);";
} }
else else
{ {
@@ -1187,9 +1290,9 @@ namespace URLNotesGrabberCORE
COALESCE(B.LikesCursor, 0), COALESCE(B.LikesCursor, 0),
COALESCE(B.LikesNewestTimestamp, 0) COALESCE(B.LikesNewestTimestamp, 0)
FROM Blogs B FROM Blogs B
INNER JOIN Notes N ON N.NoteBlogName = B.BlogName INNER JOIN Notes N ON N.NoteBlogId = B.BlogId
WHERE N.TimeStamp >= 1535778000 WHERE N.TimeStamp >= 1535778000
AND N.rootBlogName = B.BlogName AND N.RootBlogId = B.BlogId
AND B.IsActive = 1" + AndIsActive("Notes", "N", DBPath) + @" AND B.IsActive = 1" + AndIsActive("Notes", "N", DBPath) + @"
AND ( AND (
B.LikesPulled = 0 B.LikesPulled = 0
@@ -1197,7 +1300,7 @@ namespace URLNotesGrabberCORE
< (CAST(strftime('%s','now') AS INTEGER) - (@cooldownDays * 86400)) < (CAST(strftime('%s','now') AS INTEGER) - (@cooldownDays * 86400))
) )
GROUP BY B.BlogName GROUP BY B.BlogName
ORDER BY MIN(N.Timestamp);"; ORDER BY MIN(N.TimeStamp);";
} }
using (SQLiteCommand command = new SQLiteCommand(sql, connection)) using (SQLiteCommand command = new SQLiteCommand(sql, connection))
@@ -1237,11 +1340,14 @@ namespace URLNotesGrabberCORE
try try
{ {
connection.Open(); connection.Open();
// Blogs is reached in one integer hop off Blogs.BlogId, not through BlogNames --
// that would add a hop and end in the text comparison the migration removed.
// The negated form is only correct because Notes.TypeId is NOT NULL.
string sql = ""; string sql = "";
if (reblogsOnly) if (reblogsOnly)
sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive" + AndIsActive("Notes", "notes", DBPath) + " AND type IN ('reblog', 'reply', 'posted') AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top"; sql = "SELECT B.BlogName as blogName, count(*) FROM Notes N INNER JOIN Blogs B ON B.BlogId = N.NoteBlogId WHERE B.IsActive = @isActive" + AndIsActive("Notes", "N", DBPath) + " AND N.TypeId IN (SELECT TypeId FROM NoteTypes WHERE Type IN ('reblog', 'reply', 'posted')) AND B.HasBeenOutput = 0 GROUP BY N.NoteBlogId ORDER BY count(*) DESC, B.BlogName LIMIT @top";
else else
sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive" + AndIsActive("Notes", "notes", DBPath) + " AND type NOT IN ('reblog', 'reply', 'posted') AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top"; sql = "SELECT B.BlogName as blogName, count(*) FROM Notes N INNER JOIN Blogs B ON B.BlogId = N.NoteBlogId WHERE B.IsActive = @isActive" + AndIsActive("Notes", "N", DBPath) + " AND N.TypeId NOT IN (SELECT TypeId FROM NoteTypes WHERE Type IN ('reblog', 'reply', 'posted')) AND B.HasBeenOutput = 0 GROUP BY N.NoteBlogId ORDER BY count(*) DESC, B.BlogName LIMIT @top";
using (SQLiteCommand command = new SQLiteCommand(sql, connection)) using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{ {
@@ -1280,9 +1386,9 @@ namespace URLNotesGrabberCORE
connection.Open(); connection.Open();
string sql = ""; string sql = "";
if (reblogsOnly) if (reblogsOnly)
sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive" + AndIsActive("Notes", "notes", DBPath) + " AND type IN ('reblog', 'reply', 'posted') AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top"; sql = "SELECT B.BlogName as blogName, count(*) FROM Notes N INNER JOIN Blogs B ON B.BlogId = N.NoteBlogId WHERE B.IsActive = @isActive" + AndIsActive("Notes", "N", DBPath) + " AND N.TypeId IN (SELECT TypeId FROM NoteTypes WHERE Type IN ('reblog', 'reply', 'posted')) AND B.HasBeenOutput = 0 GROUP BY N.NoteBlogId ORDER BY count(*) DESC, B.BlogName LIMIT @top";
else else
sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive" + AndIsActive("Notes", "notes", DBPath) + " AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top"; sql = "SELECT B.BlogName as blogName, count(*) FROM Notes N INNER JOIN Blogs B ON B.BlogId = N.NoteBlogId WHERE B.IsActive = @isActive" + AndIsActive("Notes", "N", DBPath) + " AND B.HasBeenOutput = 0 GROUP BY N.NoteBlogId ORDER BY count(*) DESC, B.BlogName LIMIT @top";
using (SQLiteCommand command = new SQLiteCommand(sql, connection)) using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{ {
@@ -1356,7 +1462,10 @@ namespace URLNotesGrabberCORE
connection.Open(); connection.Open();
//string sql = "UPDATE Posts SET HasNotesGathered = 1, NotesGatheredDateTime = @notesGathered WHERE BlogName = @BlogName AND PostID = @PostID"; //string sql = "UPDATE Posts SET HasNotesGathered = 1, NotesGatheredDateTime = @notesGathered WHERE BlogName = @BlogName AND PostID = @PostID";
string sql = "UPDATE Posts SET HasNotesGathered = 1, NotesGatheredDateTime = @notesGathered, DateModified = @dateModified WHERE PostID = @PostID AND (IFNULL(HasNotesGathered, 0) <> 1 OR IFNULL(NotesGatheredDateTime, 0) <> @notesGathered)"; // NotesGatheredDateTime is crawl bookkeeping -- it moves on every pass and says
// nothing about the post itself, so only the HasNotesGathered flag flipping
// counts as a modification. The CASE reads the pre-UPDATE value of the flag.
string sql = "UPDATE Posts SET HasNotesGathered = 1, NotesGatheredDateTime = @notesGathered, DateModified = CASE WHEN IFNULL(HasNotesGathered, 0) <> 1 THEN @dateModified ELSE DateModified END WHERE PostID = @PostID AND (IFNULL(HasNotesGathered, 0) <> 1 OR IFNULL(NotesGatheredDateTime, 0) <> @notesGathered)";
using (SQLiteCommand command = new SQLiteCommand(sql, connection)) using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{ {
command.Parameters.AddWithValue("@notesGathered", DateTimeOffset.UtcNow.ToUnixTimeSeconds()); command.Parameters.AddWithValue("@notesGathered", DateTimeOffset.UtcNow.ToUnixTimeSeconds());
@@ -1518,7 +1627,10 @@ namespace URLNotesGrabberCORE
{ {
connection.Open(); connection.Open();
string sql = "UPDATE Notes SET timestamp = @timestamp, DateModified = @dateModified WHERE rootBlogName = @rootBlogName AND noteBlogName = @noteBlogName AND PostID = @postID AND IFNULL(timestamp, 0) <> @timestamp"; string sql = "UPDATE Notes SET TimeStamp = @timestamp, DateModified = @dateModified " +
"WHERE RootBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @rootBlogName) " +
"AND NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName) " +
"AND PostID = @postID AND IFNULL(TimeStamp, 0) <> @timestamp";
using (SQLiteCommand command = new SQLiteCommand(sql, connection)) using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{ {
command.Parameters.AddWithValue("@timestamp", timestamp); command.Parameters.AddWithValue("@timestamp", timestamp);
@@ -1532,7 +1644,7 @@ namespace URLNotesGrabberCORE
catch (Exception ex) catch (Exception ex)
{ {
// Breakpoint here // Breakpoint here
if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Notes.RootBlogName, Notes.PostID, Notes.TimeStamp, Notes.Type, Notes.NoteBlogName") if (!IsNotesDuplicateKey(ex))
{ {
Console.WriteLine(ex.Message); Console.WriteLine(ex.Message);
Console.WriteLine("^^^^^ - SHORTCUT"); Console.WriteLine("^^^^^ - SHORTCUT");
@@ -1558,49 +1670,60 @@ namespace URLNotesGrabberCORE
{ {
if (ownsConnection) connection.Open(); if (ownsConnection) connection.Open();
// "." is TraverseDirectory/ReblogRecord's sentinel for "this field had no
// matching line in this particular export file" -- not an empty value. A blog
// with two export files in different formats (e.g. an "_2" duplicate folder, or
// a Tumblr export whose field set changed over time) sends one record with a real
// Title and another with Title = "." for the same PostID, and re-importing both
// on every run must not let the "not supplied" record blank out what the other
// one has. Every content field below is CASE-guarded the same way RootBlogName/
// RootURL already were, and the change-detection ignores "." too so a "."-only
// difference doesn't fire the UPDATE (and bump DateModified) on its own. Only "."
// is treated as the sentinel -- an explicit empty string from a real field still
// overwrites, same as before.
string sql = "UPDATE Posts SET "; string sql = "UPDATE Posts SET ";
sql += "postDate = @postDate, "; sql += "postDate = CASE WHEN @postDate = '.' THEN postDate ELSE @postDate END, ";
sql += "reblogURL = @reblogURL, "; sql += "reblogURL = CASE WHEN @reblogURL = '.' THEN reblogURL ELSE @reblogURL END, ";
sql += "postURL = @postURL, "; sql += "postURL = CASE WHEN @postURL = '.' THEN postURL ELSE @postURL END, ";
sql += "slug = @slug, "; sql += "slug = CASE WHEN @slug = '.' THEN slug ELSE @slug END, ";
sql += "reblogKey = @reblogKey, "; sql += "reblogKey = CASE WHEN @reblogKey = '.' THEN reblogKey ELSE @reblogKey END, ";
sql += "reblogName = @reblogName, "; sql += "reblogName = CASE WHEN @reblogName = '.' THEN reblogName ELSE @reblogName END, ";
sql += "summary = @summary, "; sql += "summary = CASE WHEN @summary = '.' THEN summary ELSE @summary END, ";
sql += "quote = @quote, "; sql += "quote = CASE WHEN @quote = '.' THEN quote ELSE @quote END, ";
sql += "body = @body, "; sql += "body = CASE WHEN @body = '.' THEN body ELSE @body END, ";
sql += "tags = @tags, "; sql += "tags = CASE WHEN @tags = '.' THEN tags ELSE @tags END, ";
sql += "link = @link, "; sql += "link = CASE WHEN @link = '.' THEN link ELSE @link END, ";
sql += "photoURL = @photoURL, "; sql += "photoURL = CASE WHEN @photoURL = '.' THEN photoURL ELSE @photoURL END, ";
sql += "photoCaption = @photoCaption, "; sql += "photoCaption = CASE WHEN @photoCaption = '.' THEN photoCaption ELSE @photoCaption END, ";
sql += "downloadedFiles = @downloadedFiles, "; sql += "downloadedFiles = CASE WHEN @downloadedFiles = '.' THEN downloadedFiles ELSE @downloadedFiles END, ";
sql += "audioCaption = @audioCaption, "; sql += "audioCaption = CASE WHEN @audioCaption = '.' THEN audioCaption ELSE @audioCaption END, ";
sql += "question = @question, "; sql += "question = CASE WHEN @question = '.' THEN question ELSE @question END, ";
sql += "answer = @answer, "; sql += "answer = CASE WHEN @answer = '.' THEN answer ELSE @answer END, ";
sql += "title = @title, "; sql += "title = CASE WHEN @title = '.' THEN title ELSE @title END, ";
sql += "DateModified = @dateModified, "; sql += "DateModified = @dateModified, ";
sql += "RootBlogName = CASE WHEN @rootBlogName IS NULL OR @rootBlogName = '' OR @rootBlogName = '.' THEN RootBlogName ELSE @rootBlogName END, "; sql += "RootBlogName = CASE WHEN @rootBlogName IS NULL OR @rootBlogName = '' OR @rootBlogName = '.' THEN RootBlogName ELSE @rootBlogName END, ";
sql += "RootURL = CASE WHEN @rootURL IS NULL OR @rootURL = '' OR @rootURL = '.' THEN RootURL ELSE @rootURL END, "; sql += "RootURL = CASE WHEN @rootURL IS NULL OR @rootURL = '' OR @rootURL = '.' THEN RootURL ELSE @rootURL END, ";
sql += "hasImage = @hasImage, "; sql += "hasImage = @hasImage, ";
sql += "ByLikes = MAX(IFNULL(ByLikes, 0), @byLikes) "; sql += "ByLikes = MAX(IFNULL(ByLikes, 0), @byLikes) ";
sql += " WHERE BlogName = @BlogName AND PostID = @PostID AND ("; sql += " WHERE BlogName = @BlogName AND PostID = @PostID AND (";
sql += "IFNULL(postDate, '') <> @postDate OR "; sql += "(@postDate <> '.' AND IFNULL(postDate, '') <> @postDate) OR ";
sql += "IFNULL(reblogURL, '') <> @reblogURL OR "; sql += "(@reblogURL <> '.' AND IFNULL(reblogURL, '') <> @reblogURL) OR ";
sql += "IFNULL(postURL, '') <> @postURL OR "; sql += "(@postURL <> '.' AND IFNULL(postURL, '') <> @postURL) OR ";
sql += "IFNULL(slug, '') <> @slug OR "; sql += "(@slug <> '.' AND IFNULL(slug, '') <> @slug) OR ";
sql += "IFNULL(reblogKey, '') <> @reblogKey OR "; sql += "(@reblogKey <> '.' AND IFNULL(reblogKey, '') <> @reblogKey) OR ";
sql += "IFNULL(reblogName, '') <> @reblogName OR "; sql += "(@reblogName <> '.' AND IFNULL(reblogName, '') <> @reblogName) OR ";
sql += "IFNULL(summary, '') <> @summary OR "; sql += "(@summary <> '.' AND IFNULL(summary, '') <> @summary) OR ";
sql += "IFNULL(quote, '') <> @quote OR "; sql += "(@quote <> '.' AND IFNULL(quote, '') <> @quote) OR ";
sql += "IFNULL(body, '') <> @body OR "; sql += "(@body <> '.' AND IFNULL(body, '') <> @body) OR ";
sql += "IFNULL(tags, '') <> @tags OR "; sql += "(@tags <> '.' AND IFNULL(tags, '') <> @tags) OR ";
sql += "IFNULL(link, '') <> @link OR "; sql += "(@link <> '.' AND IFNULL(link, '') <> @link) OR ";
sql += "IFNULL(photoURL, '') <> @photoURL OR "; sql += "(@photoURL <> '.' AND IFNULL(photoURL, '') <> @photoURL) OR ";
sql += "IFNULL(photoCaption, '') <> @photoCaption OR "; sql += "(@photoCaption <> '.' AND IFNULL(photoCaption, '') <> @photoCaption) OR ";
sql += "IFNULL(downloadedFiles, '') <> @downloadedFiles OR "; sql += "(@downloadedFiles <> '.' AND IFNULL(downloadedFiles, '') <> @downloadedFiles) OR ";
sql += "IFNULL(audioCaption, '') <> @audioCaption OR "; sql += "(@audioCaption <> '.' AND IFNULL(audioCaption, '') <> @audioCaption) OR ";
sql += "IFNULL(question, '') <> @question OR "; sql += "(@question <> '.' AND IFNULL(question, '') <> @question) OR ";
sql += "IFNULL(answer, '') <> @answer OR "; sql += "(@answer <> '.' AND IFNULL(answer, '') <> @answer) OR ";
sql += "IFNULL(title, '') <> @title OR "; sql += "(@title <> '.' AND IFNULL(title, '') <> @title) OR ";
sql += "IFNULL(hasImage, 0) <> @hasImage OR "; sql += "IFNULL(hasImage, 0) <> @hasImage OR ";
sql += "(@byLikes = 1 AND IFNULL(ByLikes, 0) = 0) OR "; sql += "(@byLikes = 1 AND IFNULL(ByLikes, 0) = 0) OR ";
sql += "((@rootBlogName IS NOT NULL AND @rootBlogName <> '' AND @rootBlogName <> '.') AND IFNULL(RootBlogName, '') <> @rootBlogName) OR "; sql += "((@rootBlogName IS NOT NULL AND @rootBlogName <> '' AND @rootBlogName <> '.') AND IFNULL(RootBlogName, '') <> @rootBlogName) OR ";
@@ -1713,7 +1836,8 @@ namespace URLNotesGrabberCORE
string sql = @"UPDATE Blogs string sql = @"UPDATE Blogs
SET LikesNewestTimestamp = MAX(COALESCE(LikesNewestTimestamp, 0), @newest), SET LikesNewestTimestamp = MAX(COALESCE(LikesNewestTimestamp, 0), @newest),
DateModified = @modified DateModified = @modified
WHERE BlogName = @name"; WHERE BlogName = @name
AND COALESCE(LikesNewestTimestamp, 0) < @newest";
using (SQLiteCommand command = new SQLiteCommand(sql, connection)) using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{ {
command.Parameters.AddWithValue("@newest", newestTimestamp); command.Parameters.AddWithValue("@newest", newestTimestamp);
@@ -1806,10 +1930,15 @@ namespace URLNotesGrabberCORE
{ {
connection.Open(); connection.Open();
//string sql = "UPDATE Notes SET replyText = @replyText WHERE rootBlogName = @rootBlogName AND PostID = @PostID AND noteBlogName = @noteBlogName AND TimeStamp = @TimeStamp AND Type = 'reply'";
// Match on (noteBlogName, TimeStamp ±5s) only - a reply by a given blog at a given timestamp is the same reply across the original post and every reblog of it, so this fans out across reblog chains in one shot. Tolerance absorbs the ~1s drift between what -collect stored and what mode=conversation returns now. // Match on (noteBlogName, TimeStamp ±5s) only - a reply by a given blog at a given timestamp is the same reply across the original post and every reblog of it, so this fans out across reblog chains in one shot. Tolerance absorbs the ~1s drift between what -collect stored and what mode=conversation returns now.
// Only fan out to rows that match the SELECT criteria in GetRepliesWithFilledText (NULL/empty/legacy-'.'). Never overwrite '?' (confirmed-empty) or already-fetched text. // Only fan out to rows that match the SELECT criteria in GetRepliesWithFilledText (NULL/empty/legacy-'.'). Never overwrite '?' (confirmed-empty) or already-fetched text.
string sql = "UPDATE Notes SET replyText = @replyText, DateModified = @dateModified WHERE noteBlogName = @noteBlogName AND ABS(TimeStamp - @TimeStamp) <= 5 AND Type = 'reply' AND (replyText IS NULL OR replyText = '' OR replyText = '.')"; // The ABS() term cannot use an index on TimeStamp, before or after the integer schema; the NoteBlogId probe is what keeps this off a full scan.
string sql = "UPDATE Notes SET replyText = @replyText, DateModified = @dateModified " +
"WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName) " +
"AND ABS(TimeStamp - @TimeStamp) <= 5 " +
"AND TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply') " +
"AND (replyText IS NULL OR replyText = '' OR replyText = '.') " +
"AND (replyText IS NULL OR replyText <> @replyText)";
using (SQLiteCommand command = new SQLiteCommand(sql, connection)) using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{ {
command.Parameters.AddWithValue("@replyText", replyText ?? "?"); command.Parameters.AddWithValue("@replyText", replyText ?? "?");
@@ -1850,7 +1979,11 @@ namespace URLNotesGrabberCORE
{ {
connection.Open(); connection.Open();
string sql = "UPDATE Notes SET replyText = @replyText, DateModified = @dateModified WHERE rootBlogName = @rootBlogName AND PostID = @PostID AND Type = 'reply' AND IFNULL(replyText, '.') <> @replyText"; string sql = "UPDATE Notes SET replyText = @replyText, DateModified = @dateModified " +
"WHERE RootBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @rootBlogName) " +
"AND PostID = @PostID " +
"AND TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply') " +
"AND IFNULL(replyText, '.') <> @replyText";
using (SQLiteCommand command = new SQLiteCommand(sql, connection)) using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{ {
command.Parameters.AddWithValue("@replyText", replyText ?? "."); command.Parameters.AddWithValue("@replyText", replyText ?? ".");
@@ -2044,29 +2177,65 @@ namespace URLNotesGrabberCORE
if (rowsInserted == 0) if (rowsInserted == 0)
{ {
// NULL is this function's sentinel for "this file's record had no line for
// that field" (IngestMode's G(key) misses return null; LegacyPostsDbImporter
// passes null straight from a NULL source column) -- it does not mean "clear
// this field". --ingest's entire reason to exist is reconciling multiple
// export files for the same (BlogName, PostID) -- IngestMode normalizes a
// "_2"-suffixed duplicate folder onto the same blog name specifically so a
// second, differently-formatted file for a post it already has gets merged in.
// Files are walked in filesystem enumeration order, not sorted, so which
// file's UpsertPostFromTextFile call runs last for a given PostID is
// effectively arbitrary. An unconditional SET here would let whichever file
// processed last silently null out every column its own record didn't carry,
// erasing real content the other file had -- the opposite of "clean up". Each
// column is CASE-guarded to keep the existing value when this call's parameter
// is NULL, and the change-detection ignores a NULL-vs-real mismatch the same
// way, so a partial record converges into the row instead of overwriting it.
string updateSql = @"UPDATE Posts SET string updateSql = @"UPDATE Posts SET
reblogURL = @reblogURL, reblogURL = CASE WHEN @reblogURL IS NULL THEN reblogURL ELSE @reblogURL END,
PostDate = @PostDate, PostDate = CASE WHEN @PostDate IS NULL THEN PostDate ELSE @PostDate END,
PostURL = @PostURL, PostURL = CASE WHEN @PostURL IS NULL THEN PostURL ELSE @PostURL END,
Slug = @Slug, Slug = CASE WHEN @Slug IS NULL THEN Slug ELSE @Slug END,
ReblogKey = @ReblogKey, ReblogKey = CASE WHEN @ReblogKey IS NULL THEN ReblogKey ELSE @ReblogKey END,
ReblogName = @ReblogName, ReblogName = CASE WHEN @ReblogName IS NULL THEN ReblogName ELSE @ReblogName END,
Summary = @Summary, Summary = CASE WHEN @Summary IS NULL THEN Summary ELSE @Summary END,
Quote = @Quote, Quote = CASE WHEN @Quote IS NULL THEN Quote ELSE @Quote END,
Body = @Body, Body = CASE WHEN @Body IS NULL THEN Body ELSE @Body END,
Tags = @Tags, Tags = CASE WHEN @Tags IS NULL THEN Tags ELSE @Tags END,
Link = @Link, Link = CASE WHEN @Link IS NULL THEN Link ELSE @Link END,
PhotoURL = @PhotoURL, PhotoURL = CASE WHEN @PhotoURL IS NULL THEN PhotoURL ELSE @PhotoURL END,
PhotoCaption = @PhotoCaption, PhotoCaption = CASE WHEN @PhotoCaption IS NULL THEN PhotoCaption ELSE @PhotoCaption END,
DownloadedFiles = @DownloadedFiles, DownloadedFiles = CASE WHEN @DownloadedFiles IS NULL THEN DownloadedFiles ELSE @DownloadedFiles END,
AudioCaption = @AudioCaption, AudioCaption = CASE WHEN @AudioCaption IS NULL THEN AudioCaption ELSE @AudioCaption END,
Question = @Question, Question = CASE WHEN @Question IS NULL THEN Question ELSE @Question END,
Answer = @Answer, Answer = CASE WHEN @Answer IS NULL THEN Answer ELSE @Answer END,
Title = @Title, Title = CASE WHEN @Title IS NULL THEN Title ELSE @Title END,
PostType = @PostType, PostType = CASE WHEN @PostType IS NULL THEN PostType ELSE @PostType END,
HasImage = @HasImage, HasImage = @HasImage,
DateModified = @DateModified DateModified = @DateModified
WHERE BlogName = @BlogName AND PostID = @PostID"; WHERE BlogName = @BlogName AND PostID = @PostID AND (
(@reblogURL IS NOT NULL AND IFNULL(reblogURL, '') <> @reblogURL) OR
(@PostDate IS NOT NULL AND IFNULL(PostDate, '') <> @PostDate) OR
(@PostURL IS NOT NULL AND IFNULL(PostURL, '') <> @PostURL) OR
(@Slug IS NOT NULL AND IFNULL(Slug, '') <> @Slug) OR
(@ReblogKey IS NOT NULL AND IFNULL(ReblogKey, '') <> @ReblogKey) OR
(@ReblogName IS NOT NULL AND IFNULL(ReblogName, '') <> @ReblogName) OR
(@Summary IS NOT NULL AND IFNULL(Summary, '') <> @Summary) OR
(@Quote IS NOT NULL AND IFNULL(Quote, '') <> @Quote) OR
(@Body IS NOT NULL AND IFNULL(Body, '') <> @Body) OR
(@Tags IS NOT NULL AND IFNULL(Tags, '') <> @Tags) OR
(@Link IS NOT NULL AND IFNULL(Link, '') <> @Link) OR
(@PhotoURL IS NOT NULL AND IFNULL(PhotoURL, '') <> @PhotoURL) OR
(@PhotoCaption IS NOT NULL AND IFNULL(PhotoCaption, '') <> @PhotoCaption) OR
(@DownloadedFiles IS NOT NULL AND IFNULL(DownloadedFiles, '') <> @DownloadedFiles) OR
(@AudioCaption IS NOT NULL AND IFNULL(AudioCaption, '') <> @AudioCaption) OR
(@Question IS NOT NULL AND IFNULL(Question, '') <> @Question) OR
(@Answer IS NOT NULL AND IFNULL(Answer, '') <> @Answer) OR
(@Title IS NOT NULL AND IFNULL(Title, '') <> @Title) OR
(@PostType IS NOT NULL AND IFNULL(PostType, '') <> @PostType) OR
IFNULL(HasImage, 0) <> @HasImage
)";
using (var cmd = new SQLiteCommand(updateSql, connection)) using (var cmd = new SQLiteCommand(updateSql, connection))
{ {
@@ -2250,7 +2419,11 @@ namespace URLNotesGrabberCORE
}; };
} }
public static void SetBlogTTFolderPath(string blogName, string? path, string? DBPath = null) // Returns true only when a row's TTFolderPath actually changed. A false means either
// the row already held this value or no row matched the name -- callers must not
// report a write they did not get, which is how a --updatepaths run could once print
// "Updated <blog>" for every metadata file while leaving the column entirely NULL.
public static bool SetBlogTTFolderPath(string blogName, string? path, string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
try { AddBlog(blogName, false, DBPath); } catch { } try { AddBlog(blogName, false, DBPath); } catch { }
@@ -2258,24 +2431,40 @@ namespace URLNotesGrabberCORE
using var connection = new SQLiteConnection("Data Source=" + DBPath); using var connection = new SQLiteConnection("Data Source=" + DBPath);
connection.Open(); connection.Open();
using var cmd = new SQLiteCommand( using var cmd = new SQLiteCommand(
"UPDATE Blogs SET TTFolderPath = @path, DateModified = @modified WHERE BlogName = @name", "UPDATE Blogs SET TTFolderPath = @path, DateModified = @modified WHERE BlogName = @name AND IFNULL(TTFolderPath, '') <> IFNULL(@path, '')",
connection); connection);
cmd.Parameters.AddWithValue("@path", (object?)path ?? DBNull.Value); cmd.Parameters.AddWithValue("@path", (object?)path ?? DBNull.Value);
cmd.Parameters.AddWithValue("@modified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss")); cmd.Parameters.AddWithValue("@modified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
cmd.Parameters.AddWithValue("@name", blogName); cmd.Parameters.AddWithValue("@name", blogName);
cmd.ExecuteNonQuery(); return cmd.ExecuteNonQuery() > 0;
}
// Whether a Blogs row exists under this exact name. BlogName is a BINARY-collated
// primary key, so a metadata filename that differs only in case is a different blog
// as far as the UPDATE above is concerned -- worth telling the user about.
public static bool BlogExists(string blogName, string? DBPath = null)
{
DBPath ??= GetDefaultDbPath();
using var connection = new SQLiteConnection("Data Source=" + DBPath);
connection.Open();
using var cmd = new SQLiteCommand("SELECT 1 FROM Blogs WHERE BlogName = @name", connection);
cmd.Parameters.AddWithValue("@name", blogName);
return cmd.ExecuteScalar() != null;
} }
// Partial UPDATE used by the correct-apply path. fieldsToUpdate maps // Partial UPDATE used by the correct-apply path. fieldsToUpdate maps
// ThreeTxtFileHelper prefix names ("Reblog URL", "Body", etc.) to non-empty // ThreeTxtFileHelper prefix names ("Reblog URL", "Body", etc.) to non-empty
// values pulled from a BAK file. Only those columns + DateModified are written; // values pulled from a BAK file. Only those columns + DateModified are written;
// other content columns and all engagement columns are left intact. // other content columns and all engagement columns are left intact.
// Returns true if a row was matched (and therefore updated). // Returns true if a row was actually changed. A row whose columns already hold
// the incoming values is left alone, DateModified included.
public static bool UpdatePostContentFields(string blogName, string postId, IDictionary<string, string> fieldsToUpdate, string? DBPath = null) public static bool UpdatePostContentFields(string blogName, string postId, IDictionary<string, string> fieldsToUpdate, string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
var setClauses = new List<string>(); var setClauses = new List<string>();
var changedClauses = new List<string>();
var parameters = new List<(string Name, object Value)>(); var parameters = new List<(string Name, object Value)>();
foreach (var kvp in fieldsToUpdate) foreach (var kvp in fieldsToUpdate)
@@ -2285,6 +2474,7 @@ namespace URLNotesGrabberCORE
if (column == null) continue; if (column == null) continue;
string paramName = "@p" + parameters.Count; string paramName = "@p" + parameters.Count;
setClauses.Add($"{column} = {paramName}"); setClauses.Add($"{column} = {paramName}");
changedClauses.Add($"IFNULL({column}, '') <> {paramName}");
parameters.Add((paramName, kvp.Value)); parameters.Add((paramName, kvp.Value));
} }
@@ -2296,7 +2486,7 @@ namespace URLNotesGrabberCORE
using var connection = new SQLiteConnection("Data Source=" + DBPath); using var connection = new SQLiteConnection("Data Source=" + DBPath);
connection.Open(); connection.Open();
string sql = $"UPDATE Posts SET {string.Join(", ", setClauses)} WHERE BlogName = @BlogName AND PostID = @PostID"; string sql = $"UPDATE Posts SET {string.Join(", ", setClauses)} WHERE BlogName = @BlogName AND PostID = @PostID AND ({string.Join(" OR ", changedClauses)})";
using var cmd = new SQLiteCommand(sql, connection); using var cmd = new SQLiteCommand(sql, connection);
foreach (var (name, value) in parameters) foreach (var (name, value) in parameters)
cmd.Parameters.AddWithValue(name, value); cmd.Parameters.AddWithValue(name, value);
@@ -2334,24 +2524,48 @@ namespace URLNotesGrabberCORE
}; };
} }
public static List<(string BlogName, string? TTFolderPath)> GetAllBlogsWithTTFolderPath(string? DBPath = null) // Export targets only: active blogs that actually carry a TTFolderPath.
// Blogs is a 144k-row crawl registry and only the few hundred blogs downloaded
// locally have a folder, so returning the unset rows made --output print a skip
// line for every blog Tumblr has ever handed us.
public static List<(string BlogName, string TTFolderPath)> GetAllBlogsWithTTFolderPath(string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
var results = new List<(string, string?)>(); var results = new List<(string, string)>();
using var connection = new SQLiteConnection("Data Source=" + DBPath); using var connection = new SQLiteConnection("Data Source=" + DBPath);
connection.Open(); connection.Open();
using var cmd = new SQLiteCommand("SELECT BlogName, TTFolderPath FROM Blogs WHERE IsActive = 1", connection); using var cmd = new SQLiteCommand(
"SELECT BlogName, TRIM(TTFolderPath) FROM Blogs WHERE IsActive = 1 AND IFNULL(TRIM(TTFolderPath), '') <> '' ORDER BY BlogName",
connection);
using var reader = cmd.ExecuteReader(); using var reader = cmd.ExecuteReader();
while (reader.Read()) while (reader.Read())
{ results.Add((reader.GetString(0), reader.GetString(1)));
string name = reader.GetString(0);
string? path = reader.IsDBNull(1) ? null : reader.GetString(1);
results.Add((name, path));
}
return results; return results;
} }
// Companion counts for the messages --output and --updatepaths print about coverage.
public static int CountActiveBlogs(string? DBPath = null)
{
DBPath ??= GetDefaultDbPath();
using var connection = new SQLiteConnection("Data Source=" + DBPath);
connection.Open();
using var cmd = new SQLiteCommand("SELECT COUNT(*) FROM Blogs WHERE IsActive = 1", connection);
return Convert.ToInt32(cmd.ExecuteScalar());
}
public static int CountBlogsWithTTFolderPath(string? DBPath = null)
{
DBPath ??= GetDefaultDbPath();
using var connection = new SQLiteConnection("Data Source=" + DBPath);
connection.Open();
using var cmd = new SQLiteCommand(
"SELECT COUNT(*) FROM Blogs WHERE IFNULL(TRIM(TTFolderPath), '') <> ''", connection);
return Convert.ToInt32(cmd.ExecuteScalar());
}
private static string SafeStr(SQLiteDataReader reader, int ordinal) private static string SafeStr(SQLiteDataReader reader, int ordinal)
{ {
return reader.IsDBNull(ordinal) ? string.Empty : reader.GetValue(ordinal)?.ToString() ?? string.Empty; return reader.IsDBNull(ordinal) ? string.Empty : reader.GetValue(ordinal)?.ToString() ?? string.Empty;
+12 -3
View File
@@ -29,6 +29,8 @@ namespace URLNotesGrabberCORE
Console.WriteLine($"Reading legacy posts.db: {legacyDbPath}"); Console.WriteLine($"Reading legacy posts.db: {legacyDbPath}");
int blogsCopied = 0; int blogsCopied = 0;
int blogPathsWritten = 0;
int blogsWithoutPath = 0;
int postsUpserted = 0; int postsUpserted = 0;
int errors = 0; int errors = 0;
@@ -48,7 +50,13 @@ namespace URLNotesGrabberCORE
if (string.IsNullOrWhiteSpace(blogName)) continue; if (string.IsNullOrWhiteSpace(blogName)) continue;
try try
{ {
DataAccess.SetBlogTTFolderPath(blogName, ttFolderPath); // A legacy row whose TTFolderPath was already NULL copies nothing.
// Counting it as "copied" is what hid the fact that this import has
// never populated a single path.
if (string.IsNullOrWhiteSpace(ttFolderPath))
blogsWithoutPath++;
else if (DataAccess.SetBlogTTFolderPath(blogName, ttFolderPath.Trim()))
blogPathsWritten++;
blogsCopied++; blogsCopied++;
} }
catch (Exception ex) catch (Exception ex)
@@ -58,7 +66,7 @@ namespace URLNotesGrabberCORE
} }
} }
} }
Console.WriteLine($" Blogs copied: {blogsCopied}"); Console.WriteLine($" Blogs seen: {blogsCopied}, TTFolderPath written: {blogPathsWritten}, legacy rows with no path: {blogsWithoutPath}");
// 2) Copy Posts // 2) Copy Posts
try try
@@ -138,7 +146,8 @@ namespace URLNotesGrabberCORE
} }
Console.WriteLine($"\n========== Legacy import summary =========="); Console.WriteLine($"\n========== Legacy import summary ==========");
Console.WriteLine($"Blogs copied: {blogsCopied}"); Console.WriteLine($"Blogs seen: {blogsCopied}");
Console.WriteLine($"Paths written: {blogPathsWritten} (legacy rows with no path: {blogsWithoutPath})");
Console.WriteLine($"Posts upserted: {postsUpserted}"); Console.WriteLine($"Posts upserted: {postsUpserted}");
Console.WriteLine($"Errors: {errors}"); Console.WriteLine($"Errors: {errors}");
return errors == 0 ? 0 : 2; return errors == 0 ? 0 : 2;
+81 -11
View File
@@ -8,28 +8,51 @@ namespace URLNotesGrabberCORE
// field order). Reads from TL.db via DataAccess.GetAllPostsForBlog. // field order). Reads from TL.db via DataAccess.GetAllPostsForBlog.
public static class OutputMode public static class OutputMode
{ {
public static int Run(IConfiguration config) public static int Run(IConfiguration config, string[]? args = null)
{ {
DataAccess.EnsureTTFileHelperColumnsExist(); DataAccess.EnsureTTFileHelperColumnsExist();
var blogs = DataAccess.GetAllBlogsWithTTFolderPath(); string dbPath = DataAccess.GetActiveDbPath();
Console.WriteLine($"Found {blogs.Count} blog(s) to process."); Console.WriteLine($"Database: {Path.GetFullPath(dbPath)}");
foreach (var (blogName, ttFolderPath) in blogs) if (!RefreshPaths(config, args ?? Array.Empty<string>()))
return 1;
var blogs = DataAccess.GetAllBlogsWithTTFolderPath();
int activeBlogs = DataAccess.CountActiveBlogs();
Console.WriteLine($"{blogs.Count} of {activeBlogs} active blog(s) have a TTFolderPath.");
if (blogs.Count == 0)
{
Console.WriteLine($"\nNothing to export: no blog in {Path.GetFullPath(dbPath)} has a TTFolderPath.");
Console.WriteLine("Point --output at a TumblThree root so it can populate them: --output <root>,");
Console.WriteLine("or set appSettings:PathTTRoot so the refresh runs automatically.");
return 1;
}
int missingFolderCount = 0;
int writtenCount = 0;
foreach (var (blogName, folder) in blogs)
{ {
Console.WriteLine($"\nProcessing blog: {blogName}"); Console.WriteLine($"\nProcessing blog: {blogName}");
if (string.IsNullOrWhiteSpace(ttFolderPath) || !Directory.Exists(ttFolderPath)) // A stored path that this machine cannot see means the value was written on
// another machine -- re-running --updatepaths locally is the fix, so say so
// rather than lumping it in with "not set".
if (!Directory.Exists(folder))
{ {
Console.WriteLine($" TTFolderPath does not exist or is not set. Skipping."); Console.WriteLine($" TTFolderPath folder not found: {folder}. Skipping.");
missingFolderCount++;
continue; continue;
} }
Console.WriteLine($" TTFolderPath: {ttFolderPath}"); Console.WriteLine($" TTFolderPath: {folder}");
writtenCount++;
try try
{ {
foreach (var bakFile in Directory.GetFiles(ttFolderPath, "*.bak")) foreach (var bakFile in Directory.GetFiles(folder, "*.bak"))
File.Delete(bakFile); File.Delete(bakFile);
} }
catch (Exception ex) catch (Exception ex)
@@ -37,7 +60,7 @@ namespace URLNotesGrabberCORE
Console.WriteLine($" Error deleting .bak files: {ex.Message}"); Console.WriteLine($" Error deleting .bak files: {ex.Message}");
} }
RenameExistingTxtFilesToBak(ttFolderPath); RenameExistingTxtFilesToBak(folder);
var posts = DataAccess.GetAllPostsForBlog(blogName); var posts = DataAccess.GetAllPostsForBlog(blogName);
Console.WriteLine($" Found {posts.Count} post(s) for this blog."); Console.WriteLine($" Found {posts.Count} post(s) for this blog.");
@@ -46,7 +69,7 @@ namespace URLNotesGrabberCORE
foreach (var typeGroup in grouped) foreach (var typeGroup in grouped)
{ {
string postType = typeGroup.Key ?? "Unknown"; string postType = typeGroup.Key ?? "Unknown";
string outputFilePath = Path.Combine(ttFolderPath, $"{postType}.txt"); string outputFilePath = Path.Combine(folder, $"{postType}.txt");
var ordered = typeGroup.OrderBy(p => p.Date).ToList(); var ordered = typeGroup.OrderBy(p => p.Date).ToList();
Console.WriteLine($" Writing {ordered.Count} post(s) to {postType}.txt"); Console.WriteLine($" Writing {ordered.Count} post(s) to {postType}.txt");
@@ -65,10 +88,57 @@ namespace URLNotesGrabberCORE
} }
} }
Console.WriteLine("\nOutput mode complete."); Console.WriteLine($"\nOutput mode complete. {writtenCount} blog(s) exported, {missingFolderCount} skipped for a missing folder.");
if (writtenCount == 0)
Console.WriteLine("Every TTFolderPath points at a folder this machine cannot see. The paths were most likely written on another machine -- re-run --updatepaths <root> here so they match local drive letters.");
return 0; return 0;
} }
// Re-reads the TumblThree Index metadata into Blogs.TTFolderPath before exporting.
// A TL.db synced between machines cannot hold one absolute path that is valid on
// both, so the stored paths are only trustworthy on the machine that wrote them --
// which makes this refresh part of a normal export rather than a separate chore.
// Returns false only when the run should stop.
private static bool RefreshPaths(IConfiguration config, string[] args)
{
var settings = config.GetSection("appSettings");
if (args.Any(a => string.Equals(a, "--norefresh", StringComparison.OrdinalIgnoreCase)))
{
Console.WriteLine("Path refresh skipped (--norefresh); exporting to whatever paths TL.db already holds.");
return true;
}
string? root = args.FirstOrDefault(a => !a.StartsWith("--", StringComparison.Ordinal))
?? settings.GetValue<string>("PathTTRoot");
var result = UpdateBlogPathsRunner.Scan(root, verbose: false);
switch (result.Outcome)
{
case UpdateBlogPathsRunner.ScanOutcome.NoRootConfigured:
Console.WriteLine("No TumblThree root configured (appSettings:PathTTRoot is empty and none was passed),");
Console.WriteLine("so TTFolderPath was not refreshed. Pass one as --output <root> to refresh it.");
return true;
case UpdateBlogPathsRunner.ScanOutcome.IndexFolderMissing:
// Silently exporting stale paths here would defeat the point of folding
// the refresh in, so a bad root is a hard stop.
Console.WriteLine($"Index folder not found at: {result.IndexPath}");
Console.WriteLine("Fix the root (or pass --norefresh to export the paths already in TL.db).");
return false;
default:
Console.WriteLine($"Refreshed paths from {result.IndexPath}: " +
$"{result.MetadataFiles} metadata file(s), {result.Written} written, " +
$"{result.Unchanged} already correct, {result.NoLocation} without a location, " +
$"{result.NoMatchingRow} without a blog row, {result.Errors} error(s).");
return true;
}
}
private static void RenameExistingTxtFilesToBak(string folderPath) private static void RenameExistingTxtFilesToBak(string folderPath)
{ {
try try
+4 -2
View File
@@ -343,7 +343,7 @@ namespace URLNotesGrabberCORE
break; break;
case "--output": case "--output":
exitCode = OutputMode.Run(config); exitCode = OutputMode.Run(config, args.Skip(1).ToArray());
break; break;
case "--revert": case "--revert":
@@ -439,7 +439,9 @@ namespace URLNotesGrabberCORE
Console.WriteLine("--ingest [blogname]\t Ingest Tumblr .txt exports from appSettings:PathTTRoot into TL.db (all blogs, or single blog if name given)"); Console.WriteLine("--ingest [blogname]\t Ingest Tumblr .txt exports from appSettings:PathTTRoot into TL.db (all blogs, or single blog if name given)");
Console.WriteLine("--output\t Export posts from TL.db back to .txt files in each blog's TTFolderPath"); Console.WriteLine("--output [rootPath]\t Refresh Blogs.TTFolderPath from <root>\\Index (or appSettings:PathTTRoot), then export posts from TL.db back to .txt files in each blog's folder");
Console.WriteLine("--output --norefresh\t Export without refreshing TTFolderPath first");
Console.WriteLine("--revert [blogname]\t Recursively scan the PathInput tree and restore *.bak back to *.txt (current .txt saved as next-free .bkN); optional blogname filters by path substring"); Console.WriteLine("--revert [blogname]\t Recursively scan the PathInput tree and restore *.bak back to *.txt (current .txt saved as next-free .bkN); optional blogname filters by path substring");
Binary file not shown.
+411 -65
View File
@@ -4,12 +4,25 @@ The SQLite database behind **URLNotesGrabberCORE** and its sibling crawlers, and
[Rolodex](https://git.basso.land/jim/Rolodex) reads. [Rolodex](https://git.basso.land/jim/Rolodex) reads.
Everything below was read out of the live file, not inferred from code. Counts are as of Everything below was read out of the live file, not inferred from code. Counts are as of
**2026-07-29**; re-run the queries at the bottom to refresh them. **2026-08-07**; re-run the queries at the bottom to refresh them.
- Journal mode: **WAL**`TL.db-wal` and `TL.db-shm` live beside the file and are part of - Journal mode: **WAL**`TL.db-wal` and `TL.db-shm` live beside the file and are part of
the database. Copying `TL.db` alone gives you whatever was last checkpointed, not the the database. Copying `TL.db` alone gives you whatever was last checkpointed, not the
current state. current state.
- Page size: 4096. - Page size: 4096. File size: 148 MB.
> ### ⚠ Breaking change, 2026-08-07: `Notes` holds integer IDs, not names
>
> `Notes.RootBlogName`, `Notes.NoteBlogName` and `Notes.Type` **no longer exist**. They
> are now `RootBlogId`, `NoteBlogId` and `TypeId`, resolved through the new `BlogNames`
> and `NoteTypes` tables. Any query naming the old columns fails outright.
>
> There is no compatibility view. See [porting to the integer
> schema](#porting-to-the-integer-schema) for the old-to-new translation of every query
> shape the applications use.
>
> Applied by `../normalize-notes.sql`, which took the file from 207 MB to 148 MB. An
> earlier change the same day (`../shrink-db.sql`) took it from 267 MB to 207 MB.
--- ---
@@ -17,13 +30,20 @@ Everything below was read out of the live file, not inferred from code. Counts a
| Table | Rows | What it is | | Table | Rows | What it is |
|---|--:|---| |---|--:|---|
| `Blogs` | 144,367 | The crawl registry — one row per known blog, plus crawl-state flags | | `Blogs` | 188,620 | The crawl registry — one row per known blog, plus crawl-state flags |
| `Posts` | 14,589 | Stored post content. Only 3,602 blogs actually have any | | `Posts` | 22,468 | Stored post content. Only 3,867 blogs actually have any |
| `Notes` | 1,189,604 | The engagement graph: `NoteBlogName` acted on `(RootBlogName, PostID)` | | `Notes` | 1,182,333 | The engagement graph: `NoteBlogId` acted on `(RootBlogId, PostID)` |
The engagement graph is the interesting part. 31,888 distinct blogs appear as engagers — …supported by two lookup tables that exist only to keep `Notes` small:
far more than the 3,602 that have stored posts — which is what makes this a social graph
rather than a post archive. | Table | Rows | What it is |
|---|--:|---|
| `BlogNames` | 20,430 | `BlogId``BlogName`. The ID authority for everything in `Notes` |
| `NoteTypes` | 5 | `TypeId``Type`. `like`, `reblog`, `reply`, `posted`, `post_attribution` |
The engagement graph is the interesting part. 20,311 distinct blogs appear as engagers —
far more than the 3,867 that have stored posts — which is what makes this a social graph
rather than a post archive. Only 2,771 blogs appear as the *root* of a note.
### `Blogs` ### `Blogs`
@@ -42,23 +62,45 @@ CREATE TABLE "Blogs" (
LikesLastRefreshed INTEGER DEFAULT 0, LikesLastRefreshed INTEGER DEFAULT 0,
LikesLastNewCount INTEGER DEFAULT 0, LikesLastNewCount INTEGER DEFAULT 0,
TTFolderPath TEXT, TTFolderPath TEXT,
BlogId INTEGER,
PRIMARY KEY("BlogName") PRIMARY KEY("BlogName")
); );
CREATE INDEX ix_Blogs_BlogId ON Blogs (BlogId);
``` ```
`BlogName` is the primary key, so it is the only indexed way in. There is no index on any `BlogName` is the primary key, so it is the only indexed way in by name. There is no index
flag or date — filtering or sorting on those scans all 144k rows, which is affordable on any flag or date — filtering or sorting on those scans all 188k rows, which is
here and is not on `Notes`. affordable here and is not on `Notes`.
Flag distribution: `IsActive = 1` on 144,366 of 144,367 rows, `HasBeenOutput = 1` on **`BlogId` is new as of 2026-08-07 and is the join key to `Notes`.** It exists so that
5,369, `ByLikes = 1` on 2. `IsActive` carries a second meaning as of Rolodex — see `Notes` can reach `Blogs` in a single integer hop rather than going through `BlogNames`
and ending in a text comparison:
```sql
-- what you want
FROM Blogs B JOIN Notes N ON N.NoteBlogId = B.BlogId
-- not this
FROM Blogs B JOIN BlogNames BN ON BN.BlogName = B.BlogName
JOIN Notes N ON N.NoteBlogId = BN.BlogId
```
**`BlogId` is NULL on 168,202 of 188,620 rows** — every blog that has never appeared in a
note. That is the large majority, and it is not an error: the registry is far bigger than
the engagement graph. An inner join on `BlogId` therefore silently drops those blogs,
which is usually what you want for engagement queries and is wrong for registry listings.
Flag distribution: `IsActive = 1` on 188,601 of 188,620 rows, `HasBeenOutput = 1` on
5,059, `ByLikes = 1` on 2. `IsActive` carries a second meaning as of Rolodex — see
[`Blogs.IsActive`](#blogsisactive--now-written-by-two-applications) below. [`Blogs.IsActive`](#blogsisactive--now-written-by-two-applications) below.
The columns after `DateCreated` were added later by `ALTER TABLE`, which is why they carry The columns after `DateCreated` were added later by `ALTER TABLE`, which is why they carry
no quoting in the stored DDL. That is the normal way this schema grows. no quoting in the stored DDL. That is the normal way this schema grows, and `BlogId` is
the newest example.
**`DateAdded` is not written consistently.** 126,423 rows hold ISO `yyyy-MM-dd HH:mm:ss`; **`DateAdded` is not written consistently.** 170,677 rows hold ISO `yyyy-MM-dd HH:mm:ss`;
17,944 hold US-format `M/d/yy` from a bulk import. As text those two sort into different 17,943 hold US-format `M/d/yy` from a bulk import. As text those two sort into different
parts of the table, so anything ordering or range-filtering on this column has to parts of the table, so anything ordering or range-filtering on this column has to
normalise first — see `DateSql` in Rolodex. normalise first — see `DateSql` in Rolodex.
@@ -100,74 +142,352 @@ CREATE TABLE "Posts" (
); );
``` ```
**The key is `(BlogName, PostID)`, not `PostID`.** This matters more than it looks: 325 **The key is `(BlogName, PostID)`, not `PostID`.** This matters more than it looks: 345
post IDs exist under more than one blog, so an ID on its own is both ambiguous *and* post IDs exist under more than one blog, so an ID on its own is both ambiguous *and*
unindexed. Any lookup should carry the blog name, and a batch lookup should group by blog unindexed. Any lookup should carry the blog name, and a batch lookup should group by blog
so it stays on the leading column of the key. so it stays on the leading column of the key.
Notable: Notable:
- **`PostType` is `NULL` on all 14,589 rows.** The column exists but nothing has ever - **`PostType` is now mostly populated: 20,679 of 22,468 rows, leaving 1,789 `NULL`.**
populated it. Treat it as unpopulated rather than as a type discriminator. This reverses what earlier revisions of this document said — the column really was empty
- `HasImage = 1` on 14,268 rows — nearly all of them. It records that the post *had* a on every row, and something has since started writing it. Anything that treated it as
picture, not that a usable URL was kept, so it is not a reliable predictor that anything permanently unset, or derived the type from post content instead, should be re-examined
will render. against the live data. Rolodex still derives it.
- `HasImage = 1` on 14,026 rows. It records that the post *had* a picture, not that a
usable URL was kept, so it is not a reliable predictor that anything will render.
- `PhotoURL` is largely unused; in practice the image markup lives inside `Body`. - `PhotoURL` is largely unused; in practice the image markup lives inside `Body`.
- `NotFound = 1` on 4,663 rows — posts that have since been deleted upstream. - `NotFound = 1` on 4,712 rows — posts that have since been deleted upstream.
- The content columns (`Body`, `Quote`, `Question`, `Answer`, …) are the heavy ones. List - The content columns (`Body`, `Quote`, `Question`, `Answer`, …) are the heavy ones. List
views should not select them. views should not select them.
### `Notes` ### `Notes`
```sql ```sql
CREATE TABLE "Notes" ( CREATE TABLE Notes (
"RootBlogName" TEXT, RootBlogId INTEGER NOT NULL,
"PostID" INTEGER, PostID INTEGER NOT NULL,
"NoteBlogName" TEXT, NoteBlogId INTEGER NOT NULL,
"TimeStamp" INTEGER, TimeStamp INTEGER NOT NULL,
"Type" TEXT, TypeId INTEGER NOT NULL,
"replyText" TEXT DEFAULT '.', replyText TEXT,
"DatetimeCrawled" TEXT DEFAULT '2/12/26 12am', DatetimeCrawled TEXT,
"DateModified" TEXT, DateModified TEXT,
"DateCreated" TEXT, DateCreated TEXT,
PRIMARY KEY("RootBlogName","PostID","TimeStamp","Type","NoteBlogName") IsActive INTEGER NOT NULL DEFAULT 1,
); PRIMARY KEY (RootBlogId, PostID, TimeStamp, TypeId, NoteBlogId)
) WITHOUT ROWID;
CREATE INDEX "Notes_idx_06e01ae3" ON "Notes" ("TimeStamp" DESC); CREATE INDEX ix_Notes_NoteBlogId ON Notes (NoteBlogId);
CREATE INDEX "ix_NoteBlogName01" ON "Notes" ("NoteBlogName");
``` ```
**Integer IDs since 2026-08-07 — this is the breaking change.** `RootBlogName`,
`NoteBlogName` and `Type` are gone, replaced by `RootBlogId`, `NoteBlogId` and `TypeId`.
Resolve them through [`BlogNames`](#blognames) and [`NoteTypes`](#notetypes), or join
straight to `Blogs` on `BlogId`. The old names were text repeated on 1.18 million rows,
in the table *and* in every index over it; the swap took the file from 207 MB to 148 MB.
The **primary key column order is deliberately unchanged**, so the leading-prefix access
patterns callers already depend on still hold: `(RootBlogId)` and `(RootBlogId, PostID)`
remain cheap prefixes, exactly as `(RootBlogName)` and `(RootBlogName, PostID)` were.
Two nulls-and-defaults differences from the old DDL, both intentional:
- `replyText` and `DatetimeCrawled` **no longer carry column defaults**. The old table
defaulted them to `'.'` and `'2/12/26 12am'`, which is how 1.1M rows acquired
placeholder values nobody wrote. New rows now get `NULL` unless a writer supplies
something. The crawler names both columns explicitly, so its behaviour is unchanged.
- The five key columns are now `NOT NULL`. They always were in practice.
**`WITHOUT ROWID`, since earlier the same day.** The rows live in the primary key's
b-tree rather than in a rowid table with a separate key index beside it. Two consequences
matter before adding an index here:
- There is no `rowid` on this table. `SELECT rowid FROM Notes` is an error, and no
code in any of the three apps relied on it.
- A secondary index carries the whole five-column primary key as its row reference
instead of a compact rowid, so indexes here are **expensive** — though far less so
than before, now that the key is five integers rather than three integers and two
strings. `ix_Notes_NoteBlogId` costs 27 MB; its text predecessor cost 58 MB.
**`Notes_idx_06e01ae3` on `TimeStamp DESC` was dropped at the same time.** It cost
14 MB as a rowid index and would have cost 58 MB after the conversion. It was worth
neither: the crawler's only `TimeStamp` filter (`>= 1535778000`) excludes 786 rows
of 1.18M, Rolodex's default Notes sort carries a three-column tiebreaker that forces
a full sort regardless, and the reply-matching `UPDATE` uses `ABS(TimeStamp - ?) <= 5`,
which no index on `TimeStamp` can serve. The one path that got slower is Rolodex's
Notes page with a date-range filter: 60 ms to 164 ms.
See `../shrink-db.sql` for the full rationale and the applied result.
**`DatetimeCrawled` is `NULL` on 1,148,077 rows, and that is the honest value.** Those
rows previously stored the literal string `'2/12/26 12am'` — this column's own DDL
default, written as a bulk backfill placeholder rather than as a crawl time. They were
set to `NULL` on 2026-08-07, which is what consumers already displayed them as: the
string parses as a date in neither format this schema writes.
Note the trap: **the `DEFAULT '2/12/26 12am'` clause is still in the DDL above.** Any
`INSERT` that omits this column writes the placeholder straight back. The crawler names
it explicitly on every insert, so nothing reintroduces it today, but a new writer that
forgets to would — which is why consumers should keep treating an unparseable value here
as "unknown" rather than assuming `NULL` is now the only such marker.
One row per engagement event. `TimeStamp` is **unix seconds** — unlike every date column One row per engagement event. `TimeStamp` is **unix seconds** — unlike every date column
elsewhere in the schema, which are text. elsewhere in the schema, which are text.
| `Type` | Rows | Share | At 1.18M rows this is the table that dictates how the whole database has to be queried:
|---|--:|--:|
| `like` | 947,955 | 79.7% |
| `reblog` | 224,323 | 18.9% |
| `reply` | 15,201 | 1.3% |
| `posted` | 2,106 | 0.2% |
| `post_attribution` | 19 | — |
At 1.19M rows this is the table that dictates how the whole database has to be queried:
- **Nothing should run an unbounded `SELECT` or a bare `COUNT(*)` here.** A count scans - **Nothing should run an unbounded `SELECT` or a bare `COUNT(*)` here.** A count scans
the lot on every call. the lot on every call.
- The only fast access paths are the primary key's leading columns (`RootBlogName`, then - The only fast access paths are the primary key's leading columns (`RootBlogId`, then
`PostID`) and `ix_NoteBlogName01` on `NoteBlogName`. "Notes received by a blog" and `PostID`) and `ix_Notes_NoteBlogId` on `NoteBlogId`. "Notes received by a blog" and
"notes given by a blog" are both cheap; almost nothing else is. "notes given by a blog" are both cheap; almost nothing else is.
- Ordering by anything but `TimeStamp` is a full sort of whatever the filters leave. - **Every** ordering here is a full sort of whatever the filters leave, `TimeStamp`
- `replyText` is `'.'` on 1,174,706 rows — only `reply` notes carry real text. included. Filter first, then sort.
- `replyText` is `'.'` on 1,167,464 rows — only `reply` notes carry real text. Those
dots are inherited from the old column default; new rows get `NULL` instead.
**Resolve IDs by filtering the lookup, not by scanning `Notes`.** The lookup tables are
tiny and uniquely indexed, so pushing a name predicate into them costs nothing and lets
the `Notes` index do the work:
```sql
-- good: BlogNames resolves the name, then the index is searched
SELECT * FROM Notes
WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = ?);
-- also good, same plan
SELECT n.* FROM Notes n
JOIN BlogNames b ON b.BlogId = n.NoteBlogId
WHERE b.BlogName = ?;
```
### `BlogNames`
```sql
CREATE TABLE BlogNames (
BlogId INTEGER PRIMARY KEY,
BlogName TEXT NOT NULL UNIQUE
);
```
20,430 rows — every name appearing in `Notes` as either participant, and nothing else.
This is the **ID authority**: `Notes.RootBlogId` and `Notes.NoteBlogId` both point here,
and `Blogs.BlogId` is a copy of the value for the blogs that have one.
**12 of these names have no `Blogs` row.** The registry has never been a superset of the
engagement graph and still is not, so resolving an ID through `Blogs` rather than
`BlogNames` will occasionally find nothing. Use `BlogNames` when you need the name itself
and `Blogs` when you need registry columns.
IDs are assigned by SQLite and are **stable**: they are stored in over a million `Notes`
rows. Never renumber them. A blog that is renamed upstream should get a new row, not an
edit to an existing one, unless every `Notes` reference is migrated with it.
### `NoteTypes`
```sql
CREATE TABLE NoteTypes (
TypeId INTEGER PRIMARY KEY,
Type TEXT NOT NULL UNIQUE
);
```
| `TypeId` | `Type` | Rows | Share |
|--:|---|--:|--:|
| 1 | `like` | 945,167 | 79.9% |
| 2 | `reblog` | 219,203 | 18.5% |
| 3 | `reply` | 15,345 | 1.3% |
| 4 | `posted` | 2,617 | 0.2% |
| 5 | `post_attribution` | 1 | — |
The set is fixed in practice, but it is a table rather than a `CHECK` constraint so that
adding a type is an `INSERT` and not a schema migration. **The IDs above are stored in
`Notes` and must not be reassigned.**
Five rows means the lookup is effectively free; write `t.Type = 'reblog'` and let SQLite
resolve it, or hardcode the ID if you prefer — both are fine, but hardcoding ties your
code to this table's contents, so prefer the join in anything long-lived.
---
## Porting to the integer schema
Everything here was checked against the live 148 MB file. There were 14 affected call
sites in `DataAccess.cs` and 16 in `RolodexRepository.cs`. TumblThree needs no changes —
its single statement touches `Blogs.IsActive` and `BlogName` only.
**`DataAccess.cs` is ported.** All 14 sites now read the integer schema, `AddNote`
registers names and types before inserting, and `verify-db-schema.sql` reports a
pre-migration file rather than letting the app fail on it. `RolodexRepository.cs` lives in
the [Rolodex](https://git.basso.land/jim/Rolodex) repository and is not covered by that
work. One site was dropped rather than translated: the `LEFT JOIN Notes` in `GetPosts`
selected nothing and was collapsed by the query's own `GROUP BY`, so it could not affect
the result.
### Column mapping
| Was | Is now | Resolve via |
|---|---|---|
| `Notes.RootBlogName` | `Notes.RootBlogId` | `BlogNames.BlogId``.BlogName` |
| `Notes.NoteBlogName` | `Notes.NoteBlogId` | `BlogNames.BlogId``.BlogName` |
| `Notes.Type` | `Notes.TypeId` | `NoteTypes.TypeId``.Type` |
| `ix_NoteBlogName01` | `ix_Notes_NoteBlogId` | — |
`PostID`, `TimeStamp`, `replyText`, `DatetimeCrawled`, `DateModified`, `DateCreated` and
`IsActive` are unchanged.
### Filtering by a blog name
```sql
-- was
WHERE NoteBlogName = @Name
-- now, either form; both search ix_Notes_NoteBlogId after a unique-index lookup
WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @Name)
-- or
JOIN BlogNames b ON b.BlogId = n.NoteBlogId WHERE b.BlogName = @Name
```
Measured 73 ms against 63 ms for the old text form on the busiest blog — the extra hop is
a unique-index probe on a 20k-row table and does not show.
### Joining `Notes` to `Blogs`
This is the join to get right; it is the most common shape in both applications.
```sql
-- was
FROM Blogs B INNER JOIN Notes N ON N.NoteBlogName = B.BlogName
-- now: one integer hop, using the new Blogs.BlogId
FROM Blogs B INNER JOIN Notes N ON N.NoteBlogId = B.BlogId
```
Do **not** route this through `BlogNames` — that adds a hop and ends in the text
comparison the change was meant to remove.
### Selecting a name back out
```sql
-- was
SELECT NoteBlogName AS blogName, COUNT(*) FROM Notes ... GROUP BY NoteBlogName
-- now
SELECT bn.BlogName AS blogName, COUNT(*)
FROM Notes n JOIN BlogNames bn ON bn.BlogId = n.NoteBlogId
... GROUP BY bn.BlogName
```
Group by `n.NoteBlogId` instead of `bn.BlogName` when you only need the name for display —
grouping on the integer is cheaper and the name comes along for free.
### Filtering by type
```sql
-- was
WHERE type IN ('reblog', 'reply', 'posted')
-- now
WHERE TypeId IN (SELECT TypeId FROM NoteTypes WHERE Type IN ('reblog','reply','posted'))
-- or, equivalently
JOIN NoteTypes t ON t.TypeId = n.TypeId WHERE t.Type IN ('reblog','reply','posted')
```
`WHERE TypeId IN (2,3,4)` also works and is marginally faster, but hardcodes this table's
contents into application code. Prefer the lookup outside of hot paths.
Note the negated form needs care: `type NOT IN ('reblog','reply','posted')` becomes
`TypeId NOT IN (SELECT TypeId FROM NoteTypes WHERE Type IN (...))`, which is correct only
because `TypeId` is `NOT NULL`.
### Inserting a note
The crawler must ensure both names have IDs first. `INSERT OR IGNORE` on `BlogNames` is
the whole of it — no read-back, no round trip, safe to run every time:
```sql
INSERT OR IGNORE INTO BlogNames (BlogName) VALUES (@rootBlogName);
INSERT OR IGNORE INTO BlogNames (BlogName) VALUES (@noteBlogName);
INSERT OR IGNORE INTO Notes
(RootBlogId, PostID, NoteBlogId, TimeStamp, TypeId,
DatetimeCrawled, DateModified, DateCreated)
SELECT (SELECT BlogId FROM BlogNames WHERE BlogName = @rootBlogName),
@PostID,
(SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName),
@TimeStamp,
(SELECT TypeId FROM NoteTypes WHERE Type = @Type),
@DatetimeCrawled, @DateModified, @DateCreated;
```
Verified: a genuinely new note inserts, and re-running the identical statement inserts 0.
Run all three statements in one transaction so a crash cannot leave a name registered
with no note.
**The duplicate-key error message has changed.** `DataAccess.cs` compares against the
literal string
```
UNIQUE constraint failed: Notes.RootBlogName, Notes.PostID, Notes.TimeStamp, Notes.Type, Notes.NoteBlogName
```
at two call sites to decide whether to swallow an exception. SQLite now emits the *new*
column names, so those comparisons no longer match and real errors will surface where
they used to be silently ignored — or vice versa.
Both sites now go through `IsNotesDuplicateKey` in `DataAccess.cs`, which matches on
`UNIQUE constraint failed` plus `Notes.` rather than on the column list. A literal
comparison is what broke here; the next rename should not break it again.
### Updating notes
Predicates translate the same way. The reply-matching update, which cannot use an index
on `TimeStamp` either before or after:
```sql
-- now
UPDATE Notes SET replyText = @replyText, DateModified = @dateModified
WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName)
AND ABS(TimeStamp - @TimeStamp) <= 5
AND TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')
AND (replyText IS NULL OR replyText = '' OR replyText = '.')
AND (replyText IS NULL OR replyText <> @replyText);
```
Rolodex's soft-delete updates need no change beyond the `WHERE` clause — they set
`IsActive`, which is untouched.
### Three traps
**`Blogs.BlogId` is NULL on 168,202 of 188,620 rows.** Any inner join on it silently drops
every blog that has never appeared in a note. Correct for engagement queries; wrong for
registry listings, which need a `LEFT JOIN` or no join at all.
**12 names in `BlogNames` have no `Blogs` row.** Resolving an ID to a name through `Blogs`
will occasionally find nothing. Use `BlogNames` for names and `Blogs` for registry columns.
**IDs are stable and must stay so.** `BlogNames.BlogId` and `NoteTypes.TypeId` are stored
in over a million `Notes` rows. Never renumber. A blog renamed upstream gets a new row,
not an edited one, unless every `Notes` reference migrates with it.
---
### Referential integrity ### Referential integrity
There are no foreign keys, and the tables do not perfectly agree: There are no foreign keys, and the tables do not perfectly agree:
- 4 `Posts` rows name a blog with no `Blogs` row. - 4 `Posts` rows name a blog with no `Blogs` row.
- 15 of the 31,888 distinct engagers have no `Blogs` row. - 12 of the 20,430 names in `BlogNames` have no `Blogs` row.
So a name appearing in `Notes` or `Posts` is not a guarantee that the registry knows about So a name appearing in `Notes` or `Posts` is not a guarantee that the registry knows about
it. Joins from those tables back to `Blogs` should tolerate a miss. it. Joins from those tables back to `Blogs` should tolerate a miss.
The integer schema does not fix this and was not meant to. `BlogNames` is deliberately
built from `Notes` rather than from `Blogs`, precisely so that the 12 unregistered
engagers keep their IDs and their rows. Had it been built from the registry, those notes
would have been dropped by the migration's inner joins.
--- ---
## The `'.'` placeholder convention ## The `'.'` placeholder convention
@@ -178,10 +498,14 @@ consumer.
| Column | `'.'` rows | | Column | `'.'` rows |
|---|--:| |---|--:|
| `Notes.replyText` | 1,174,706 | | `Notes.replyText` | 1,167,464 |
| `Posts.Title` | 13,144 | | `Posts.Title` | 12,562 |
| `Posts.Body` | 172 | | `Posts.Body` | 172 |
`Notes.replyText` and `Notes.DatetimeCrawled` **no longer carry column defaults** as of
the integer migration, so new note rows get `NULL` rather than a placeholder. The dots
already in `replyText` were not rewritten — cleaning is still required on read.
Any query whose output reaches a human should collapse it: Any query whose output reaches a human should collapse it:
```sql ```sql
@@ -214,9 +538,12 @@ Crawler bookkeeping. Rolodex ignores all of these.
`DataAccess.cs` joins on it to decide what to collect: `DataAccess.cs` joins on it to decide what to collect:
```sql ```sql
SELECT NoteBlogName, count(*) FROM notes -- shape only; the ported GetBlogs joins Blogs directly on BlogId and needs no BlogNames hop
INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName SELECT bn.BlogName, count(*)
WHERE blogs.IsActive = @isActive AND ... FROM Notes n
JOIN Blogs b ON b.BlogId = n.NoteBlogId
JOIN BlogNames bn ON bn.BlogId = n.NoteBlogId
WHERE b.IsActive = @isActive AND ...
``` ```
Nothing inside the crawler *writes* it — it is an input, set from outside. Nothing inside the crawler *writes* it — it is an input, set from outside.
@@ -262,12 +589,16 @@ under its Posts and Notes pages. Removing a blog hides the blog, not what it col
--- ---
## `Posts.IsActive` and `Notes.IsActive` — optional, and not in this database yet ## `Posts.IsActive` and `Notes.IsActive` — present, and written from outside
The same flag is being extended to the two content tables, with the same meaning: `0` is The same flag extends to the two content tables, with the same meaning: `0` is removed,
removed, anything else — including `NULL` — is live. **Neither column exists in the live anything else — including `NULL` — is live. **Both columns now exist in the live `TL.db`**
`TL.db` as of 2026-07-29**; the DDL quoted above for `Posts` and `Notes` is complete. Like and are included in the DDL quoted above. As of 2026-08-07, `Posts.IsActive = 0` on 5,900
`Blogs.IsActive`, they are written from outside this crawler. rows and `Notes.IsActive = 0` on none. Like `Blogs.IsActive`, they are written from
outside this crawler.
On `Notes` the column is `INTEGER NOT NULL DEFAULT 1`, so a `NULL` cannot occur there;
`Posts` and `Blogs` are laxer, which is why the predicate below still uses `COALESCE`.
The crawler therefore treats both as optional, and as nothing it owns: The crawler therefore treats both as optional, and as nothing it owns:
@@ -307,11 +638,19 @@ handled:
```sql ```sql
SELECT 'Blogs', COUNT(*) FROM Blogs SELECT 'Blogs', COUNT(*) FROM Blogs
UNION ALL SELECT 'Posts', COUNT(*) FROM Posts UNION ALL SELECT 'Posts', COUNT(*) FROM Posts
UNION ALL SELECT 'Notes', COUNT(*) FROM Notes; UNION ALL SELECT 'Notes', COUNT(*) FROM Notes
UNION ALL SELECT 'BlogNames', COUNT(*) FROM BlogNames;
-- note type mix -- note type mix (joins NoteTypes; Notes.Type no longer exists)
SELECT Type, COUNT(*) FROM Notes GROUP BY Type ORDER BY 2 DESC; SELECT t.Type, COUNT(*)
FROM Notes n JOIN NoteTypes t ON t.TypeId = n.TypeId
GROUP BY t.Type ORDER BY 2 DESC;
-- how much of the registry participates in the engagement graph
SELECT COUNT(*) FILTER (WHERE BlogId IS NOT NULL) AS with_notes,
COUNT(*) FILTER (WHERE BlogId IS NULL) AS without_notes
FROM Blogs;
-- the two date shapes in Blogs.DateAdded -- the two date shapes in Blogs.DateAdded
SELECT CASE WHEN DateAdded LIKE '____-__-__%' THEN 'ISO' ELSE 'US' END, COUNT(*) SELECT CASE WHEN DateAdded LIKE '____-__-__%' THEN 'ISO' ELSE 'US' END, COUNT(*)
@@ -324,6 +663,13 @@ SELECT COUNT(*) FROM (
-- rows that reference a blog the registry does not have -- rows that reference a blog the registry does not have
SELECT COUNT(*) FROM Posts p SELECT COUNT(*) FROM Posts p
WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = p.BlogName); WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = p.BlogName);
SELECT COUNT(*) FROM BlogNames bn
WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = bn.BlogName);
-- space by object, to see where the file actually goes
SELECT name, SUM(pgsize)/1024/1024 AS mb
FROM dbstat GROUP BY name ORDER BY SUM(pgsize) DESC;
``` ```
Open the file read-only so an inspection can never disturb a running crawl: Open the file read-only so an inspection can never disturb a running crawl:
+111 -17
View File
@@ -4,34 +4,64 @@ namespace URLNotesGrabberCORE
{ {
// Port of ThreeTxtFileHelper/UpdateBlogPaths.cs. Reads .tumblr / .tmblrpriv metadata // Port of ThreeTxtFileHelper/UpdateBlogPaths.cs. Reads .tumblr / .tmblrpriv metadata
// files from a root\Index folder and populates Blogs.TTFolderPath in TL.db. // files from a root\Index folder and populates Blogs.TTFolderPath in TL.db.
//
// Scan() is the reusable engine: --updatepaths wraps it as a standalone command and
// --output calls it as a refresh step, because a TL.db synced between machines cannot
// hold one absolute path that is correct on both.
public static class UpdateBlogPathsRunner public static class UpdateBlogPathsRunner
{ {
public static int Run(string rootPath) public enum ScanOutcome
{
Completed,
NoRootConfigured,
IndexFolderMissing
}
public sealed class ScanResult
{
public ScanOutcome Outcome { get; init; }
public string RootPath { get; init; } = string.Empty;
public string IndexPath { get; init; } = string.Empty;
public int MetadataFiles { get; init; }
public int Written { get; init; }
public int Unchanged { get; init; }
public int NoLocation { get; init; }
public int NoMatchingRow { get; init; }
public int Errors { get; init; }
}
// verbose: log a line per metadata file. --updatepaths wants that detail; --output
// only wants the counts, since a few hundred lines before the export would bury it.
public static ScanResult Scan(string? rootPath, bool verbose)
{ {
if (string.IsNullOrWhiteSpace(rootPath)) if (string.IsNullOrWhiteSpace(rootPath))
{ return new ScanResult { Outcome = ScanOutcome.NoRootConfigured };
Console.WriteLine("UpdateBlogPaths: rootPath is required.");
return 1;
}
DataAccess.EnsureTTFileHelperColumnsExist(); DataAccess.EnsureTTFileHelperColumnsExist();
string indexPath = Path.Combine(rootPath, "Index"); string indexPath = Path.Combine(rootPath, "Index");
if (!Directory.Exists(indexPath)) if (!Directory.Exists(indexPath))
{ {
Console.WriteLine($"Index folder not found at: {indexPath}"); return new ScanResult
return 1; {
Outcome = ScanOutcome.IndexFolderMissing,
RootPath = rootPath,
IndexPath = indexPath
};
} }
Console.WriteLine($"Scanning Index folder: {indexPath}");
var blogFiles = Directory.GetFiles(indexPath, "*.tumblr") var blogFiles = Directory.GetFiles(indexPath, "*.tumblr")
.Concat(Directory.GetFiles(indexPath, "*.tmblrpriv")) .Concat(Directory.GetFiles(indexPath, "*.tmblrpriv"))
.ToList(); .ToList();
Console.WriteLine($"Found {blogFiles.Count} blog metadata files"); if (verbose)
Console.WriteLine($"Found {blogFiles.Count} blog metadata files");
int updatedCount = 0; int updatedCount = 0;
int unchangedCount = 0;
int noLocationCount = 0;
int noRowCount = 0;
int errorCount = 0;
foreach (var blogFile in blogFiles) foreach (var blogFile in blogFiles)
{ {
@@ -44,27 +74,91 @@ namespace URLNotesGrabberCORE
if (root.TryGetProperty("FileDownloadLocation", out JsonElement locationElement)) if (root.TryGetProperty("FileDownloadLocation", out JsonElement locationElement))
{ {
string? fileDownloadLocation = locationElement.GetString(); string? fileDownloadLocation = locationElement.GetString()?.Trim();
if (!string.IsNullOrWhiteSpace(fileDownloadLocation)) if (!string.IsNullOrWhiteSpace(fileDownloadLocation))
{ {
DataAccess.SetBlogTTFolderPath(blogName, fileDownloadLocation); // Report the database's answer, not the fact that the file parsed.
updatedCount++; if (DataAccess.SetBlogTTFolderPath(blogName, fileDownloadLocation))
Console.WriteLine($"Updated {blogName}: {fileDownloadLocation}"); {
updatedCount++;
if (verbose)
Console.WriteLine($"Updated {blogName}: {fileDownloadLocation}");
}
else if (DataAccess.BlogExists(blogName))
{
unchangedCount++;
}
else
{
noRowCount++;
Console.WriteLine($"No Blogs row named '{blogName}' -- path not stored (name may differ in case)");
}
}
else
{
noLocationCount++;
if (verbose)
Console.WriteLine($"Empty FileDownloadLocation in {blogFile}");
} }
} }
else else
{ {
Console.WriteLine($"No FileDownloadLocation found in {blogFile}"); noLocationCount++;
if (verbose)
Console.WriteLine($"No FileDownloadLocation found in {blogFile}");
} }
} }
catch (Exception ex) catch (Exception ex)
{ {
errorCount++;
Console.WriteLine($"Error processing {blogFile}: {ex.Message}"); Console.WriteLine($"Error processing {blogFile}: {ex.Message}");
} }
} }
Console.WriteLine($"\nUpdated {updatedCount} blogs with TTFolderPath"); return new ScanResult
return 0; {
Outcome = ScanOutcome.Completed,
RootPath = rootPath,
IndexPath = indexPath,
MetadataFiles = blogFiles.Count,
Written = updatedCount,
Unchanged = unchangedCount,
NoLocation = noLocationCount,
NoMatchingRow = noRowCount,
Errors = errorCount
};
}
public static int Run(string rootPath)
{
if (string.IsNullOrWhiteSpace(rootPath))
{
Console.WriteLine("UpdateBlogPaths: rootPath is required.");
return 1;
}
string indexPath = Path.Combine(rootPath, "Index");
Console.WriteLine($"Scanning Index folder: {indexPath}");
var result = Scan(rootPath, verbose: true);
if (result.Outcome == ScanOutcome.IndexFolderMissing)
{
Console.WriteLine($"Index folder not found at: {result.IndexPath}");
return 1;
}
Console.WriteLine($"\n========== UpdateBlogPaths summary ==========");
Console.WriteLine($"Metadata files: {result.MetadataFiles}");
Console.WriteLine($"TTFolderPath written: {result.Written}");
Console.WriteLine($"Already correct: {result.Unchanged}");
Console.WriteLine($"No FileDownloadLocation: {result.NoLocation}");
Console.WriteLine($"No matching blog row: {result.NoMatchingRow}");
Console.WriteLine($"Errors: {result.Errors}");
Console.WriteLine($"\nBlogs now holding a TTFolderPath: {DataAccess.CountBlogsWithTTFolderPath()}");
return result.Errors == 0 ? 0 : 2;
} }
} }
} }
+143
View File
@@ -0,0 +1,143 @@
-- normalize-notes.sql
-- Replaces the repeated blog-name and type TEXT in Notes with integer IDs.
-- Reduces TL.db from ~207 MB to ~148 MB (-29%).
--
-- THIS IS A BREAKING SCHEMA CHANGE. There is no compatibility layer. Every
-- query in URLNotesGrabberCORE and Rolodex that names Notes.RootBlogName,
-- Notes.NoteBlogName or Notes.Type stops working the moment this runs, and
-- stays broken until those queries are rewritten. This was a deliberate choice
-- over a view-plus-triggers shim, which was measured to work but cost 194 ms ->
-- 321 ms on Rolodex's unfiltered Notes page.
--
-- TumblThree is unaffected. It touches only Blogs, and the column added to
-- Blogs here is additive.
--
-- HOW TO RUN (DB Browser for SQLite):
-- 1. Stop all three apps. Pause NextCloud sync.
-- 2. Back up TL.db.
-- 3. Execute SQL, paste this file, run. Then Write Changes.
-- 4. Tools > Compact Database (VACUUM). Nothing shrinks until this finishes.
--------------------------------------------------------------------------
-- The shape this produces
--------------------------------------------------------------------------
-- BlogNames(BlogId, BlogName) the ID authority: every name appearing in
-- Notes as either participant. 20,430 rows.
-- 12 of these have no Blogs row -- the
-- registry has never been a superset of the
-- engagement graph, and still is not.
--
-- NoteTypes(TypeId, Type) 5 rows. Fixed set, but written as a table
-- rather than a CHECK so a new type is an
-- INSERT and not a schema migration.
--
-- Notes(...Id columns...) integer FKs in place of text. WITHOUT ROWID,
-- same 5-column key in the same column order.
--
-- Blogs.BlogId NEW additive column. Lets Notes join Blogs in
-- one integer hop instead of going through
-- BlogNames and comparing text at the end.
-- NULL on the 168,202 blogs with no notes.
PRAGMA foreign_keys = off;
BEGIN;
--------------------------------------------------------------------------
-- STEP 1: the ID authority
--------------------------------------------------------------------------
CREATE TABLE BlogNames (
BlogId INTEGER PRIMARY KEY,
BlogName TEXT NOT NULL UNIQUE
);
INSERT INTO BlogNames (BlogName)
SELECT RootBlogName FROM Notes
UNION
SELECT NoteBlogName FROM Notes;
--------------------------------------------------------------------------
-- STEP 2: the type lookup
--------------------------------------------------------------------------
CREATE TABLE NoteTypes (
TypeId INTEGER PRIMARY KEY,
Type TEXT NOT NULL UNIQUE
);
-- IDs are assigned explicitly and must stay stable: they are stored in Notes.
INSERT INTO NoteTypes (TypeId, Type) VALUES
(1, 'like'),
(2, 'reblog'),
(3, 'reply'),
(4, 'posted'),
(5, 'post_attribution');
--------------------------------------------------------------------------
-- STEP 3: rebuild Notes with integer keys
--------------------------------------------------------------------------
-- Column order of the primary key is unchanged from the text version, so the
-- leading-column access patterns callers already rely on still hold:
-- (RootBlogId) and (RootBlogId, PostID) remain cheap prefixes.
CREATE TABLE NotesN (
RootBlogId INTEGER NOT NULL,
PostID INTEGER NOT NULL,
NoteBlogId INTEGER NOT NULL,
TimeStamp INTEGER NOT NULL,
TypeId INTEGER NOT NULL,
replyText TEXT,
DatetimeCrawled TEXT,
DateModified TEXT,
DateCreated TEXT,
IsActive INTEGER NOT NULL DEFAULT 1,
PRIMARY KEY (RootBlogId, PostID, TimeStamp, TypeId, NoteBlogId)
) WITHOUT ROWID;
-- Inner joins are safe here: BlogNames was just built from these very columns,
-- and NoteTypes covers all 5 values present. A row that failed to match would
-- be silently dropped, which is what the row-count check at the bottom is for.
INSERT INTO NotesN
SELECT r.BlogId, n.PostID, b.BlogId, n.TimeStamp, t.TypeId,
n.replyText, n.DatetimeCrawled, n.DateModified, n.DateCreated, n.IsActive
FROM Notes n
JOIN BlogNames r ON r.BlogName = n.RootBlogName
JOIN BlogNames b ON b.BlogName = n.NoteBlogName
JOIN NoteTypes t ON t.Type = n.Type;
DROP TABLE Notes;
ALTER TABLE NotesN RENAME TO Notes;
-- Replaces ix_NoteBlogName01. Renamed because it indexes a different column now.
CREATE INDEX ix_Notes_NoteBlogId ON Notes (NoteBlogId);
--------------------------------------------------------------------------
-- STEP 4: give Blogs the matching id
--------------------------------------------------------------------------
-- Additive: no existing column changes, so TumblThree's
-- "UPDATE Blogs SET IsActive = 0 ... WHERE BlogName = ?" is untouched.
ALTER TABLE Blogs ADD COLUMN BlogId INTEGER;
UPDATE Blogs
SET BlogId = (SELECT bn.BlogId FROM BlogNames bn WHERE bn.BlogName = Blogs.BlogName);
CREATE INDEX ix_Blogs_BlogId ON Blogs (BlogId);
COMMIT;
--------------------------------------------------------------------------
-- STEP 5: Write Changes, then Tools > Compact Database
--------------------------------------------------------------------------
-- From the CLI instead: sqlite3 TL.db "VACUUM;"
--------------------------------------------------------------------------
-- VERIFY
--------------------------------------------------------------------------
-- PRAGMA integrity_check; -- expect: ok
-- SELECT COUNT(*) FROM Notes; -- expect: 1182333, unchanged
-- SELECT COUNT(*) FROM BlogNames; -- expect: 20430
-- SELECT COUNT(*) FROM Blogs WHERE BlogId IS NOT NULL; -- expect: 20418
--
-- Losslessness was proven before this ran, by reconstructing the old text shape
-- from the new schema and diffing it against the original both ways:
-- SELECT COUNT(*) FROM (SELECT * FROM old.Notes EXCEPT SELECT * FROM Rebuilt);
-- SELECT COUNT(*) FROM (SELECT * FROM Rebuilt EXCEPT SELECT * FROM old.Notes);
-- Both returned 0 across all 1,182,333 rows and all 10 columns.
+159
View File
@@ -0,0 +1,159 @@
-- shrink-db.sql
-- Reduces TL.db from ~267 MB to ~207 MB (-22%) with no application changes,
-- and no visible change in any of the three apps that touch this file.
--
-- The three consumers, and what each one uses:
-- URLNotesGrabberCORE System.Data.SQLite 1.0.119 writes Notes, Posts, Blogs
-- Rolodex (web) Microsoft.Data.Sqlite 10.0 reads all three; soft-deletes via IsActive
-- TumblThree System.Data.SQLite.Core 1.0.119
-- one statement only, ManagerController.cs:972 --
-- "UPDATE Blogs SET IsActive = 0, DateModified = @DateModified
-- WHERE BlogName = @BlogName"
-- Nothing below touches the Blogs table, so TumblThree is
-- unaffected. (Its GlobalDatabaseService talks to TumblThree's
-- own separate FileEntries/BlogFiles database, not this file.)
--
-- WITHOUT ROWID needs SQLite >= 3.8.2 (Dec 2013). All three providers above are
-- 2024-25 builds, an order of magnitude newer, so STEP 2 is readable by all of them.
--
-- Every figure below was measured on a copy of the live 267 MB file, and the
-- result was checked against all three apps' access patterns:
-- PRAGMA integrity_check ....... ok
-- row counts ................... Notes 1182333, Posts 22468, Blogs 188620 (unchanged)
-- Rolodex soft-delete UPDATE ... works
-- Rolodex NoteBlogName filter .. still uses ix_NoteBlogName01
-- crawler INSERT OR IGNORE ..... still dedupes (0 dupes admitted)
-- TumblThree's UPDATE Blogs ..... untouched -- Blogs is not modified by this script
--
-- HOW TO RUN (DB Browser for SQLite):
-- 1. Stop ALL THREE apps: the crawler, the Rolodex web app, and TumblThree.
-- Rolodex holds the file open and checkpoints the WAL, so it must be down,
-- not just idle. TumblThree only opens the file for an instant when you
-- delete a blog, but it can also launch the crawler on its own
-- (UrlNotesGrabberService) -- so close it rather than merely avoiding it.
-- 2. Back up TL.db (copy the 267 MB file somewhere safe).
-- 3. Open TL.db, go to Execute SQL, paste STEP 1-3, run.
-- 4. Click "Write Changes".
-- 5. Run Tools > Compact Database. This is VACUUM; it will not run from the
-- Execute SQL tab because DB Browser keeps a transaction open there.
-- NOTHING SHRINKS ON DISK UNTIL THIS FINISHES.
--
-- Expected: steps 1-3 a couple of minutes, Compact a couple more.
-- Free disk needed during Compact: ~270 MB for the temp copy.
--------------------------------------------------------------------------
-- STEP 1: drop the TimeStamp index (required by STEP 2, not optional)
--------------------------------------------------------------------------
-- Measured cost/benefit:
--
-- * Rolodex's DEFAULT Notes view does not use it. Its sort carries the
-- tiebreaker "RootBlogName, PostID, NoteBlogName", which forces a full sort
-- regardless -- the query plan is byte-identical with and without the index.
-- Sorting.cs:138 already assumes as much, and is right in practice.
-- * The crawler's collect query (DataAccess.cs:1183) filters
-- TimeStamp >= 1535778000, which excludes 786 of 1,182,333 rows (0.07%).
-- A full index scan wearing a disguise. Same measured time without it.
-- * The reply-matching UPDATE uses ABS(TimeStamp - ?) <= 5, which can never
-- use an index on TimeStamp.
-- * It DOES help exactly one path: Rolodex's Notes page with a date-range
-- filter applied. 60 ms -> 164 ms. That is the whole of what is lost.
--
-- And it must go, because after STEP 2 it stops being cheap. A secondary index
-- on a WITHOUT ROWID table carries the full 5-column primary key instead of a
-- compact rowid, so this index grows 14 MB -> 58 MB. Keeping it lands the file
-- at 265 MB instead of 207 MB -- i.e. it cancels the entire exercise to save
-- 100 ms on one filtered view.
DROP INDEX IF EXISTS Notes_idx_06e01ae3;
--------------------------------------------------------------------------
-- STEP 2: rebuild Notes as WITHOUT ROWID (-32 MB)
--------------------------------------------------------------------------
-- Notes has a 5-column composite primary key. In a rowid table SQLite stores
-- that key twice: once in the table, once in sqlite_autoindex_Notes_1 (62 MB).
-- WITHOUT ROWID stores the rows *in* the key's b-tree, so the copy disappears.
--
-- ix_NoteBlogName01 grows 25 -> 58 MB for the reason described above. Net -32 MB.
-- It is kept because Rolodex filters on NoteBlogName and the crawler joins on it.
--
-- Safe: neither codebase references rowid on Notes (grep across both trees,
-- zero matches). IsActive keeps its exact current declaration, which is what
-- Rolodex's ActiveFlag predicate reads.
PRAGMA foreign_keys = off;
CREATE TABLE Notes_new (
"RootBlogName" TEXT,
"PostID" INTEGER,
"NoteBlogName" TEXT,
"TimeStamp" INTEGER,
"Type" TEXT,
"replyText" TEXT DEFAULT '.',
"DatetimeCrawled" TEXT DEFAULT '2/12/26 12am',
"DateModified" TEXT,
"DateCreated" TEXT,
IsActive INTEGER NOT NULL DEFAULT 1,
PRIMARY KEY("RootBlogName","PostID","TimeStamp","Type","NoteBlogName")
) WITHOUT ROWID;
INSERT INTO Notes_new
SELECT RootBlogName, PostID, NoteBlogName, TimeStamp, Type,
replyText, DatetimeCrawled, DateModified, DateCreated, IsActive
FROM Notes;
DROP TABLE Notes;
ALTER TABLE Notes_new RENAME TO Notes;
CREATE INDEX "ix_NoteBlogName01" ON "Notes" ("NoteBlogName");
--------------------------------------------------------------------------
-- STEP 3: clear the DatetimeCrawled placeholder (-13 MB)
--------------------------------------------------------------------------
-- 1,148,077 of 1,182,333 rows hold the literal DDL default '2/12/26 12am' --
-- a backfill placeholder, not a crawl time. SQLite stores all 12 bytes of it
-- on every one of those rows.
--
-- This is UI-NEUTRAL in Rolodex, which is why it is safe despite Rolodex
-- displaying the column. Rolodex reads and sorts it through DateSql.Sortable
-- (DateRange.cs:67), whose CASE matches '____-__-__%' or the 8-character
-- '__/__/__'. The 12-character '2/12/26 12am' matches neither, so Sortable
-- already returns NULL for these rows and the page already renders an em dash
-- and sorts them to the bottom. RolodexRepository.cs:957-963 documents exactly
-- this. Writing a real NULL changes the bytes on disk, not the screen.
--
-- The crawler never reads the column back -- it only writes it on INSERT
-- (DataAccess.cs:713, 721).
UPDATE Notes SET DatetimeCrawled = NULL WHERE DatetimeCrawled = '2/12/26 12am';
--------------------------------------------------------------------------
-- DELIBERATELY NOT DONE: nulling Notes.DateCreated
--------------------------------------------------------------------------
-- An earlier draft of this script also cleared DateCreated = '2026-04-13'
-- (a further -12 MB). Do not. Unlike DatetimeCrawled, that value DOES match
-- Sortable's '____-__-__%' branch, so Rolodex renders it as a real date in the
-- "Created" column on the Notes page and Post detail, and sorts by it. Nulling
-- it would turn visible dates into em dashes and move rows in the sort order.
--------------------------------------------------------------------------
-- STEP 4: Write Changes, then Tools > Compact Database
--------------------------------------------------------------------------
-- Nothing above reclaims disk until VACUUM runs. From the sqlite3 CLI instead:
-- sqlite3 TL.db "VACUUM;"
--------------------------------------------------------------------------
-- VERIFY (run after compacting; file should be ~207 MB)
--------------------------------------------------------------------------
-- PRAGMA integrity_check;
--
-- SELECT 'Notes' t, COUNT(*) n FROM Notes
-- UNION ALL SELECT 'Posts', COUNT(*) FROM Posts
-- UNION ALL SELECT 'Blogs', COUNT(*) FROM Blogs;
-- -- expect 1182333 / 22468 / 188620, unchanged
--
-- SELECT name, SUM(pgsize)/1024/1024 AS mb
-- FROM dbstat GROUP BY name ORDER BY SUM(pgsize) DESC;
-- -- expect Notes 78, ix_NoteBlogName01 58, Posts 50, Blogs 14
--
-- APPLIED 2026-08-07. Actual result: 267.32 MB -> 207.17 MB, integrity_check ok,
-- row counts unchanged, journal_mode still wal. VACUUM took 6 seconds.
+54 -9
View File
@@ -79,18 +79,35 @@ WITH expected(tbl, col, alter_stmt) AS (
('Blogs','LikesLastRefreshed', 'ALTER TABLE Blogs ADD COLUMN LikesLastRefreshed INTEGER DEFAULT 0;'), ('Blogs','LikesLastRefreshed', 'ALTER TABLE Blogs ADD COLUMN LikesLastRefreshed INTEGER DEFAULT 0;'),
('Blogs','LikesLastNewCount', 'ALTER TABLE Blogs ADD COLUMN LikesLastNewCount INTEGER DEFAULT 0;'), ('Blogs','LikesLastNewCount', 'ALTER TABLE Blogs ADD COLUMN LikesLastNewCount INTEGER DEFAULT 0;'),
('Blogs','TTFolderPath', 'ALTER TABLE Blogs ADD COLUMN TTFolderPath TEXT;'), ('Blogs','TTFolderPath', 'ALTER TABLE Blogs ADD COLUMN TTFolderPath TEXT;'),
-- Blogs.BlogId (2026-08-07) is the single-hop join key into Notes. Deliberately NOT
-- auto-fixable: an added-but-empty BlogId makes every engagement join return zero
-- rows silently, which is worse than the hard error a missing column gives.
('Blogs','BlogId', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
-- Notes (base columns: manual review if missing) -- Notes (base columns: manual review if missing)
('Notes','RootBlogName', 'MANUAL REVIEW - base/PK column missing'), -- Integer IDs since 2026-08-07. RootBlogName/NoteBlogName/Type are GONE, not renamed
-- in place -- a backup that still has them needs normalize-notes.sql, not an ALTER.
-- Query 1d below reports exactly that case.
('Notes','RootBlogId', 'MANUAL REVIEW - see query 1d: pre-2026-08-07 name schema, or damaged'),
('Notes','PostID', 'MANUAL REVIEW - base/PK column missing'), ('Notes','PostID', 'MANUAL REVIEW - base/PK column missing'),
('Notes','NoteBlogName', 'MANUAL REVIEW - base/PK column missing'), ('Notes','NoteBlogId', 'MANUAL REVIEW - see query 1d: pre-2026-08-07 name schema, or damaged'),
('Notes','TimeStamp', 'MANUAL REVIEW - base/PK column missing'), ('Notes','TimeStamp', 'MANUAL REVIEW - base/PK column missing'),
('Notes','Type', 'MANUAL REVIEW - base/PK column missing'), ('Notes','TypeId', 'MANUAL REVIEW - see query 1d: pre-2026-08-07 name schema, or damaged'),
('Notes','DatetimeCrawled', 'MANUAL REVIEW - base column missing'), ('Notes','DatetimeCrawled', 'MANUAL REVIEW - base column missing'),
('Notes','DateModified', 'MANUAL REVIEW - base column missing'), ('Notes','DateModified', 'MANUAL REVIEW - base column missing'),
('Notes','DateCreated', 'MANUAL REVIEW - base column missing'), ('Notes','DateCreated', 'MANUAL REVIEW - base column missing'),
-- Notes (additive migration column, auto-fixable) -- Notes (additive migration column, auto-fixable)
('Notes','replyText', 'ALTER TABLE Notes ADD COLUMN replyText TEXT DEFAULT ''.'';'), -- No DEFAULT: the migrated schema dropped it, so new rows get NULL rather than a
-- placeholder. EnsureReplyTextColumnExists in DataAccess.cs adds it the same way.
('Notes','replyText', 'ALTER TABLE Notes ADD COLUMN replyText TEXT;'),
-- BlogNames / NoteTypes (the lookup tables Notes resolves its IDs through, 2026-08-07).
-- Not auto-fixable: an empty BlogNames does not mean "add the table", it means the
-- Notes rows have nothing to resolve against. Rebuild with normalize-notes.sql.
('BlogNames','BlogId', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
('BlogNames','BlogName', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
('NoteTypes','TypeId', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
('NoteTypes','Type', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
-- DailyAPICount (base columns) -- DailyAPICount (base columns)
('DailyAPICount','Date', 'MANUAL REVIEW - base/PK column missing'), ('DailyAPICount','Date', 'MANUAL REVIEW - base/PK column missing'),
@@ -106,6 +123,8 @@ actual(tbl, col) AS (
SELECT 'Posts', name FROM pragma_table_info('Posts') SELECT 'Posts', name FROM pragma_table_info('Posts')
UNION ALL SELECT 'Blogs', name FROM pragma_table_info('Blogs') UNION ALL SELECT 'Blogs', name FROM pragma_table_info('Blogs')
UNION ALL SELECT 'Notes', name FROM pragma_table_info('Notes') UNION ALL SELECT 'Notes', name FROM pragma_table_info('Notes')
UNION ALL SELECT 'BlogNames', name FROM pragma_table_info('BlogNames')
UNION ALL SELECT 'NoteTypes', name FROM pragma_table_info('NoteTypes')
UNION ALL SELECT 'DailyAPICount', name FROM pragma_table_info('DailyAPICount') UNION ALL SELECT 'DailyAPICount', name FROM pragma_table_info('DailyAPICount')
UNION ALL SELECT 'ApiKeyPoolState', name FROM pragma_table_info('ApiKeyPoolState') UNION ALL SELECT 'ApiKeyPoolState', name FROM pragma_table_info('ApiKeyPoolState')
UNION ALL SELECT 'ApiKeyPoolMeta', name FROM pragma_table_info('ApiKeyPoolMeta') UNION ALL SELECT 'ApiKeyPoolMeta', name FROM pragma_table_info('ApiKeyPoolMeta')
@@ -127,7 +146,7 @@ ORDER BY (e.alter_stmt LIKE 'ALTER%') DESC, e.tbl, e.col;
-- 1b. MISSING TABLES: expected tables that don't exist at all in this DB. -- 1b. MISSING TABLES: expected tables that don't exist at all in this DB.
-- Zero rows = good. -- Zero rows = good.
WITH expected_tables(tbl) AS ( WITH expected_tables(tbl) AS (
VALUES ('Posts'),('Blogs'),('Notes'),('DailyAPICount'), VALUES ('Posts'),('Blogs'),('Notes'),('BlogNames'),('NoteTypes'),('DailyAPICount'),
('ApiKeyPoolState'),('ApiKeyPoolMeta') ('ApiKeyPoolState'),('ApiKeyPoolMeta')
) )
SELECT et.tbl AS missing_table SELECT et.tbl AS missing_table
@@ -157,10 +176,12 @@ WITH expected(tbl, col) AS (
('Blogs','BlogName'),('Blogs','HasBeenOutput'),('Blogs','IsActive'),('Blogs','DateAdded'), ('Blogs','BlogName'),('Blogs','HasBeenOutput'),('Blogs','IsActive'),('Blogs','DateAdded'),
('Blogs','ByLikes'),('Blogs','DateModified'),('Blogs','DateCreated'),('Blogs','LikesPulled'), ('Blogs','ByLikes'),('Blogs','DateModified'),('Blogs','DateCreated'),('Blogs','LikesPulled'),
('Blogs','LikesCursor'),('Blogs','LikesNewestTimestamp'),('Blogs','LikesLastRefreshed'), ('Blogs','LikesCursor'),('Blogs','LikesNewestTimestamp'),('Blogs','LikesLastRefreshed'),
('Blogs','LikesLastNewCount'),('Blogs','TTFolderPath'), ('Blogs','LikesLastNewCount'),('Blogs','TTFolderPath'),('Blogs','BlogId'),
('Notes','RootBlogName'),('Notes','PostID'),('Notes','NoteBlogName'),('Notes','TimeStamp'), ('Notes','RootBlogId'),('Notes','PostID'),('Notes','NoteBlogId'),('Notes','TimeStamp'),
('Notes','Type'),('Notes','DatetimeCrawled'),('Notes','DateModified'),('Notes','DateCreated'), ('Notes','TypeId'),('Notes','DatetimeCrawled'),('Notes','DateModified'),('Notes','DateCreated'),
('Notes','replyText'),('Notes','IsActive'), ('Notes','replyText'),('Notes','IsActive'),
('BlogNames','BlogId'),('BlogNames','BlogName'),
('NoteTypes','TypeId'),('NoteTypes','Type'),
('DailyAPICount','Date'),('DailyAPICount','APICount'), ('DailyAPICount','Date'),('DailyAPICount','APICount'),
('ApiKeyPoolState','KeyName'),('ApiKeyPoolState','RetryUntil'), ('ApiKeyPoolState','KeyName'),('ApiKeyPoolState','RetryUntil'),
('ApiKeyPoolMeta','Id'),('ApiKeyPoolMeta','LastIndex') ('ApiKeyPoolMeta','Id'),('ApiKeyPoolMeta','LastIndex')
@@ -169,6 +190,8 @@ actual(tbl, col) AS (
SELECT 'Posts', name FROM pragma_table_info('Posts') SELECT 'Posts', name FROM pragma_table_info('Posts')
UNION ALL SELECT 'Blogs', name FROM pragma_table_info('Blogs') UNION ALL SELECT 'Blogs', name FROM pragma_table_info('Blogs')
UNION ALL SELECT 'Notes', name FROM pragma_table_info('Notes') UNION ALL SELECT 'Notes', name FROM pragma_table_info('Notes')
UNION ALL SELECT 'BlogNames', name FROM pragma_table_info('BlogNames')
UNION ALL SELECT 'NoteTypes', name FROM pragma_table_info('NoteTypes')
UNION ALL SELECT 'DailyAPICount', name FROM pragma_table_info('DailyAPICount') UNION ALL SELECT 'DailyAPICount', name FROM pragma_table_info('DailyAPICount')
UNION ALL SELECT 'ApiKeyPoolState', name FROM pragma_table_info('ApiKeyPoolState') UNION ALL SELECT 'ApiKeyPoolState', name FROM pragma_table_info('ApiKeyPoolState')
UNION ALL SELECT 'ApiKeyPoolMeta', name FROM pragma_table_info('ApiKeyPoolMeta') UNION ALL SELECT 'ApiKeyPoolMeta', name FROM pragma_table_info('ApiKeyPoolMeta')
@@ -181,6 +204,25 @@ WHERE e.col IS NULL
ORDER BY a.tbl, a.col; ORDER BY a.tbl, a.col;
-- 1d. PRE-MIGRATION DATABASE: a backup from before 2026-08-07, when Notes still
-- stored names. Zero rows = good.
--
-- This is the one failure SECTION 2 cannot fix. Notes.RootBlogName /
-- NoteBlogName / Type were replaced by RootBlogId / NoteBlogId / TypeId
-- resolving through BlogNames and NoteTypes -- a data migration, not an
-- ADD COLUMN. There is no compatibility view, so the current code fails
-- outright ("no such column: RootBlogId") against such a file.
--
-- Fix: run normalize-notes.sql against a COPY of the backup, then re-run
-- SECTION 1. Do not hand-add the ID columns: they would be empty, and an
-- empty NoteBlogId is indistinguishable from a note by blog #0.
SELECT 'Notes still stores names -- run normalize-notes.sql on a copy' AS pre_migration_schema,
group_concat(name, ', ') AS legacy_columns_found
FROM pragma_table_info('Notes')
WHERE lower(name) IN ('rootblogname','noteblogname','type')
HAVING COUNT(*) > 0;
-- ============================================================================ -- ============================================================================
-- SECTION 2 -- FIX (opt-in, additive only) -- SECTION 2 -- FIX (opt-in, additive only)
-- --
@@ -190,6 +232,9 @@ ORDER BY a.tbl, a.col;
-- "duplicate column name" error and changes nothing -- just run the flagged -- "duplicate column name" error and changes nothing -- just run the flagged
-- subset. These are the 8 additive migration columns and nothing else; the -- subset. These are the 8 additive migration columns and nothing else; the
-- likes high-water-mark reset is intentionally NOT included. -- likes high-water-mark reset is intentionally NOT included.
--
-- Nothing here addresses query 1d. The Notes integer schema is a data migration
-- (normalize-notes.sql) and cannot be reached by adding columns.
-- ============================================================================ -- ============================================================================
-- ALTER TABLE Posts ADD COLUMN PostType TEXT; -- ALTER TABLE Posts ADD COLUMN PostType TEXT;
@@ -199,4 +244,4 @@ ORDER BY a.tbl, a.col;
-- ALTER TABLE Blogs ADD COLUMN LikesLastRefreshed INTEGER DEFAULT 0; -- ALTER TABLE Blogs ADD COLUMN LikesLastRefreshed INTEGER DEFAULT 0;
-- ALTER TABLE Blogs ADD COLUMN LikesLastNewCount INTEGER DEFAULT 0; -- ALTER TABLE Blogs ADD COLUMN LikesLastNewCount INTEGER DEFAULT 0;
-- ALTER TABLE Blogs ADD COLUMN TTFolderPath TEXT; -- ALTER TABLE Blogs ADD COLUMN TTFolderPath TEXT;
-- ALTER TABLE Notes ADD COLUMN replyText TEXT DEFAULT '.'; -- ALTER TABLE Notes ADD COLUMN replyText TEXT;