Compare commits
28
Commits
16147b273e
..
master
| Author | SHA1 | Date | |
|---|---|---|---|
  jimandClaude Opus 5
|
b31d5842cc | ||
|
jim
|
c3cf89c3f1 | ||
|
jimandClaude Opus 5
|
ab36085ba8 | ||
|
jimandClaude Opus 5
|
4d37999f8e | ||
|
jim
|
387c023900 | ||
|
jimandClaude Opus 5
|
a9bd5a4c37 | ||
|
jim
|
70b32dfc89 | ||
|
jimandClaude Opus 5
|
8f4177a0c9 | ||
|
jim
|
a2763d0026 | ||
|
jimandClaude Opus 5
|
721224bc13 | ||
|
jim
|
ef6629d86a | ||
|
jimandClaude Opus 5
|
6320e2c0c9 | ||
|
jim
|
6136901cc7 | ||
|
jimandClaude Opus 5
|
83e35a2323 | ||
|
jim
|
f0ccac6503 | ||
|
jimandClaude Opus 5
|
e1d2eb48c2 | ||
|
jim
|
3c85a05afc | ||
|
jim
|
60912c882d | ||
|
jim
|
8a4ab2402d | ||
|
jimandClaude Opus 5
|
05ec465f74 | ||
|
jimandClaude Opus 5
|
eded5271ea | ||
|
jimandClaude Opus 5
|
a14debd5ed | ||
|
jimandClaude Opus 5
|
d6637266b7 | ||
|
jimandClaude Opus 4.8
|
2a02811003 | ||
|
jimandClaude Opus 4.8
|
a73b597381 | ||
|
jim
|
f9e1d2100b | ||
|
jimandClaude Opus 4.8
|
3e2b287737 | ||
|
jimandClaude Opus 4.8
|
003a504d5e |
Binary file not shown.
@@ -28,6 +28,162 @@
|
||||
- Preserve console color state: use save/restore pattern for temporary color changes
|
||||
- API rate limits must use `ApiKeyPool.MarkRateLimited()`/`MarkAvailable()`
|
||||
|
||||
### API Failure Classification
|
||||
Tumblr sits behind a CDN that returns HTML error pages (403, 5xx) which never reach the API. These
|
||||
say nothing about the item being fetched, so they must not be recorded as per-item failures.
|
||||
|
||||
- A response body that will not parse as JSON did not come from the API. Flag it with
|
||||
`Root.transientFailure`, never as `FAILURE`
|
||||
- Transient failures retry in place (`TransientBackoffSeconds`) before the item is skipped; a skipped
|
||||
item stays unmarked in the DB so a later launch retries it
|
||||
- `MaxConsecutiveTransient` consecutive transient failures aborts the pass rather than skipping
|
||||
item-by-item against an edge that is refusing all traffic
|
||||
- Only call `ApiKeyPool.MarkAvailable()` on a response that actually reached the API. A transport or
|
||||
CDN failure says nothing about the key's standing and must not clear its flag
|
||||
- Only a real HTTP 429 (or `meta.status == 429`) counts as a rate limit. Do not infer one from the
|
||||
presence of `X-RateLimit-*` headers, which Tumblr sends on every response
|
||||
- Rate limiters must pace with `await AcquireAsync()`. `AttemptAcquire()` does not wait, so a
|
||||
saturated window aborts the run instead of throttling it
|
||||
- Long-running commands return exit 3 when a pass ends incomplete (rate-limit pause, breaker trip, or
|
||||
skipped items), so a caller can distinguish that from a clean run
|
||||
|
||||
### `Notes` Stores Integer IDs, Not Names
|
||||
As of 2026-08-07 `Notes.RootBlogName`, `NoteBlogName` and `Type` are gone, replaced by
|
||||
`RootBlogId`, `NoteBlogId` and `TypeId` resolving through the `BlogNames` and `NoteTypes`
|
||||
lookup tables. There is no compatibility view — naming an old column is a hard SQLite
|
||||
error, so unlike `IsActive` this is a hard cut with no runtime probe. Full detail in
|
||||
`URLNotesGrabberCORE/TL.db.md`.
|
||||
|
||||
- **Joining `Notes` to `Blogs` goes through `Blogs.BlogId`**, not `BlogNames`:
|
||||
`FROM Blogs B INNER JOIN Notes N ON N.NoteBlogId = B.BlogId`. Routing it through
|
||||
`BlogNames` adds a hop and ends in the text comparison the migration removed
|
||||
- **Joining `Notes` to `Posts` is the opposite** — `Posts` has only `BlogName`, so it must
|
||||
go through `BlogNames` (`GetRepliesWithFilledText`). This is the only such join
|
||||
- **Resolve a name by filtering the lookup, never by scanning `Notes`**:
|
||||
`WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @name)`. The subquery
|
||||
is a unique-index probe on 20k rows and does not show against the 1.18M-row table
|
||||
- **`AddNote` registers both blog names *and* the note type** with `INSERT OR IGNORE`
|
||||
before inserting, all in one transaction. `NoteTypes` is a table rather than a `CHECK`
|
||||
constraint precisely so an unseen type is an `INSERT`; without that registration it
|
||||
would resolve to `NULL` and fail the `NOT NULL` on `TypeId`, losing the note
|
||||
- **`Blogs.BlogId` is NULL on 168,202 of 188,620 rows** — every blog that has never
|
||||
appeared in a note. An inner join on it silently drops them. Correct for engagement
|
||||
queries, wrong for anything listing the registry
|
||||
- **IDs are stable and must never be renumbered.** They are stored in 1.18M `Notes` rows.
|
||||
A blog renamed upstream gets a new `BlogNames` row, not an edited one
|
||||
- Prefer `TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')` over a hardcoded
|
||||
ID. A negated `TypeId NOT IN (SELECT …)` is only correct because `TypeId` is `NOT NULL`
|
||||
- Duplicate-key detection uses `IsNotesDuplicateKey`, which matches the constraint and the
|
||||
table rather than an exact column list. The old literal string comparison broke silently
|
||||
on this rename — do not reintroduce one
|
||||
|
||||
### `IsActive` Is Not Ours To Write
|
||||
`Blogs.IsActive`, `Posts.IsActive` and `Notes.IsActive` are removal flags set by other tools
|
||||
(Rolodex). `0` means removed; anything else, including `NULL`, means live. Full detail in
|
||||
`URLNotesGrabberCORE/TL.db.md`.
|
||||
|
||||
- **Never write any `IsActive` column.** Not in an `INSERT` column list, not in an `UPDATE`,
|
||||
and never via `INSERT OR REPLACE` on these tables — that resets the column default and
|
||||
un-removes the row. Re-crawling a removed row must refresh its content and leave the flag
|
||||
where it was
|
||||
- **Filter at selection, not at write.** Every query that *selects* posts, notes or blogs
|
||||
excludes removed rows. Update statements stay keyed on a row the caller already selected;
|
||||
filtering them would spend API quota and then fail to persist the result
|
||||
- `Posts.IsActive` and `Notes.IsActive` are **optional** — they are absent from databases
|
||||
that predate them, and naming a missing column is a hard SQLite error. Compose the filter
|
||||
with `AndIsActive`/`WhereIsActive` in `DataAccess.cs`, which return
|
||||
`COALESCE(IsActive, 1) = 1` only when `HasIsActiveColumn` finds the column. `Blogs.IsActive`
|
||||
is not optional and is filtered directly
|
||||
- Do not add these columns from this app, and do not add them to the missing-column list in
|
||||
`verify-db-schema.sql`
|
||||
|
||||
### `DateModified` Tracks Real Changes Only
|
||||
`Blogs.DateModified`, `Posts.DateModified` and `Notes.DateModified` must move only when a
|
||||
column beside `DateModified` itself actually changed. Re-crawling or re-ingesting identical
|
||||
content has to leave the row — and its timestamp — untouched, or downstream consumers cannot
|
||||
tell a refreshed row from a rewritten one.
|
||||
|
||||
- Enforce it in the `WHERE` clause, not in C#. Every `UPDATE` that sets `DateModified` ends
|
||||
with an `AND (<col> <> @param OR ...)` term covering every column in its `SET` list, so
|
||||
SQLite matches zero rows on a no-op and never writes
|
||||
- Compare NULL-safely: `IFNULL(col, '') <> IFNULL(@param, '')` for text,
|
||||
`IFNULL(col, 0) <> @param` for integer flags. A bare `col <> @param` is NULL on a NULL
|
||||
column and silently skips the row that most needs writing
|
||||
- Where NULL is not equivalent to the default, spell it out. The `HasBeenOutput = 0` stamps
|
||||
use `(HasBeenOutput IS NULL OR HasBeenOutput <> 0)` because the selection queries test
|
||||
`HasBeenOutput = 0`, which a NULL would never match
|
||||
- Dynamic `SET` lists (`UpdatePostContentFields`) build the guard alongside the assignments
|
||||
so the two lists cannot drift apart
|
||||
- These statements now return 0 rows for "found but unchanged" as well as "not found".
|
||||
Callers that read `ExecuteNonQuery()` must not treat 0 as "row missing"
|
||||
|
||||
**`Posts.NotesGatheredDateTime` is crawl bookkeeping, not content.** It moves on every
|
||||
`-collect` pass and says nothing about the post, so it must never move `DateModified` on its
|
||||
own. `UpdatePostMarkNotesCollected` still writes it every pass but wraps the timestamp in
|
||||
`DateModified = CASE WHEN IFNULL(HasNotesGathered, 0) <> 1 THEN @dateModified ELSE
|
||||
DateModified END` — SQLite evaluates `SET` expressions against the pre-`UPDATE` row, so only
|
||||
the flag flipping counts as a modification. Use this shape for any column that has to be
|
||||
refreshed unconditionally without being a change. `Blogs.LikesLastRefreshed` is the
|
||||
deliberate exception: a refresh pass is treated as a real event on the blog row.
|
||||
|
||||
**`Blogs.DateAdded` is write-once.** `AddBlog`'s `INSERT` is the only place that sets it. A
|
||||
new post arriving for a known blog reopens `HasBeenOutput` but must leave `DateAdded` alone —
|
||||
a new post is not a new blog, and rewriting the column both destroys the registration date
|
||||
and makes every insert look like a change.
|
||||
|
||||
**`"."` in a `Posts` content field means "not supplied", not "empty".** `ReblogRecord`
|
||||
(`TraverseDirectory`'s parser for the local `.txt` export tree) and the `--likes` API path
|
||||
both default every content field to the literal string `"."` when their source has no value
|
||||
for it, then pass that straight to `UpdatePost`. A blog with two export folders in different
|
||||
field formats (a duplicate `_2` folder, or a Tumblr export whose field set changed over time)
|
||||
sends one record with a real `Title`/`Tags`/`Slug` and another with those fields `"."`
|
||||
because that format never had a line for them — and without a guard, re-importing both on
|
||||
every run flips the row back and forth forever, bumping `DateModified` on every pass even
|
||||
though the true content never changes.
|
||||
|
||||
- Every content column in `UpdatePost`'s `SET` list is guarded the same way `RootBlogName`/
|
||||
`RootURL` already were: `col = CASE WHEN @col = '.' THEN col ELSE @col END`. A `"."`
|
||||
parameter leaves the existing value alone instead of overwriting it
|
||||
- The change-detection `WHERE` clause carries the same exception —
|
||||
`(@col <> '.' AND IFNULL(col, '') <> @col) OR ...` — so a `"."`-only difference does not
|
||||
make the statement fire at all, and `DateModified` stays put
|
||||
- Deliberately narrow: only the literal `"."` is the sentinel. An explicit empty string from
|
||||
a real record still overwrites, same as before this fix. `postID`, `BlogName`, `hasImage`,
|
||||
`ByLikes` are not part of this convention and are unaffected
|
||||
- If a new content field is added to `Posts`/`UpdatePost`, decide explicitly whether its
|
||||
source can legitimately supply `"."` as "field absent" before deciding whether it needs
|
||||
the same `CASE` treatment — don't assume every column needs it
|
||||
|
||||
**`--ingest` (`UpsertPostFromTextFile`) uses `NULL`, not `"."`, for the same "field absent"
|
||||
convention, and reconciling exactly this kind of duplicate IS the feature's job.**
|
||||
`IngestMode` strips a trailing `_N` from the folder name before it ever reaches
|
||||
`UpsertPostFromTextFile`, so a duplicate export folder collapses onto the same `BlogName` on
|
||||
purpose — the whole point is to merge multiple differently-formatted files for the same post
|
||||
into one row. `IngestMode.G(key)` returns `null` (not `"."`) when a field's line is absent
|
||||
from a given file, `LegacyPostsDbImporter` passes `null` straight from a `NULL` source column,
|
||||
and files are walked in raw filesystem enumeration order — never sorted — so which file's call
|
||||
lands last for a given `(BlogName, PostID)` is arbitrary.
|
||||
|
||||
- Before the fix, the `UPDATE` branch set every column unconditionally, so whichever file
|
||||
processed last for a `PostID` would null out every field its own record didn't carry —
|
||||
silently erasing real `Title`/`Slug`/`Tags`/… another file had, the opposite of what
|
||||
`--ingest` exists to do. This is worse than the `"."` case above: that one only caused
|
||||
churn (the two writes canceled out); this one loses data, and which posts lose which
|
||||
fields depends on filesystem enumeration order
|
||||
- Same shape of fix, `NULL` instead of `"."` as the sentinel: `col = CASE WHEN @col IS NULL
|
||||
THEN col ELSE @col END` in the `SET` list, `(@col IS NOT NULL AND IFNULL(col, '') <> @col)
|
||||
OR ...` in the change-detection
|
||||
- Same narrow rule: only `NULL` (the field's line was never present in this file) is the
|
||||
sentinel. `G()` already distinguishes this from "present but blank" — a dictionary miss is
|
||||
`null`, an empty value after the prefix is `""` — so an explicitly blank field still
|
||||
overwrites
|
||||
- `HasImage` is **not** guarded and remains a known gap: `IngestMode` always computes a
|
||||
concrete `bool` (defaulting `false` when a file has no `Has Image:` line), so there is no
|
||||
way for this function to tell "this format says no image" from "this format doesn't report
|
||||
it at all" without changing the parameter to `bool?` and threading that through
|
||||
`IngestMode`/`LegacyPostsDbImporter`. Fix this the same way if `--ingest` is observed
|
||||
downgrading a post's `HasImage` from `1` to `0`
|
||||
|
||||
### Testing
|
||||
- No existing test suite; use xUnit if adding tests
|
||||
- Test critical logic: `ApiKeyPool` init, color parsing, config persistence
|
||||
|
||||
+39
-6
@@ -1,4 +1,4 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?><sqlb_project><db path="C:/Users/jim/Nextcloud/C#/URLNotesGrabberCORE/URLNotesGrabberCORE/TL.db" readonly="0" foreign_keys="1" case_sensitive_like="0" temp_store="0" wal_autocheckpoint="1000" synchronous="2"/><attached/><window><main_tabs open="structure browser pragmas query" current="3"/></window><tab_structure><column_width id="0" width="300"/><column_width id="1" width="0"/><column_width id="2" width="100"/><column_width id="3" width="4305"/><column_width id="4" width="0"/><expanded_item id="0" parent="1"/><expanded_item id="1" parent="1"/><expanded_item id="2" parent="1"/><expanded_item id="3" parent="1"/></tab_structure><tab_browse><table title="Posts" custom_title="0" dock_id="4" table="4,5:mainPosts"/><dock_state state="000000ff00000000fd00000001000000020000077200000379fc0100000006fb000000160064006f0063006b00420072006f00770073006500310100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500320100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500330100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500350100000000000005f40000000000000000fb000000160064006f0063006b00420072006f00770073006500340100000000000007720000011700fffffffb000000160064006f0063006b00420072006f00770073006500340100000000000005f40000000000000000000007720000000000000004000000040000000800000008fc00000000"/><default_encoding codec=""/><browse_table_settings><table schema="main" name="ApiKeyPoolMeta" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="29"/><column index="2" value="64"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Blogs" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="257"/><column index="2" value="95"/><column index="3" value="54"/><column index="4" value="156"/><column index="5" value="51"/><column index="6" value="71"/><column index="7" value="85"/><column index="8" value="156"/><column index="9" value="156"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Posts" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="28" mode="1"/></sort><column_widths><column index="1" value="241"/><column index="2" value="148"/><column index="3" value="126"/><column index="4" value="300"/><column index="5" value="75"/><column index="6" value="187"/><column index="7" value="159"/><column index="8" value="75"/><column index="9" value="300"/><column index="10" value="300"/><column index="11" value="78"/><column index="12" value="249"/><column index="13" value="300"/><column index="14" value="53"/><column index="15" value="300"/><column index="16" value="300"/><column index="17" value="41"/><column index="18" value="75"/><column index="19" value="96"/><column index="20" value="300"/><column index="21" value="96"/><column index="22" value="300"/><column index="23" value="300"/><column index="24" value="42"/><column index="25" value="60"/><column index="26" value="218"/><column index="27" value="920"/><column index="28" value="156"/><column index="29" value="156"/></column_widths><filter_values><column index="24" value="=1"/><column index="28" value=">2026-05-06 20:00:01"/></filter_values><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table></browse_table_settings></tab_browse><tab_sql><sql name="SQL 1">UPDATE Posts
|
||||
<?xml version="1.0" encoding="UTF-8"?><sqlb_project><db path="D:/NextCloud/C#/URLNotesGrabberCORE/URLNotesGrabberCORE/TL.db" readonly="0" foreign_keys="1" case_sensitive_like="0" temp_store="0" wal_autocheckpoint="1000" synchronous="2"/><attached/><window><main_tabs open="structure browser pragmas query" current="3"/></window><tab_structure><column_width id="0" width="300"/><column_width id="1" width="0"/><column_width id="2" width="100"/><column_width id="3" width="4486"/><column_width id="4" width="0"/><expanded_item id="0" parent="1"/><expanded_item id="1" parent="1"/><expanded_item id="2" parent="1"/><expanded_item id="3" parent="1"/></tab_structure><tab_browse><table title="Posts" custom_title="0" dock_id="4" table="4,5:mainPosts"/><dock_state state="000000ff00000000fd0000000100000002000005470000029afc0100000006fb000000160064006f0063006b00420072006f00770073006500310100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500320100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500330100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500350100000000000005f40000000000000000fb000000160064006f0063006b00420072006f00770073006500340100000000000005470000011100fffffffb000000160064006f0063006b00420072006f00770073006500340100000000000005f40000000000000000000005470000000000000004000000040000000800000008fc00000000"/><default_encoding codec=""/><browse_table_settings><table schema="main" name="ApiKeyPoolMeta" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="29"/><column index="2" value="64"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Blogs" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="7" mode="1"/></sort><column_widths><column index="1" value="257"/><column index="2" value="108"/><column index="3" value="63"/><column index="4" value="156"/><column index="5" value="60"/><column index="6" value="81"/><column index="7" value="85"/><column index="8" value="156"/><column index="9" value="156"/><column index="10" value="151"/><column index="11" value="125"/><column index="12" value="129"/></column_widths><filter_values><column index="4" value="1"/><column index="7" value=">2026-05-27 17:22:36"/></filter_values><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Notes" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="198"/><column index="2" value="144"/><column index="3" value="251"/><column index="4" value="84"/><column index="5" value="53"/><column index="6" value="300"/><column index="7" value="116"/><column index="8" value="152"/><column index="9" value="89"/><column index="10" value="63"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Posts" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="14" mode="1"/></sort><column_widths><column index="1" value="236"/><column index="2" value="144"/><column index="3" value="126"/><column index="4" value="32"/><column index="5" value="32"/><column index="6" value="32"/><column index="7" value="32"/><column index="8" value="32"/><column index="9" value="0"/><column index="10" value="0"/><column index="11" value="0"/><column index="12" value="243"/><column index="13" value="300"/><column index="14" value="53"/><column index="15" value="37351"/><column index="16" value="300"/><column index="17" value="41"/><column index="18" value="75"/><column index="19" value="96"/><column index="20" value="300"/><column index="21" value="96"/><column index="22" value="300"/><column index="23" value="300"/><column index="24" value="548"/><column index="25" value="60"/><column index="26" value="213"/><column index="27" value="532"/><column index="28" value="152"/><column index="29" value="152"/><column index="30" value="69"/><column index="31" value="63"/></column_widths><filter_values><column index="2" value="0"/><column index="1" value="734568371821084672"/></filter_values><conditional_formats/><row_id_formats/><display_formats/><hidden_columns><column index="9" value="1"/><column index="10" value="1"/><column index="11" value="1"/></hidden_columns><plot_y_axes/><global_filter/></table></browse_table_settings></tab_browse><tab_sql><sql name="SQL 1">UPDATE Posts
|
||||
SET HasNotesGathered = 0
|
||||
WHERE (BlogName, PostID) IN (
|
||||
SELECT p.BlogName, p.PostID
|
||||
@@ -29,11 +29,12 @@ blogname in
|
||||
'nudenymph',
|
||||
'caylachief'
|
||||
|
||||
)</sql><sql name="New Notes">select RootBlogName, PostID, NoteBlogName || '.tumblr.com' as NoteBlogName, DatetimeCrawled, TimeStamp, type, RootBlogName || '.tumblr.com/post/' || postid, datetime(timestamp, 'unixepoch')
|
||||
from Notes
|
||||
)</sql><sql name="New Notes">select P.slug, N.replyText, n.RootBlogName, n.PostID, NoteBlogName || '.tumblr.com' as NoteBlogName, DatetimeCrawled, TimeStamp, type, n.RootBlogName || '.tumblr.com/post/' || n.postid, datetime(timestamp, 'unixepoch')
|
||||
from Notes N inner join Posts P on p.PostID = n.PostID
|
||||
where
|
||||
DatetimeCrawled > '2026-05-14 02:50:05' --and type like 'r%'
|
||||
order by DatetimeCrawled desc</sql><sql name="Pull Blogs*">SELECT distinct␍
|
||||
DatetimeCrawled > '2026-08-07 11:47:22' and type like 'r%'
|
||||
and P.IsActive = 1
|
||||
order by n.DatetimeCrawled</sql><sql name="Pull Blogs">SELECT distinct
|
||||
'''' || blogname || ''',',
|
||||
blogs.*
|
||||
, blogname || '.tumblr.com'
|
||||
@@ -64,4 +65,36 @@ JOIN ReplyCounts c ON n.NoteBlogName = c.NoteBlogName
|
||||
where replyText <> '.' and type <> 'reply'
|
||||
--AND N.NoteBlogName NOT IN ( 'roadblocker21', 'thesaddemon666', 'edwardabbeyhoffman', 'tattedsoldier20', 'zomb-eh', 'animalistic13', 'indken', 'maccloud1592',
|
||||
--'moss-wizard', 'supertrucker12682', 'exploringthrupics', 'padeyepete' )
|
||||
order by c.DistinctReplyCount desc, n.NoteBlogName, n.DateModified desc, replyText, RootBlogName, PostID</sql><current_tab id="3"/></tab_sql></sqlb_project>
|
||||
order by c.DistinctReplyCount desc, n.NoteBlogName, n.DateModified desc, replyText, RootBlogName, PostID</sql><sql name="Collect">WITH PostsWithCount AS ( SELECT P.BlogName, P.PostID, 1925013599 AS LatestNoteTimestamp, P.NotesGatheredDateTime, COUNT(P.PostID) OVER(PARTITION BY P.BlogName) AS CNT, P.HasNotesGathered, P.NotFound, P.PostDate FROM Posts P WHERE COALESCE(P.IsActive, 1) = 1 ), Unioned AS ( SELECT BlogName, PostID, LatestNoteTimestamp, NotesGatheredDateTime, CNT, PostDate FROM PostsWithCount WHERE NotFound = 0 AND HasNotesGathered = 0 UNION SELECT BlogName, PostID, LatestNoteTimestamp, NotesGatheredDateTime, CNT, PostDate FROM PostsWithCount WHERE BlogName = 'zomb-eh' AND NotFound = 0 AND NotesGatheredDateTime < unixepoch('now', 'localtime', '-3 days') ) SELECT U.BlogName, U.PostID, U.LatestNoteTimestamp, U.NotesGatheredDateTime, U.CNT FROM Unioned U WHERE (U.NotesGatheredDateTime < 1786134037 OR U.NotesGatheredDateTime IS NULL) ORDER BY U.NotesGatheredDateTime, U.PostDate DESC, U.BlogName, U.PostID;</sql><sql name="Del Posts">delete from posts where postid in
|
||||
(
|
||||
'741662499571728384',
|
||||
178892849664,
|
||||
178264721139,
|
||||
177012868749,
|
||||
169950081964,
|
||||
755440787056099328
|
||||
)</sql><sql name="notes NO post*">select *
|
||||
-- delete
|
||||
from notes
|
||||
where postid not in (select distinct postid from posts where IsActive = 1)</sql><sql name="SQL 9">SELECT
|
||||
*
|
||||
FROM
|
||||
POSTS P
|
||||
WHERE
|
||||
P.ByLikes = 1
|
||||
AND
|
||||
P.DateCreated > '2026-05-26 17:47:32'
|
||||
ORDER BY
|
||||
P.DateCreated desc</sql><sql name="SQL 13">update posts set IsActive = 0 where blogname IN ( 'shoebiedoo', 'redheaded-girlygirl', 'xlittle-ghost' )</sql><sql name="SQL 14*">update Posts␍
|
||||
set IsActive = 0␍
|
||||
where postid in␍
|
||||
(␍
|
||||
␍
|
||||
␍
|
||||
'731937314675310592'␍
|
||||
␍
|
||||
␍
|
||||
␍
|
||||
)␍
|
||||
␍
|
||||
</sql><current_tab id="7"/></tab_sql></sqlb_project>
|
||||
|
||||
Binary file not shown.
+512
-236
File diff suppressed because it is too large
Load Diff
@@ -29,6 +29,8 @@ namespace URLNotesGrabberCORE
|
||||
Console.WriteLine($"Reading legacy posts.db: {legacyDbPath}");
|
||||
|
||||
int blogsCopied = 0;
|
||||
int blogPathsWritten = 0;
|
||||
int blogsWithoutPath = 0;
|
||||
int postsUpserted = 0;
|
||||
int errors = 0;
|
||||
|
||||
@@ -48,7 +50,13 @@ namespace URLNotesGrabberCORE
|
||||
if (string.IsNullOrWhiteSpace(blogName)) continue;
|
||||
try
|
||||
{
|
||||
DataAccess.SetBlogTTFolderPath(blogName, ttFolderPath);
|
||||
// A legacy row whose TTFolderPath was already NULL copies nothing.
|
||||
// Counting it as "copied" is what hid the fact that this import has
|
||||
// never populated a single path.
|
||||
if (string.IsNullOrWhiteSpace(ttFolderPath))
|
||||
blogsWithoutPath++;
|
||||
else if (DataAccess.SetBlogTTFolderPath(blogName, ttFolderPath.Trim()))
|
||||
blogPathsWritten++;
|
||||
blogsCopied++;
|
||||
}
|
||||
catch (Exception ex)
|
||||
@@ -58,7 +66,7 @@ namespace URLNotesGrabberCORE
|
||||
}
|
||||
}
|
||||
}
|
||||
Console.WriteLine($" Blogs copied: {blogsCopied}");
|
||||
Console.WriteLine($" Blogs seen: {blogsCopied}, TTFolderPath written: {blogPathsWritten}, legacy rows with no path: {blogsWithoutPath}");
|
||||
|
||||
// 2) Copy Posts
|
||||
try
|
||||
@@ -138,7 +146,8 @@ namespace URLNotesGrabberCORE
|
||||
}
|
||||
|
||||
Console.WriteLine($"\n========== Legacy import summary ==========");
|
||||
Console.WriteLine($"Blogs copied: {blogsCopied}");
|
||||
Console.WriteLine($"Blogs seen: {blogsCopied}");
|
||||
Console.WriteLine($"Paths written: {blogPathsWritten} (legacy rows with no path: {blogsWithoutPath})");
|
||||
Console.WriteLine($"Posts upserted: {postsUpserted}");
|
||||
Console.WriteLine($"Errors: {errors}");
|
||||
return errors == 0 ? 0 : 2;
|
||||
|
||||
@@ -8,28 +8,51 @@ namespace URLNotesGrabberCORE
|
||||
// field order). Reads from TL.db via DataAccess.GetAllPostsForBlog.
|
||||
public static class OutputMode
|
||||
{
|
||||
public static int Run(IConfiguration config)
|
||||
public static int Run(IConfiguration config, string[]? args = null)
|
||||
{
|
||||
DataAccess.EnsureTTFileHelperColumnsExist();
|
||||
|
||||
var blogs = DataAccess.GetAllBlogsWithTTFolderPath();
|
||||
Console.WriteLine($"Found {blogs.Count} blog(s) to process.");
|
||||
string dbPath = DataAccess.GetActiveDbPath();
|
||||
Console.WriteLine($"Database: {Path.GetFullPath(dbPath)}");
|
||||
|
||||
foreach (var (blogName, ttFolderPath) in blogs)
|
||||
if (!RefreshPaths(config, args ?? Array.Empty<string>()))
|
||||
return 1;
|
||||
|
||||
var blogs = DataAccess.GetAllBlogsWithTTFolderPath();
|
||||
int activeBlogs = DataAccess.CountActiveBlogs();
|
||||
Console.WriteLine($"{blogs.Count} of {activeBlogs} active blog(s) have a TTFolderPath.");
|
||||
|
||||
if (blogs.Count == 0)
|
||||
{
|
||||
Console.WriteLine($"\nNothing to export: no blog in {Path.GetFullPath(dbPath)} has a TTFolderPath.");
|
||||
Console.WriteLine("Point --output at a TumblThree root so it can populate them: --output <root>,");
|
||||
Console.WriteLine("or set appSettings:PathTTRoot so the refresh runs automatically.");
|
||||
return 1;
|
||||
}
|
||||
|
||||
int missingFolderCount = 0;
|
||||
int writtenCount = 0;
|
||||
|
||||
foreach (var (blogName, folder) in blogs)
|
||||
{
|
||||
Console.WriteLine($"\nProcessing blog: {blogName}");
|
||||
|
||||
if (string.IsNullOrWhiteSpace(ttFolderPath) || !Directory.Exists(ttFolderPath))
|
||||
// A stored path that this machine cannot see means the value was written on
|
||||
// another machine -- re-running --updatepaths locally is the fix, so say so
|
||||
// rather than lumping it in with "not set".
|
||||
if (!Directory.Exists(folder))
|
||||
{
|
||||
Console.WriteLine($" TTFolderPath does not exist or is not set. Skipping.");
|
||||
Console.WriteLine($" TTFolderPath folder not found: {folder}. Skipping.");
|
||||
missingFolderCount++;
|
||||
continue;
|
||||
}
|
||||
|
||||
Console.WriteLine($" TTFolderPath: {ttFolderPath}");
|
||||
Console.WriteLine($" TTFolderPath: {folder}");
|
||||
writtenCount++;
|
||||
|
||||
try
|
||||
{
|
||||
foreach (var bakFile in Directory.GetFiles(ttFolderPath, "*.bak"))
|
||||
foreach (var bakFile in Directory.GetFiles(folder, "*.bak"))
|
||||
File.Delete(bakFile);
|
||||
}
|
||||
catch (Exception ex)
|
||||
@@ -37,7 +60,7 @@ namespace URLNotesGrabberCORE
|
||||
Console.WriteLine($" Error deleting .bak files: {ex.Message}");
|
||||
}
|
||||
|
||||
RenameExistingTxtFilesToBak(ttFolderPath);
|
||||
RenameExistingTxtFilesToBak(folder);
|
||||
|
||||
var posts = DataAccess.GetAllPostsForBlog(blogName);
|
||||
Console.WriteLine($" Found {posts.Count} post(s) for this blog.");
|
||||
@@ -46,7 +69,7 @@ namespace URLNotesGrabberCORE
|
||||
foreach (var typeGroup in grouped)
|
||||
{
|
||||
string postType = typeGroup.Key ?? "Unknown";
|
||||
string outputFilePath = Path.Combine(ttFolderPath, $"{postType}.txt");
|
||||
string outputFilePath = Path.Combine(folder, $"{postType}.txt");
|
||||
var ordered = typeGroup.OrderBy(p => p.Date).ToList();
|
||||
Console.WriteLine($" Writing {ordered.Count} post(s) to {postType}.txt");
|
||||
|
||||
@@ -65,10 +88,57 @@ namespace URLNotesGrabberCORE
|
||||
}
|
||||
}
|
||||
|
||||
Console.WriteLine("\nOutput mode complete.");
|
||||
Console.WriteLine($"\nOutput mode complete. {writtenCount} blog(s) exported, {missingFolderCount} skipped for a missing folder.");
|
||||
|
||||
if (writtenCount == 0)
|
||||
Console.WriteLine("Every TTFolderPath points at a folder this machine cannot see. The paths were most likely written on another machine -- re-run --updatepaths <root> here so they match local drive letters.");
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
// Re-reads the TumblThree Index metadata into Blogs.TTFolderPath before exporting.
|
||||
// A TL.db synced between machines cannot hold one absolute path that is valid on
|
||||
// both, so the stored paths are only trustworthy on the machine that wrote them --
|
||||
// which makes this refresh part of a normal export rather than a separate chore.
|
||||
// Returns false only when the run should stop.
|
||||
private static bool RefreshPaths(IConfiguration config, string[] args)
|
||||
{
|
||||
var settings = config.GetSection("appSettings");
|
||||
|
||||
if (args.Any(a => string.Equals(a, "--norefresh", StringComparison.OrdinalIgnoreCase)))
|
||||
{
|
||||
Console.WriteLine("Path refresh skipped (--norefresh); exporting to whatever paths TL.db already holds.");
|
||||
return true;
|
||||
}
|
||||
|
||||
string? root = args.FirstOrDefault(a => !a.StartsWith("--", StringComparison.Ordinal))
|
||||
?? settings.GetValue<string>("PathTTRoot");
|
||||
|
||||
var result = UpdateBlogPathsRunner.Scan(root, verbose: false);
|
||||
|
||||
switch (result.Outcome)
|
||||
{
|
||||
case UpdateBlogPathsRunner.ScanOutcome.NoRootConfigured:
|
||||
Console.WriteLine("No TumblThree root configured (appSettings:PathTTRoot is empty and none was passed),");
|
||||
Console.WriteLine("so TTFolderPath was not refreshed. Pass one as --output <root> to refresh it.");
|
||||
return true;
|
||||
|
||||
case UpdateBlogPathsRunner.ScanOutcome.IndexFolderMissing:
|
||||
// Silently exporting stale paths here would defeat the point of folding
|
||||
// the refresh in, so a bad root is a hard stop.
|
||||
Console.WriteLine($"Index folder not found at: {result.IndexPath}");
|
||||
Console.WriteLine("Fix the root (or pass --norefresh to export the paths already in TL.db).");
|
||||
return false;
|
||||
|
||||
default:
|
||||
Console.WriteLine($"Refreshed paths from {result.IndexPath}: " +
|
||||
$"{result.MetadataFiles} metadata file(s), {result.Written} written, " +
|
||||
$"{result.Unchanged} already correct, {result.NoLocation} without a location, " +
|
||||
$"{result.NoMatchingRow} without a blog row, {result.Errors} error(s).");
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
private static void RenameExistingTxtFilesToBak(string folderPath)
|
||||
{
|
||||
try
|
||||
|
||||
+113
-56
@@ -281,7 +281,7 @@ namespace URLNotesGrabberCORE
|
||||
managedCollectRun = true;
|
||||
}
|
||||
|
||||
CollectNotes(settings.GetValue<string>("PathOutput"), withoutNotesOnly, beforeDate, managedCollectRun).GetAwaiter().GetResult();
|
||||
exitCode = CollectNotes(settings.GetValue<string>("PathOutput"), withoutNotesOnly, beforeDate, managedCollectRun).GetAwaiter().GetResult();
|
||||
break;
|
||||
|
||||
case "--blogsR": //collect notes from all posts
|
||||
@@ -343,7 +343,7 @@ namespace URLNotesGrabberCORE
|
||||
break;
|
||||
|
||||
case "--output":
|
||||
exitCode = OutputMode.Run(config);
|
||||
exitCode = OutputMode.Run(config, args.Skip(1).ToArray());
|
||||
break;
|
||||
|
||||
case "--revert":
|
||||
@@ -439,7 +439,9 @@ namespace URLNotesGrabberCORE
|
||||
|
||||
Console.WriteLine("--ingest [blogname]\t Ingest Tumblr .txt exports from appSettings:PathTTRoot into TL.db (all blogs, or single blog if name given)");
|
||||
|
||||
Console.WriteLine("--output\t Export posts from TL.db back to .txt files in each blog's TTFolderPath");
|
||||
Console.WriteLine("--output [rootPath]\t Refresh Blogs.TTFolderPath from <root>\\Index (or appSettings:PathTTRoot), then export posts from TL.db back to .txt files in each blog's folder");
|
||||
|
||||
Console.WriteLine("--output --norefresh\t Export without refreshing TTFolderPath first");
|
||||
|
||||
Console.WriteLine("--revert [blogname]\t Recursively scan the PathInput tree and restore *.bak back to *.txt (current .txt saved as next-free .bkN); optional blogname filters by path substring");
|
||||
|
||||
@@ -452,7 +454,7 @@ namespace URLNotesGrabberCORE
|
||||
Console.WriteLine("--importposts [path-to-posts.db]\t One-time migration: copy legacy ThreeTxtFileHelper posts.db rows into TL.db");
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine("Exit status: 0 = success; 1 = unexpected error; 2 = usage error (unknown command or bad/missing arguments)");
|
||||
Console.WriteLine("Exit status: 0 = success; 1 = unexpected error; 2 = usage error (unknown command or bad/missing arguments); 3 = incomplete (--collect paused on a rate limit, or skipped posts after transient API failures) - relaunch to resume");
|
||||
}
|
||||
|
||||
static void WritePostBlogsToFile(string outPath)
|
||||
@@ -582,16 +584,11 @@ namespace URLNotesGrabberCORE
|
||||
|
||||
protected static string NormalizeBlogFolderName(string folderName)
|
||||
{
|
||||
return folderName
|
||||
.Replace("_1", "")
|
||||
.Replace("_2", "")
|
||||
.Replace("_3", "")
|
||||
.Replace("_4", "")
|
||||
.Replace("_5", "")
|
||||
.Replace("_6", "")
|
||||
.Replace("_7", "")
|
||||
.Replace("_8", "")
|
||||
.Replace("_9", "");
|
||||
// Archive tools suffix duplicate blog folders with _1, _2, ... _10 and beyond. Strip only a
|
||||
// trailing numeric suffix: unanchored substring removal ate the "_1" inside "_10" and left the
|
||||
// "0" welded to the name (zomb-eh_10 -> zomb-eh0), and mangled any blog whose real name
|
||||
// contains "_1". A blog name is never a prefix of itself plus "_<digits>", so this is safe.
|
||||
return System.Text.RegularExpressions.Regex.Replace(folderName, @"_\d+$", "");
|
||||
}
|
||||
|
||||
static async Task FetchAndStoreReplyText(string blogName, long postID, long timestamp)
|
||||
@@ -847,7 +844,9 @@ namespace URLNotesGrabberCORE
|
||||
|
||||
RateLimiter limiter = new SlidingWindowRateLimiter(new SlidingWindowRateLimiterOptions
|
||||
{
|
||||
PermitLimit = 300,
|
||||
// 1/sec average, matching --collect: the CDN reacts to aggregate traffic from the IP,
|
||||
// not to per-command rates.
|
||||
PermitLimit = 60,
|
||||
QueueProcessingOrder = QueueProcessingOrder.OldestFirst,
|
||||
QueueLimit = 1,
|
||||
Window = TimeSpan.FromMinutes(1),
|
||||
@@ -883,7 +882,9 @@ namespace URLNotesGrabberCORE
|
||||
|
||||
while (hasMoreLikes)
|
||||
{
|
||||
using RateLimitLease lease = limiter.AttemptAcquire(1);
|
||||
// Wait for a permit rather than giving up on one: the limiter paces the loop, it is
|
||||
// not a failure condition. Only one acquire is ever pending, so QueueLimit = 1 suffices.
|
||||
using RateLimitLease lease = await limiter.AcquireAsync(1);
|
||||
if (!lease.IsAcquired)
|
||||
{
|
||||
Console.WriteLine("!@@@@@@@ - Rate Limited Exceeded: No Lease Available");
|
||||
@@ -1129,6 +1130,43 @@ if (shouldInsert)
|
||||
}
|
||||
}
|
||||
|
||||
// Backoff between in-place retries of a transient infrastructure failure. Most CDN 403s and edge
|
||||
// 5xxs clear within a few seconds, so retrying here saves the post its single attempt for the pass.
|
||||
static readonly int[] TransientBackoffSeconds = { 1, 4, 10 };
|
||||
|
||||
// Fetches one page, retrying transient failures in place. Rate limits are returned to the caller
|
||||
// untouched — those are handled by pausing the whole run, not by retrying this post.
|
||||
static async Task<Root> FetchNotesPage(Tuple<string, long, long, long> post, string beforeTimestamp)
|
||||
{
|
||||
Root response = null!;
|
||||
|
||||
for (int attempt = 0; ; attempt++)
|
||||
{
|
||||
var key = ApiKeyPool.GetCurrentKey();
|
||||
response = await APIAccess.GrabNotes(key, post.Item1, post.Item2, beforeTimestamp);
|
||||
|
||||
if (response.statusCode == "TooManyRequests")
|
||||
{
|
||||
ApiKeyPool.MarkRateLimited(key, response.retryInSeconds > 0 ? response.retryInSeconds : 60);
|
||||
return response;
|
||||
}
|
||||
|
||||
if (!response.transientFailure)
|
||||
{
|
||||
// Only a response that actually reached the API says anything about the key's standing.
|
||||
ApiKeyPool.MarkAvailable(key);
|
||||
return response;
|
||||
}
|
||||
|
||||
if (attempt >= TransientBackoffSeconds.Length)
|
||||
return response;
|
||||
|
||||
int delay = TransientBackoffSeconds[attempt];
|
||||
Console.WriteLine($"[Transient] retry {attempt + 1}/{TransientBackoffSeconds.Length} in {delay}s");
|
||||
await Task.Delay(delay * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
static async Task<string> GrabNotes(Tuple<string, long, long, long> post)
|
||||
{
|
||||
try
|
||||
@@ -1146,18 +1184,16 @@ if (shouldInsert)
|
||||
string beforeTimestamp = post.Item3.ToString();
|
||||
bool hasReplies = false;
|
||||
const int maxPages = 500;
|
||||
var key = ApiKeyPool.GetCurrentKey();
|
||||
var response = await APIAccess.GrabNotes(key, post.Item1, post.Item2, beforeTimestamp);
|
||||
var response = await FetchNotesPage(post, beforeTimestamp);
|
||||
|
||||
if (response.statusCode == "TooManyRequests")
|
||||
{
|
||||
int retry = response.retryInSeconds > 0 ? response.retryInSeconds : 60;
|
||||
ApiKeyPool.MarkRateLimited(key, retry);
|
||||
return "TooManyRequests";
|
||||
}
|
||||
|
||||
if (response.meta?.status != 429)
|
||||
ApiKeyPool.MarkAvailable(key);
|
||||
if (response.transientFailure)
|
||||
{
|
||||
Console.WriteLine($"[Skip] {post.Item1}/{post.Item2} — {response.statusCode} after {TransientBackoffSeconds.Length} retries");
|
||||
return "Transient";
|
||||
}
|
||||
|
||||
if (IsNotFound(response))
|
||||
{
|
||||
@@ -1166,19 +1202,6 @@ if (shouldInsert)
|
||||
Thread.Sleep(1000);
|
||||
return "NotFound";
|
||||
}
|
||||
if (response == null)
|
||||
{
|
||||
Console.WriteLine("##### Response is null - API Failure? ###");
|
||||
return "FAILURE";
|
||||
}
|
||||
if (response.statusCode == "TooManyRequests")
|
||||
{
|
||||
int retry = response.retryInSeconds > 0 ? response.retryInSeconds : 60;
|
||||
ApiKeyPool.MarkRateLimited(key, retry);
|
||||
|
||||
ApiKeyPool.SleepUntilAnyAvailable(30);
|
||||
return response.statusCode;
|
||||
}
|
||||
|
||||
// Pagination loop
|
||||
while (true)
|
||||
@@ -1228,16 +1251,15 @@ if (shouldInsert)
|
||||
break;
|
||||
}
|
||||
|
||||
key = ApiKeyPool.GetCurrentKey();
|
||||
response = await APIAccess.GrabNotes(key, post.Item1, post.Item2, beforeTimestamp);
|
||||
response = await FetchNotesPage(post, beforeTimestamp);
|
||||
if (response.statusCode == "TooManyRequests")
|
||||
{
|
||||
int retry = response.retryInSeconds > 0 ? response.retryInSeconds : 60;
|
||||
ApiKeyPool.MarkRateLimited(key, retry);
|
||||
return "TooManyRequests";
|
||||
|
||||
if (response.transientFailure)
|
||||
{
|
||||
Console.WriteLine($"[Skip] {post.Item1}/{post.Item2} — {response.statusCode} on page {page} after {TransientBackoffSeconds.Length} retries");
|
||||
return "Transient";
|
||||
}
|
||||
if (response.meta?.status != 429)
|
||||
ApiKeyPool.MarkAvailable(key);
|
||||
|
||||
if (IsNotFound(response))
|
||||
{
|
||||
@@ -1268,7 +1290,11 @@ if (shouldInsert)
|
||||
return "UNKNOWN";
|
||||
}
|
||||
|
||||
static async Task CollectNotes(string outPath, bool withoutNotesOnly = true, DateTime? beforeDate = null, bool managedRun = false)
|
||||
// Consecutive transient failures that mean the API edge is rejecting traffic wholesale rather than
|
||||
// blipping on one post. Past this, skipping post-by-post would just hammer a closed door.
|
||||
const int MaxConsecutiveTransient = 10;
|
||||
|
||||
static async Task<int> CollectNotes(string outPath, bool withoutNotesOnly = true, DateTime? beforeDate = null, bool managedRun = false)
|
||||
{
|
||||
List<Tuple<string, long, long, long>> posts = DataAccess.GetPosts(withoutNotesOnly, beforeDate);
|
||||
|
||||
@@ -1277,9 +1303,14 @@ if (shouldInsert)
|
||||
// post that keeps returning FAILURE/UNKNOWN. Successful/NotFound posts drop out via the DB filter anyway.
|
||||
HashSet<(string, long)> attempted = new HashSet<(string, long)>();
|
||||
|
||||
int skipped = 0;
|
||||
int consecutiveTransient = 0;
|
||||
|
||||
RateLimiter limiter = new SlidingWindowRateLimiter(new SlidingWindowRateLimiterOptions
|
||||
{
|
||||
PermitLimit = 300,
|
||||
// 1/sec average. Sustained higher rates draw CDN-level 403s that the API's own rate-limit
|
||||
// headers never warn about, so this sits well under the per-key quota on purpose.
|
||||
PermitLimit = 60,
|
||||
QueueProcessingOrder = QueueProcessingOrder.OldestFirst,
|
||||
QueueLimit = 1,
|
||||
Window = TimeSpan.FromMinutes(1),
|
||||
@@ -1304,19 +1335,35 @@ if (shouldInsert)
|
||||
|
||||
ApiKeyPool.SleepUntilAnyAvailable(30);
|
||||
|
||||
string status;
|
||||
|
||||
using RateLimitLease lease = limiter.AttemptAcquire(1);
|
||||
if (lease.IsAcquired)
|
||||
// Wait for a permit rather than giving up on one: the limiter paces the loop, it is
|
||||
// not a failure condition. Only one acquire is ever pending, so QueueLimit = 1 suffices.
|
||||
using RateLimitLease lease = await limiter.AcquireAsync(1);
|
||||
if (!lease.IsAcquired)
|
||||
{
|
||||
Console.WriteLine("!@@@@@@@ - Rate Limited Exceeded: No Lease Available");
|
||||
return 3; // abort without completing the run so a later launch resumes
|
||||
}
|
||||
|
||||
Console.WriteLine("{0,32} - {1,15} - {2}", post.Item1, post.Item2, DateTimeOffset.FromUnixTimeSeconds(post.Item4).ToString());
|
||||
status = await GrabNotes(post);
|
||||
string status = await GrabNotes(post);
|
||||
|
||||
if (status == "Transient")
|
||||
{
|
||||
// Retries in GrabNotes are already exhausted. Skip the post so the pass can make
|
||||
// progress; it stays unmarked in the DB, so the next launch picks it up again.
|
||||
attempted.Add((post.Item1, post.Item2));
|
||||
skipped++;
|
||||
consecutiveTransient++;
|
||||
|
||||
if (consecutiveTransient >= MaxConsecutiveTransient)
|
||||
{
|
||||
Console.WriteLine($"[Abort] {consecutiveTransient} consecutive transient failures - the API edge is rejecting traffic. Pausing run; relaunch to resume. ({skipped} post(s) skipped)");
|
||||
return 3;
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
Console.WriteLine("!@@@@@@@ - Rate Limited Exceeded: No Lease Available");
|
||||
return; // throttle: abort without completing the run so a later launch resumes
|
||||
}
|
||||
consecutiveTransient = 0;
|
||||
|
||||
if (status == "Success")
|
||||
{
|
||||
@@ -1333,8 +1380,8 @@ if (shouldInsert)
|
||||
{
|
||||
// Throttle, not a real per-post failure: don't consume this post's single attempt.
|
||||
// Abort the pass without completing so a later launch resumes against the same cutoff.
|
||||
Console.WriteLine("GrabNotes Result: TooManyRequests - pausing run; relaunch to resume.");
|
||||
return;
|
||||
Console.WriteLine($"GrabNotes Result: TooManyRequests - pausing run; relaunch to resume. ({skipped} post(s) skipped)");
|
||||
return 3;
|
||||
}
|
||||
else
|
||||
{
|
||||
@@ -1342,6 +1389,7 @@ if (shouldInsert)
|
||||
attempted.Add((post.Item1, post.Item2));
|
||||
Console.WriteLine("GrabNotes Result: " + status);
|
||||
}
|
||||
}
|
||||
|
||||
// Re-fetch the updated list after processing the current post
|
||||
posts = DataAccess.GetPosts(withoutNotesOnly, beforeDate);
|
||||
@@ -1354,10 +1402,19 @@ if (shouldInsert)
|
||||
DataAccess.CompleteCollectRun();
|
||||
Console.WriteLine("Full re-check run complete.");
|
||||
}
|
||||
|
||||
if (skipped > 0)
|
||||
{
|
||||
Console.WriteLine($"Pass finished with {skipped} post(s) skipped after transient failures; relaunch to retry them.");
|
||||
return 3;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
Console.WriteLine(ex.ToString());
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -85,6 +85,10 @@ namespace URLNotesGrabberCORE
|
||||
public int retryInSeconds { get; set; }
|
||||
|
||||
public string rawJson { get; set; }
|
||||
|
||||
// The request never reached the Tumblr API (transport error, or an edge/CDN response with a
|
||||
// non-JSON body). Says nothing about the post, so the caller should retry rather than fail it.
|
||||
public bool transientFailure { get; set; }
|
||||
}
|
||||
|
||||
// Classes for Posts API endpoint (for reply_text)
|
||||
|
||||
Binary file not shown.
@@ -0,0 +1,679 @@
|
||||
# `TL.db` — schema notes
|
||||
|
||||
The SQLite database behind **URLNotesGrabberCORE** and its sibling crawlers, and the one
|
||||
[Rolodex](https://git.basso.land/jim/Rolodex) reads.
|
||||
|
||||
Everything below was read out of the live file, not inferred from code. Counts are as of
|
||||
**2026-08-07**; re-run the queries at the bottom to refresh them.
|
||||
|
||||
- Journal mode: **WAL** — `TL.db-wal` and `TL.db-shm` live beside the file and are part of
|
||||
the database. Copying `TL.db` alone gives you whatever was last checkpointed, not the
|
||||
current state.
|
||||
- Page size: 4096. File size: 148 MB.
|
||||
|
||||
> ### ⚠ Breaking change, 2026-08-07: `Notes` holds integer IDs, not names
|
||||
>
|
||||
> `Notes.RootBlogName`, `Notes.NoteBlogName` and `Notes.Type` **no longer exist**. They
|
||||
> are now `RootBlogId`, `NoteBlogId` and `TypeId`, resolved through the new `BlogNames`
|
||||
> and `NoteTypes` tables. Any query naming the old columns fails outright.
|
||||
>
|
||||
> There is no compatibility view. See [porting to the integer
|
||||
> schema](#porting-to-the-integer-schema) for the old-to-new translation of every query
|
||||
> shape the applications use.
|
||||
>
|
||||
> Applied by `../normalize-notes.sql`, which took the file from 207 MB to 148 MB. An
|
||||
> earlier change the same day (`../shrink-db.sql`) took it from 267 MB to 207 MB.
|
||||
|
||||
---
|
||||
|
||||
## The three content tables
|
||||
|
||||
| Table | Rows | What it is |
|
||||
|---|--:|---|
|
||||
| `Blogs` | 188,620 | The crawl registry — one row per known blog, plus crawl-state flags |
|
||||
| `Posts` | 22,468 | Stored post content. Only 3,867 blogs actually have any |
|
||||
| `Notes` | 1,182,333 | The engagement graph: `NoteBlogId` acted on `(RootBlogId, PostID)` |
|
||||
|
||||
…supported by two lookup tables that exist only to keep `Notes` small:
|
||||
|
||||
| Table | Rows | What it is |
|
||||
|---|--:|---|
|
||||
| `BlogNames` | 20,430 | `BlogId` ⇄ `BlogName`. The ID authority for everything in `Notes` |
|
||||
| `NoteTypes` | 5 | `TypeId` ⇄ `Type`. `like`, `reblog`, `reply`, `posted`, `post_attribution` |
|
||||
|
||||
The engagement graph is the interesting part. 20,311 distinct blogs appear as engagers —
|
||||
far more than the 3,867 that have stored posts — which is what makes this a social graph
|
||||
rather than a post archive. Only 2,771 blogs appear as the *root* of a note.
|
||||
|
||||
### `Blogs`
|
||||
|
||||
```sql
|
||||
CREATE TABLE "Blogs" (
|
||||
"BlogName" TEXT,
|
||||
"HasBeenOutput" INTEGER DEFAULT 0,
|
||||
"IsActive" INTEGER DEFAULT 1,
|
||||
"DateAdded" TEXT NOT NULL DEFAULT '12/24/25',
|
||||
"ByLikes" INTEGER NOT NULL DEFAULT 0,
|
||||
"LikesPulled" INTEGER NOT NULL DEFAULT 0,
|
||||
"LikesCursor" INTEGER DEFAULT 0,
|
||||
"DateModified" TEXT,
|
||||
"DateCreated" TEXT,
|
||||
LikesNewestTimestamp INTEGER DEFAULT 0,
|
||||
LikesLastRefreshed INTEGER DEFAULT 0,
|
||||
LikesLastNewCount INTEGER DEFAULT 0,
|
||||
TTFolderPath TEXT,
|
||||
BlogId INTEGER,
|
||||
PRIMARY KEY("BlogName")
|
||||
);
|
||||
|
||||
CREATE INDEX ix_Blogs_BlogId ON Blogs (BlogId);
|
||||
```
|
||||
|
||||
`BlogName` is the primary key, so it is the only indexed way in by name. There is no index
|
||||
on any flag or date — filtering or sorting on those scans all 188k rows, which is
|
||||
affordable here and is not on `Notes`.
|
||||
|
||||
**`BlogId` is new as of 2026-08-07 and is the join key to `Notes`.** It exists so that
|
||||
`Notes` can reach `Blogs` in a single integer hop rather than going through `BlogNames`
|
||||
and ending in a text comparison:
|
||||
|
||||
```sql
|
||||
-- what you want
|
||||
FROM Blogs B JOIN Notes N ON N.NoteBlogId = B.BlogId
|
||||
|
||||
-- not this
|
||||
FROM Blogs B JOIN BlogNames BN ON BN.BlogName = B.BlogName
|
||||
JOIN Notes N ON N.NoteBlogId = BN.BlogId
|
||||
```
|
||||
|
||||
**`BlogId` is NULL on 168,202 of 188,620 rows** — every blog that has never appeared in a
|
||||
note. That is the large majority, and it is not an error: the registry is far bigger than
|
||||
the engagement graph. An inner join on `BlogId` therefore silently drops those blogs,
|
||||
which is usually what you want for engagement queries and is wrong for registry listings.
|
||||
|
||||
Flag distribution: `IsActive = 1` on 188,601 of 188,620 rows, `HasBeenOutput = 1` on
|
||||
5,059, `ByLikes = 1` on 2. `IsActive` carries a second meaning as of Rolodex — see
|
||||
[`Blogs.IsActive`](#blogsisactive--now-written-by-two-applications) below.
|
||||
|
||||
The columns after `DateCreated` were added later by `ALTER TABLE`, which is why they carry
|
||||
no quoting in the stored DDL. That is the normal way this schema grows, and `BlogId` is
|
||||
the newest example.
|
||||
|
||||
**`DateAdded` is not written consistently.** 170,677 rows hold ISO `yyyy-MM-dd HH:mm:ss`;
|
||||
17,943 hold US-format `M/d/yy` from a bulk import. As text those two sort into different
|
||||
parts of the table, so anything ordering or range-filtering on this column has to
|
||||
normalise first — see `DateSql` in Rolodex.
|
||||
|
||||
### `Posts`
|
||||
|
||||
```sql
|
||||
CREATE TABLE "Posts" (
|
||||
"BlogName" TEXT,
|
||||
"PostID" INTEGER,
|
||||
"HasNotesGathered" INTEGER DEFAULT 0,
|
||||
"reblogURL" TEXT,
|
||||
"NotFound" INTEGER DEFAULT 0,
|
||||
"PostDate" TEXT,
|
||||
"NotesGatheredDateTime" INTEGER NOT NULL DEFAULT 1729746000,
|
||||
"HasImage" INTEGER NOT NULL DEFAULT 0,
|
||||
"PostURL" TEXT,
|
||||
"Slug" TEXT,
|
||||
"ReblogKey" TEXT,
|
||||
"ReblogName" TEXT,
|
||||
"Summary" TEXT,
|
||||
"Quote" TEXT,
|
||||
"Body" TEXT,
|
||||
"Tags" TEXT,
|
||||
"Link" TEXT,
|
||||
"PhotoURL" TEXT,
|
||||
"PhotoCaption" TEXT,
|
||||
"DownloadedFiles" TEXT,
|
||||
"AudioCaption" TEXT,
|
||||
"Question" TEXT,
|
||||
"Answer" TEXT,
|
||||
"Title" TEXT,
|
||||
"ByLikes" INTEGER NOT NULL DEFAULT 0,
|
||||
"RootBlogName" TEXT,
|
||||
"RootURL" TEXT,
|
||||
"DateModified" TEXT,
|
||||
"DateCreated" TEXT,
|
||||
PostType TEXT,
|
||||
PRIMARY KEY("BlogName","PostID")
|
||||
);
|
||||
```
|
||||
|
||||
**The key is `(BlogName, PostID)`, not `PostID`.** This matters more than it looks: 345
|
||||
post IDs exist under more than one blog, so an ID on its own is both ambiguous *and*
|
||||
unindexed. Any lookup should carry the blog name, and a batch lookup should group by blog
|
||||
so it stays on the leading column of the key.
|
||||
|
||||
Notable:
|
||||
|
||||
- **`PostType` is now mostly populated: 20,679 of 22,468 rows, leaving 1,789 `NULL`.**
|
||||
This reverses what earlier revisions of this document said — the column really was empty
|
||||
on every row, and something has since started writing it. Anything that treated it as
|
||||
permanently unset, or derived the type from post content instead, should be re-examined
|
||||
against the live data. Rolodex still derives it.
|
||||
- `HasImage = 1` on 14,026 rows. It records that the post *had* a picture, not that a
|
||||
usable URL was kept, so it is not a reliable predictor that anything will render.
|
||||
- `PhotoURL` is largely unused; in practice the image markup lives inside `Body`.
|
||||
- `NotFound = 1` on 4,712 rows — posts that have since been deleted upstream.
|
||||
- The content columns (`Body`, `Quote`, `Question`, `Answer`, …) are the heavy ones. List
|
||||
views should not select them.
|
||||
|
||||
### `Notes`
|
||||
|
||||
```sql
|
||||
CREATE TABLE Notes (
|
||||
RootBlogId INTEGER NOT NULL,
|
||||
PostID INTEGER NOT NULL,
|
||||
NoteBlogId INTEGER NOT NULL,
|
||||
TimeStamp INTEGER NOT NULL,
|
||||
TypeId INTEGER NOT NULL,
|
||||
replyText TEXT,
|
||||
DatetimeCrawled TEXT,
|
||||
DateModified TEXT,
|
||||
DateCreated TEXT,
|
||||
IsActive INTEGER NOT NULL DEFAULT 1,
|
||||
PRIMARY KEY (RootBlogId, PostID, TimeStamp, TypeId, NoteBlogId)
|
||||
) WITHOUT ROWID;
|
||||
|
||||
CREATE INDEX ix_Notes_NoteBlogId ON Notes (NoteBlogId);
|
||||
```
|
||||
|
||||
**Integer IDs since 2026-08-07 — this is the breaking change.** `RootBlogName`,
|
||||
`NoteBlogName` and `Type` are gone, replaced by `RootBlogId`, `NoteBlogId` and `TypeId`.
|
||||
Resolve them through [`BlogNames`](#blognames) and [`NoteTypes`](#notetypes), or join
|
||||
straight to `Blogs` on `BlogId`. The old names were text repeated on 1.18 million rows,
|
||||
in the table *and* in every index over it; the swap took the file from 207 MB to 148 MB.
|
||||
|
||||
The **primary key column order is deliberately unchanged**, so the leading-prefix access
|
||||
patterns callers already depend on still hold: `(RootBlogId)` and `(RootBlogId, PostID)`
|
||||
remain cheap prefixes, exactly as `(RootBlogName)` and `(RootBlogName, PostID)` were.
|
||||
|
||||
Two nulls-and-defaults differences from the old DDL, both intentional:
|
||||
|
||||
- `replyText` and `DatetimeCrawled` **no longer carry column defaults**. The old table
|
||||
defaulted them to `'.'` and `'2/12/26 12am'`, which is how 1.1M rows acquired
|
||||
placeholder values nobody wrote. New rows now get `NULL` unless a writer supplies
|
||||
something. The crawler names both columns explicitly, so its behaviour is unchanged.
|
||||
- The five key columns are now `NOT NULL`. They always were in practice.
|
||||
|
||||
**`WITHOUT ROWID`, since earlier the same day.** The rows live in the primary key's
|
||||
b-tree rather than in a rowid table with a separate key index beside it. Two consequences
|
||||
matter before adding an index here:
|
||||
|
||||
- There is no `rowid` on this table. `SELECT rowid FROM Notes` is an error, and no
|
||||
code in any of the three apps relied on it.
|
||||
- A secondary index carries the whole five-column primary key as its row reference
|
||||
instead of a compact rowid, so indexes here are **expensive** — though far less so
|
||||
than before, now that the key is five integers rather than three integers and two
|
||||
strings. `ix_Notes_NoteBlogId` costs 27 MB; its text predecessor cost 58 MB.
|
||||
|
||||
**`Notes_idx_06e01ae3` on `TimeStamp DESC` was dropped at the same time.** It cost
|
||||
14 MB as a rowid index and would have cost 58 MB after the conversion. It was worth
|
||||
neither: the crawler's only `TimeStamp` filter (`>= 1535778000`) excludes 786 rows
|
||||
of 1.18M, Rolodex's default Notes sort carries a three-column tiebreaker that forces
|
||||
a full sort regardless, and the reply-matching `UPDATE` uses `ABS(TimeStamp - ?) <= 5`,
|
||||
which no index on `TimeStamp` can serve. The one path that got slower is Rolodex's
|
||||
Notes page with a date-range filter: 60 ms to 164 ms.
|
||||
|
||||
See `../shrink-db.sql` for the full rationale and the applied result.
|
||||
|
||||
**`DatetimeCrawled` is `NULL` on 1,148,077 rows, and that is the honest value.** Those
|
||||
rows previously stored the literal string `'2/12/26 12am'` — this column's own DDL
|
||||
default, written as a bulk backfill placeholder rather than as a crawl time. They were
|
||||
set to `NULL` on 2026-08-07, which is what consumers already displayed them as: the
|
||||
string parses as a date in neither format this schema writes.
|
||||
|
||||
Note the trap: **the `DEFAULT '2/12/26 12am'` clause is still in the DDL above.** Any
|
||||
`INSERT` that omits this column writes the placeholder straight back. The crawler names
|
||||
it explicitly on every insert, so nothing reintroduces it today, but a new writer that
|
||||
forgets to would — which is why consumers should keep treating an unparseable value here
|
||||
as "unknown" rather than assuming `NULL` is now the only such marker.
|
||||
|
||||
One row per engagement event. `TimeStamp` is **unix seconds** — unlike every date column
|
||||
elsewhere in the schema, which are text.
|
||||
|
||||
At 1.18M rows this is the table that dictates how the whole database has to be queried:
|
||||
|
||||
- **Nothing should run an unbounded `SELECT` or a bare `COUNT(*)` here.** A count scans
|
||||
the lot on every call.
|
||||
- The only fast access paths are the primary key's leading columns (`RootBlogId`, then
|
||||
`PostID`) and `ix_Notes_NoteBlogId` on `NoteBlogId`. "Notes received by a blog" and
|
||||
"notes given by a blog" are both cheap; almost nothing else is.
|
||||
- **Every** ordering here is a full sort of whatever the filters leave, `TimeStamp`
|
||||
included. Filter first, then sort.
|
||||
- `replyText` is `'.'` on 1,167,464 rows — only `reply` notes carry real text. Those
|
||||
dots are inherited from the old column default; new rows get `NULL` instead.
|
||||
|
||||
**Resolve IDs by filtering the lookup, not by scanning `Notes`.** The lookup tables are
|
||||
tiny and uniquely indexed, so pushing a name predicate into them costs nothing and lets
|
||||
the `Notes` index do the work:
|
||||
|
||||
```sql
|
||||
-- good: BlogNames resolves the name, then the index is searched
|
||||
SELECT * FROM Notes
|
||||
WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = ?);
|
||||
|
||||
-- also good, same plan
|
||||
SELECT n.* FROM Notes n
|
||||
JOIN BlogNames b ON b.BlogId = n.NoteBlogId
|
||||
WHERE b.BlogName = ?;
|
||||
```
|
||||
|
||||
### `BlogNames`
|
||||
|
||||
```sql
|
||||
CREATE TABLE BlogNames (
|
||||
BlogId INTEGER PRIMARY KEY,
|
||||
BlogName TEXT NOT NULL UNIQUE
|
||||
);
|
||||
```
|
||||
|
||||
20,430 rows — every name appearing in `Notes` as either participant, and nothing else.
|
||||
This is the **ID authority**: `Notes.RootBlogId` and `Notes.NoteBlogId` both point here,
|
||||
and `Blogs.BlogId` is a copy of the value for the blogs that have one.
|
||||
|
||||
**12 of these names have no `Blogs` row.** The registry has never been a superset of the
|
||||
engagement graph and still is not, so resolving an ID through `Blogs` rather than
|
||||
`BlogNames` will occasionally find nothing. Use `BlogNames` when you need the name itself
|
||||
and `Blogs` when you need registry columns.
|
||||
|
||||
IDs are assigned by SQLite and are **stable**: they are stored in over a million `Notes`
|
||||
rows. Never renumber them. A blog that is renamed upstream should get a new row, not an
|
||||
edit to an existing one, unless every `Notes` reference is migrated with it.
|
||||
|
||||
### `NoteTypes`
|
||||
|
||||
```sql
|
||||
CREATE TABLE NoteTypes (
|
||||
TypeId INTEGER PRIMARY KEY,
|
||||
Type TEXT NOT NULL UNIQUE
|
||||
);
|
||||
```
|
||||
|
||||
| `TypeId` | `Type` | Rows | Share |
|
||||
|--:|---|--:|--:|
|
||||
| 1 | `like` | 945,167 | 79.9% |
|
||||
| 2 | `reblog` | 219,203 | 18.5% |
|
||||
| 3 | `reply` | 15,345 | 1.3% |
|
||||
| 4 | `posted` | 2,617 | 0.2% |
|
||||
| 5 | `post_attribution` | 1 | — |
|
||||
|
||||
The set is fixed in practice, but it is a table rather than a `CHECK` constraint so that
|
||||
adding a type is an `INSERT` and not a schema migration. **The IDs above are stored in
|
||||
`Notes` and must not be reassigned.**
|
||||
|
||||
Five rows means the lookup is effectively free; write `t.Type = 'reblog'` and let SQLite
|
||||
resolve it, or hardcode the ID if you prefer — both are fine, but hardcoding ties your
|
||||
code to this table's contents, so prefer the join in anything long-lived.
|
||||
|
||||
---
|
||||
|
||||
## Porting to the integer schema
|
||||
|
||||
Everything here was checked against the live 148 MB file. There were 14 affected call
|
||||
sites in `DataAccess.cs` and 16 in `RolodexRepository.cs`. TumblThree needs no changes —
|
||||
its single statement touches `Blogs.IsActive` and `BlogName` only.
|
||||
|
||||
**`DataAccess.cs` is ported.** All 14 sites now read the integer schema, `AddNote`
|
||||
registers names and types before inserting, and `verify-db-schema.sql` reports a
|
||||
pre-migration file rather than letting the app fail on it. `RolodexRepository.cs` lives in
|
||||
the [Rolodex](https://git.basso.land/jim/Rolodex) repository and is not covered by that
|
||||
work. One site was dropped rather than translated: the `LEFT JOIN Notes` in `GetPosts`
|
||||
selected nothing and was collapsed by the query's own `GROUP BY`, so it could not affect
|
||||
the result.
|
||||
|
||||
### Column mapping
|
||||
|
||||
| Was | Is now | Resolve via |
|
||||
|---|---|---|
|
||||
| `Notes.RootBlogName` | `Notes.RootBlogId` | `BlogNames.BlogId` → `.BlogName` |
|
||||
| `Notes.NoteBlogName` | `Notes.NoteBlogId` | `BlogNames.BlogId` → `.BlogName` |
|
||||
| `Notes.Type` | `Notes.TypeId` | `NoteTypes.TypeId` → `.Type` |
|
||||
| `ix_NoteBlogName01` | `ix_Notes_NoteBlogId` | — |
|
||||
|
||||
`PostID`, `TimeStamp`, `replyText`, `DatetimeCrawled`, `DateModified`, `DateCreated` and
|
||||
`IsActive` are unchanged.
|
||||
|
||||
### Filtering by a blog name
|
||||
|
||||
```sql
|
||||
-- was
|
||||
WHERE NoteBlogName = @Name
|
||||
|
||||
-- now, either form; both search ix_Notes_NoteBlogId after a unique-index lookup
|
||||
WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @Name)
|
||||
-- or
|
||||
JOIN BlogNames b ON b.BlogId = n.NoteBlogId WHERE b.BlogName = @Name
|
||||
```
|
||||
|
||||
Measured 73 ms against 63 ms for the old text form on the busiest blog — the extra hop is
|
||||
a unique-index probe on a 20k-row table and does not show.
|
||||
|
||||
### Joining `Notes` to `Blogs`
|
||||
|
||||
This is the join to get right; it is the most common shape in both applications.
|
||||
|
||||
```sql
|
||||
-- was
|
||||
FROM Blogs B INNER JOIN Notes N ON N.NoteBlogName = B.BlogName
|
||||
|
||||
-- now: one integer hop, using the new Blogs.BlogId
|
||||
FROM Blogs B INNER JOIN Notes N ON N.NoteBlogId = B.BlogId
|
||||
```
|
||||
|
||||
Do **not** route this through `BlogNames` — that adds a hop and ends in the text
|
||||
comparison the change was meant to remove.
|
||||
|
||||
### Selecting a name back out
|
||||
|
||||
```sql
|
||||
-- was
|
||||
SELECT NoteBlogName AS blogName, COUNT(*) FROM Notes ... GROUP BY NoteBlogName
|
||||
|
||||
-- now
|
||||
SELECT bn.BlogName AS blogName, COUNT(*)
|
||||
FROM Notes n JOIN BlogNames bn ON bn.BlogId = n.NoteBlogId
|
||||
... GROUP BY bn.BlogName
|
||||
```
|
||||
|
||||
Group by `n.NoteBlogId` instead of `bn.BlogName` when you only need the name for display —
|
||||
grouping on the integer is cheaper and the name comes along for free.
|
||||
|
||||
### Filtering by type
|
||||
|
||||
```sql
|
||||
-- was
|
||||
WHERE type IN ('reblog', 'reply', 'posted')
|
||||
|
||||
-- now
|
||||
WHERE TypeId IN (SELECT TypeId FROM NoteTypes WHERE Type IN ('reblog','reply','posted'))
|
||||
-- or, equivalently
|
||||
JOIN NoteTypes t ON t.TypeId = n.TypeId WHERE t.Type IN ('reblog','reply','posted')
|
||||
```
|
||||
|
||||
`WHERE TypeId IN (2,3,4)` also works and is marginally faster, but hardcodes this table's
|
||||
contents into application code. Prefer the lookup outside of hot paths.
|
||||
|
||||
Note the negated form needs care: `type NOT IN ('reblog','reply','posted')` becomes
|
||||
`TypeId NOT IN (SELECT TypeId FROM NoteTypes WHERE Type IN (...))`, which is correct only
|
||||
because `TypeId` is `NOT NULL`.
|
||||
|
||||
### Inserting a note
|
||||
|
||||
The crawler must ensure both names have IDs first. `INSERT OR IGNORE` on `BlogNames` is
|
||||
the whole of it — no read-back, no round trip, safe to run every time:
|
||||
|
||||
```sql
|
||||
INSERT OR IGNORE INTO BlogNames (BlogName) VALUES (@rootBlogName);
|
||||
INSERT OR IGNORE INTO BlogNames (BlogName) VALUES (@noteBlogName);
|
||||
|
||||
INSERT OR IGNORE INTO Notes
|
||||
(RootBlogId, PostID, NoteBlogId, TimeStamp, TypeId,
|
||||
DatetimeCrawled, DateModified, DateCreated)
|
||||
SELECT (SELECT BlogId FROM BlogNames WHERE BlogName = @rootBlogName),
|
||||
@PostID,
|
||||
(SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName),
|
||||
@TimeStamp,
|
||||
(SELECT TypeId FROM NoteTypes WHERE Type = @Type),
|
||||
@DatetimeCrawled, @DateModified, @DateCreated;
|
||||
```
|
||||
|
||||
Verified: a genuinely new note inserts, and re-running the identical statement inserts 0.
|
||||
Run all three statements in one transaction so a crash cannot leave a name registered
|
||||
with no note.
|
||||
|
||||
**The duplicate-key error message has changed.** `DataAccess.cs` compares against the
|
||||
literal string
|
||||
|
||||
```
|
||||
UNIQUE constraint failed: Notes.RootBlogName, Notes.PostID, Notes.TimeStamp, Notes.Type, Notes.NoteBlogName
|
||||
```
|
||||
|
||||
at two call sites to decide whether to swallow an exception. SQLite now emits the *new*
|
||||
column names, so those comparisons no longer match and real errors will surface where
|
||||
they used to be silently ignored — or vice versa.
|
||||
|
||||
Both sites now go through `IsNotesDuplicateKey` in `DataAccess.cs`, which matches on
|
||||
`UNIQUE constraint failed` plus `Notes.` rather than on the column list. A literal
|
||||
comparison is what broke here; the next rename should not break it again.
|
||||
|
||||
### Updating notes
|
||||
|
||||
Predicates translate the same way. The reply-matching update, which cannot use an index
|
||||
on `TimeStamp` either before or after:
|
||||
|
||||
```sql
|
||||
-- now
|
||||
UPDATE Notes SET replyText = @replyText, DateModified = @dateModified
|
||||
WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName)
|
||||
AND ABS(TimeStamp - @TimeStamp) <= 5
|
||||
AND TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')
|
||||
AND (replyText IS NULL OR replyText = '' OR replyText = '.')
|
||||
AND (replyText IS NULL OR replyText <> @replyText);
|
||||
```
|
||||
|
||||
Rolodex's soft-delete updates need no change beyond the `WHERE` clause — they set
|
||||
`IsActive`, which is untouched.
|
||||
|
||||
### Three traps
|
||||
|
||||
**`Blogs.BlogId` is NULL on 168,202 of 188,620 rows.** Any inner join on it silently drops
|
||||
every blog that has never appeared in a note. Correct for engagement queries; wrong for
|
||||
registry listings, which need a `LEFT JOIN` or no join at all.
|
||||
|
||||
**12 names in `BlogNames` have no `Blogs` row.** Resolving an ID to a name through `Blogs`
|
||||
will occasionally find nothing. Use `BlogNames` for names and `Blogs` for registry columns.
|
||||
|
||||
**IDs are stable and must stay so.** `BlogNames.BlogId` and `NoteTypes.TypeId` are stored
|
||||
in over a million `Notes` rows. Never renumber. A blog renamed upstream gets a new row,
|
||||
not an edited one, unless every `Notes` reference migrates with it.
|
||||
|
||||
---
|
||||
|
||||
### Referential integrity
|
||||
|
||||
There are no foreign keys, and the tables do not perfectly agree:
|
||||
|
||||
- 4 `Posts` rows name a blog with no `Blogs` row.
|
||||
- 12 of the 20,430 names in `BlogNames` have no `Blogs` row.
|
||||
|
||||
So a name appearing in `Notes` or `Posts` is not a guarantee that the registry knows about
|
||||
it. Joins from those tables back to `Blogs` should tolerate a miss.
|
||||
|
||||
The integer schema does not fix this and was not meant to. `BlogNames` is deliberately
|
||||
built from `Notes` rather than from `Blogs`, precisely so that the 12 unregistered
|
||||
engagers keep their IDs and their rows. Had it been built from the registry, those notes
|
||||
would have been dropped by the migration's inner joins.
|
||||
|
||||
---
|
||||
|
||||
## The `'.'` placeholder convention
|
||||
|
||||
**The crawler writes a single dot into text columns it has no value for, rather than
|
||||
`NULL`.** This is the single most surprising thing about the schema and it affects every
|
||||
consumer.
|
||||
|
||||
| Column | `'.'` rows |
|
||||
|---|--:|
|
||||
| `Notes.replyText` | 1,167,464 |
|
||||
| `Posts.Title` | 12,562 |
|
||||
| `Posts.Body` | 172 |
|
||||
|
||||
`Notes.replyText` and `Notes.DatetimeCrawled` **no longer carry column defaults** as of
|
||||
the integer migration, so new note rows get `NULL` rather than a placeholder. The dots
|
||||
already in `replyText` were not rewritten — cleaning is still required on read.
|
||||
|
||||
Any query whose output reaches a human should collapse it:
|
||||
|
||||
```sql
|
||||
NULLIF(NULLIF(SomeColumn, '.'), '') AS SomeColumn
|
||||
```
|
||||
|
||||
Empty string turns up too, hence the double `NULLIF`. Not every column is affected —
|
||||
`Blogs.TTFolderPath` and `Blogs.DateModified` currently have zero dot rows — but new
|
||||
columns tend to acquire them, so treat cleaning as the default for any text column
|
||||
rendered to a user.
|
||||
|
||||
---
|
||||
|
||||
## Supporting tables
|
||||
|
||||
Crawler bookkeeping. Rolodex ignores all of these.
|
||||
|
||||
| Table | Rows | What it is |
|
||||
|---|--:|---|
|
||||
| `DailyAPICount` | 133 | `(Date TEXT PK, APICount INTEGER)` — per-day API call tally against the rate limit |
|
||||
| `ApiKeyPoolState` | 2 | `(KeyName TEXT PK, RetryUntil INTEGER)` — per-key backoff; `RetryUntil` is unix seconds |
|
||||
| `ApiKeyPoolMeta` | 1 | `(Id PK CHECK (Id = 1), LastIndex)` — round-robin cursor. Singleton by check constraint |
|
||||
| `CollectRunState` | 1 | `(Id PK CHECK (Id = 1), RunCutoff, RunComplete, RunStarted, RunCompletedAt)` — resume state for an interrupted collection run. Also a singleton |
|
||||
|
||||
---
|
||||
|
||||
## `Blogs.IsActive` — now written by two applications
|
||||
|
||||
`IsActive` has always been the crawler's work-selection flag. `GetBlogs` in
|
||||
`DataAccess.cs` joins on it to decide what to collect:
|
||||
|
||||
```sql
|
||||
-- shape only; the ported GetBlogs joins Blogs directly on BlogId and needs no BlogNames hop
|
||||
SELECT bn.BlogName, count(*)
|
||||
FROM Notes n
|
||||
JOIN Blogs b ON b.BlogId = n.NoteBlogId
|
||||
JOIN BlogNames bn ON bn.BlogId = n.NoteBlogId
|
||||
WHERE b.IsActive = @isActive AND ...
|
||||
```
|
||||
|
||||
Nothing inside the crawler *writes* it — it is an input, set from outside.
|
||||
|
||||
**Rolodex is now one of the things that sets it.** Removing a blog through the Rolodex UI
|
||||
runs exactly this:
|
||||
|
||||
```sql
|
||||
UPDATE Blogs SET IsActive = 0 WHERE BlogName = ?;
|
||||
```
|
||||
|
||||
Rolodex adds no column and changes no schema. It reuses this flag because the two meanings
|
||||
were judged to be one decision: a blog you do not want in the browsing UI is a blog you do
|
||||
not want to keep crawling. Removal therefore stops collection, and the Rolodex
|
||||
confirmation screen says so before anyone commits.
|
||||
|
||||
- `1` (or absent/NULL) — live. Crawled, and visible in Rolodex.
|
||||
- `0` — removed. Not crawled, hidden from the Rolodex registry, dashboard counts and
|
||||
engagement rollups.
|
||||
|
||||
Restoring is the same `UPDATE` with a `1`. Nothing is destroyed either way: the blog's
|
||||
`Posts` and `Notes` rows are never touched, and Rolodex deliberately keeps showing them
|
||||
under its Posts and Notes pages. Removing a blog hides the blog, not what it collected.
|
||||
|
||||
### What other tools need to know
|
||||
|
||||
1. **Setting `IsActive = 0` now also hides the blog from Rolodex**, and setting it back to
|
||||
`1` makes it reappear. If another tool deactivates blogs in bulk, it is also removing
|
||||
them from the browsing UI — which may be exactly right, but it is no longer a
|
||||
crawler-only decision.
|
||||
2. **Re-crawling a removed blog will not bring it back**, since nothing in the crawler
|
||||
writes the flag. An `INSERT OR REPLACE` on the `Blogs` row *would*, by resetting it to
|
||||
the column default of `1`. Prefer an `UPDATE` of the specific columns, or
|
||||
`INSERT … ON CONFLICT DO UPDATE SET` naming only the columns being refreshed.
|
||||
3. **NULL is treated as live.** The column is `INTEGER DEFAULT 1` with no `NOT NULL`, so
|
||||
Rolodex reads it through `COALESCE(IsActive, 1)`. A NULL therefore leaves the blog
|
||||
visible rather than stranding it outside both the registry and the removed list, where
|
||||
no screen could reach it. Write `0` or `1`, not NULL.
|
||||
4. **Backing the feature out is a configuration change, not a migration.** Because there is
|
||||
no Rolodex-owned column, setting `Rolodex__EnableBlogDeletion=false` is the whole of it;
|
||||
there is nothing to drop. Any blogs already at `IsActive = 0` simply go back to being
|
||||
ordinary inactive blogs.
|
||||
|
||||
---
|
||||
|
||||
## `Posts.IsActive` and `Notes.IsActive` — present, and written from outside
|
||||
|
||||
The same flag extends to the two content tables, with the same meaning: `0` is removed,
|
||||
anything else — including `NULL` — is live. **Both columns now exist in the live `TL.db`**
|
||||
and are included in the DDL quoted above. As of 2026-08-07, `Posts.IsActive = 0` on 5,900
|
||||
rows and `Notes.IsActive = 0` on none. Like `Blogs.IsActive`, they are written from
|
||||
outside this crawler.
|
||||
|
||||
On `Notes` the column is `INTEGER NOT NULL DEFAULT 1`, so a `NULL` cannot occur there;
|
||||
`Posts` and `Blogs` are laxer, which is why the predicate below still uses `COALESCE`.
|
||||
|
||||
The crawler therefore treats both as optional, and as nothing it owns:
|
||||
|
||||
- **It never writes them.** No `INSERT` column list names `IsActive`, no `UPDATE` sets it,
|
||||
and `MapPrefixToColumn` — the only place a column name is chosen at runtime — cannot map
|
||||
to it. Re-crawling a removed post or note refreshes its content and leaves the flag at
|
||||
`0`. There is no `INSERT OR REPLACE` on `Posts` or `Notes` for a default to be reset by.
|
||||
- **It filters on them only when they exist.** `HasIsActiveColumn` in `DataAccess.cs` asks
|
||||
`PRAGMA table_info` once per table per database path and caches the answer; the filter
|
||||
is `COALESCE(IsActive, 1) = 1`, and it is dropped entirely when the column is absent.
|
||||
Naming a missing column is a hard SQLite error, so this is what lets one build run
|
||||
against databases on both sides of the change. The cache lives for the process — adding
|
||||
the columns to a live database takes effect on the next run.
|
||||
|
||||
Every read that selects posts or notes carries the filter: `GetPosts`, `GetReplies`,
|
||||
`GetRepliesWithMissingText`, `GetRepliesWithFilledText`, `GetAllPostTextColumns`,
|
||||
`GetAllPostsForBlog`, `GetPost`, `GetPostByIdAnyBlog`, and the engagement queries that
|
||||
count or join `Notes` (`GetBlogs`, `GetBlogsAll`, `GetBlogsForLikes`). The one deliberate
|
||||
omission is the `LEFT JOIN Notes` in `GetPosts`: nothing is selected from it and it can
|
||||
neither add nor remove a row, so filtering it would buy nothing.
|
||||
|
||||
`LegacyPostsDbImporter` is unfiltered too — it reads a foreign legacy database whose
|
||||
`Posts` table is not this schema.
|
||||
|
||||
Two consequences worth stating plainly, both inherited from how `Blogs.IsActive` is
|
||||
handled:
|
||||
|
||||
1. **Removal hides a row; it does not freeze it.** The write paths are keyed on a post the
|
||||
caller already selected, so an ingest or a correction run still overwrites the content
|
||||
of a removed post. Only selection is filtered.
|
||||
2. **`NULL` is live.** Write `0` or `1`, not `NULL`, but a `NULL` leaves the row visible
|
||||
rather than stranding it.
|
||||
|
||||
---
|
||||
|
||||
## Reproducing the numbers
|
||||
|
||||
```sql
|
||||
SELECT 'Blogs', COUNT(*) FROM Blogs
|
||||
UNION ALL SELECT 'Posts', COUNT(*) FROM Posts
|
||||
UNION ALL SELECT 'Notes', COUNT(*) FROM Notes
|
||||
UNION ALL SELECT 'BlogNames', COUNT(*) FROM BlogNames;
|
||||
|
||||
-- note type mix (joins NoteTypes; Notes.Type no longer exists)
|
||||
SELECT t.Type, COUNT(*)
|
||||
FROM Notes n JOIN NoteTypes t ON t.TypeId = n.TypeId
|
||||
GROUP BY t.Type ORDER BY 2 DESC;
|
||||
|
||||
-- how much of the registry participates in the engagement graph
|
||||
SELECT COUNT(*) FILTER (WHERE BlogId IS NOT NULL) AS with_notes,
|
||||
COUNT(*) FILTER (WHERE BlogId IS NULL) AS without_notes
|
||||
FROM Blogs;
|
||||
|
||||
-- the two date shapes in Blogs.DateAdded
|
||||
SELECT CASE WHEN DateAdded LIKE '____-__-__%' THEN 'ISO' ELSE 'US' END, COUNT(*)
|
||||
FROM Blogs GROUP BY 1;
|
||||
|
||||
-- post IDs that are ambiguous without a blog name
|
||||
SELECT COUNT(*) FROM (
|
||||
SELECT PostID FROM Posts GROUP BY PostID HAVING COUNT(DISTINCT BlogName) > 1);
|
||||
|
||||
-- rows that reference a blog the registry does not have
|
||||
SELECT COUNT(*) FROM Posts p
|
||||
WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = p.BlogName);
|
||||
|
||||
SELECT COUNT(*) FROM BlogNames bn
|
||||
WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = bn.BlogName);
|
||||
|
||||
-- space by object, to see where the file actually goes
|
||||
SELECT name, SUM(pgsize)/1024/1024 AS mb
|
||||
FROM dbstat GROUP BY name ORDER BY SUM(pgsize) DESC;
|
||||
```
|
||||
|
||||
Open the file read-only so an inspection can never disturb a running crawl:
|
||||
|
||||
```bash
|
||||
sqlite3 "file:TL.db?mode=ro" ".schema"
|
||||
```
|
||||
@@ -4,34 +4,64 @@ namespace URLNotesGrabberCORE
|
||||
{
|
||||
// Port of ThreeTxtFileHelper/UpdateBlogPaths.cs. Reads .tumblr / .tmblrpriv metadata
|
||||
// files from a root\Index folder and populates Blogs.TTFolderPath in TL.db.
|
||||
//
|
||||
// Scan() is the reusable engine: --updatepaths wraps it as a standalone command and
|
||||
// --output calls it as a refresh step, because a TL.db synced between machines cannot
|
||||
// hold one absolute path that is correct on both.
|
||||
public static class UpdateBlogPathsRunner
|
||||
{
|
||||
public static int Run(string rootPath)
|
||||
public enum ScanOutcome
|
||||
{
|
||||
Completed,
|
||||
NoRootConfigured,
|
||||
IndexFolderMissing
|
||||
}
|
||||
|
||||
public sealed class ScanResult
|
||||
{
|
||||
public ScanOutcome Outcome { get; init; }
|
||||
public string RootPath { get; init; } = string.Empty;
|
||||
public string IndexPath { get; init; } = string.Empty;
|
||||
public int MetadataFiles { get; init; }
|
||||
public int Written { get; init; }
|
||||
public int Unchanged { get; init; }
|
||||
public int NoLocation { get; init; }
|
||||
public int NoMatchingRow { get; init; }
|
||||
public int Errors { get; init; }
|
||||
}
|
||||
|
||||
// verbose: log a line per metadata file. --updatepaths wants that detail; --output
|
||||
// only wants the counts, since a few hundred lines before the export would bury it.
|
||||
public static ScanResult Scan(string? rootPath, bool verbose)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(rootPath))
|
||||
{
|
||||
Console.WriteLine("UpdateBlogPaths: rootPath is required.");
|
||||
return 1;
|
||||
}
|
||||
return new ScanResult { Outcome = ScanOutcome.NoRootConfigured };
|
||||
|
||||
DataAccess.EnsureTTFileHelperColumnsExist();
|
||||
|
||||
string indexPath = Path.Combine(rootPath, "Index");
|
||||
if (!Directory.Exists(indexPath))
|
||||
{
|
||||
Console.WriteLine($"Index folder not found at: {indexPath}");
|
||||
return 1;
|
||||
return new ScanResult
|
||||
{
|
||||
Outcome = ScanOutcome.IndexFolderMissing,
|
||||
RootPath = rootPath,
|
||||
IndexPath = indexPath
|
||||
};
|
||||
}
|
||||
|
||||
Console.WriteLine($"Scanning Index folder: {indexPath}");
|
||||
|
||||
var blogFiles = Directory.GetFiles(indexPath, "*.tumblr")
|
||||
.Concat(Directory.GetFiles(indexPath, "*.tmblrpriv"))
|
||||
.ToList();
|
||||
|
||||
if (verbose)
|
||||
Console.WriteLine($"Found {blogFiles.Count} blog metadata files");
|
||||
|
||||
int updatedCount = 0;
|
||||
int unchangedCount = 0;
|
||||
int noLocationCount = 0;
|
||||
int noRowCount = 0;
|
||||
int errorCount = 0;
|
||||
|
||||
foreach (var blogFile in blogFiles)
|
||||
{
|
||||
@@ -44,27 +74,91 @@ namespace URLNotesGrabberCORE
|
||||
|
||||
if (root.TryGetProperty("FileDownloadLocation", out JsonElement locationElement))
|
||||
{
|
||||
string? fileDownloadLocation = locationElement.GetString();
|
||||
string? fileDownloadLocation = locationElement.GetString()?.Trim();
|
||||
if (!string.IsNullOrWhiteSpace(fileDownloadLocation))
|
||||
{
|
||||
DataAccess.SetBlogTTFolderPath(blogName, fileDownloadLocation);
|
||||
// Report the database's answer, not the fact that the file parsed.
|
||||
if (DataAccess.SetBlogTTFolderPath(blogName, fileDownloadLocation))
|
||||
{
|
||||
updatedCount++;
|
||||
if (verbose)
|
||||
Console.WriteLine($"Updated {blogName}: {fileDownloadLocation}");
|
||||
}
|
||||
else if (DataAccess.BlogExists(blogName))
|
||||
{
|
||||
unchangedCount++;
|
||||
}
|
||||
else
|
||||
{
|
||||
noRowCount++;
|
||||
Console.WriteLine($"No Blogs row named '{blogName}' -- path not stored (name may differ in case)");
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
noLocationCount++;
|
||||
if (verbose)
|
||||
Console.WriteLine($"Empty FileDownloadLocation in {blogFile}");
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
noLocationCount++;
|
||||
if (verbose)
|
||||
Console.WriteLine($"No FileDownloadLocation found in {blogFile}");
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
errorCount++;
|
||||
Console.WriteLine($"Error processing {blogFile}: {ex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
Console.WriteLine($"\nUpdated {updatedCount} blogs with TTFolderPath");
|
||||
return 0;
|
||||
return new ScanResult
|
||||
{
|
||||
Outcome = ScanOutcome.Completed,
|
||||
RootPath = rootPath,
|
||||
IndexPath = indexPath,
|
||||
MetadataFiles = blogFiles.Count,
|
||||
Written = updatedCount,
|
||||
Unchanged = unchangedCount,
|
||||
NoLocation = noLocationCount,
|
||||
NoMatchingRow = noRowCount,
|
||||
Errors = errorCount
|
||||
};
|
||||
}
|
||||
|
||||
public static int Run(string rootPath)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(rootPath))
|
||||
{
|
||||
Console.WriteLine("UpdateBlogPaths: rootPath is required.");
|
||||
return 1;
|
||||
}
|
||||
|
||||
string indexPath = Path.Combine(rootPath, "Index");
|
||||
Console.WriteLine($"Scanning Index folder: {indexPath}");
|
||||
|
||||
var result = Scan(rootPath, verbose: true);
|
||||
|
||||
if (result.Outcome == ScanOutcome.IndexFolderMissing)
|
||||
{
|
||||
Console.WriteLine($"Index folder not found at: {result.IndexPath}");
|
||||
return 1;
|
||||
}
|
||||
|
||||
Console.WriteLine($"\n========== UpdateBlogPaths summary ==========");
|
||||
Console.WriteLine($"Metadata files: {result.MetadataFiles}");
|
||||
Console.WriteLine($"TTFolderPath written: {result.Written}");
|
||||
Console.WriteLine($"Already correct: {result.Unchanged}");
|
||||
Console.WriteLine($"No FileDownloadLocation: {result.NoLocation}");
|
||||
Console.WriteLine($"No matching blog row: {result.NoMatchingRow}");
|
||||
Console.WriteLine($"Errors: {result.Errors}");
|
||||
|
||||
Console.WriteLine($"\nBlogs now holding a TTFolderPath: {DataAccess.CountBlogsWithTTFolderPath()}");
|
||||
|
||||
return result.Errors == 0 ? 0 : 2;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
-- normalize-notes.sql
|
||||
-- Replaces the repeated blog-name and type TEXT in Notes with integer IDs.
|
||||
-- Reduces TL.db from ~207 MB to ~148 MB (-29%).
|
||||
--
|
||||
-- THIS IS A BREAKING SCHEMA CHANGE. There is no compatibility layer. Every
|
||||
-- query in URLNotesGrabberCORE and Rolodex that names Notes.RootBlogName,
|
||||
-- Notes.NoteBlogName or Notes.Type stops working the moment this runs, and
|
||||
-- stays broken until those queries are rewritten. This was a deliberate choice
|
||||
-- over a view-plus-triggers shim, which was measured to work but cost 194 ms ->
|
||||
-- 321 ms on Rolodex's unfiltered Notes page.
|
||||
--
|
||||
-- TumblThree is unaffected. It touches only Blogs, and the column added to
|
||||
-- Blogs here is additive.
|
||||
--
|
||||
-- HOW TO RUN (DB Browser for SQLite):
|
||||
-- 1. Stop all three apps. Pause NextCloud sync.
|
||||
-- 2. Back up TL.db.
|
||||
-- 3. Execute SQL, paste this file, run. Then Write Changes.
|
||||
-- 4. Tools > Compact Database (VACUUM). Nothing shrinks until this finishes.
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
-- The shape this produces
|
||||
--------------------------------------------------------------------------
|
||||
-- BlogNames(BlogId, BlogName) the ID authority: every name appearing in
|
||||
-- Notes as either participant. 20,430 rows.
|
||||
-- 12 of these have no Blogs row -- the
|
||||
-- registry has never been a superset of the
|
||||
-- engagement graph, and still is not.
|
||||
--
|
||||
-- NoteTypes(TypeId, Type) 5 rows. Fixed set, but written as a table
|
||||
-- rather than a CHECK so a new type is an
|
||||
-- INSERT and not a schema migration.
|
||||
--
|
||||
-- Notes(...Id columns...) integer FKs in place of text. WITHOUT ROWID,
|
||||
-- same 5-column key in the same column order.
|
||||
--
|
||||
-- Blogs.BlogId NEW additive column. Lets Notes join Blogs in
|
||||
-- one integer hop instead of going through
|
||||
-- BlogNames and comparing text at the end.
|
||||
-- NULL on the 168,202 blogs with no notes.
|
||||
|
||||
PRAGMA foreign_keys = off;
|
||||
|
||||
BEGIN;
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
-- STEP 1: the ID authority
|
||||
--------------------------------------------------------------------------
|
||||
CREATE TABLE BlogNames (
|
||||
BlogId INTEGER PRIMARY KEY,
|
||||
BlogName TEXT NOT NULL UNIQUE
|
||||
);
|
||||
|
||||
INSERT INTO BlogNames (BlogName)
|
||||
SELECT RootBlogName FROM Notes
|
||||
UNION
|
||||
SELECT NoteBlogName FROM Notes;
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
-- STEP 2: the type lookup
|
||||
--------------------------------------------------------------------------
|
||||
CREATE TABLE NoteTypes (
|
||||
TypeId INTEGER PRIMARY KEY,
|
||||
Type TEXT NOT NULL UNIQUE
|
||||
);
|
||||
|
||||
-- IDs are assigned explicitly and must stay stable: they are stored in Notes.
|
||||
INSERT INTO NoteTypes (TypeId, Type) VALUES
|
||||
(1, 'like'),
|
||||
(2, 'reblog'),
|
||||
(3, 'reply'),
|
||||
(4, 'posted'),
|
||||
(5, 'post_attribution');
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
-- STEP 3: rebuild Notes with integer keys
|
||||
--------------------------------------------------------------------------
|
||||
-- Column order of the primary key is unchanged from the text version, so the
|
||||
-- leading-column access patterns callers already rely on still hold:
|
||||
-- (RootBlogId) and (RootBlogId, PostID) remain cheap prefixes.
|
||||
CREATE TABLE NotesN (
|
||||
RootBlogId INTEGER NOT NULL,
|
||||
PostID INTEGER NOT NULL,
|
||||
NoteBlogId INTEGER NOT NULL,
|
||||
TimeStamp INTEGER NOT NULL,
|
||||
TypeId INTEGER NOT NULL,
|
||||
replyText TEXT,
|
||||
DatetimeCrawled TEXT,
|
||||
DateModified TEXT,
|
||||
DateCreated TEXT,
|
||||
IsActive INTEGER NOT NULL DEFAULT 1,
|
||||
PRIMARY KEY (RootBlogId, PostID, TimeStamp, TypeId, NoteBlogId)
|
||||
) WITHOUT ROWID;
|
||||
|
||||
-- Inner joins are safe here: BlogNames was just built from these very columns,
|
||||
-- and NoteTypes covers all 5 values present. A row that failed to match would
|
||||
-- be silently dropped, which is what the row-count check at the bottom is for.
|
||||
INSERT INTO NotesN
|
||||
SELECT r.BlogId, n.PostID, b.BlogId, n.TimeStamp, t.TypeId,
|
||||
n.replyText, n.DatetimeCrawled, n.DateModified, n.DateCreated, n.IsActive
|
||||
FROM Notes n
|
||||
JOIN BlogNames r ON r.BlogName = n.RootBlogName
|
||||
JOIN BlogNames b ON b.BlogName = n.NoteBlogName
|
||||
JOIN NoteTypes t ON t.Type = n.Type;
|
||||
|
||||
DROP TABLE Notes;
|
||||
ALTER TABLE NotesN RENAME TO Notes;
|
||||
|
||||
-- Replaces ix_NoteBlogName01. Renamed because it indexes a different column now.
|
||||
CREATE INDEX ix_Notes_NoteBlogId ON Notes (NoteBlogId);
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
-- STEP 4: give Blogs the matching id
|
||||
--------------------------------------------------------------------------
|
||||
-- Additive: no existing column changes, so TumblThree's
|
||||
-- "UPDATE Blogs SET IsActive = 0 ... WHERE BlogName = ?" is untouched.
|
||||
ALTER TABLE Blogs ADD COLUMN BlogId INTEGER;
|
||||
|
||||
UPDATE Blogs
|
||||
SET BlogId = (SELECT bn.BlogId FROM BlogNames bn WHERE bn.BlogName = Blogs.BlogName);
|
||||
|
||||
CREATE INDEX ix_Blogs_BlogId ON Blogs (BlogId);
|
||||
|
||||
COMMIT;
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
-- STEP 5: Write Changes, then Tools > Compact Database
|
||||
--------------------------------------------------------------------------
|
||||
-- From the CLI instead: sqlite3 TL.db "VACUUM;"
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
-- VERIFY
|
||||
--------------------------------------------------------------------------
|
||||
-- PRAGMA integrity_check; -- expect: ok
|
||||
-- SELECT COUNT(*) FROM Notes; -- expect: 1182333, unchanged
|
||||
-- SELECT COUNT(*) FROM BlogNames; -- expect: 20430
|
||||
-- SELECT COUNT(*) FROM Blogs WHERE BlogId IS NOT NULL; -- expect: 20418
|
||||
--
|
||||
-- Losslessness was proven before this ran, by reconstructing the old text shape
|
||||
-- from the new schema and diffing it against the original both ways:
|
||||
-- SELECT COUNT(*) FROM (SELECT * FROM old.Notes EXCEPT SELECT * FROM Rebuilt);
|
||||
-- SELECT COUNT(*) FROM (SELECT * FROM Rebuilt EXCEPT SELECT * FROM old.Notes);
|
||||
-- Both returned 0 across all 1,182,333 rows and all 10 columns.
|
||||
+159
@@ -0,0 +1,159 @@
|
||||
-- shrink-db.sql
|
||||
-- Reduces TL.db from ~267 MB to ~207 MB (-22%) with no application changes,
|
||||
-- and no visible change in any of the three apps that touch this file.
|
||||
--
|
||||
-- The three consumers, and what each one uses:
|
||||
-- URLNotesGrabberCORE System.Data.SQLite 1.0.119 writes Notes, Posts, Blogs
|
||||
-- Rolodex (web) Microsoft.Data.Sqlite 10.0 reads all three; soft-deletes via IsActive
|
||||
-- TumblThree System.Data.SQLite.Core 1.0.119
|
||||
-- one statement only, ManagerController.cs:972 --
|
||||
-- "UPDATE Blogs SET IsActive = 0, DateModified = @DateModified
|
||||
-- WHERE BlogName = @BlogName"
|
||||
-- Nothing below touches the Blogs table, so TumblThree is
|
||||
-- unaffected. (Its GlobalDatabaseService talks to TumblThree's
|
||||
-- own separate FileEntries/BlogFiles database, not this file.)
|
||||
--
|
||||
-- WITHOUT ROWID needs SQLite >= 3.8.2 (Dec 2013). All three providers above are
|
||||
-- 2024-25 builds, an order of magnitude newer, so STEP 2 is readable by all of them.
|
||||
--
|
||||
-- Every figure below was measured on a copy of the live 267 MB file, and the
|
||||
-- result was checked against all three apps' access patterns:
|
||||
-- PRAGMA integrity_check ....... ok
|
||||
-- row counts ................... Notes 1182333, Posts 22468, Blogs 188620 (unchanged)
|
||||
-- Rolodex soft-delete UPDATE ... works
|
||||
-- Rolodex NoteBlogName filter .. still uses ix_NoteBlogName01
|
||||
-- crawler INSERT OR IGNORE ..... still dedupes (0 dupes admitted)
|
||||
-- TumblThree's UPDATE Blogs ..... untouched -- Blogs is not modified by this script
|
||||
--
|
||||
-- HOW TO RUN (DB Browser for SQLite):
|
||||
-- 1. Stop ALL THREE apps: the crawler, the Rolodex web app, and TumblThree.
|
||||
-- Rolodex holds the file open and checkpoints the WAL, so it must be down,
|
||||
-- not just idle. TumblThree only opens the file for an instant when you
|
||||
-- delete a blog, but it can also launch the crawler on its own
|
||||
-- (UrlNotesGrabberService) -- so close it rather than merely avoiding it.
|
||||
-- 2. Back up TL.db (copy the 267 MB file somewhere safe).
|
||||
-- 3. Open TL.db, go to Execute SQL, paste STEP 1-3, run.
|
||||
-- 4. Click "Write Changes".
|
||||
-- 5. Run Tools > Compact Database. This is VACUUM; it will not run from the
|
||||
-- Execute SQL tab because DB Browser keeps a transaction open there.
|
||||
-- NOTHING SHRINKS ON DISK UNTIL THIS FINISHES.
|
||||
--
|
||||
-- Expected: steps 1-3 a couple of minutes, Compact a couple more.
|
||||
-- Free disk needed during Compact: ~270 MB for the temp copy.
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
-- STEP 1: drop the TimeStamp index (required by STEP 2, not optional)
|
||||
--------------------------------------------------------------------------
|
||||
-- Measured cost/benefit:
|
||||
--
|
||||
-- * Rolodex's DEFAULT Notes view does not use it. Its sort carries the
|
||||
-- tiebreaker "RootBlogName, PostID, NoteBlogName", which forces a full sort
|
||||
-- regardless -- the query plan is byte-identical with and without the index.
|
||||
-- Sorting.cs:138 already assumes as much, and is right in practice.
|
||||
-- * The crawler's collect query (DataAccess.cs:1183) filters
|
||||
-- TimeStamp >= 1535778000, which excludes 786 of 1,182,333 rows (0.07%).
|
||||
-- A full index scan wearing a disguise. Same measured time without it.
|
||||
-- * The reply-matching UPDATE uses ABS(TimeStamp - ?) <= 5, which can never
|
||||
-- use an index on TimeStamp.
|
||||
-- * It DOES help exactly one path: Rolodex's Notes page with a date-range
|
||||
-- filter applied. 60 ms -> 164 ms. That is the whole of what is lost.
|
||||
--
|
||||
-- And it must go, because after STEP 2 it stops being cheap. A secondary index
|
||||
-- on a WITHOUT ROWID table carries the full 5-column primary key instead of a
|
||||
-- compact rowid, so this index grows 14 MB -> 58 MB. Keeping it lands the file
|
||||
-- at 265 MB instead of 207 MB -- i.e. it cancels the entire exercise to save
|
||||
-- 100 ms on one filtered view.
|
||||
|
||||
DROP INDEX IF EXISTS Notes_idx_06e01ae3;
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
-- STEP 2: rebuild Notes as WITHOUT ROWID (-32 MB)
|
||||
--------------------------------------------------------------------------
|
||||
-- Notes has a 5-column composite primary key. In a rowid table SQLite stores
|
||||
-- that key twice: once in the table, once in sqlite_autoindex_Notes_1 (62 MB).
|
||||
-- WITHOUT ROWID stores the rows *in* the key's b-tree, so the copy disappears.
|
||||
--
|
||||
-- ix_NoteBlogName01 grows 25 -> 58 MB for the reason described above. Net -32 MB.
|
||||
-- It is kept because Rolodex filters on NoteBlogName and the crawler joins on it.
|
||||
--
|
||||
-- Safe: neither codebase references rowid on Notes (grep across both trees,
|
||||
-- zero matches). IsActive keeps its exact current declaration, which is what
|
||||
-- Rolodex's ActiveFlag predicate reads.
|
||||
|
||||
PRAGMA foreign_keys = off;
|
||||
|
||||
CREATE TABLE Notes_new (
|
||||
"RootBlogName" TEXT,
|
||||
"PostID" INTEGER,
|
||||
"NoteBlogName" TEXT,
|
||||
"TimeStamp" INTEGER,
|
||||
"Type" TEXT,
|
||||
"replyText" TEXT DEFAULT '.',
|
||||
"DatetimeCrawled" TEXT DEFAULT '2/12/26 12am',
|
||||
"DateModified" TEXT,
|
||||
"DateCreated" TEXT,
|
||||
IsActive INTEGER NOT NULL DEFAULT 1,
|
||||
PRIMARY KEY("RootBlogName","PostID","TimeStamp","Type","NoteBlogName")
|
||||
) WITHOUT ROWID;
|
||||
|
||||
INSERT INTO Notes_new
|
||||
SELECT RootBlogName, PostID, NoteBlogName, TimeStamp, Type,
|
||||
replyText, DatetimeCrawled, DateModified, DateCreated, IsActive
|
||||
FROM Notes;
|
||||
|
||||
DROP TABLE Notes;
|
||||
ALTER TABLE Notes_new RENAME TO Notes;
|
||||
|
||||
CREATE INDEX "ix_NoteBlogName01" ON "Notes" ("NoteBlogName");
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
-- STEP 3: clear the DatetimeCrawled placeholder (-13 MB)
|
||||
--------------------------------------------------------------------------
|
||||
-- 1,148,077 of 1,182,333 rows hold the literal DDL default '2/12/26 12am' --
|
||||
-- a backfill placeholder, not a crawl time. SQLite stores all 12 bytes of it
|
||||
-- on every one of those rows.
|
||||
--
|
||||
-- This is UI-NEUTRAL in Rolodex, which is why it is safe despite Rolodex
|
||||
-- displaying the column. Rolodex reads and sorts it through DateSql.Sortable
|
||||
-- (DateRange.cs:67), whose CASE matches '____-__-__%' or the 8-character
|
||||
-- '__/__/__'. The 12-character '2/12/26 12am' matches neither, so Sortable
|
||||
-- already returns NULL for these rows and the page already renders an em dash
|
||||
-- and sorts them to the bottom. RolodexRepository.cs:957-963 documents exactly
|
||||
-- this. Writing a real NULL changes the bytes on disk, not the screen.
|
||||
--
|
||||
-- The crawler never reads the column back -- it only writes it on INSERT
|
||||
-- (DataAccess.cs:713, 721).
|
||||
|
||||
UPDATE Notes SET DatetimeCrawled = NULL WHERE DatetimeCrawled = '2/12/26 12am';
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
-- DELIBERATELY NOT DONE: nulling Notes.DateCreated
|
||||
--------------------------------------------------------------------------
|
||||
-- An earlier draft of this script also cleared DateCreated = '2026-04-13'
|
||||
-- (a further -12 MB). Do not. Unlike DatetimeCrawled, that value DOES match
|
||||
-- Sortable's '____-__-__%' branch, so Rolodex renders it as a real date in the
|
||||
-- "Created" column on the Notes page and Post detail, and sorts by it. Nulling
|
||||
-- it would turn visible dates into em dashes and move rows in the sort order.
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
-- STEP 4: Write Changes, then Tools > Compact Database
|
||||
--------------------------------------------------------------------------
|
||||
-- Nothing above reclaims disk until VACUUM runs. From the sqlite3 CLI instead:
|
||||
-- sqlite3 TL.db "VACUUM;"
|
||||
|
||||
--------------------------------------------------------------------------
|
||||
-- VERIFY (run after compacting; file should be ~207 MB)
|
||||
--------------------------------------------------------------------------
|
||||
-- PRAGMA integrity_check;
|
||||
--
|
||||
-- SELECT 'Notes' t, COUNT(*) n FROM Notes
|
||||
-- UNION ALL SELECT 'Posts', COUNT(*) FROM Posts
|
||||
-- UNION ALL SELECT 'Blogs', COUNT(*) FROM Blogs;
|
||||
-- -- expect 1182333 / 22468 / 188620, unchanged
|
||||
--
|
||||
-- SELECT name, SUM(pgsize)/1024/1024 AS mb
|
||||
-- FROM dbstat GROUP BY name ORDER BY SUM(pgsize) DESC;
|
||||
-- -- expect Notes 78, ix_NoteBlogName01 58, Posts 50, Blogs 14
|
||||
--
|
||||
-- APPLIED 2026-08-07. Actual result: 267.32 MB -> 207.17 MB, integrity_check ok,
|
||||
-- row counts unchanged, journal_mode still wal. VACUUM took 6 seconds.
|
||||
+59
-11
@@ -79,18 +79,35 @@ WITH expected(tbl, col, alter_stmt) AS (
|
||||
('Blogs','LikesLastRefreshed', 'ALTER TABLE Blogs ADD COLUMN LikesLastRefreshed INTEGER DEFAULT 0;'),
|
||||
('Blogs','LikesLastNewCount', 'ALTER TABLE Blogs ADD COLUMN LikesLastNewCount INTEGER DEFAULT 0;'),
|
||||
('Blogs','TTFolderPath', 'ALTER TABLE Blogs ADD COLUMN TTFolderPath TEXT;'),
|
||||
-- Blogs.BlogId (2026-08-07) is the single-hop join key into Notes. Deliberately NOT
|
||||
-- auto-fixable: an added-but-empty BlogId makes every engagement join return zero
|
||||
-- rows silently, which is worse than the hard error a missing column gives.
|
||||
('Blogs','BlogId', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
|
||||
|
||||
-- Notes (base columns: manual review if missing)
|
||||
('Notes','RootBlogName', 'MANUAL REVIEW - base/PK column missing'),
|
||||
-- Integer IDs since 2026-08-07. RootBlogName/NoteBlogName/Type are GONE, not renamed
|
||||
-- in place -- a backup that still has them needs normalize-notes.sql, not an ALTER.
|
||||
-- Query 1d below reports exactly that case.
|
||||
('Notes','RootBlogId', 'MANUAL REVIEW - see query 1d: pre-2026-08-07 name schema, or damaged'),
|
||||
('Notes','PostID', 'MANUAL REVIEW - base/PK column missing'),
|
||||
('Notes','NoteBlogName', 'MANUAL REVIEW - base/PK column missing'),
|
||||
('Notes','NoteBlogId', 'MANUAL REVIEW - see query 1d: pre-2026-08-07 name schema, or damaged'),
|
||||
('Notes','TimeStamp', 'MANUAL REVIEW - base/PK column missing'),
|
||||
('Notes','Type', 'MANUAL REVIEW - base/PK column missing'),
|
||||
('Notes','TypeId', 'MANUAL REVIEW - see query 1d: pre-2026-08-07 name schema, or damaged'),
|
||||
('Notes','DatetimeCrawled', 'MANUAL REVIEW - base column missing'),
|
||||
('Notes','DateModified', 'MANUAL REVIEW - base column missing'),
|
||||
('Notes','DateCreated', 'MANUAL REVIEW - base column missing'),
|
||||
-- Notes (additive migration column, auto-fixable)
|
||||
('Notes','replyText', 'ALTER TABLE Notes ADD COLUMN replyText TEXT DEFAULT ''.'';'),
|
||||
-- No DEFAULT: the migrated schema dropped it, so new rows get NULL rather than a
|
||||
-- placeholder. EnsureReplyTextColumnExists in DataAccess.cs adds it the same way.
|
||||
('Notes','replyText', 'ALTER TABLE Notes ADD COLUMN replyText TEXT;'),
|
||||
|
||||
-- BlogNames / NoteTypes (the lookup tables Notes resolves its IDs through, 2026-08-07).
|
||||
-- Not auto-fixable: an empty BlogNames does not mean "add the table", it means the
|
||||
-- Notes rows have nothing to resolve against. Rebuild with normalize-notes.sql.
|
||||
('BlogNames','BlogId', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
|
||||
('BlogNames','BlogName', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
|
||||
('NoteTypes','TypeId', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
|
||||
('NoteTypes','Type', 'MANUAL REVIEW - see query 1d: run normalize-notes.sql'),
|
||||
|
||||
-- DailyAPICount (base columns)
|
||||
('DailyAPICount','Date', 'MANUAL REVIEW - base/PK column missing'),
|
||||
@@ -106,6 +123,8 @@ actual(tbl, col) AS (
|
||||
SELECT 'Posts', name FROM pragma_table_info('Posts')
|
||||
UNION ALL SELECT 'Blogs', name FROM pragma_table_info('Blogs')
|
||||
UNION ALL SELECT 'Notes', name FROM pragma_table_info('Notes')
|
||||
UNION ALL SELECT 'BlogNames', name FROM pragma_table_info('BlogNames')
|
||||
UNION ALL SELECT 'NoteTypes', name FROM pragma_table_info('NoteTypes')
|
||||
UNION ALL SELECT 'DailyAPICount', name FROM pragma_table_info('DailyAPICount')
|
||||
UNION ALL SELECT 'ApiKeyPoolState', name FROM pragma_table_info('ApiKeyPoolState')
|
||||
UNION ALL SELECT 'ApiKeyPoolMeta', name FROM pragma_table_info('ApiKeyPoolMeta')
|
||||
@@ -127,7 +146,7 @@ ORDER BY (e.alter_stmt LIKE 'ALTER%') DESC, e.tbl, e.col;
|
||||
-- 1b. MISSING TABLES: expected tables that don't exist at all in this DB.
|
||||
-- Zero rows = good.
|
||||
WITH expected_tables(tbl) AS (
|
||||
VALUES ('Posts'),('Blogs'),('Notes'),('DailyAPICount'),
|
||||
VALUES ('Posts'),('Blogs'),('Notes'),('BlogNames'),('NoteTypes'),('DailyAPICount'),
|
||||
('ApiKeyPoolState'),('ApiKeyPoolMeta')
|
||||
)
|
||||
SELECT et.tbl AS missing_table
|
||||
@@ -142,6 +161,9 @@ ORDER BY et.tbl;
|
||||
-- 1c. EXTRA / UNEXPECTED COLUMNS: present in the DB but not in the expected
|
||||
-- list above. Informational only -- e.g. a NEWER backup, or a column this
|
||||
-- script's expected-list hasn't been updated for. Not an error by itself.
|
||||
-- Posts.IsActive and Notes.IsActive are listed here and NOT in 1a on
|
||||
-- purpose: they are written by other tools, the app only reads them when
|
||||
-- present, and it must not be told to add them. See TL.db.md.
|
||||
WITH expected(tbl, col) AS (
|
||||
VALUES
|
||||
('Posts','BlogName'),('Posts','PostID'),('Posts','HasNotesGathered'),('Posts','reblogURL'),
|
||||
@@ -150,14 +172,16 @@ WITH expected(tbl, col) AS (
|
||||
('Posts','Quote'),('Posts','Body'),('Posts','Tags'),('Posts','Link'),('Posts','PhotoURL'),
|
||||
('Posts','PhotoCaption'),('Posts','DownloadedFiles'),('Posts','AudioCaption'),('Posts','Question'),
|
||||
('Posts','Answer'),('Posts','Title'),('Posts','ByLikes'),('Posts','RootBlogName'),('Posts','RootURL'),
|
||||
('Posts','DateModified'),('Posts','DateCreated'),('Posts','PostType'),
|
||||
('Posts','DateModified'),('Posts','DateCreated'),('Posts','PostType'),('Posts','IsActive'),
|
||||
('Blogs','BlogName'),('Blogs','HasBeenOutput'),('Blogs','IsActive'),('Blogs','DateAdded'),
|
||||
('Blogs','ByLikes'),('Blogs','DateModified'),('Blogs','DateCreated'),('Blogs','LikesPulled'),
|
||||
('Blogs','LikesCursor'),('Blogs','LikesNewestTimestamp'),('Blogs','LikesLastRefreshed'),
|
||||
('Blogs','LikesLastNewCount'),('Blogs','TTFolderPath'),
|
||||
('Notes','RootBlogName'),('Notes','PostID'),('Notes','NoteBlogName'),('Notes','TimeStamp'),
|
||||
('Notes','Type'),('Notes','DatetimeCrawled'),('Notes','DateModified'),('Notes','DateCreated'),
|
||||
('Notes','replyText'),
|
||||
('Blogs','LikesLastNewCount'),('Blogs','TTFolderPath'),('Blogs','BlogId'),
|
||||
('Notes','RootBlogId'),('Notes','PostID'),('Notes','NoteBlogId'),('Notes','TimeStamp'),
|
||||
('Notes','TypeId'),('Notes','DatetimeCrawled'),('Notes','DateModified'),('Notes','DateCreated'),
|
||||
('Notes','replyText'),('Notes','IsActive'),
|
||||
('BlogNames','BlogId'),('BlogNames','BlogName'),
|
||||
('NoteTypes','TypeId'),('NoteTypes','Type'),
|
||||
('DailyAPICount','Date'),('DailyAPICount','APICount'),
|
||||
('ApiKeyPoolState','KeyName'),('ApiKeyPoolState','RetryUntil'),
|
||||
('ApiKeyPoolMeta','Id'),('ApiKeyPoolMeta','LastIndex')
|
||||
@@ -166,6 +190,8 @@ actual(tbl, col) AS (
|
||||
SELECT 'Posts', name FROM pragma_table_info('Posts')
|
||||
UNION ALL SELECT 'Blogs', name FROM pragma_table_info('Blogs')
|
||||
UNION ALL SELECT 'Notes', name FROM pragma_table_info('Notes')
|
||||
UNION ALL SELECT 'BlogNames', name FROM pragma_table_info('BlogNames')
|
||||
UNION ALL SELECT 'NoteTypes', name FROM pragma_table_info('NoteTypes')
|
||||
UNION ALL SELECT 'DailyAPICount', name FROM pragma_table_info('DailyAPICount')
|
||||
UNION ALL SELECT 'ApiKeyPoolState', name FROM pragma_table_info('ApiKeyPoolState')
|
||||
UNION ALL SELECT 'ApiKeyPoolMeta', name FROM pragma_table_info('ApiKeyPoolMeta')
|
||||
@@ -178,6 +204,25 @@ WHERE e.col IS NULL
|
||||
ORDER BY a.tbl, a.col;
|
||||
|
||||
|
||||
-- 1d. PRE-MIGRATION DATABASE: a backup from before 2026-08-07, when Notes still
|
||||
-- stored names. Zero rows = good.
|
||||
--
|
||||
-- This is the one failure SECTION 2 cannot fix. Notes.RootBlogName /
|
||||
-- NoteBlogName / Type were replaced by RootBlogId / NoteBlogId / TypeId
|
||||
-- resolving through BlogNames and NoteTypes -- a data migration, not an
|
||||
-- ADD COLUMN. There is no compatibility view, so the current code fails
|
||||
-- outright ("no such column: RootBlogId") against such a file.
|
||||
--
|
||||
-- Fix: run normalize-notes.sql against a COPY of the backup, then re-run
|
||||
-- SECTION 1. Do not hand-add the ID columns: they would be empty, and an
|
||||
-- empty NoteBlogId is indistinguishable from a note by blog #0.
|
||||
SELECT 'Notes still stores names -- run normalize-notes.sql on a copy' AS pre_migration_schema,
|
||||
group_concat(name, ', ') AS legacy_columns_found
|
||||
FROM pragma_table_info('Notes')
|
||||
WHERE lower(name) IN ('rootblogname','noteblogname','type')
|
||||
HAVING COUNT(*) > 0;
|
||||
|
||||
|
||||
-- ============================================================================
|
||||
-- SECTION 2 -- FIX (opt-in, additive only)
|
||||
--
|
||||
@@ -187,6 +232,9 @@ ORDER BY a.tbl, a.col;
|
||||
-- "duplicate column name" error and changes nothing -- just run the flagged
|
||||
-- subset. These are the 8 additive migration columns and nothing else; the
|
||||
-- likes high-water-mark reset is intentionally NOT included.
|
||||
--
|
||||
-- Nothing here addresses query 1d. The Notes integer schema is a data migration
|
||||
-- (normalize-notes.sql) and cannot be reached by adding columns.
|
||||
-- ============================================================================
|
||||
|
||||
-- ALTER TABLE Posts ADD COLUMN PostType TEXT;
|
||||
@@ -196,4 +244,4 @@ ORDER BY a.tbl, a.col;
|
||||
-- ALTER TABLE Blogs ADD COLUMN LikesLastRefreshed INTEGER DEFAULT 0;
|
||||
-- ALTER TABLE Blogs ADD COLUMN LikesLastNewCount INTEGER DEFAULT 0;
|
||||
-- ALTER TABLE Blogs ADD COLUMN TTFolderPath TEXT;
|
||||
-- ALTER TABLE Notes ADD COLUMN replyText TEXT DEFAULT '.';
|
||||
-- ALTER TABLE Notes ADD COLUMN replyText TEXT;
|
||||
|
||||
Reference in New Issue
Block a user