jim/URLNotesGrabberCore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: jim/URLNotesGrabberCore:e6a3efba5b7806e2eb3b0040754025c9449f960a
Branches Tags
jim/URLNotesGrabberCore:master jim/URLNotesGrabberCore:claude/collect-command-issue-09fcb8 jim/URLNotesGrabberCore:claude/dbbrowser-broken-queries-e2bee5 jim/URLNotesGrabberCore:claude/posttype-noargs-backfill jim/URLNotesGrabberCore:claude/posttype-unknown-fix jim/URLNotesGrabberCore:claude/ttfolderpath-not-set-8c9b45 jim/URLNotesGrabberCore:claude/jovial-mayer-77c2b5 jim/URLNotesGrabberCore:claude/youthful-austin-cc753e
..
compare: jim/URLNotesGrabberCore:claude/dbbrowser-broken-queries-e2bee5
Branches Tags
jim/URLNotesGrabberCore:master jim/URLNotesGrabberCore:claude/collect-command-issue-09fcb8 jim/URLNotesGrabberCore:claude/dbbrowser-broken-queries-e2bee5 jim/URLNotesGrabberCore:claude/posttype-noargs-backfill jim/URLNotesGrabberCore:claude/posttype-unknown-fix jim/URLNotesGrabberCore:claude/ttfolderpath-not-set-8c9b45 jim/URLNotesGrabberCore:claude/jovial-mayer-77c2b5 jim/URLNotesGrabberCore:claude/youthful-austin-cc753e
3 Commits
e6a3efba5b .. claude/dbbrowser-broken-queries-e2bee5
Author SHA1 Message Date
jimandClaude Sonnet 5 a28c5cc9ec fix(dbbrowser): rewrite saved queries for the Notes integer schema 
Notes.RootBlogName/NoteBlogName/Type were replaced by RootBlogId/NoteBlogId/
TypeId (resolved via BlogNames and NoteTypes) back on 2026-08-07. The saved
DB Browser for SQLite queries in RERUN.sqbpro and TL.sqbpro still referenced
the old text columns and failed against the migrated TL.db.

Rewrote the 5 affected queries per TL.db.md's porting guide: Notes<->Blogs
joins go through Blogs.BlogId in one hop, Notes<->Posts joins route through
BlogNames (Posts has no BlogId), and type filters resolve through NoteTypes.
Verified read-only against the live TL.db -- all 5 execute without error.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
 2026-08-20 09:07:08 -05:00
jim d6a96f7885 Merge branch 'claude/posttype-noargs-backfill' into master 2026-08-20 08:15:54 -05:00
jimandClaude Opus 5 864b468d96 fix(posttype): run the migration and backfill in no-args mode 
The backfill hung off --ingest, --output, --correct, --importposts and
--updatepaths, because those were where EnsureTTFileHelperColumnsExist was
already being called. But the no-argument traversal is the mode that
actually gets run day to day, and it called none of them -- so the command
used most often was the one command that never repaired an untyped row.

TraverseDirectory already types the posts it inserts, from the filename it
is reading. This closes the other half: the rows already sitting untyped
now get fixed by an ordinary run, with no separate maintenance command.

Called before BeginImportSession so it uses its own connection rather than
contending with the import session's, and before the traversal so existing
rows are typed first and newly inserted ones arrive already typed.

Co-Authored-By: Claude Opus 5 <[email protected]>
 2026-08-20 08:15:54 -05:00
3 changed files with 145 additions and 124 deletions
Expand all files Collapse all files
RERUN.sqbpro
+109 -100
View File
@@ -1,100 +1,109 @@
<?xml version="1.0" encoding="UTF-8"?><sqlb_project><db path="D:/NextCloud/C#/URLNotesGrabberCORE/URLNotesGrabberCORE/TL.db" readonly="0" foreign_keys="1" case_sensitive_like="0" temp_store="0" wal_autocheckpoint="1000" synchronous="2"/><attached/><window><main_tabs open="structure browser pragmas query" current="3"/></window><tab_structure><column_width id="0" width="300"/><column_width id="1" width="0"/><column_width id="2" width="100"/><column_width id="3" width="4486"/><column_width id="4" width="0"/><expanded_item id="0" parent="1"/><expanded_item id="1" parent="1"/><expanded_item id="2" parent="1"/><expanded_item id="3" parent="1"/></tab_structure><tab_browse><table title="Posts" custom_title="0" dock_id="4" table="4,5:mainPosts"/><dock_state state="000000ff00000000fd0000000100000002000005470000029afc0100000006fb000000160064006f0063006b00420072006f00770073006500310100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500320100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500330100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500350100000000000005f40000000000000000fb000000160064006f0063006b00420072006f00770073006500340100000000000005470000011100fffffffb000000160064006f0063006b00420072006f00770073006500340100000000000005f40000000000000000000005470000000000000004000000040000000800000008fc00000000"/><default_encoding codec=""/><browse_table_settings><table schema="main" name="ApiKeyPoolMeta" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="29"/><column index="2" value="64"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Blogs" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="7" mode="1"/></sort><column_widths><column index="1" value="257"/><column index="2" value="108"/><column index="3" value="63"/><column index="4" value="156"/><column index="5" value="60"/><column index="6" value="81"/><column index="7" value="85"/><column index="8" value="156"/><column index="9" value="156"/><column index="10" value="151"/><column index="11" value="125"/><column index="12" value="129"/></column_widths><filter_values><column index="4" value="1"/><column index="7" value="&gt;2026-05-27 17:22:36"/></filter_values><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Notes" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="198"/><column index="2" value="144"/><column index="3" value="251"/><column index="4" value="84"/><column index="5" value="53"/><column index="6" value="300"/><column index="7" value="116"/><column index="8" value="152"/><column index="9" value="89"/><column index="10" value="63"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Posts" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="14" mode="1"/></sort><column_widths><column index="1" value="236"/><column index="2" value="144"/><column index="3" value="126"/><column index="4" value="32"/><column index="5" value="32"/><column index="6" value="32"/><column index="7" value="32"/><column index="8" value="32"/><column index="9" value="0"/><column index="10" value="0"/><column index="11" value="0"/><column index="12" value="243"/><column index="13" value="300"/><column index="14" value="53"/><column index="15" value="37351"/><column index="16" value="300"/><column index="17" value="41"/><column index="18" value="75"/><column index="19" value="96"/><column index="20" value="300"/><column index="21" value="96"/><column index="22" value="300"/><column index="23" value="300"/><column index="24" value="548"/><column index="25" value="60"/><column index="26" value="213"/><column index="27" value="532"/><column index="28" value="152"/><column index="29" value="152"/><column index="30" value="69"/><column index="31" value="63"/></column_widths><filter_values><column index="2" value="0"/><column index="1" value="734568371821084672"/></filter_values><conditional_formats/><row_id_formats/><display_formats/><hidden_columns><column index="9" value="1"/><column index="10" value="1"/><column index="11" value="1"/></hidden_columns><plot_y_axes/><global_filter/></table></browse_table_settings></tab_browse><tab_sql><sql name="SQL 1">UPDATE Posts <?xml version="1.0" encoding="UTF-8"?><sqlb_project><db path="D:/NextCloud/C#/URLNotesGrabberCORE/URLNotesGrabberCORE/TL.db" readonly="0" foreign_keys="1" case_sensitive_like="0" temp_store="0" wal_autocheckpoint="1000" synchronous="2"/><attached/><window><main_tabs open="structure browser pragmas query" current="3"/></window><tab_structure><column_width id="0" width="300"/><column_width id="1" width="0"/><column_width id="2" width="100"/><column_width id="3" width="4486"/><column_width id="4" width="0"/><expanded_item id="0" parent="1"/><expanded_item id="1" parent="1"/><expanded_item id="2" parent="1"/><expanded_item id="3" parent="1"/></tab_structure><tab_browse><table title="Posts" custom_title="0" dock_id="4" table="4,5:mainPosts"/><dock_state state="000000ff00000000fd0000000100000002000005470000029afc0100000006fb000000160064006f0063006b00420072006f00770073006500310100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500320100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500330100000000000004a10000000000000000fb000000160064006f0063006b00420072006f00770073006500350100000000000005f40000000000000000fb000000160064006f0063006b00420072006f00770073006500340100000000000005470000011100fffffffb000000160064006f0063006b00420072006f00770073006500340100000000000005f40000000000000000000005470000000000000004000000040000000800000008fc00000000"/><default_encoding codec=""/><browse_table_settings><table schema="main" name="ApiKeyPoolMeta" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="29"/><column index="2" value="64"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Blogs" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="7" mode="1"/></sort><column_widths><column index="1" value="257"/><column index="2" value="108"/><column index="3" value="63"/><column index="4" value="156"/><column index="5" value="60"/><column index="6" value="81"/><column index="7" value="85"/><column index="8" value="156"/><column index="9" value="156"/><column index="10" value="151"/><column index="11" value="125"/><column index="12" value="129"/></column_widths><filter_values><column index="4" value="1"/><column index="7" value="&gt;2026-05-27 17:22:36"/></filter_values><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Notes" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort/><column_widths><column index="1" value="198"/><column index="2" value="144"/><column index="3" value="251"/><column index="4" value="84"/><column index="5" value="53"/><column index="6" value="300"/><column index="7" value="116"/><column index="8" value="152"/><column index="9" value="89"/><column index="10" value="63"/></column_widths><filter_values/><conditional_formats/><row_id_formats/><display_formats/><hidden_columns/><plot_y_axes/><global_filter/></table><table schema="main" name="Posts" show_row_id="0" encoding="" plot_x_axis="" unlock_view_pk="_rowid_" freeze_columns="0"><sort><column index="14" mode="1"/></sort><column_widths><column index="1" value="236"/><column index="2" value="144"/><column index="3" value="126"/><column index="4" value="32"/><column index="5" value="32"/><column index="6" value="32"/><column index="7" value="32"/><column index="8" value="32"/><column index="9" value="0"/><column index="10" value="0"/><column index="11" value="0"/><column index="12" value="243"/><column index="13" value="300"/><column index="14" value="53"/><column index="15" value="37351"/><column index="16" value="300"/><column index="17" value="41"/><column index="18" value="75"/><column index="19" value="96"/><column index="20" value="300"/><column index="21" value="96"/><column index="22" value="300"/><column index="23" value="300"/><column index="24" value="548"/><column index="25" value="60"/><column index="26" value="213"/><column index="27" value="532"/><column index="28" value="152"/><column index="29" value="152"/><column index="30" value="69"/><column index="31" value="63"/></column_widths><filter_values><column index="2" value="0"/><column index="1" value="734568371821084672"/></filter_values><conditional_formats/><row_id_formats/><display_formats/><hidden_columns><column index="9" value="1"/><column index="10" value="1"/><column index="11" value="1"/></hidden_columns><plot_y_axes/><global_filter/></table></browse_table_settings></tab_browse><tab_sql><sql name="SQL 1">UPDATE Posts
SET HasNotesGathered = 0 SET HasNotesGathered = 0
WHERE (BlogName, PostID) IN ( WHERE (BlogName, PostID) IN (
SELECT p.BlogName, p.PostID SELECT p.BlogName, p.PostID
FROM Posts p FROM Posts p
WHERE p.HasNotesGathered = 1 WHERE p.HasNotesGathered = 1
AND P.notesGatheredDatetime &lt; 1774294520 AND P.notesGatheredDatetime &lt; 1774294520
AND EXISTS ( AND EXISTS (
SELECT 1 SELECT 1
FROM Notes n FROM Notes n
WHERE n.PostID = p.PostID WHERE n.PostID = p.PostID
AND n.RootBlogName = p.BlogName AND n.RootBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = p.BlogName)
--AND n.Type NOT IN ('reblog', 'reply') --AND n.TypeId NOT IN (SELECT TypeId FROM NoteTypes WHERE Type IN ('reblog', 'reply'))
) )
ORDER BY P.PostDate ASC ORDER BY P.PostDate ASC
--LIMIT 500 --LIMIT 500
);</sql><sql name="Mark Blogs">select * );</sql><sql name="Mark Blogs">select *
from Blogs from Blogs
--update blogs set HasBeenOutput = 1 --update blogs set HasBeenOutput = 1
where HasBeenOutput = 0 where HasBeenOutput = 0
AND AND
blogname in blogname in
( (
'teaberrybee', 'teaberrybee',
'reddevilgoddesstoo', 'reddevilgoddesstoo',
'waywardog13', 'waywardog13',
'wzjustbrowsing-blog', 'wzjustbrowsing-blog',
'lewerta', 'lewerta',
'nudenymph', 'nudenymph',
'caylachief' 'caylachief'
)</sql><sql name="New Notes">select P.slug, N.replyText, n.RootBlogName, n.PostID, NoteBlogName || '.tumblr.com' as NoteBlogName, DatetimeCrawled, TimeStamp, type, n.RootBlogName || '.tumblr.com/post/' || n.postid, datetime(timestamp, 'unixepoch') )</sql><sql name="New Notes">select P.slug, N.replyText, rbn.BlogName as RootBlogName, n.PostID, nbn.BlogName || '.tumblr.com' as NoteBlogName, DatetimeCrawled, TimeStamp, nt.Type, rbn.BlogName || '.tumblr.com/post/' || n.postid, datetime(timestamp, 'unixepoch')
from Notes N inner join Posts P on p.PostID = n.PostID from Notes N
where inner join Posts P on p.PostID = n.PostID
DatetimeCrawled &gt; '2026-08-07 11:47:22' and type like 'r%' inner join BlogNames rbn on rbn.BlogId = n.RootBlogId
and P.IsActive = 1 inner join BlogNames nbn on nbn.BlogId = n.NoteBlogId
order by n.DatetimeCrawled</sql><sql name="Pull Blogs">SELECT distinct inner join NoteTypes nt on nt.TypeId = n.TypeId
'''' || blogname || ''',', where
blogs.* DatetimeCrawled &gt; '2026-08-07 11:47:22' and nt.Type like 'r%'
, blogname || '.tumblr.com' and P.IsActive = 1
FROM order by n.DatetimeCrawled</sql><sql name="Pull Blogs">SELECT distinct
Blogs '''' || blogname || ''',',
inner JOIN blogs.*
Notes on notes.noteBlogName = blogs.BlogName , blogname || '.tumblr.com'
WHERE FROM
HasBeenOutput = 0 and type = 'reblog' Blogs
order by inner JOIN
Notes.Type desc, Notes on notes.noteBlogId = blogs.BlogId
DateAdded desc inner JOIN
LIMIT 100;</sql><sql name="SQL 7">WITH ReplyCounts AS ( NoteTypes on NoteTypes.TypeId = Notes.TypeId
SELECT WHERE
NoteBlogName, HasBeenOutput = 0 and NoteTypes.Type = 'reblog'
COUNT(DISTINCT replyText) AS DistinctReplyCount order by
FROM Notes NoteTypes.Type desc,
where replyText &lt;&gt; '.' DateAdded desc
GROUP BY NoteBlogName LIMIT 100;</sql><sql name="SQL 7">WITH ReplyCounts AS (
) SELECT
SELECT NoteBlogId,
n.RootBlogName || '.tumblr.com/post/' || n.PostID AS PostURL, postid, COUNT(DISTINCT replyText) AS DistinctReplyCount
n.NoteBlogName, FROM Notes
n.replyText, where replyText &lt;&gt; '.'
c.DistinctReplyCount GROUP BY NoteBlogId
FROM Notes n )
JOIN ReplyCounts c ON n.NoteBlogName = c.NoteBlogName SELECT
where replyText &lt;&gt; '.' and type &lt;&gt; 'reply' rbn.BlogName || '.tumblr.com/post/' || n.PostID AS PostURL, postid,
--AND N.NoteBlogName NOT IN ( 'roadblocker21', 'thesaddemon666', 'edwardabbeyhoffman', 'tattedsoldier20', 'zomb-eh', 'animalistic13', 'indken', 'maccloud1592', nbn.BlogName AS NoteBlogName,
--'moss-wizard', 'supertrucker12682', 'exploringthrupics', 'padeyepete' ) n.replyText,
order by c.DistinctReplyCount desc, n.NoteBlogName, n.DateModified desc, replyText, RootBlogName, PostID</sql><sql name="Collect">WITH PostsWithCount AS ( SELECT P.BlogName, P.PostID, 1925013599 AS LatestNoteTimestamp, P.NotesGatheredDateTime, COUNT(P.PostID) OVER(PARTITION BY P.BlogName) AS CNT, P.HasNotesGathered, P.NotFound, P.PostDate FROM Posts P WHERE COALESCE(P.IsActive, 1) = 1 ), Unioned AS ( SELECT BlogName, PostID, LatestNoteTimestamp, NotesGatheredDateTime, CNT, PostDate FROM PostsWithCount WHERE NotFound = 0 AND HasNotesGathered = 0 UNION SELECT BlogName, PostID, LatestNoteTimestamp, NotesGatheredDateTime, CNT, PostDate FROM PostsWithCount WHERE BlogName = 'zomb-eh' AND NotFound = 0 AND NotesGatheredDateTime &lt; unixepoch('now', 'localtime', '-3 days') ) SELECT U.BlogName, U.PostID, U.LatestNoteTimestamp, U.NotesGatheredDateTime, U.CNT FROM Unioned U WHERE (U.NotesGatheredDateTime &lt; 1786134037 OR U.NotesGatheredDateTime IS NULL) ORDER BY U.NotesGatheredDateTime, U.PostDate DESC, U.BlogName, U.PostID;</sql><sql name="Del Posts">delete from posts where postid in c.DistinctReplyCount
( FROM Notes n
'741662499571728384', JOIN ReplyCounts c ON n.NoteBlogId = c.NoteBlogId
178892849664, JOIN BlogNames rbn ON rbn.BlogId = n.RootBlogId
178264721139, JOIN BlogNames nbn ON nbn.BlogId = n.NoteBlogId
177012868749, JOIN NoteTypes t ON t.TypeId = n.TypeId
169950081964, where replyText &lt;&gt; '.' and t.Type &lt;&gt; 'reply'
755440787056099328 --AND nbn.BlogName NOT IN ( 'roadblocker21', 'thesaddemon666', 'edwardabbeyhoffman', 'tattedsoldier20', 'zomb-eh', 'animalistic13', 'indken', 'maccloud1592',
)</sql><sql name="notes NO post*">select * --'moss-wizard', 'supertrucker12682', 'exploringthrupics', 'padeyepete' )
-- delete order by c.DistinctReplyCount desc, nbn.BlogName, n.DateModified desc, replyText, rbn.BlogName, PostID</sql><sql name="Collect">WITH PostsWithCount AS ( SELECT P.BlogName, P.PostID, 1925013599 AS LatestNoteTimestamp, P.NotesGatheredDateTime, COUNT(P.PostID) OVER(PARTITION BY P.BlogName) AS CNT, P.HasNotesGathered, P.NotFound, P.PostDate FROM Posts P WHERE COALESCE(P.IsActive, 1) = 1 ), Unioned AS ( SELECT BlogName, PostID, LatestNoteTimestamp, NotesGatheredDateTime, CNT, PostDate FROM PostsWithCount WHERE NotFound = 0 AND HasNotesGathered = 0 UNION SELECT BlogName, PostID, LatestNoteTimestamp, NotesGatheredDateTime, CNT, PostDate FROM PostsWithCount WHERE BlogName = 'zomb-eh' AND NotFound = 0 AND NotesGatheredDateTime &lt; unixepoch('now', 'localtime', '-3 days') ) SELECT U.BlogName, U.PostID, U.LatestNoteTimestamp, U.NotesGatheredDateTime, U.CNT FROM Unioned U WHERE (U.NotesGatheredDateTime &lt; 1786134037 OR U.NotesGatheredDateTime IS NULL) ORDER BY U.NotesGatheredDateTime, U.PostDate DESC, U.BlogName, U.PostID;</sql><sql name="Del Posts">delete from posts where postid in
from notes (
where postid not in (select distinct postid from posts where IsActive = 1)</sql><sql name="SQL 9">SELECT '741662499571728384',
* 178892849664,
FROM 178264721139,
POSTS P 177012868749,
WHERE 169950081964,
P.ByLikes = 1 755440787056099328
AND )</sql><sql name="notes NO post*">select *
P.DateCreated &gt; '2026-05-26 17:47:32' -- delete
ORDER BY from notes
P.DateCreated desc</sql><sql name="SQL 13">update posts set IsActive = 0 where blogname IN ( 'shoebiedoo', 'redheaded-girlygirl', 'xlittle-ghost' )</sql><sql name="SQL 14*">update Posts␍ where postid not in (select distinct postid from posts where IsActive = 1)</sql><sql name="SQL 9">SELECT
set IsActive = 0␍ *
where postid in␍ FROM
(␍ POSTS P
␍ WHERE
␍ P.ByLikes = 1
'731937314675310592'␍ AND
␍ P.DateCreated &gt; '2026-05-26 17:47:32'
␍ ORDER BY
␍ P.DateCreated desc</sql><sql name="SQL 13">update posts set IsActive = 0 where blogname IN ( 'shoebiedoo', 'redheaded-girlygirl', 'xlittle-ghost' )</sql><sql name="SQL 14*">update Posts
)␍ set IsActive = 0
␍ where postid in
</sql><current_tab id="7"/></tab_sql></sqlb_project> (
'731937314675310592'
)
</sql><current_tab id="7"/></tab_sql></sqlb_project>
TL.sqbpro
+30 -24
View File
@@ -1,24 +1,30 @@
<?xml version="1.0" encoding="UTF-8"?><sqlb_project><db path="" readonly="1" foreign_keys="" case_sensitive_like="" temp_store="" wal_autocheckpoint="" synchronous=""/><attached/><window><main_tabs open="structure browser pragmas query" current="3"/></window><tab_structure><column_width id="0" width="300"/><column_width id="1" width="0"/><column_width id="2" width="100"/><column_width id="3" width="3571"/><column_width id="4" width="0"/></tab_structure><tab_browse><table title="." custom_title="0" dock_id="4" table="0,0:"/><dock_state state="000000ff00000000fd0000000100000002000005f40000030ffc0100000002fb000000160064006f0063006b00420072006f00770073006500310100000000000005f40000000000000000fb000000160064006f0063006b00420072006f00770073006500340100000000ffffffff0000011700ffffff000005f40000000000000004000000040000000800000008fc00000000"/><default_encoding codec=""/><browse_table_settings/></tab_browse><tab_sql><sql name="SQL 1">SELECT <?xml version="1.0" encoding="UTF-8"?><sqlb_project><db path="" readonly="1" foreign_keys="" case_sensitive_like="" temp_store="" wal_autocheckpoint="" synchronous=""/><attached/><window><main_tabs open="structure browser pragmas query" current="3"/></window><tab_structure><column_width id="0" width="300"/><column_width id="1" width="0"/><column_width id="2" width="100"/><column_width id="3" width="3571"/><column_width id="4" width="0"/></tab_structure><tab_browse><table title="." custom_title="0" dock_id="4" table="0,0:"/><dock_state state="000000ff00000000fd0000000100000002000005f40000030ffc0100000002fb000000160064006f0063006b00420072006f00770073006500310100000000000005f40000000000000000fb000000160064006f0063006b00420072006f00770073006500340100000000ffffffff0000011700ffffff000005f40000000000000004000000040000000800000008fc00000000"/><default_encoding codec=""/><browse_table_settings/></tab_browse><tab_sql><sql name="SQL 1">SELECT
BlogName || '.tumblr.com/post/' || postID, BlogName || '.tumblr.com/post/' || postID,
datetime(NotesGatheredDateTime, 'unixepoch'), * datetime(NotesGatheredDateTime, 'unixepoch'), *
FROM FROM
Posts Posts
WHERE WHERE
NotesGatheredDateTime &lt;&gt; 0 NotesGatheredDateTime &lt;&gt; 0
ORDER BY ORDER BY
postdate desc</sql><sql name="SQL 2*">SELECT postdate desc</sql><sql name="SQL 2*">SELECT
datetime(TimeStamp, 'unixepoch'), datetime(TimeStamp, 'unixepoch'),
RootBlogName || '.tumblr.com/post/' || N.postid, rbn.BlogName || '.tumblr.com/post/' || N.postid,
*, *,
NoteBlogName || '.tumblr.com' nbn.BlogName || '.tumblr.com'
FROM FROM
Notes N␍ Notes N
inner JOIN␍ inner JOIN
Posts P on P.PostID = N.PostID and P.BlogName = N.RootBlogName BlogNames rbn on rbn.BlogId = N.RootBlogId
WHERE RootBlogName NOT IN ('xlittle-ghost', 'glimmerin-darlin', 'vvenus-child')␍ inner JOIN
and type like 'r%'␍ BlogNames nbn on nbn.BlogId = N.NoteBlogId
and RootBlogName = 'zomb-eh'␍ inner JOIN
and P.HasImage = 1 NoteTypes t on t.TypeId = N.TypeId
ORDER BY inner JOIN
TimeStamp desc</sql><current_tab id="1"/></tab_sql></sqlb_project> Posts P on P.PostID = N.PostID and P.BlogName = rbn.BlogName
WHERE rbn.BlogName NOT IN ('xlittle-ghost', 'glimmerin-darlin', 'vvenus-child')
and t.Type like 'r%'
and rbn.BlogName = 'zomb-eh'
and P.HasImage = 1
ORDER BY
TimeStamp desc</sql><current_tab id="1"/></tab_sql></sqlb_project>
URLNotesGrabberCORE/Program.cs
+6
View File
@@ -148,6 +148,12 @@ namespace URLNotesGrabberCORE
if (args.Length == 0) //Traverse folder structure to add posts and thus blogs to DB if (args.Length == 0) //Traverse folder structure to add posts and thus blogs to DB
{ {
int postsAdded = 0; int postsAdded = 0;
// This is the mode that actually gets run day to day, so the schema migration and
// the PostType backfill have to happen here too. They used to hang off --ingest,
// --output and friends only, which meant the untyped rows this traversal creates
// could sit unrepaired indefinitely while the one command everyone runs skipped
// the fix entirely. Idempotent, so paying it on every run costs nothing.
DataAccess.EnsureTTFileHelperColumnsExist();
try try
{ {
DataAccess.EnableImportModePragmas(); DataAccess.EnableImportModePragmas();