jim/URLNotesGrabberCore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(db): make Blogs.BlogId the only blog ID and drop BlogNames

Browse Source 
Blogs.BlogId was a one-time copy of BlogNames and nothing kept it
current: 12,238 blogs first seen after 2026-08-07 had a BlogNames ID
but a NULL Blogs.BlogId, so GetBlogs' join on BlogId silently skipped
them and their 23,148 notes.

- retire-blognames.sql: stub Blogs rows for the 17 unregistered note
  participants, backfill IDs (none renumbered), make ix_Blogs_BlogId
  UNIQUE, drop BlogNames, and add triggers that stop a Blogs row with
  a BlogId from being deleted, renamed or renumbered
- AddNote registers both blogs via RegisterBlog (Blogs row + MAX+1 ID)
  and every query resolves names through Blogs instead of BlogNames
- verify-db-schema.sql reports a DB that still has BlogNames (1e)
- Update TL.db.md, AGENTS.md and the DB Browser saved queries

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jim
2026-09-28 11:33:04 -05:00
co-authored by Claude Opus 5.5
parent c8c43c4918
commit 8fe2ffeb96
8 changed files with 751 additions and 232 deletions
Download Patch File Download Diff File Expand all files Collapse all files
URLNotesGrabberCORE/DataAccess.cs
+53 -37
View File
@@ -216,18 +216,19 @@ namespace URLNotesGrabberCORE
#region Notes integer schema
// Notes stopped storing names on 2026-08-07: RootBlogName/NoteBlogName/Type became
// RootBlogId/NoteBlogId/TypeId, resolved through BlogNames and NoteTypes. There is no
// RootBlogId/NoteBlogId/TypeId, resolved through Blogs.BlogId and NoteTypes. There is no
// compatibility view -- a query naming an old column fails outright, so this is a hard
// cut rather than an optional column like IsActive. See TL.db.md.
//
// Blogs.BlogId is the only ID authority as of 2026-09-28; the BlogNames table that used
// to hold the IDs is gone. Every name in Notes has a Blogs row, created by RegisterBlog.
//
// Two shapes recur below and are spelled out inline rather than hidden behind a helper,
// so that every statement reads as the SQL it actually runs:
// (SELECT BlogId FROM BlogNames WHERE BlogName = @name) -- unique-index probe, 20k rows
// (SELECT BlogId FROM Blogs WHERE BlogName = @name) -- primary-key probe
// (SELECT TypeId FROM NoteTypes WHERE Type = 'reply') -- 5 rows, effectively free
// Joining Notes to Blogs is the one case that must NOT route through BlogNames: Blogs
// carries its own BlogId, so N.NoteBlogId = B.BlogId is a single integer hop. Joining
// Notes to Posts is the opposite case -- Posts has only BlogName, so it has to go
// through BlogNames.
// Joining Notes to Blogs is N.NoteBlogId = B.BlogId, a single hop on the unique index.
// Joining Notes to Posts goes through Blogs too -- Posts has only BlogName.
/// <summary>
/// True when the exception is a duplicate-key collision on Notes. The message embeds the
@@ -242,14 +243,30 @@ namespace URLNotesGrabberCORE
}
/// <summary>
/// Gives a blog name an ID if it does not have one. No read-back and no round trip -- a
/// name that is already registered keeps the ID that 1.18M Notes rows point at.
/// Ensures a blog has a Blogs row and a BlogId, so a note can point at it. No read-back
/// and no round trip -- a blog that already has an ID keeps the one Notes rows point at.
/// Unlike AddBlog this does not skip "deact" names: a note by a deactivated blog still
/// needs an ID, and Blogs is the only place one can live.
/// Assigning the ID is bookkeeping, not a content change, so DateModified is not touched.
/// MAX(BlogId) + 1 cannot hand out a used ID because trg_Blogs_BlogId_NoDelete stops any
/// row that holds one from being deleted.
/// </summary>
private static void RegisterBlogName(SQLiteConnection connection, SQLiteTransaction? transaction, string blogName)
private static void RegisterBlog(SQLiteConnection connection, SQLiteTransaction? transaction, string blogName)
{
using SQLiteCommand command = new SQLiteCommand("INSERT OR IGNORE INTO BlogNames (BlogName) VALUES (@BlogName)", connection, transaction);
command.Parameters.AddWithValue("@BlogName", blogName);
command.ExecuteNonQuery();
string now = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss");
using (SQLiteCommand command = new SQLiteCommand("INSERT OR IGNORE INTO Blogs (BlogName, DateAdded, DateModified, DateCreated) VALUES (@BlogName, @Now, @Now, @Now)", connection, transaction))
{
command.Parameters.AddWithValue("@BlogName", blogName);
command.Parameters.AddWithValue("@Now", now);
command.ExecuteNonQuery();
}
using (SQLiteCommand command = new SQLiteCommand("UPDATE Blogs SET BlogId = (SELECT IFNULL(MAX(BlogId), 0) + 1 FROM Blogs) WHERE BlogName = @BlogName AND BlogId IS NULL", connection, transaction))
{
command.Parameters.AddWithValue("@BlogName", blogName);
command.ExecuteNonQuery();
}
}
/// <summary>
@@ -762,7 +779,6 @@ namespace URLNotesGrabberCORE
{
DBPath ??= GetDefaultDbPath();
//try { AddPost(rootBlogName, postID, DBPath); } catch { }
try { AddBlog(noteBlogName, false, DBPath); } catch { }
using SQLiteConnection connection2 = new SQLiteConnection("Data Source=" + DBPath);
int rowsInserted = 0;
@@ -771,23 +787,24 @@ namespace URLNotesGrabberCORE
{
connection2.Open();
// Notes stores integer IDs, so both participants and the type have to exist in
// their lookup table before the note can point at them.
// Notes stores integer IDs, so both participants need a Blogs row with a BlogId,
// and the type a NoteTypes row, before the note can point at them. RegisterBlog
// also does what the AddBlog call here used to: register the note's blog.
//
// All four statements run in one transaction so a crash cannot leave a name or a
// All statements run in one transaction so a crash cannot leave a blog or a
// type registered with no note. The transaction is committed before the console
// output below, which sleeps -- a write lock must not be held across that.
using (SQLiteTransaction transaction = connection2.BeginTransaction())
{
RegisterBlogName(connection2, transaction, rootBlogName);
RegisterBlogName(connection2, transaction, noteBlogName);
RegisterBlog(connection2, transaction, rootBlogName);
RegisterBlog(connection2, transaction, noteBlogName);
RegisterNoteType(connection2, transaction, type ?? string.Empty);
// INSERT OR IGNORE, and no IsActive in the column list: re-crawling a note
// that was removed elsewhere leaves the existing row -- and its flag -- alone.
string sql = "INSERT OR IGNORE INTO Notes (RootBlogId, NoteBlogId, PostID, TimeStamp, TypeId, DatetimeCrawled, DateModified, DateCreated) " +
"SELECT (SELECT BlogId FROM BlogNames WHERE BlogName = @rootBlogName), " +
" (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName), " +
"SELECT (SELECT BlogId FROM Blogs WHERE BlogName = @rootBlogName), " +
" (SELECT BlogId FROM Blogs WHERE BlogName = @noteBlogName), " +
" @PostID, @TimeStamp, " +
" (SELECT TypeId FROM NoteTypes WHERE Type = @Type), " +
" @DatetimeCrawled, @DateModified, @DateCreated";
@@ -1126,11 +1143,11 @@ namespace URLNotesGrabberCORE
{
connection.Open();
string sql = "SELECT DISTINCT BN.BlogName as blogName, N.PostID" +
string sql = "SELECT DISTINCT RB.BlogName as blogName, N.PostID" +
" FROM Notes N" +
" INNER JOIN BlogNames BN ON BN.BlogId = N.RootBlogId" +
" INNER JOIN Blogs RB ON RB.BlogId = N.RootBlogId" +
" WHERE N.TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')" + AndIsActive("Notes", "N", DBPath) +
" ORDER BY BN.BlogName, N.PostID";
" ORDER BY RB.BlogName, N.PostID";
using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{
@@ -1171,11 +1188,11 @@ namespace URLNotesGrabberCORE
connection.Open();
// Grouped on the integer rather than the name: the group key is what gets sorted,
// and BN.BlogName comes along for free off the join.
string sql = @"SELECT BN.BlogName as blogName, N.PostID,
// and RB.BlogName comes along for free off the join.
string sql = @"SELECT RB.BlogName as blogName, N.PostID,
MAX(N.TimeStamp) as LatestTimestamp
FROM Notes N
INNER JOIN BlogNames BN ON BN.BlogId = N.RootBlogId
INNER JOIN Blogs RB ON RB.BlogId = N.RootBlogId
WHERE N.TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')
AND (N.replyText IS NULL OR N.replyText = '' OR N.replyText = '.')" + AndIsActive("Notes", "N", DBPath) + @"
GROUP BY N.RootBlogId, N.PostID
@@ -1220,13 +1237,13 @@ namespace URLNotesGrabberCORE
{
connection.Open();
// Posts carries only BlogName, so this is the one join to Notes that has to go
// through BlogNames -- there is no Posts.BlogId to hop on. The name predicate is
// pushed into the 20k-row lookup, which then feeds integers to the Notes key.
// Posts carries only BlogName, so the join to Notes goes through Blogs -- there is
// no Posts.BlogId to hop on. Each post's name is a primary-key probe on Blogs,
// which then feeds an integer to the Notes key.
string sql = @"SELECT DISTINCT P.BlogName, P.PostID, MAX(N.TimeStamp) as LatestTimestamp
FROM Posts P
INNER JOIN BlogNames RBN ON RBN.BlogName = P.BlogName
INNER JOIN Notes N ON N.RootBlogId = RBN.BlogId AND N.PostID = P.PostID
INNER JOIN Blogs RB ON RB.BlogName = P.BlogName
INNER JOIN Notes N ON N.RootBlogId = RB.BlogId AND N.PostID = P.PostID
WHERE P.NotFound = 0
AND N.TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')
AND (N.replyText IS NULL OR N.replyText = '' OR N.replyText = '.')" + AndIsActive("Posts", "P", DBPath) + AndIsActive("Notes", "N", DBPath) + @"
@@ -1415,8 +1432,7 @@ namespace URLNotesGrabberCORE
try
{
connection.Open();
// Blogs is reached in one integer hop off Blogs.BlogId, not through BlogNames --
// that would add a hop and end in the text comparison the migration removed.
// Blogs is reached in one integer hop off Blogs.BlogId.
// The negated form is only correct because Notes.TypeId is NOT NULL.
string sql = "";
if (reblogsOnly)
@@ -1703,8 +1719,8 @@ namespace URLNotesGrabberCORE
connection.Open();
string sql = "UPDATE Notes SET TimeStamp = @timestamp, DateModified = @dateModified " +
"WHERE RootBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @rootBlogName) " +
"AND NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName) " +
"WHERE RootBlogId = (SELECT BlogId FROM Blogs WHERE BlogName = @rootBlogName) " +
"AND NoteBlogId = (SELECT BlogId FROM Blogs WHERE BlogName = @noteBlogName) " +
"AND PostID = @postID AND IFNULL(TimeStamp, 0) <> @timestamp";
using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{
@@ -2018,7 +2034,7 @@ namespace URLNotesGrabberCORE
// Only fan out to rows that match the SELECT criteria in GetRepliesWithFilledText (NULL/empty/legacy-'.'). Never overwrite '?' (confirmed-empty) or already-fetched text.
// The ABS() term cannot use an index on TimeStamp, before or after the integer schema; the NoteBlogId probe is what keeps this off a full scan.
string sql = "UPDATE Notes SET replyText = @replyText, DateModified = @dateModified " +
"WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName) " +
"WHERE NoteBlogId = (SELECT BlogId FROM Blogs WHERE BlogName = @noteBlogName) " +
"AND ABS(TimeStamp - @TimeStamp) <= 5 " +
"AND TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply') " +
"AND (replyText IS NULL OR replyText = '' OR replyText = '.') " +
@@ -2064,7 +2080,7 @@ namespace URLNotesGrabberCORE
connection.Open();
string sql = "UPDATE Notes SET replyText = @replyText, DateModified = @dateModified " +
"WHERE RootBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @rootBlogName) " +
"WHERE RootBlogId = (SELECT BlogId FROM Blogs WHERE BlogName = @rootBlogName) " +
"AND PostID = @PostID " +
"AND TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply') " +
"AND IFNULL(replyText, '.') <> @replyText";