jim/URLNotesGrabberCore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(db)!: port DataAccess to the Notes integer schema

Browse Source 
Notes.RootBlogName/NoteBlogName/Type became RootBlogId/NoteBlogId/TypeId
on 2026-08-07, resolved through the new BlogNames and NoteTypes tables.
There is no compatibility view, so every affected statement is a hard cut.

All 14 call sites in DataAccess.cs are ported:

- Notes->Blogs joins go through Blogs.BlogId in one integer hop; the
  Notes->Posts join in GetRepliesWithFilledText is the only one that must
  route through BlogNames, since Posts carries no BlogId
- AddNote registers both blog names and the note type with INSERT OR
  IGNORE before inserting, in one transaction committed before the console
  sleep. Registering the type matters: an unseen type would resolve to
  NULL and fail NOT NULL on TypeId, silently losing the note
- The LEFT JOIN Notes in GetPosts is dropped rather than translated. It
  selected nothing, could not remove a row, and its duplicates were
  collapsed by the query's own GROUP BY
- Duplicate-key detection moves to IsNotesDuplicateKey, matching the
  constraint and table instead of an exact column list. The old literal
  string is what broke on this rename
- EnsureReplyTextColumnExists drops DEFAULT '.', matching the migrated
  schema: new rows get NULL, not a placeholder nobody wrote

verify-db-schema.sql gains BlogNames, NoteTypes, Blogs.BlogId and the new
Notes columns, plus query 1d naming a pre-migration file and pointing at
normalize-notes.sql. Blogs.BlogId is deliberately not auto-fixable -- an
added-but-empty column makes engagement joins return zero rows silently.

Verified against the live 148 MB file: query plans hit the intended
indexes, and the BlogId join matches an independent name-resolved
formulation exactly on all 4,267 GetBlogs and 2,637 GetBlogsForLikes rows.

RolodexRepository.cs (16 sites) lives in the Rolodex repo and is not
covered here.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jim
2026-08-07 21:23:13 -05:00
co-authored by Claude Opus 5
parent c3cf89c3f1
commit b31d5842cc
4 changed files with 277 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files
URLNotesGrabberCORE/DataAccess.cs
+176 -65
View File
@@ -213,6 +213,59 @@ namespace URLNotesGrabberCORE
#endregion IsActive
#region Notes integer schema
// Notes stopped storing names on 2026-08-07: RootBlogName/NoteBlogName/Type became
// RootBlogId/NoteBlogId/TypeId, resolved through BlogNames and NoteTypes. There is no
// compatibility view -- a query naming an old column fails outright, so this is a hard
// cut rather than an optional column like IsActive. See TL.db.md.
//
// Two shapes recur below and are spelled out inline rather than hidden behind a helper,
// so that every statement reads as the SQL it actually runs:
// (SELECT BlogId FROM BlogNames WHERE BlogName = @name) -- unique-index probe, 20k rows
// (SELECT TypeId FROM NoteTypes WHERE Type = 'reply') -- 5 rows, effectively free
// Joining Notes to Blogs is the one case that must NOT route through BlogNames: Blogs
// carries its own BlogId, so N.NoteBlogId = B.BlogId is a single integer hop. Joining
// Notes to Posts is the opposite case -- Posts has only BlogName, so it has to go
// through BlogNames.
/// <summary>
/// True when the exception is a duplicate-key collision on Notes. The message embeds the
/// primary key's column names, which the integer migration renamed, so this matches on the
/// constraint and the table instead of on an exact column list -- a literal comparison
/// silently inverts into "log every error" the next time a column is renamed.
/// </summary>
private static bool IsNotesDuplicateKey(Exception ex)
{
return ex.Message.Contains("UNIQUE constraint failed", StringComparison.OrdinalIgnoreCase)
&& ex.Message.Contains("Notes.", StringComparison.OrdinalIgnoreCase);
}
/// <summary>
/// Gives a blog name an ID if it does not have one. No read-back and no round trip -- a
/// name that is already registered keeps the ID that 1.18M Notes rows point at.
/// </summary>
private static void RegisterBlogName(SQLiteConnection connection, SQLiteTransaction? transaction, string blogName)
{
using SQLiteCommand command = new SQLiteCommand("INSERT OR IGNORE INTO BlogNames (BlogName) VALUES (@BlogName)", connection, transaction);
command.Parameters.AddWithValue("@BlogName", blogName);
command.ExecuteNonQuery();
}
/// <summary>
/// Same, for a note type. NoteTypes is a table rather than a CHECK constraint precisely so
/// that a type this crawler has not seen before is an INSERT and not a schema migration --
/// without this the type would resolve to NULL and fail the NOT NULL on Notes.TypeId.
/// </summary>
private static void RegisterNoteType(SQLiteConnection connection, SQLiteTransaction? transaction, string type)
{
using SQLiteCommand command = new SQLiteCommand("INSERT OR IGNORE INTO NoteTypes (Type) VALUES (@Type)", connection, transaction);
command.Parameters.AddWithValue("@Type", type);
command.ExecuteNonQuery();
}
#endregion Notes integer schema
public static string Q(string input)
{
return "'" + input.Replace("'", "''") + "'";
@@ -371,7 +424,10 @@ namespace URLNotesGrabberCORE
connection.Close();
connection.Open();
string addColumnSql = "ALTER TABLE Notes ADD COLUMN replyText TEXT DEFAULT '.';";
// No column default: the migrated schema dropped the DEFAULT '.' that
// is how 1.1M rows acquired a placeholder nobody wrote. New rows get
// NULL, which every reader here already treats as "no reply text".
string addColumnSql = "ALTER TABLE Notes ADD COLUMN replyText TEXT;";
using (SQLiteCommand addCommand = new SQLiteCommand(addColumnSql, connection))
{
addCommand.ExecuteNonQuery();
@@ -703,63 +759,85 @@ namespace URLNotesGrabberCORE
try { AddBlog(noteBlogName, false, DBPath); } catch { }
using SQLiteConnection connection2 = new SQLiteConnection("Data Source=" + DBPath);
int rowsInserted = 0;
try
{
connection2.Open();
// INSERT OR IGNORE, and no IsActive in the column list: re-crawling a note
// that was removed elsewhere leaves the existing row -- and its flag -- alone.
string sql = "INSERT OR IGNORE INTO Notes (rootBlogName, noteBlogName, PostID, TimeStamp, Type, DatetimeCrawled, DateModified, DateCreated) values(@rootBlogName, @noteBlogName, @PostID, @TimeStamp, @Type, @DatetimeCrawled, @DateModified, @DateCreated)";
using (SQLiteCommand command = new SQLiteCommand(sql, connection2))
// Notes stores integer IDs, so both participants and the type have to exist in
// their lookup table before the note can point at them.
//
// All four statements run in one transaction so a crash cannot leave a name or a
// type registered with no note. The transaction is committed before the console
// output below, which sleeps -- a write lock must not be held across that.
using (SQLiteTransaction transaction = connection2.BeginTransaction())
{
command.Parameters.AddWithValue("@rootBlogName", rootBlogName);
command.Parameters.AddWithValue("@noteBlogName", noteBlogName);
command.Parameters.AddWithValue("@PostID", postID);
command.Parameters.AddWithValue("@TimeStamp", timestamp);
command.Parameters.AddWithValue("@Type", type ?? string.Empty);
command.Parameters.AddWithValue("@DatetimeCrawled", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
command.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
command.Parameters.AddWithValue("@DateCreated", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
RegisterBlogName(connection2, transaction, rootBlogName);
RegisterBlogName(connection2, transaction, noteBlogName);
RegisterNoteType(connection2, transaction, type ?? string.Empty);
int rowsInserted = command.ExecuteNonQuery();
// INSERT OR IGNORE, and no IsActive in the column list: re-crawling a note
// that was removed elsewhere leaves the existing row -- and its flag -- alone.
string sql = "INSERT OR IGNORE INTO Notes (RootBlogId, NoteBlogId, PostID, TimeStamp, TypeId, DatetimeCrawled, DateModified, DateCreated) " +
"SELECT (SELECT BlogId FROM BlogNames WHERE BlogName = @rootBlogName), " +
" (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName), " +
" @PostID, @TimeStamp, " +
" (SELECT TypeId FROM NoteTypes WHERE Type = @Type), " +
" @DatetimeCrawled, @DateModified, @DateCreated";
if (rowsInserted == 1)
using (SQLiteCommand command = new SQLiteCommand(sql, connection2, transaction))
{
ConsoleColor previousColor = Console.ForegroundColor;
Console.ForegroundColor = ConsoleColor.Green;
Console.WriteLine("{2}\t{0}\t{1}", UnixTimeStampToDateTime(timestamp), noteBlogName, type);
Console.ForegroundColor = previousColor;
Thread.Sleep(250); // Brief pause to make new notes more noticeable in the console output
}
else
{
Console.WriteLine("{2}\t{0}\t{1}", UnixTimeStampToDateTime(timestamp), noteBlogName, type);
command.Parameters.AddWithValue("@rootBlogName", rootBlogName);
command.Parameters.AddWithValue("@noteBlogName", noteBlogName);
command.Parameters.AddWithValue("@PostID", postID);
command.Parameters.AddWithValue("@TimeStamp", timestamp);
command.Parameters.AddWithValue("@Type", type ?? string.Empty);
command.Parameters.AddWithValue("@DatetimeCrawled", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
command.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
command.Parameters.AddWithValue("@DateCreated", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
rowsInserted = command.ExecuteNonQuery();
}
// Only update HasBeenOutput if a new note was inserted
if (rowsInserted == 1)
transaction.Commit();
}
if (rowsInserted == 1)
{
ConsoleColor previousColor = Console.ForegroundColor;
Console.ForegroundColor = ConsoleColor.Green;
Console.WriteLine("{2}\t{0}\t{1}", UnixTimeStampToDateTime(timestamp), noteBlogName, type);
Console.ForegroundColor = previousColor;
Thread.Sleep(250); // Brief pause to make new notes more noticeable in the console output
}
else
{
Console.WriteLine("{2}\t{0}\t{1}", UnixTimeStampToDateTime(timestamp), noteBlogName, type);
}
// Only update HasBeenOutput if a new note was inserted
if (rowsInserted == 1)
{
try
{
try
// HasBeenOutput IS NULL still counts as a change: the selection queries
// test HasBeenOutput = 0, which a NULL would never match.
string updateSql = "UPDATE Blogs SET HasBeenOutput = 0, DateModified = @DateModified WHERE BlogName = @BlogName AND (HasBeenOutput IS NULL OR HasBeenOutput <> 0)";
using (var updateCommand = new SQLiteCommand(updateSql, connection2))
{
// HasBeenOutput IS NULL still counts as a change: the selection queries
// test HasBeenOutput = 0, which a NULL would never match.
string updateSql = "UPDATE Blogs SET HasBeenOutput = 0, DateModified = @DateModified WHERE BlogName = @BlogName AND (HasBeenOutput IS NULL OR HasBeenOutput <> 0)";
using (var updateCommand = new SQLiteCommand(updateSql, connection2))
{
updateCommand.Parameters.AddWithValue("@BlogName", noteBlogName);
updateCommand.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
updateCommand.ExecuteNonQuery();
}
updateCommand.Parameters.AddWithValue("@BlogName", noteBlogName);
updateCommand.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
updateCommand.ExecuteNonQuery();
}
catch { }
}
catch { }
}
}
catch (Exception ex)
{
// Breakpoint here
if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Notes.RootBlogName, Notes.PostID, Notes.TimeStamp, Notes.Type, Notes.NoteBlogName")
if (!IsNotesDuplicateKey(ex))
{
Console.WriteLine(ex.Message);
Console.WriteLine("^^^^^ - SHORTCUT");
@@ -850,6 +928,10 @@ namespace URLNotesGrabberCORE
}
else
{
// The LEFT OUTER JOIN to Notes that used to sit here has been dropped rather
// than ported. Nothing was selected from it, a LEFT JOIN cannot remove a row,
// and the GROUP BY below collapsed the rows it duplicated -- so it could not
// affect the result, and it cost a join against 1.18M rows on every pass.
sql = "SELECT " +
" MAX(Posts.BlogName) as BlogName, " + Environment.NewLine +
" Posts.PostID, " + Environment.NewLine +
@@ -859,8 +941,6 @@ namespace URLNotesGrabberCORE
"FROM " + Environment.NewLine +
" Posts " + Environment.NewLine +
" LEFT OUTER JOIN " + Environment.NewLine +
" Notes ON Notes.RootBlogName = Posts.BlogName AND Notes.PostID = Posts.PostID " + Environment.NewLine +
" LEFT OUTER JOIN " + Environment.NewLine +
" ( select BlogName, count(PostID) as CNT from Posts" + WhereIsActive("Posts", "", DBPath) + " group by BlogName) CNT on CNT.blogName = Posts.BlogName " +
"WHERE NotFound = 0 " + AndIsActive("Posts", "Posts", DBPath) + Environment.NewLine;
@@ -971,7 +1051,11 @@ namespace URLNotesGrabberCORE
{
connection.Open();
string sql = "SELECT distinct RootBlogName as blogName, postID FROM Notes WHERE Notes.type = 'reply'" + AndIsActive("Notes", "Notes", DBPath) + " order by RootBlogName, PostID";
string sql = "SELECT DISTINCT BN.BlogName as blogName, N.PostID" +
" FROM Notes N" +
" INNER JOIN BlogNames BN ON BN.BlogId = N.RootBlogId" +
" WHERE N.TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')" + AndIsActive("Notes", "N", DBPath) +
" ORDER BY BN.BlogName, N.PostID";
using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{
@@ -1011,12 +1095,15 @@ namespace URLNotesGrabberCORE
{
connection.Open();
string sql = @"SELECT DISTINCT Notes.RootBlogName as blogName, Notes.PostID,
MAX(Notes.timestamp) as LatestTimestamp
FROM Notes
WHERE Notes.type = 'reply'
AND (Notes.replyText IS NULL OR Notes.replyText = '' OR Notes.replyText = '.')" + AndIsActive("Notes", "Notes", DBPath) + @"
GROUP BY Notes.RootBlogName, Notes.PostID
// Grouped on the integer rather than the name: the group key is what gets sorted,
// and BN.BlogName comes along for free off the join.
string sql = @"SELECT BN.BlogName as blogName, N.PostID,
MAX(N.TimeStamp) as LatestTimestamp
FROM Notes N
INNER JOIN BlogNames BN ON BN.BlogId = N.RootBlogId
WHERE N.TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')
AND (N.replyText IS NULL OR N.replyText = '' OR N.replyText = '.')" + AndIsActive("Notes", "N", DBPath) + @"
GROUP BY N.RootBlogId, N.PostID
ORDER BY LatestTimestamp ASC
LIMIT @limit";
@@ -1058,11 +1145,15 @@ namespace URLNotesGrabberCORE
{
connection.Open();
string sql = @"SELECT DISTINCT P.BlogName, P.PostID, MAX(N.timestamp) as LatestTimestamp
// Posts carries only BlogName, so this is the one join to Notes that has to go
// through BlogNames -- there is no Posts.BlogId to hop on. The name predicate is
// pushed into the 20k-row lookup, which then feeds integers to the Notes key.
string sql = @"SELECT DISTINCT P.BlogName, P.PostID, MAX(N.TimeStamp) as LatestTimestamp
FROM Posts P
INNER JOIN Notes N ON N.PostID = P.PostID AND N.RootBlogName = P.BlogName
INNER JOIN BlogNames RBN ON RBN.BlogName = P.BlogName
INNER JOIN Notes N ON N.RootBlogId = RBN.BlogId AND N.PostID = P.PostID
WHERE P.NotFound = 0
AND N.type = 'reply'
AND N.TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply')
AND (N.replyText IS NULL OR N.replyText = '' OR N.replyText = '.')" + AndIsActive("Posts", "P", DBPath) + AndIsActive("Notes", "N", DBPath) + @"
GROUP BY P.BlogName, P.PostID
ORDER BY LatestTimestamp ASC";
@@ -1160,6 +1251,11 @@ namespace URLNotesGrabberCORE
{
connection.Open();
string sql;
// The two Notes branches below join on Blogs.BlogId, which is NULL for the 168k
// registry rows that have never appeared in a note. The inner join drops them,
// which is correct here -- both branches already require a note to exist -- but
// it is the wrong shape for anything that lists the registry.
if (!string.IsNullOrEmpty(specificBlog))
{
// Specific blog: always process, bypass cooldown
@@ -1179,12 +1275,12 @@ namespace URLNotesGrabberCORE
COALESCE(B.LikesCursor, 0),
COALESCE(B.LikesNewestTimestamp, 0)
FROM Blogs B
INNER JOIN Notes N ON N.NoteBlogName = B.BlogName
INNER JOIN Notes N ON N.NoteBlogId = B.BlogId
WHERE N.TimeStamp >= 1535778000
AND N.rootBlogName = B.BlogName
AND N.RootBlogId = B.BlogId
AND B.IsActive = 1" + AndIsActive("Notes", "N", DBPath) + @"
GROUP BY B.BlogName
ORDER BY MIN(N.Timestamp);";
ORDER BY MIN(N.TimeStamp);";
}
else
{
@@ -1194,9 +1290,9 @@ namespace URLNotesGrabberCORE
COALESCE(B.LikesCursor, 0),
COALESCE(B.LikesNewestTimestamp, 0)
FROM Blogs B
INNER JOIN Notes N ON N.NoteBlogName = B.BlogName
INNER JOIN Notes N ON N.NoteBlogId = B.BlogId
WHERE N.TimeStamp >= 1535778000
AND N.rootBlogName = B.BlogName
AND N.RootBlogId = B.BlogId
AND B.IsActive = 1" + AndIsActive("Notes", "N", DBPath) + @"
AND (
B.LikesPulled = 0
@@ -1204,7 +1300,7 @@ namespace URLNotesGrabberCORE
< (CAST(strftime('%s','now') AS INTEGER) - (@cooldownDays * 86400))
)
GROUP BY B.BlogName
ORDER BY MIN(N.Timestamp);";
ORDER BY MIN(N.TimeStamp);";
}
using (SQLiteCommand command = new SQLiteCommand(sql, connection))
@@ -1244,11 +1340,14 @@ namespace URLNotesGrabberCORE
try
{
connection.Open();
// Blogs is reached in one integer hop off Blogs.BlogId, not through BlogNames --
// that would add a hop and end in the text comparison the migration removed.
// The negated form is only correct because Notes.TypeId is NOT NULL.
string sql = "";
if (reblogsOnly)
sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive" + AndIsActive("Notes", "notes", DBPath) + " AND type IN ('reblog', 'reply', 'posted') AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top";
sql = "SELECT B.BlogName as blogName, count(*) FROM Notes N INNER JOIN Blogs B ON B.BlogId = N.NoteBlogId WHERE B.IsActive = @isActive" + AndIsActive("Notes", "N", DBPath) + " AND N.TypeId IN (SELECT TypeId FROM NoteTypes WHERE Type IN ('reblog', 'reply', 'posted')) AND B.HasBeenOutput = 0 GROUP BY N.NoteBlogId ORDER BY count(*) DESC, B.BlogName LIMIT @top";
else
sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive" + AndIsActive("Notes", "notes", DBPath) + " AND type NOT IN ('reblog', 'reply', 'posted') AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top";
sql = "SELECT B.BlogName as blogName, count(*) FROM Notes N INNER JOIN Blogs B ON B.BlogId = N.NoteBlogId WHERE B.IsActive = @isActive" + AndIsActive("Notes", "N", DBPath) + " AND N.TypeId NOT IN (SELECT TypeId FROM NoteTypes WHERE Type IN ('reblog', 'reply', 'posted')) AND B.HasBeenOutput = 0 GROUP BY N.NoteBlogId ORDER BY count(*) DESC, B.BlogName LIMIT @top";
using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{
@@ -1287,9 +1386,9 @@ namespace URLNotesGrabberCORE
connection.Open();
string sql = "";
if (reblogsOnly)
sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive" + AndIsActive("Notes", "notes", DBPath) + " AND type IN ('reblog', 'reply', 'posted') AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top";
sql = "SELECT B.BlogName as blogName, count(*) FROM Notes N INNER JOIN Blogs B ON B.BlogId = N.NoteBlogId WHERE B.IsActive = @isActive" + AndIsActive("Notes", "N", DBPath) + " AND N.TypeId IN (SELECT TypeId FROM NoteTypes WHERE Type IN ('reblog', 'reply', 'posted')) AND B.HasBeenOutput = 0 GROUP BY N.NoteBlogId ORDER BY count(*) DESC, B.BlogName LIMIT @top";
else
sql = "SELECT NoteBlogName as blogName, count(*) FROM notes INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName WHERE blogs.IsActive = @isActive" + AndIsActive("Notes", "notes", DBPath) + " AND HasBeenOutput = 0 GROUP BY NoteBlogName ORDER BY count(*) DESC, BlogName LIMIT @top";
sql = "SELECT B.BlogName as blogName, count(*) FROM Notes N INNER JOIN Blogs B ON B.BlogId = N.NoteBlogId WHERE B.IsActive = @isActive" + AndIsActive("Notes", "N", DBPath) + " AND B.HasBeenOutput = 0 GROUP BY N.NoteBlogId ORDER BY count(*) DESC, B.BlogName LIMIT @top";
using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{
@@ -1528,7 +1627,10 @@ namespace URLNotesGrabberCORE
{
connection.Open();
string sql = "UPDATE Notes SET timestamp = @timestamp, DateModified = @dateModified WHERE rootBlogName = @rootBlogName AND noteBlogName = @noteBlogName AND PostID = @postID AND IFNULL(timestamp, 0) <> @timestamp";
string sql = "UPDATE Notes SET TimeStamp = @timestamp, DateModified = @dateModified " +
"WHERE RootBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @rootBlogName) " +
"AND NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName) " +
"AND PostID = @postID AND IFNULL(TimeStamp, 0) <> @timestamp";
using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{
command.Parameters.AddWithValue("@timestamp", timestamp);
@@ -1542,7 +1644,7 @@ namespace URLNotesGrabberCORE
catch (Exception ex)
{
// Breakpoint here
if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Notes.RootBlogName, Notes.PostID, Notes.TimeStamp, Notes.Type, Notes.NoteBlogName")
if (!IsNotesDuplicateKey(ex))
{
Console.WriteLine(ex.Message);
Console.WriteLine("^^^^^ - SHORTCUT");
@@ -1828,10 +1930,15 @@ namespace URLNotesGrabberCORE
{
connection.Open();
//string sql = "UPDATE Notes SET replyText = @replyText WHERE rootBlogName = @rootBlogName AND PostID = @PostID AND noteBlogName = @noteBlogName AND TimeStamp = @TimeStamp AND Type = 'reply'";
// Match on (noteBlogName, TimeStamp ±5s) only - a reply by a given blog at a given timestamp is the same reply across the original post and every reblog of it, so this fans out across reblog chains in one shot. Tolerance absorbs the ~1s drift between what -collect stored and what mode=conversation returns now.
// Only fan out to rows that match the SELECT criteria in GetRepliesWithFilledText (NULL/empty/legacy-'.'). Never overwrite '?' (confirmed-empty) or already-fetched text.
string sql = "UPDATE Notes SET replyText = @replyText, DateModified = @dateModified WHERE noteBlogName = @noteBlogName AND ABS(TimeStamp - @TimeStamp) <= 5 AND Type = 'reply' AND (replyText IS NULL OR replyText = '' OR replyText = '.') AND (replyText IS NULL OR replyText <> @replyText)";
// The ABS() term cannot use an index on TimeStamp, before or after the integer schema; the NoteBlogId probe is what keeps this off a full scan.
string sql = "UPDATE Notes SET replyText = @replyText, DateModified = @dateModified " +
"WHERE NoteBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @noteBlogName) " +
"AND ABS(TimeStamp - @TimeStamp) <= 5 " +
"AND TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply') " +
"AND (replyText IS NULL OR replyText = '' OR replyText = '.') " +
"AND (replyText IS NULL OR replyText <> @replyText)";
using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{
command.Parameters.AddWithValue("@replyText", replyText ?? "?");
@@ -1872,7 +1979,11 @@ namespace URLNotesGrabberCORE
{
connection.Open();
string sql = "UPDATE Notes SET replyText = @replyText, DateModified = @dateModified WHERE rootBlogName = @rootBlogName AND PostID = @PostID AND Type = 'reply' AND IFNULL(replyText, '.') <> @replyText";
string sql = "UPDATE Notes SET replyText = @replyText, DateModified = @dateModified " +
"WHERE RootBlogId = (SELECT BlogId FROM BlogNames WHERE BlogName = @rootBlogName) " +
"AND PostID = @PostID " +
"AND TypeId = (SELECT TypeId FROM NoteTypes WHERE Type = 'reply') " +
"AND IFNULL(replyText, '.') <> @replyText";
using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{
command.Parameters.AddWithValue("@replyText", replyText ?? ".");