jim/URLNotesGrabberCore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: make --output and --updatepaths tell the truth about TTFolderPath

Browse Source 
--output iterated all 156k active Blogs rows and printed a "does not exist or is
not set" skip line for each, which is nearly every blog in the crawl registry --
only the few hundred downloaded locally ever have a folder. The signal was
buried in six figures of noise.

GetAllBlogsWithTTFolderPath now selects only active blogs carrying a non-empty
path, so --output processes export targets and nothing else. When none exist it
says so once, names the database it read, points at --updatepaths, and returns
non-zero instead of reporting success. A stored path this machine cannot see is
now reported separately from an unset one, with the path shown, because the two
are fixed in different places. Paths are trimmed before Directory.Exists, which
stray whitespace in a .tumblr FileDownloadLocation would otherwise defeat.

Both writers counted optimistically. UpdateBlogPathsRunner printed its
per-blog success line and incremented its total from the metadata file parsing,
never checking whether the UPDATE matched a row; LegacyPostsDbImporter counted a
blog as copied even when the legacy TTFolderPath was NULL. Either could report
full success having written nothing -- which is consistent with TL.db holding
zero populated paths across all 156,492 active blogs despite 20,679 posts having
merged. SetBlogTTFolderPath now returns whether a row changed, and both callers
report written / already-correct / no-matching-row separately.

Verified against a throwaway database: no-paths case, export case (stale .txt
rotated to .bak, per-PostType files, date-sorted), missing-folder case,
--updatepaths honest counts, and an idempotent rerun reporting already-correct.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jim
2026-08-05 12:02:49 -05:00
co-authored by Claude Opus 5
parent ef6629d86a
commit 721224bc13
4 changed files with 141 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
URLNotesGrabberCORE/UpdateBlogPathsRunner.cs
+39 -6
View File
@@ -32,6 +32,10 @@ namespace URLNotesGrabberCORE
Console.WriteLine($"Found {blogFiles.Count} blog metadata files");
int updatedCount = 0;
int unchangedCount = 0;
int noLocationCount = 0;
int noRowCount = 0;
int errorCount = 0;
foreach (var blogFile in blogFiles)
{
@@ -44,27 +48,56 @@ namespace URLNotesGrabberCORE
if (root.TryGetProperty("FileDownloadLocation", out JsonElement locationElement))
{
string? fileDownloadLocation = locationElement.GetString();
string? fileDownloadLocation = locationElement.GetString()?.Trim();
if (!string.IsNullOrWhiteSpace(fileDownloadLocation))
{
DataAccess.SetBlogTTFolderPath(blogName, fileDownloadLocation);
updatedCount++;
Console.WriteLine($"Updated {blogName}: {fileDownloadLocation}");
// Report the database's answer, not the fact that the file parsed.
if (DataAccess.SetBlogTTFolderPath(blogName, fileDownloadLocation))
{
updatedCount++;
Console.WriteLine($"Updated {blogName}: {fileDownloadLocation}");
}
else if (DataAccess.BlogExists(blogName))
{
unchangedCount++;
}
else
{
noRowCount++;
Console.WriteLine($"No Blogs row named '{blogName}' -- path not stored (name may differ in case)");
}
}
else
{
noLocationCount++;
Console.WriteLine($"Empty FileDownloadLocation in {blogFile}");
}
}
else
{
noLocationCount++;
Console.WriteLine($"No FileDownloadLocation found in {blogFile}");
}
}
catch (Exception ex)
{
errorCount++;
Console.WriteLine($"Error processing {blogFile}: {ex.Message}");
}
}
Console.WriteLine($"\nUpdated {updatedCount} blogs with TTFolderPath");
return 0;
Console.WriteLine($"\n========== UpdateBlogPaths summary ==========");
Console.WriteLine($"Metadata files: {blogFiles.Count}");
Console.WriteLine($"TTFolderPath written: {updatedCount}");
Console.WriteLine($"Already correct: {unchangedCount}");
Console.WriteLine($"No FileDownloadLocation: {noLocationCount}");
Console.WriteLine($"No matching blog row: {noRowCount}");
Console.WriteLine($"Errors: {errorCount}");
int stored = DataAccess.CountBlogsWithTTFolderPath();
Console.WriteLine($"\nBlogs now holding a TTFolderPath: {stored}");
return errorCount == 0 ? 0 : 2;
}
}
}