Author SHA1 Message Date
jim f541ec4260 fix: correctly detect rate-limit state for single-key API pools
IsAllRateLimited() short-circuited true for any pool with 0 or 1
keys, with minRetrySeconds left at 0 regardless of whether that key
was actually rate-limited. SleepUntilAnyAvailable() checks
"minRetry <= 0" to decide whether to skip sleeping, so with exactly
one key it always skipped the wait and let callers hammer the API
again immediately after a 429, even mid-cooldown.

The per-key loop already computes this correctly for any key count;
the special case only needs to cover the true no-keys edge case,
where there's nothing to wait on.
2026-06-30 21:22:12 -05:00
jim f549f020e1 refactor: standardize SQLiteConnection disposal via using; guard config
Replace the try/finally { connection.Close(); } pattern used across
most of DataAccess.cs with using declarations, so disposal happens
automatically and can't be skipped by a future edit that adds an
early return before the finally. Left the shared-connection
(ownsConnection) call sites alone since those intentionally outlive
a single method call.

Also drop a stray unused `using static ... JSType` import, and make
a missing ContainsList config setting fail with a clear
InvalidOperationException instead of a NullReferenceException from
Split(',') on null.
2026-06-30 20:54:18 -05:00
jim 0ff80a0fd3 fix: parameterize AddPost fallback UPDATE, guard args indexing
Posts.hasImage/DateModified fallback update built its WHERE clause via
raw string concatenation of blogName/postID, unlike every other query
in this method — a blog name containing a single quote would break or
inject into the query. Switch it to parameters.

--parse, --blogsO, and --bop indexed args[1..3] before checking
args.Length, so a missing argument threw IndexOutOfRangeException
instead of hitting the intended usage message.
2026-06-30 20:41:51 -05:00
jimandClaude Opus 4.8 4df73367fb BREAKING: switch all multi-char commands to POSIX --double-dash
Rename every multi-character option/command from single-dash to double-dash (--likes, --collect, --force, etc.) to follow the POSIX long-option convention. Single-character short options (-h, -V, -?) keep their single dash, as POSIX prescribes.

Breaking: existing invocations/scripts using single-dash forms now report Unknown Command and must be updated. Run profile (launchSettings.json) and CLI docs (copilot-instructions.md) updated to match.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-09 15:26:38 -05:00
jimandClaude Opus 4.8 a437fa87d3 Document -post and -bop commands in --help
These two commands were handled by the switch but never listed in help.
--help now covers every command the program accepts.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-09 15:19:51 -05:00
jimandClaude Opus 4.8 32a1583efd Document exit-status codes in --help output
The new 0/1/2 exit codes had no footprint in --help; add an Exit status
line so the documented behavior matches what the program now returns.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-09 15:18:25 -05:00
jimandClaude Opus 4.8 03676432bd Add POSIX-friendly CLI handling: --, --help/--version, exit codes
Keep the existing single-dash switch style and case-insensitive matching,
but add the cheap, non-breaking POSIX wins:

- `--` end-of-options: tokens after a bare `--` are treated as literal operands
- `--help`/`-h` (alongside `-?`) and `-V`/`--version`
- Main returns a real exit code: 2 for usage errors, propagates handler
  return codes, and a top-level catch yields a quiet 1 on unhandled errors

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-09 15:12:10 -05:00
4 changed files with 206 additions and 251 deletions
+9 -9
View File
@@ -22,18 +22,18 @@ This document provides essential context for AI agents working with URLNotesGrab
```powershell ```powershell
dotnet build dotnet build
dotnet run # Process all files in input directory dotnet run # Process all files in input directory
dotnet run -- -parse [blogname] # Process specific blog dotnet run -- --parse [blogname] # Process specific blog
dotnet run -- -test [blogname] [postID] # Test API for specific post dotnet run -- --test [blogname] [postID] # Test API for specific post
``` ```
### Command-Line Interface ### Command-Line Interface
- `-parse [blogname]`: Parse text files for specific blog - `--parse [blogname]`: Parse text files for specific blog
- `-test [blogname] [postID]`: Test API note collection - `--test [blogname] [postID]`: Test API note collection
- `-posts`: Export post blogs to file - `--posts`: Export post blogs to file
- `-blogs`: Export blog list to file - `--blogs`: Export blog list to file
- `-collect`: Collect notes for all posts in DB - `--collect`: Collect notes for all posts in DB
- `-blogsR`: Export reply blogs to file - `--blogsR`: Export reply blogs to file
- `-blogsO [start] [stop]`: Export blogs within range - `--blogsO [start] [stop]`: Export blogs within range
## Project Conventions ## Project Conventions
+33 -134
View File
@@ -139,7 +139,7 @@ namespace URLNotesGrabberCORE
if (_savedImportPragmas != null) if (_savedImportPragmas != null)
return; return;
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
connection.Open(); connection.Open();
@@ -175,10 +175,6 @@ namespace URLNotesGrabberCORE
{ {
Console.WriteLine($"[SQLite Import Mode] Failed to enable import pragmas: {ex.Message}"); Console.WriteLine($"[SQLite Import Mode] Failed to enable import pragmas: {ex.Message}");
} }
finally
{
connection.Close();
}
} }
} }
@@ -190,7 +186,7 @@ namespace URLNotesGrabberCORE
if (_savedImportPragmas == null) if (_savedImportPragmas == null)
return; return;
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
connection.Open(); connection.Open();
@@ -217,7 +213,6 @@ namespace URLNotesGrabberCORE
finally finally
{ {
_savedImportPragmas = null; _savedImportPragmas = null;
connection.Close();
} }
} }
} }
@@ -252,7 +247,7 @@ namespace URLNotesGrabberCORE
public static void EnsureReplyTextColumnExists(string? DBPath = null) public static void EnsureReplyTextColumnExists(string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
@@ -296,16 +291,12 @@ namespace URLNotesGrabberCORE
// Breakpoint here // Breakpoint here
Console.WriteLine($"Error checking/creating replyText column: {ex.Message}"); Console.WriteLine($"Error checking/creating replyText column: {ex.Message}");
} }
finally
{
connection.Close();
}
} }
public static void EnsureBlogsLikesColumnsExist(string? DBPath = null) public static void EnsureBlogsLikesColumnsExist(string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
@@ -391,10 +382,6 @@ namespace URLNotesGrabberCORE
{ {
Console.WriteLine($"Error mapping Blogs likes columns: {ex.Message}"); Console.WriteLine($"Error mapping Blogs likes columns: {ex.Message}");
} }
finally
{
connection.Close();
}
} }
#region Adds #region Adds
@@ -518,8 +505,12 @@ namespace URLNotesGrabberCORE
{ {
if (ownsConnection) connection.Open(); if (ownsConnection) connection.Open();
string updateSql = "UPDATE Posts SET hasImage = " + (hasImage ? 1 : 0) + ", DateModified = '" + DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + "' WHERE blogName = '" + blogName + "' AND postID = '" + postID + "'"; string updateSql = "UPDATE Posts SET hasImage = @hasImage, DateModified = @DateModified WHERE blogName = @blogName AND postID = @postID";
SQLiteCommand updateCommand = new SQLiteCommand(updateSql, connection); using SQLiteCommand updateCommand = new SQLiteCommand(updateSql, connection);
updateCommand.Parameters.AddWithValue("@hasImage", hasImage ? 1 : 0);
updateCommand.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
updateCommand.Parameters.AddWithValue("@blogName", blogName);
updateCommand.Parameters.AddWithValue("@postID", postID);
updateCommand.ExecuteNonQuery(); updateCommand.ExecuteNonQuery();
} }
@@ -565,7 +556,7 @@ namespace URLNotesGrabberCORE
public static void AddAPICount(string? DBPath = null) public static void AddAPICount(string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
@@ -586,10 +577,6 @@ namespace URLNotesGrabberCORE
// Breakpoint here // Breakpoint here
//Console.WriteLine(ex.Message); //Console.WriteLine(ex.Message);
} }
finally
{
connection.Close();
}
} }
public static bool AddNote(string rootBlogName, string noteBlogName, long postID, long timestamp, string type, string? DBPath = null) public static bool AddNote(string rootBlogName, string noteBlogName, long postID, long timestamp, string type, string? DBPath = null)
@@ -598,7 +585,7 @@ namespace URLNotesGrabberCORE
//try { AddPost(rootBlogName, postID, DBPath); } catch { } //try { AddPost(rootBlogName, postID, DBPath); } catch { }
try { AddBlog(noteBlogName, false, DBPath); } catch { } try { AddBlog(noteBlogName, false, DBPath); } catch { }
SQLiteConnection connection2 = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection2 = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
@@ -657,10 +644,6 @@ namespace URLNotesGrabberCORE
Console.WriteLine("^^^^^ - SHORTCUT"); Console.WriteLine("^^^^^ - SHORTCUT");
} }
} }
finally
{
connection2.Close();
}
return false; return false;
} }
#endregion Adds #endregion Adds
@@ -676,7 +659,7 @@ namespace URLNotesGrabberCORE
public static List<Tuple<string, long, long, long>> GetPosts(bool withoutNotesOnly = false, DateTime? beforeDate = null, string? DBPath = null) public static List<Tuple<string, long, long, long>> GetPosts(bool withoutNotesOnly = false, DateTime? beforeDate = null, string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
List<Tuple<string, long, long, long>> posts = new List<Tuple<string, long, long, long>>(); List<Tuple<string, long, long, long>> posts = new List<Tuple<string, long, long, long>>();
try try
@@ -854,17 +837,13 @@ namespace URLNotesGrabberCORE
if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID") if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
Console.WriteLine(ex.Message); Console.WriteLine(ex.Message);
} }
finally
{
connection.Close();
}
return posts; return posts;
} }
public static List<Tuple<string, long>> GetReplies(string? DBPath = null) public static List<Tuple<string, long>> GetReplies(string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
List<Tuple<string, long>> posts = new List<Tuple<string, long>>(); List<Tuple<string, long>> posts = new List<Tuple<string, long>>();
try try
@@ -898,17 +877,13 @@ namespace URLNotesGrabberCORE
if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID") if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
Console.WriteLine(ex.Message); Console.WriteLine(ex.Message);
} }
finally
{
connection.Close();
}
return posts; return posts;
} }
public static List<Tuple<string, long>> GetRepliesWithMissingText(string? DBPath = null, int limit = 50) public static List<Tuple<string, long>> GetRepliesWithMissingText(string? DBPath = null, int limit = 50)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
List<Tuple<string, long>> posts = new List<Tuple<string, long>>(); List<Tuple<string, long>> posts = new List<Tuple<string, long>>();
try try
@@ -949,17 +924,13 @@ namespace URLNotesGrabberCORE
// Breakpoint here // Breakpoint here
Console.WriteLine($"Error getting replies with missing text: {ex.Message}"); Console.WriteLine($"Error getting replies with missing text: {ex.Message}");
} }
finally
{
connection.Close();
}
return posts; return posts;
} }
public static List<Tuple<string, long, long>> GetRepliesWithFilledText(string? DBPath = null, int? limit = null) public static List<Tuple<string, long, long>> GetRepliesWithFilledText(string? DBPath = null, int? limit = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
List<Tuple<string, long, long>> posts = new List<Tuple<string, long, long>>(); List<Tuple<string, long, long>> posts = new List<Tuple<string, long, long>>();
try try
@@ -1011,17 +982,13 @@ namespace URLNotesGrabberCORE
// Breakpoint here // Breakpoint here
Console.WriteLine($"Error getting replies with filled text: {ex.Message}"); Console.WriteLine($"Error getting replies with filled text: {ex.Message}");
} }
finally
{
connection.Close();
}
return posts; return posts;
} }
public static int GetAPICount(string? DBPath = null) public static int GetAPICount(string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
int count = 0; int count = 0;
try { AddAPICount(); } catch { } try { AddAPICount(); } catch { }
@@ -1050,17 +1017,13 @@ namespace URLNotesGrabberCORE
if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID") if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
Console.WriteLine(ex.Message); Console.WriteLine(ex.Message);
} }
finally
{
connection.Close();
}
return count; return count;
} }
public static List<Tuple<string, int, long, long>> GetBlogsForLikes(string specificBlog = null, int cooldownDays = 7, bool ignoreCooldown = false, string? DBPath = null) public static List<Tuple<string, int, long, long>> GetBlogsForLikes(string specificBlog = null, int cooldownDays = 7, bool ignoreCooldown = false, string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
List<Tuple<string, int, long, long>> blogs = new List<Tuple<string, int, long, long>>(); List<Tuple<string, int, long, long>> blogs = new List<Tuple<string, int, long, long>>();
try try
@@ -1136,17 +1099,13 @@ namespace URLNotesGrabberCORE
{ {
Console.WriteLine($"Error fetching blogs for likes: {ex.Message}"); Console.WriteLine($"Error fetching blogs for likes: {ex.Message}");
} }
finally
{
connection.Close();
}
return blogs; return blogs;
} }
public static List<string> GetBlogs(bool reblogsOnly, int from, int to, int top, string? DBPath = null) public static List<string> GetBlogs(bool reblogsOnly, int from, int to, int top, string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
List<string> blogs = new List<string>(); List<string> blogs = new List<string>();
try try
@@ -1181,17 +1140,13 @@ namespace URLNotesGrabberCORE
if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID") if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
Console.WriteLine(ex.Message); Console.WriteLine(ex.Message);
} }
finally
{
connection.Close();
}
return blogs; return blogs;
} }
public static List<string> GetBlogsAll(bool reblogsOnly, int from, int to, int top, string? DBPath = null) public static List<string> GetBlogsAll(bool reblogsOnly, int from, int to, int top, string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
List<string> blogs = new List<string>(); List<string> blogs = new List<string>();
try try
@@ -1226,19 +1181,13 @@ namespace URLNotesGrabberCORE
if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID") if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
Console.WriteLine(ex.Message); Console.WriteLine(ex.Message);
} }
finally
{
connection.Close();
}
return blogs; return blogs;
} }
public static IEnumerable<List<string>> GetAllPostTextColumns(string? DBPath = null) public static IEnumerable<List<string>> GetAllPostTextColumns(string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
connection.Open(); connection.Open();
try
{
string sql = "SELECT BlogName, reblogURL, PostURL, Slug, ReblogKey, ReblogName, Summary, Quote, Body, Tags, Link, PhotoURL, PhotoCaption, DownloadedFiles, AudioCaption, Question, Answer, Title, RootBlogName, RootURL FROM Posts WHERE IFNULL(DownloadedFiles, '.') = '.'"; string sql = "SELECT BlogName, reblogURL, PostURL, Slug, ReblogKey, ReblogName, Summary, Quote, Body, Tags, Link, PhotoURL, PhotoCaption, DownloadedFiles, AudioCaption, Question, Answer, Title, RootBlogName, RootURL FROM Posts WHERE IFNULL(DownloadedFiles, '.') = '.'";
using (SQLiteCommand command = new SQLiteCommand(sql, connection)) using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{ {
@@ -1266,12 +1215,6 @@ namespace URLNotesGrabberCORE
} }
} }
} }
finally
{
connection.Close();
connection.Dispose();
}
}
#endregion Gets #endregion Gets
#region Updates #region Updates
@@ -1280,7 +1223,7 @@ namespace URLNotesGrabberCORE
public static void UpdatePostMarkNotesCollected(string blogName, long postID, string? DBPath = null) public static void UpdatePostMarkNotesCollected(string blogName, long postID, string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
@@ -1303,16 +1246,12 @@ namespace URLNotesGrabberCORE
if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID") if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
Console.WriteLine(ex.Message); Console.WriteLine(ex.Message);
} }
finally
{
connection.Close();
}
} }
public static void UpdatePostMarkNotFound(string blogName, long postID, string? DBPath = null) public static void UpdatePostMarkNotFound(string blogName, long postID, string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
@@ -1336,10 +1275,6 @@ namespace URLNotesGrabberCORE
if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID") if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
Console.WriteLine(ex.Message); Console.WriteLine(ex.Message);
} }
finally
{
connection.Close();
}
} }
// ----- CollectRunState: tracks the frozen cutoff + completion flag for a managed "-collect 0" full re-check run ----- // ----- CollectRunState: tracks the frozen cutoff + completion flag for a managed "-collect 0" full re-check run -----
@@ -1451,7 +1386,7 @@ namespace URLNotesGrabberCORE
Console.WriteLine("{2}\t{0}\t{1}", UnixTimeStampToDateTime(timestamp), noteBlogName, type); Console.WriteLine("{2}\t{0}\t{1}", UnixTimeStampToDateTime(timestamp), noteBlogName, type);
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
@@ -1478,10 +1413,6 @@ namespace URLNotesGrabberCORE
return true; return true;
} }
} }
finally
{
connection.Close();
}
return false; return false;
} }
@@ -1597,7 +1528,7 @@ namespace URLNotesGrabberCORE
public static void UpdateBlogOutput(string blogName, string? DBPath = null) public static void UpdateBlogOutput(string blogName, string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
@@ -1617,16 +1548,12 @@ namespace URLNotesGrabberCORE
if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID") if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
Console.WriteLine(ex.Message); Console.WriteLine(ex.Message);
} }
finally
{
connection.Close();
}
} }
public static void UpdateBlogLikesStatus(string blogName, int likesPulled, long likesCursor, string? DBPath = null) public static void UpdateBlogLikesStatus(string blogName, int likesPulled, long likesCursor, string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
connection.Open(); connection.Open();
@@ -1644,10 +1571,6 @@ namespace URLNotesGrabberCORE
{ {
Console.WriteLine($"Error updating blog likes status: {ex.Message}"); Console.WriteLine($"Error updating blog likes status: {ex.Message}");
} }
finally
{
connection.Close();
}
} }
// Bumps the high-water mark for a blog. Used during Branch A (initial backfill) when we // Bumps the high-water mark for a blog. Used during Branch A (initial backfill) when we
@@ -1657,7 +1580,7 @@ namespace URLNotesGrabberCORE
{ {
if (newestTimestamp <= 0) return; if (newestTimestamp <= 0) return;
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
connection.Open(); connection.Open();
@@ -1677,10 +1600,6 @@ namespace URLNotesGrabberCORE
{ {
Console.WriteLine($"Error updating blog likes newest timestamp: {ex.Message}"); Console.WriteLine($"Error updating blog likes newest timestamp: {ex.Message}");
} }
finally
{
connection.Close();
}
} }
// Called at the end of a refresh pass (Branch B). Bumps the high-water mark, stamps the // Called at the end of a refresh pass (Branch B). Bumps the high-water mark, stamps the
@@ -1688,7 +1607,7 @@ namespace URLNotesGrabberCORE
public static void UpdateBlogLikesRefreshStatus(string blogName, long newestTimestamp, int newCount, string? DBPath = null) public static void UpdateBlogLikesRefreshStatus(string blogName, long newestTimestamp, int newCount, string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
connection.Open(); connection.Open();
@@ -1711,16 +1630,12 @@ namespace URLNotesGrabberCORE
{ {
Console.WriteLine($"Error updating blog likes refresh status: {ex.Message}"); Console.WriteLine($"Error updating blog likes refresh status: {ex.Message}");
} }
finally
{
connection.Close();
}
} }
public static int UpdateAPICount(string? DBPath = null) public static int UpdateAPICount(string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
int APICount = DataAccess.GetAPICount(); int APICount = DataAccess.GetAPICount();
APICount++; APICount++;
@@ -1743,10 +1658,6 @@ namespace URLNotesGrabberCORE
if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID") if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
Console.WriteLine(ex.Message); Console.WriteLine(ex.Message);
} }
finally
{
connection.Close();
}
return APICount; return APICount;
} }
@@ -1754,7 +1665,7 @@ namespace URLNotesGrabberCORE
public static int UpdateNoteReplyText(string rootBlogName, long postID, string noteBlogName, long timestamp, string replyText, string? DBPath = null) public static int UpdateNoteReplyText(string rootBlogName, long postID, string noteBlogName, long timestamp, string replyText, string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
int rowsAffected = 0; int rowsAffected = 0;
// A bare "." collides with the "needs processing" sentinel in GetRepliesWithFilledText, which would loop the post forever. Store as ". " so the data is preserved but no longer matches the sentinel. // A bare "." collides with the "needs processing" sentinel in GetRepliesWithFilledText, which would loop the post forever. Store as ". " so the data is preserved but no longer matches the sentinel.
@@ -1797,17 +1708,13 @@ namespace URLNotesGrabberCORE
Console.WriteLine($"[UpdateNoteReplyText] Error updating reply text: {ex.Message}"); Console.WriteLine($"[UpdateNoteReplyText] Error updating reply text: {ex.Message}");
Console.WriteLine($"[UpdateNoteReplyText] StackTrace: {ex.StackTrace}"); Console.WriteLine($"[UpdateNoteReplyText] StackTrace: {ex.StackTrace}");
} }
finally
{
connection.Close();
}
return rowsAffected; return rowsAffected;
} }
public static int UpdateAllNoteReplyTextForPost(string rootBlogName, long postID, string replyText, string? DBPath = null) public static int UpdateAllNoteReplyTextForPost(string rootBlogName, long postID, string replyText, string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
@@ -1843,10 +1750,6 @@ namespace URLNotesGrabberCORE
Console.WriteLine($"[UpdateAllNoteReplyTextForPost] StackTrace: {ex.StackTrace}"); Console.WriteLine($"[UpdateAllNoteReplyTextForPost] StackTrace: {ex.StackTrace}");
return 0; return 0;
} }
finally
{
connection.Close();
}
} }
#endregion Updates #endregion Updates
@@ -1857,7 +1760,7 @@ namespace URLNotesGrabberCORE
public static void EnsureTTFileHelperColumnsExist(string? DBPath = null) public static void EnsureTTFileHelperColumnsExist(string? DBPath = null)
{ {
DBPath ??= GetDefaultDbPath(); DBPath ??= GetDefaultDbPath();
SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath); using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
try try
{ {
@@ -1903,10 +1806,6 @@ namespace URLNotesGrabberCORE
{ {
Console.WriteLine($"Error ensuring TTFileHelper columns: {ex.Message}"); Console.WriteLine($"Error ensuring TTFileHelper columns: {ex.Message}");
} }
finally
{
connection.Close();
}
} }
// INSERT-or-UPDATE for a post arriving from a Tumblr text-file export. // INSERT-or-UPDATE for a post arriving from a Tumblr text-file export.
@@ -2727,7 +2626,7 @@ namespace URLNotesGrabberCORE
{ {
minRetrySeconds = 0; minRetrySeconds = 0;
if (!_usePool || _overrideKey != null) return false; if (!_usePool || _overrideKey != null) return false;
if (_keys.Count <= 1) return true; if (_keys.Count == 0) return false;
var now = DateTimeOffset.UtcNow.ToUnixTimeSeconds(); var now = DateTimeOffset.UtcNow.ToUnixTimeSeconds();
bool all = true; bool all = true;
+148 -92
View File
@@ -5,7 +5,6 @@ using Microsoft.Extensions.Configuration;
using System.Configuration; using System.Configuration;
using System.Threading; using System.Threading;
using Microsoft.Extensions.Diagnostics.Latency; using Microsoft.Extensions.Diagnostics.Latency;
using static System.Runtime.InteropServices.JavaScript.JSType;
using System.Text.RegularExpressions; using System.Text.RegularExpressions;
namespace URLNotesGrabberCORE namespace URLNotesGrabberCORE
@@ -13,12 +12,27 @@ namespace URLNotesGrabberCORE
internal class Program internal class Program
{ {
static void Main(string[] args) static int Main(string[] args)
{ {
// Reset console color on exit (including Ctrl+C) // Reset console color on exit (including Ctrl+C)
Console.CancelKeyPress += (s, e) => Console.ResetColor(); Console.CancelKeyPress += (s, e) => Console.ResetColor();
AppDomain.CurrentDomain.ProcessExit += (s, e) => Console.ResetColor(); AppDomain.CurrentDomain.ProcessExit += (s, e) => Console.ResetColor();
try
{
return Run(args);
}
catch (Exception ex)
{
// Turn configuration errors / unguarded indexers into a quiet, deterministic exit code.
Console.Error.WriteLine(ex.Message);
return 1;
}
}
static int Run(string[] args)
{
int exitCode = 0;
IConfiguration config = new ConfigurationBuilder() IConfiguration config = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory()) .SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: true, reloadOnChange: false) .AddJsonFile("appsettings.json", optional: true, reloadOnChange: false)
@@ -33,31 +47,34 @@ namespace URLNotesGrabberCORE
List<string> filteredArgs = new List<string>(); List<string> filteredArgs = new List<string>();
for (int i = 0; i < args.Length; i++) for (int i = 0; i < args.Length; i++)
{ {
if (string.Equals(args[i], "-force", StringComparison.OrdinalIgnoreCase) || if (args[i] == "--")
string.Equals(args[i], "--force", StringComparison.OrdinalIgnoreCase)) {
// POSIX end-of-options: everything after is a literal operand.
for (int j = i + 1; j < args.Length; j++) filteredArgs.Add(args[j]);
break;
}
if (string.Equals(args[i], "--force", StringComparison.OrdinalIgnoreCase))
{ {
forceIgnoreCooldown = true; forceIgnoreCooldown = true;
continue; continue;
} }
if (string.Equals(args[i], "-api3", StringComparison.OrdinalIgnoreCase) || if (string.Equals(args[i], "--api3", StringComparison.OrdinalIgnoreCase))
string.Equals(args[i], "--api3", StringComparison.OrdinalIgnoreCase))
{ {
apiSectionName = "TumblrApi3"; apiSectionName = "TumblrApi3";
apiExplicitlySet = true; apiExplicitlySet = true;
continue; continue;
} }
if (string.Equals(args[i], "-api4", StringComparison.OrdinalIgnoreCase) || if (string.Equals(args[i], "--api4", StringComparison.OrdinalIgnoreCase))
string.Equals(args[i], "--api4", StringComparison.OrdinalIgnoreCase))
{ {
apiSectionName = "TumblrApi4"; apiSectionName = "TumblrApi4";
apiExplicitlySet = true; apiExplicitlySet = true;
continue; continue;
} }
if (string.Equals(args[i], "-api", StringComparison.OrdinalIgnoreCase) || if (string.Equals(args[i], "--api", StringComparison.OrdinalIgnoreCase))
string.Equals(args[i], "--api", StringComparison.OrdinalIgnoreCase))
{ {
if (i + 1 < args.Length && !string.IsNullOrWhiteSpace(args[i + 1])) if (i + 1 < args.Length && !string.IsNullOrWhiteSpace(args[i + 1]))
{ {
@@ -67,13 +84,12 @@ namespace URLNotesGrabberCORE
} }
else else
{ {
Console.WriteLine("--Missing API section after -api/--api. Using default TumblrApi.--"); Console.WriteLine("--Missing API section after --api. Using default TumblrApi.--");
} }
continue; continue;
} }
if (string.Equals(args[i], "-start", StringComparison.OrdinalIgnoreCase) || if (string.Equals(args[i], "--start", StringComparison.OrdinalIgnoreCase))
string.Equals(args[i], "--start", StringComparison.OrdinalIgnoreCase))
{ {
if (i + 1 < args.Length && !string.IsNullOrWhiteSpace(args[i + 1])) if (i + 1 < args.Length && !string.IsNullOrWhiteSpace(args[i + 1]))
{ {
@@ -82,7 +98,7 @@ namespace URLNotesGrabberCORE
} }
else else
{ {
Console.WriteLine("--Missing blog name after -start/--start. Ignoring.--"); Console.WriteLine("--Missing blog name after --start. Ignoring.--");
} }
continue; continue;
} }
@@ -123,7 +139,10 @@ namespace URLNotesGrabberCORE
Console.SetOut(dualLogger); Console.SetOut(dualLogger);
} }
List<string> contains = settings.GetValue<string>("ContainsList").Split(',').ToList(); string? containsListSetting = settings.GetValue<string>("ContainsList");
if (string.IsNullOrEmpty(containsListSetting))
throw new InvalidOperationException("ContainsList is not configured in appsettings.json");
List<string> contains = containsListSetting.Split(',').ToList();
bool logTraversalRecordImports = settings.GetValue("LogTraversalRecordImports", false); bool logTraversalRecordImports = settings.GetValue("LogTraversalRecordImports", false);
if (args.Length == 0) //Traverse folder structure to add posts and thus blogs to DB if (args.Length == 0) //Traverse folder structure to add posts and thus blogs to DB
@@ -147,57 +166,23 @@ namespace URLNotesGrabberCORE
switch (args[0]) switch (args[0])
{ {
case "-?": case "-?":
Console.WriteLine("\t Parse .txt files to find blogs"); case "-h":
case "--help":
Console.WriteLine("-?\t Usage help"); PrintHelp();
Console.WriteLine("-parse\t Parse .txt files with specified blogname");
Console.WriteLine("-test\t Calls API for given blogname and postID");
Console.WriteLine("-posts\t For each Post in DB, write blogname to file");
Console.WriteLine("-blogs\t For each Blog in DB, write blogname to file");
Console.WriteLine("-collect [0|1] [datetime]\t Collect Notes from API. 1=only posts without notes. 0=full re-check of all posts: a single resumable pass (interrupt & relaunch to resume; stops when complete, retrigger for a new pass). Optional datetime overrides the cutoff and runs as a one-off (bypasses resume tracking).");
Console.WriteLine("-blogsR\t For each Note that is a REPLY, write blogname to file ");
Console.WriteLine("-blogsO\t For each Blog in DB, write blogname to file, but limit via a passed start and stop range ");
Console.WriteLine("-replies\t Fetch and update missing reply text for all replies in database");
Console.WriteLine("-likes\t Fetch likes: initial backfill for new blogs, incremental refresh for blogs past cooldown. Optional blog name forces single-blog run.");
Console.WriteLine("-force\t (with -likes) Ignore cooldown and refresh every fully-backfilled blog");
Console.WriteLine("-urldump\t Scan all posts' text columns and extract suspected URLs to configured file");
Console.WriteLine("-api3\t Use TumblrApi3 settings from appsettings.json");
Console.WriteLine("-api4\t Use TumblrApi4 settings from appsettings.json");
Console.WriteLine("-start [blogname]\t Start traversal alphabetically at this blog name");
Console.WriteLine("-api [section]\t Use a specific API settings section from appsettings.json (e.g. TumblrApi3)");
Console.WriteLine("-ingest [blogname]\t Ingest Tumblr .txt exports from appSettings:PathTTRoot into TL.db (all blogs, or single blog if name given)");
Console.WriteLine("-output\t Export posts from TL.db back to .txt files in each blog's TTFolderPath");
Console.WriteLine("-revert [blogname]\t Recursively scan the PathInput tree and restore *.bak back to *.txt (current .txt saved as next-free .bkN); optional blogname filters by path substring");
Console.WriteLine("-correct [bakPath]\t Dry-run: report multi-line field updates available from a BAK directory");
Console.WriteLine("-correct -apply [bakPath]\t Apply BAK-file corrections to matching posts (prompts yes/no)");
Console.WriteLine("-updatepaths [rootPath]\t Read .tumblr/.tmblrpriv metadata from <root>\\Index and set Blogs.TTFolderPath");
Console.WriteLine("-importposts [path-to-posts.db]\t One-time migration: copy legacy ThreeTxtFileHelper posts.db rows into TL.db");
break; break;
case "-parse": case "-V":
case "--version":
Console.WriteLine(System.Reflection.Assembly.GetExecutingAssembly().GetName().Version?.ToString() ?? "unknown");
break;
case "--parse":
if (args.Length < 2)
{
Console.WriteLine("Usage: --parse <blogname>");
exitCode = 2;
break;
}
string blogNameToParse = args[1]; string blogNameToParse = args[1];
int postsAdded = 0; int postsAdded = 0;
try try
@@ -212,23 +197,23 @@ namespace URLNotesGrabberCORE
Console.WriteLine($"Total posts added: {postsAdded}"); Console.WriteLine($"Total posts added: {postsAdded}");
break; break;
case "-test": case "--test":
Console.WriteLine("Test command not implemented"); Console.WriteLine("Test command not implemented");
break; break;
case "-post": case "--post":
TraverseDirectoryForCorruption(settings.GetValue<string>("PathInput"), settings.GetValue<string>("PathOutputBlogs"), contains); TraverseDirectoryForCorruption(settings.GetValue<string>("PathInput"), settings.GetValue<string>("PathOutputBlogs"), contains);
break; break;
case "-posts": //write post's blogs to file case "--posts": //write post's blogs to file
WritePostBlogsToFile(settings.GetValue<string>("PathOutputPosts")); WritePostBlogsToFile(settings.GetValue<string>("PathOutputPosts"));
break; break;
case "-blogs": //write blogs to file case "--blogs": //write blogs to file
WriteBlogsToFile(settings.GetValue<string>("PathOutputBlogs")); WriteBlogsToFile(settings.GetValue<string>("PathOutputBlogs"));
break; break;
case "-collect": //collect notes from all posts case "--collect": //collect notes from all posts
bool withoutNotesOnly = true; bool withoutNotesOnly = true;
DateTime? beforeDate = DateTime.Now; DateTime? beforeDate = DateTime.Now;
bool explicitDateSupplied = false; bool explicitDateSupplied = false;
@@ -236,6 +221,7 @@ namespace URLNotesGrabberCORE
if (args.Length < 2) if (args.Length < 2)
{ {
Console.WriteLine("--Expected WITHOUTNOTESONLY (0, 1) [OPTIONAL: BEFOREDATE]--"); Console.WriteLine("--Expected WITHOUTNOTESONLY (0, 1) [OPTIONAL: BEFOREDATE]--");
exitCode = 2;
break; break;
} }
@@ -268,6 +254,7 @@ namespace URLNotesGrabberCORE
else else
{ {
Console.WriteLine($"ERROR: Invalid date format '{args[2]}'"); Console.WriteLine($"ERROR: Invalid date format '{args[2]}'");
exitCode = 2;
break; break;
} }
} }
@@ -297,14 +284,14 @@ namespace URLNotesGrabberCORE
CollectNotes(settings.GetValue<string>("PathOutput"), withoutNotesOnly, beforeDate, managedCollectRun).GetAwaiter().GetResult(); CollectNotes(settings.GetValue<string>("PathOutput"), withoutNotesOnly, beforeDate, managedCollectRun).GetAwaiter().GetResult();
break; break;
case "-blogsR": //collect notes from all posts case "--blogsR": //collect notes from all posts
WriteBlogsToFile(settings.GetValue<string>("PathOutputBlogs"), true); WriteBlogsToFile(settings.GetValue<string>("PathOutputBlogs"), true);
break; break;
case "-blogsO": //collect notes from all posts case "--blogsO": //collect notes from all posts
int from = 1, to = 999999, top = 100; int from = 1, to = 999999, top = 100;
if (args[1] is not null && args[2] is not null && args[3] is not null) if (args.Length >= 4 && args[1] is not null && args[2] is not null && args[3] is not null)
{ {
from = int.Parse(args[1]); from = int.Parse(args[1]);
to = int.Parse(args[2]); to = int.Parse(args[2]);
@@ -313,14 +300,16 @@ namespace URLNotesGrabberCORE
else else
{ {
Console.WriteLine("--Expected FROM TO--"); Console.WriteLine("--Expected FROM TO--");
exitCode = 2;
break;
} }
WriteBlogsToFile(settings.GetValue<string>("PathOutputBlogs"), false, from, to, top); WriteBlogsToFile(settings.GetValue<string>("PathOutputBlogs"), false, from, to, top);
break; break;
case "-bop": //collect notes from all posts case "--bop": //collect notes from all posts
from = 1; to = 999999; top = 100; from = 1; to = 999999; top = 100;
if (args[1] is not null && args[2] is not null && args[3] is not null) if (args.Length >= 4 && args[1] is not null && args[2] is not null && args[3] is not null)
{ {
from = int.Parse(args[1]); from = int.Parse(args[1]);
to = int.Parse(args[2]); to = int.Parse(args[2]);
@@ -329,66 +318,70 @@ namespace URLNotesGrabberCORE
else else
{ {
Console.WriteLine("--Expected FROM TO--"); Console.WriteLine("--Expected FROM TO--");
exitCode = 2;
break;
} }
WriteBlogsToFileAll(settings.GetValue<string>("PathOutputBlogs"), false, from, to, top); WriteBlogsToFileAll(settings.GetValue<string>("PathOutputBlogs"), false, from, to, top);
break; break;
case "-replies": //update reply text case "--replies": //update reply text
CollectMissingReplyText().GetAwaiter().GetResult(); CollectMissingReplyText().GetAwaiter().GetResult();
break; break;
case "-likes": case "--likes":
string likeBlog = args.Length > 1 ? args[1] : null; string likeBlog = args.Length > 1 ? args[1] : null;
int cooldownDays = settings.GetValue("LikesRefreshCooldownDays", 7); int cooldownDays = settings.GetValue("LikesRefreshCooldownDays", 7);
CollectLikes(likeBlog, contains, cooldownDays, forceIgnoreCooldown).GetAwaiter().GetResult(); CollectLikes(likeBlog, contains, cooldownDays, forceIgnoreCooldown).GetAwaiter().GetResult();
break; break;
case "-urldump": case "--urldump":
DumpUrls(settings.GetValue<string>("PathOutputUrls")); DumpUrls(settings.GetValue<string>("PathOutputUrls"));
break; break;
case "-ingest": case "--ingest":
IngestMode.Run(config, args.Skip(1).ToArray()); exitCode = IngestMode.Run(config, args.Skip(1).ToArray());
break; break;
case "-output": case "--output":
OutputMode.Run(config); exitCode = OutputMode.Run(config);
break; break;
case "-revert": case "--revert":
RevertMode.Run(config, args.Length > 1 ? args[1] : null); exitCode = RevertMode.Run(config, args.Length > 1 ? args[1] : null);
break; break;
case "-correct": case "--correct":
{ {
bool applyChanges = args.Skip(1).Any(a => string.Equals(a, "-apply", StringComparison.OrdinalIgnoreCase) || string.Equals(a, "--apply", StringComparison.OrdinalIgnoreCase)); bool applyChanges = args.Skip(1).Any(a => string.Equals(a, "--apply", StringComparison.OrdinalIgnoreCase));
var correctArgs = args.Skip(1) var correctArgs = args.Skip(1)
.Where(a => !string.Equals(a, "-apply", StringComparison.OrdinalIgnoreCase) && !string.Equals(a, "--apply", StringComparison.OrdinalIgnoreCase)) .Where(a => !string.Equals(a, "--apply", StringComparison.OrdinalIgnoreCase))
.ToArray(); .ToArray();
CorrectMode.Run(config, correctArgs, applyChanges); exitCode = CorrectMode.Run(config, correctArgs, applyChanges);
break; break;
} }
case "-updatepaths": case "--updatepaths":
{ {
string rootPath = args.Length > 1 ? args[1] : (settings.GetValue<string>("PathTTRoot") ?? settings.GetValue<string>("PathInput") ?? string.Empty); string rootPath = args.Length > 1 ? args[1] : (settings.GetValue<string>("PathTTRoot") ?? settings.GetValue<string>("PathInput") ?? string.Empty);
UpdateBlogPathsRunner.Run(rootPath); exitCode = UpdateBlogPathsRunner.Run(rootPath);
break; break;
} }
case "-importposts": case "--importposts":
{ {
if (args.Length < 2) if (args.Length < 2)
{ {
Console.WriteLine("Usage: -importposts <path-to-legacy-posts.db>"); Console.WriteLine("Usage: --importposts <path-to-legacy-posts.db>");
exitCode = 2;
break; break;
} }
LegacyPostsDbImporter.Run(args[1]); exitCode = LegacyPostsDbImporter.Run(args[1]);
break; break;
} }
default: default:
Console.WriteLine("** Unknown Command ** " + args[0]); Console.WriteLine("** Unknown Command ** " + args[0]);
exitCode = 2;
break; break;
} }
} }
@@ -397,6 +390,69 @@ namespace URLNotesGrabberCORE
System.Console.WriteLine("<fin>:/"); System.Console.WriteLine("<fin>:/");
//System.Console.ReadKey(); //System.Console.ReadKey();
return exitCode;
}
static void PrintHelp()
{
Console.WriteLine("\t Parse .txt files to find blogs");
Console.WriteLine("-?, -h, --help\t Usage help");
Console.WriteLine("-V, --version\t Print the application version");
Console.WriteLine("--\t End of options: treat every following token as a literal operand");
Console.WriteLine("--parse\t Parse .txt files with specified blogname");
Console.WriteLine("--test\t Calls API for given blogname and postID");
Console.WriteLine("--post\t Traverse the input directory tree checking .txt files for corruption");
Console.WriteLine("--posts\t For each Post in DB, write blogname to file");
Console.WriteLine("--blogs\t For each Blog in DB, write blogname to file");
Console.WriteLine("--collect [0|1] [datetime]\t Collect Notes from API. 1=only posts without notes. 0=full re-check of all posts: a single resumable pass (interrupt & relaunch to resume; stops when complete, retrigger for a new pass). Optional datetime overrides the cutoff and runs as a one-off (bypasses resume tracking).");
Console.WriteLine("--blogsR\t For each Note that is a REPLY, write blogname to file ");
Console.WriteLine("--blogsO\t For each Blog in DB, write blogname to file, but limit via a passed start and stop range ");
Console.WriteLine("--bop [from] [to] [top]\t Write ALL blog names to file, limited by FROM TO TOP range arguments");
Console.WriteLine("--replies\t Fetch and update missing reply text for all replies in database");
Console.WriteLine("--likes\t Fetch likes: initial backfill for new blogs, incremental refresh for blogs past cooldown. Optional blog name forces single-blog run.");
Console.WriteLine("--force\t (with --likes) Ignore cooldown and refresh every fully-backfilled blog");
Console.WriteLine("--urldump\t Scan all posts' text columns and extract suspected URLs to configured file");
Console.WriteLine("--api3\t Use TumblrApi3 settings from appsettings.json");
Console.WriteLine("--api4\t Use TumblrApi4 settings from appsettings.json");
Console.WriteLine("--start [blogname]\t Start traversal alphabetically at this blog name");
Console.WriteLine("--api [section]\t Use a specific API settings section from appsettings.json (e.g. TumblrApi3)");
Console.WriteLine("--ingest [blogname]\t Ingest Tumblr .txt exports from appSettings:PathTTRoot into TL.db (all blogs, or single blog if name given)");
Console.WriteLine("--output\t Export posts from TL.db back to .txt files in each blog's TTFolderPath");
Console.WriteLine("--revert [blogname]\t Recursively scan the PathInput tree and restore *.bak back to *.txt (current .txt saved as next-free .bkN); optional blogname filters by path substring");
Console.WriteLine("--correct [bakPath]\t Dry-run: report multi-line field updates available from a BAK directory");
Console.WriteLine("--correct --apply [bakPath]\t Apply BAK-file corrections to matching posts (prompts yes/no)");
Console.WriteLine("--updatepaths [rootPath]\t Read .tumblr/.tmblrpriv metadata from <root>\\Index and set Blogs.TTFolderPath");
Console.WriteLine("--importposts [path-to-posts.db]\t One-time migration: copy legacy ThreeTxtFileHelper posts.db rows into TL.db");
Console.WriteLine();
Console.WriteLine("Exit status: 0 = success; 1 = unexpected error; 2 = usage error (unknown command or bad/missing arguments)");
} }
static void WritePostBlogsToFile(string outPath) static void WritePostBlogsToFile(string outPath)
@@ -2,7 +2,7 @@
"profiles": { "profiles": {
"URLNotesGrabberCORE": { "URLNotesGrabberCORE": {
"commandName": "Project", "commandName": "Project",
"commandLineArgs": "-collect 1 -api4" "commandLineArgs": "--collect 1 --api4"
} }
} }
} }