fix: correctly detect rate-limit state for single-key API pools

IsAllRateLimited() short-circuited true for any pool with 0 or 1 keys, with minRetrySeconds left at 0 regardless of whether that key was actually rate-limited. SleepUntilAnyAvailable() checks "minRetry <= 0" to decide whether to skip sleeping, so with exactly one key it always skipped the wait and let callers hammer the API again immediately after a 429, even mid-cooldown. The per-key loop already computes this correctly for any key count; the special case only needs to cover the true no-keys edge case, where there's nothing to wait on.
refactor: standardize SQLiteConnection disposal via using; guard config
2026-06-30 21:22:12 -05:00 · 2026-06-30 20:54:18 -05:00 · 2026-06-30 20:41:51 -05:00 · 2026-06-09 15:26:38 -05:00 · 2026-06-09 15:19:51 -05:00 · 2026-06-09 15:18:25 -05:00
4 changed files with 206 additions and 251 deletions
@@ -22,18 +22,18 @@ This document provides essential context for AI agents working with URLNotesGrab
 ```powershell
 dotnet build
 dotnet run            # Process all files in input directory
-dotnet run -- -parse [blogname]  # Process specific blog
-dotnet run -- -test [blogname] [postID]  # Test API for specific post
+dotnet run -- --parse [blogname]  # Process specific blog
+dotnet run -- --test [blogname] [postID]  # Test API for specific post
 ```

 ### Command-Line Interface
- `-parse [blogname]`: Parse text files for specific blog
- `-test [blogname] [postID]`: Test API note collection
- `-posts`: Export post blogs to file
- `-blogs`: Export blog list to file
- `-collect`: Collect notes for all posts in DB
- `-blogsR`: Export reply blogs to file
- `-blogsO [start] [stop]`: Export blogs within range
+- `--parse [blogname]`: Parse text files for specific blog
+- `--test [blogname] [postID]`: Test API note collection
+- `--posts`: Export post blogs to file
+- `--blogs`: Export blog list to file
+- `--collect`: Collect notes for all posts in DB
+- `--blogsR`: Export reply blogs to file
+- `--blogsO [start] [stop]`: Export blogs within range

 ## Project Conventions

@@ -139,7 +139,7 @@ namespace URLNotesGrabberCORE
                if (_savedImportPragmas != null)
                    return;

-                SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+                using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
                try
                {
                    connection.Open();
@@ -175,10 +175,6 @@ namespace URLNotesGrabberCORE
                {
                    Console.WriteLine($"[SQLite Import Mode] Failed to enable import pragmas: {ex.Message}");
                }
-                finally
-                {
-                    connection.Close();
-                }
            }
        }

@@ -190,7 +186,7 @@ namespace URLNotesGrabberCORE
                if (_savedImportPragmas == null)
                    return;

-                SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+                using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
                try
                {
                    connection.Open();
@@ -217,7 +213,6 @@ namespace URLNotesGrabberCORE
                finally
                {
                    _savedImportPragmas = null;
-                    connection.Close();
                }
            }
        }
@@ -252,7 +247,7 @@ namespace URLNotesGrabberCORE
        public static void EnsureReplyTextColumnExists(string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);

            try
            {
@@ -296,16 +291,12 @@ namespace URLNotesGrabberCORE
                // Breakpoint here
                Console.WriteLine($"Error checking/creating replyText column: {ex.Message}");
            }
-            finally
-            {
-                connection.Close();
-            }
        }

        public static void EnsureBlogsLikesColumnsExist(string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);

            try
            {
@@ -391,10 +382,6 @@ namespace URLNotesGrabberCORE
            {
                Console.WriteLine($"Error mapping Blogs likes columns: {ex.Message}");
            }
-            finally
-            {
-                connection.Close();
-            }
        }

        #region Adds
@@ -518,8 +505,12 @@ namespace URLNotesGrabberCORE
                        {
                            if (ownsConnection) connection.Open();

-                            string updateSql = "UPDATE Posts SET hasImage = " + (hasImage ? 1 : 0) + ", DateModified = '" + DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + "' WHERE blogName = '" + blogName + "' AND postID = '" + postID + "'";
-                            SQLiteCommand updateCommand = new SQLiteCommand(updateSql, connection);
+                            string updateSql = "UPDATE Posts SET hasImage = @hasImage, DateModified = @DateModified WHERE blogName = @blogName AND postID = @postID";
+                            using SQLiteCommand updateCommand = new SQLiteCommand(updateSql, connection);
+                            updateCommand.Parameters.AddWithValue("@hasImage", hasImage ? 1 : 0);
+                            updateCommand.Parameters.AddWithValue("@DateModified", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
+                            updateCommand.Parameters.AddWithValue("@blogName", blogName);
+                            updateCommand.Parameters.AddWithValue("@postID", postID);

                            updateCommand.ExecuteNonQuery();
                        }
@@ -565,7 +556,7 @@ namespace URLNotesGrabberCORE
        public static void AddAPICount(string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);

            try
            {
@@ -586,10 +577,6 @@ namespace URLNotesGrabberCORE
                // Breakpoint here
                //Console.WriteLine(ex.Message);
            }
-            finally
-            {
-                connection.Close();
-            }
        }

        public static bool AddNote(string rootBlogName, string noteBlogName, long postID, long timestamp, string type, string? DBPath = null)
@@ -598,7 +585,7 @@ namespace URLNotesGrabberCORE
            //try { AddPost(rootBlogName, postID, DBPath); } catch { }
            try { AddBlog(noteBlogName, false, DBPath); } catch { }

-            SQLiteConnection connection2 = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection2 = new SQLiteConnection("Data Source=" + DBPath);

            try
            {
@@ -657,10 +644,6 @@ namespace URLNotesGrabberCORE
                    Console.WriteLine("^^^^^ - SHORTCUT");
                }
            }
-            finally
-            {
-                connection2.Close();
-            }
            return false;
        }
        #endregion Adds
@@ -676,7 +659,7 @@ namespace URLNotesGrabberCORE
        public static List<Tuple<string, long, long, long>> GetPosts(bool withoutNotesOnly = false, DateTime? beforeDate = null, string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
            List<Tuple<string, long, long, long>> posts = new List<Tuple<string, long, long, long>>();

            try
@@ -854,17 +837,13 @@ namespace URLNotesGrabberCORE
                if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
                    Console.WriteLine(ex.Message);
            }
-            finally
-            {
-                connection.Close();
-            }
            return posts;
        }

        public static List<Tuple<string, long>> GetReplies(string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
            List<Tuple<string, long>> posts = new List<Tuple<string, long>>();

            try
@@ -898,17 +877,13 @@ namespace URLNotesGrabberCORE
                if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
                    Console.WriteLine(ex.Message);
            }
-            finally
-            {
-                connection.Close();
-            }
            return posts;
        }

        public static List<Tuple<string, long>> GetRepliesWithMissingText(string? DBPath = null, int limit = 50)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
            List<Tuple<string, long>> posts = new List<Tuple<string, long>>();

            try
@@ -949,17 +924,13 @@ namespace URLNotesGrabberCORE
                // Breakpoint here
                Console.WriteLine($"Error getting replies with missing text: {ex.Message}");
            }
-            finally
-            {
-                connection.Close();
-            }
            return posts;
        }

        public static List<Tuple<string, long, long>> GetRepliesWithFilledText(string? DBPath = null, int? limit = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
            List<Tuple<string, long, long>> posts = new List<Tuple<string, long, long>>();

            try
@@ -1011,17 +982,13 @@ namespace URLNotesGrabberCORE
                // Breakpoint here
                Console.WriteLine($"Error getting replies with filled text: {ex.Message}");
            }
-            finally
-            {
-                connection.Close();
-            }
            return posts;
        }

        public static int GetAPICount(string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
            int count = 0;

            try { AddAPICount(); } catch { }
@@ -1050,17 +1017,13 @@ namespace URLNotesGrabberCORE
                if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
                    Console.WriteLine(ex.Message);
            }
-            finally
-            {
-                connection.Close();
-            }
            return count;
        }

        public static List<Tuple<string, int, long, long>> GetBlogsForLikes(string specificBlog = null, int cooldownDays = 7, bool ignoreCooldown = false, string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
            List<Tuple<string, int, long, long>> blogs = new List<Tuple<string, int, long, long>>();

            try
@@ -1136,17 +1099,13 @@ namespace URLNotesGrabberCORE
            {
                Console.WriteLine($"Error fetching blogs for likes: {ex.Message}");
            }
-            finally
-            {
-                connection.Close();
-            }
            return blogs;
        }

        public static List<string> GetBlogs(bool reblogsOnly, int from, int to, int top, string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
            List<string> blogs = new List<string>();

            try
@@ -1181,17 +1140,13 @@ namespace URLNotesGrabberCORE
                if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
                    Console.WriteLine(ex.Message);
            }
-            finally
-            {
-                connection.Close();
-            }
            return blogs;
        }

        public static List<string> GetBlogsAll(bool reblogsOnly, int from, int to, int top, string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
            List<string> blogs = new List<string>();

            try
@@ -1226,19 +1181,13 @@ namespace URLNotesGrabberCORE
                if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
                    Console.WriteLine(ex.Message);
            }
-            finally
-            {
-                connection.Close();
-            }
            return blogs;
        }
        public static IEnumerable<List<string>> GetAllPostTextColumns(string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
            connection.Open();
-            try
-            {
            string sql = "SELECT BlogName, reblogURL, PostURL, Slug, ReblogKey, ReblogName, Summary, Quote, Body, Tags, Link, PhotoURL, PhotoCaption, DownloadedFiles, AudioCaption, Question, Answer, Title, RootBlogName, RootURL FROM Posts WHERE IFNULL(DownloadedFiles, '.') = '.'";
            using (SQLiteCommand command = new SQLiteCommand(sql, connection))
            {
@@ -1266,12 +1215,6 @@ namespace URLNotesGrabberCORE
                }
            }
        }
-            finally
-            {
-                connection.Close();
-                connection.Dispose();
-            }
-        }
        #endregion Gets

        #region Updates
@@ -1280,7 +1223,7 @@ namespace URLNotesGrabberCORE
        public static void UpdatePostMarkNotesCollected(string blogName, long postID, string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);

            try
            {
@@ -1303,16 +1246,12 @@ namespace URLNotesGrabberCORE
                if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
                    Console.WriteLine(ex.Message);
            }
-            finally
-            {
-                connection.Close();
-            }
        }

        public static void UpdatePostMarkNotFound(string blogName, long postID, string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);

            try
            {
@@ -1336,10 +1275,6 @@ namespace URLNotesGrabberCORE
                if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
                    Console.WriteLine(ex.Message);
            }
-            finally
-            {
-                connection.Close();
-            }
        }

        // ----- CollectRunState: tracks the frozen cutoff + completion flag for a managed "-collect 0" full re-check run -----
@@ -1451,7 +1386,7 @@ namespace URLNotesGrabberCORE

            Console.WriteLine("{2}\t{0}\t{1}", UnixTimeStampToDateTime(timestamp), noteBlogName, type);

-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);

            try
            {
@@ -1478,10 +1413,6 @@ namespace URLNotesGrabberCORE
                    return true;
                }
            }
-            finally
-            {
-                connection.Close();
-            }
            return false;
        }

@@ -1597,7 +1528,7 @@ namespace URLNotesGrabberCORE
        public static void UpdateBlogOutput(string blogName, string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);

            try
            {
@@ -1617,16 +1548,12 @@ namespace URLNotesGrabberCORE
                if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
                    Console.WriteLine(ex.Message);
            }
-            finally
-            {
-                connection.Close();
-            }
        }

        public static void UpdateBlogLikesStatus(string blogName, int likesPulled, long likesCursor, string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
            try
            {
                connection.Open();
@@ -1644,10 +1571,6 @@ namespace URLNotesGrabberCORE
            {
                Console.WriteLine($"Error updating blog likes status: {ex.Message}");
            }
-            finally
-            {
-                connection.Close();
-            }
        }

        // Bumps the high-water mark for a blog. Used during Branch A (initial backfill) when we
@@ -1657,7 +1580,7 @@ namespace URLNotesGrabberCORE
        {
            if (newestTimestamp <= 0) return;
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
            try
            {
                connection.Open();
@@ -1677,10 +1600,6 @@ namespace URLNotesGrabberCORE
            {
                Console.WriteLine($"Error updating blog likes newest timestamp: {ex.Message}");
            }
-            finally
-            {
-                connection.Close();
-            }
        }

        // Called at the end of a refresh pass (Branch B). Bumps the high-water mark, stamps the
@@ -1688,7 +1607,7 @@ namespace URLNotesGrabberCORE
        public static void UpdateBlogLikesRefreshStatus(string blogName, long newestTimestamp, int newCount, string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
            try
            {
                connection.Open();
@@ -1711,16 +1630,12 @@ namespace URLNotesGrabberCORE
            {
                Console.WriteLine($"Error updating blog likes refresh status: {ex.Message}");
            }
-            finally
-            {
-                connection.Close();
-            }
        }

        public static int UpdateAPICount(string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);

            int APICount = DataAccess.GetAPICount();
            APICount++;
@@ -1743,10 +1658,6 @@ namespace URLNotesGrabberCORE
                if (ex.Message != "constraint failed\r\nUNIQUE constraint failed: Posts.BlogName, Posts.PostID")
                    Console.WriteLine(ex.Message);
            }
-            finally
-            {
-                connection.Close();
-            }

            return APICount;
        }
@@ -1754,7 +1665,7 @@ namespace URLNotesGrabberCORE
        public static int UpdateNoteReplyText(string rootBlogName, long postID, string noteBlogName, long timestamp, string replyText, string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
            int rowsAffected = 0;

            // A bare "." collides with the "needs processing" sentinel in GetRepliesWithFilledText, which would loop the post forever. Store as ". " so the data is preserved but no longer matches the sentinel.
@@ -1797,17 +1708,13 @@ namespace URLNotesGrabberCORE
                Console.WriteLine($"[UpdateNoteReplyText] Error updating reply text: {ex.Message}");
                Console.WriteLine($"[UpdateNoteReplyText] StackTrace: {ex.StackTrace}");
            }
-            finally
-            {
-                connection.Close();
-            }
            return rowsAffected;
        }

        public static int UpdateAllNoteReplyTextForPost(string rootBlogName, long postID, string replyText, string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);

            try
            {
@@ -1843,10 +1750,6 @@ namespace URLNotesGrabberCORE
                Console.WriteLine($"[UpdateAllNoteReplyTextForPost] StackTrace: {ex.StackTrace}");
                return 0;
            }
-            finally
-            {
-                connection.Close();
-            }
        }
        #endregion Updates

@@ -1857,7 +1760,7 @@ namespace URLNotesGrabberCORE
        public static void EnsureTTFileHelperColumnsExist(string? DBPath = null)
        {
            DBPath ??= GetDefaultDbPath();
-            SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);
+            using SQLiteConnection connection = new SQLiteConnection("Data Source=" + DBPath);

            try
            {
@@ -1903,10 +1806,6 @@ namespace URLNotesGrabberCORE
            {
                Console.WriteLine($"Error ensuring TTFileHelper columns: {ex.Message}");
            }
-            finally
-            {
-                connection.Close();
-            }
        }

        // INSERT-or-UPDATE for a post arriving from a Tumblr text-file export.
@@ -2727,7 +2626,7 @@ namespace URLNotesGrabberCORE
        {
            minRetrySeconds = 0;
            if (!_usePool || _overrideKey != null) return false;
-            if (_keys.Count <= 1) return true;
+            if (_keys.Count == 0) return false;

            var now = DateTimeOffset.UtcNow.ToUnixTimeSeconds();
            bool all = true;
@@ -5,7 +5,6 @@ using Microsoft.Extensions.Configuration;
 using System.Configuration;
 using System.Threading;
 using Microsoft.Extensions.Diagnostics.Latency;
-using static System.Runtime.InteropServices.JavaScript.JSType;
 using System.Text.RegularExpressions;

 namespace URLNotesGrabberCORE
@@ -13,12 +12,27 @@ namespace URLNotesGrabberCORE
    internal class Program
    {

-        static void Main(string[] args)
+        static int Main(string[] args)
        {
            // Reset console color on exit (including Ctrl+C)
            Console.CancelKeyPress += (s, e) => Console.ResetColor();
            AppDomain.CurrentDomain.ProcessExit += (s, e) => Console.ResetColor();

+            try
+            {
+                return Run(args);
+            }
+            catch (Exception ex)
+            {
+                // Turn configuration errors / unguarded indexers into a quiet, deterministic exit code.
+                Console.Error.WriteLine(ex.Message);
+                return 1;
+            }
+        }
+
+        static int Run(string[] args)
+        {
+            int exitCode = 0;
            IConfiguration config = new ConfigurationBuilder()
                                    .SetBasePath(Directory.GetCurrentDirectory())
                                    .AddJsonFile("appsettings.json", optional: true, reloadOnChange: false)
@@ -33,31 +47,34 @@ namespace URLNotesGrabberCORE
            List<string> filteredArgs = new List<string>();
            for (int i = 0; i < args.Length; i++)
            {
-                if (string.Equals(args[i], "-force", StringComparison.OrdinalIgnoreCase) ||
-                    string.Equals(args[i], "--force", StringComparison.OrdinalIgnoreCase))
+                if (args[i] == "--")
+                {
+                    // POSIX end-of-options: everything after is a literal operand.
+                    for (int j = i + 1; j < args.Length; j++) filteredArgs.Add(args[j]);
+                    break;
+                }
+
+                if (string.Equals(args[i], "--force", StringComparison.OrdinalIgnoreCase))
                {
                    forceIgnoreCooldown = true;
                    continue;
                }

-                if (string.Equals(args[i], "-api3", StringComparison.OrdinalIgnoreCase) ||
-                    string.Equals(args[i], "--api3", StringComparison.OrdinalIgnoreCase))
+                if (string.Equals(args[i], "--api3", StringComparison.OrdinalIgnoreCase))
                {
                    apiSectionName = "TumblrApi3";
                    apiExplicitlySet = true;
                    continue;
                }

-                if (string.Equals(args[i], "-api4", StringComparison.OrdinalIgnoreCase) ||
-                    string.Equals(args[i], "--api4", StringComparison.OrdinalIgnoreCase))
+                if (string.Equals(args[i], "--api4", StringComparison.OrdinalIgnoreCase))
                {
                    apiSectionName = "TumblrApi4";
                    apiExplicitlySet = true;
                    continue;
                }

-                if (string.Equals(args[i], "-api", StringComparison.OrdinalIgnoreCase) ||
-                    string.Equals(args[i], "--api", StringComparison.OrdinalIgnoreCase))
+                if (string.Equals(args[i], "--api", StringComparison.OrdinalIgnoreCase))
                {
                    if (i + 1 < args.Length && !string.IsNullOrWhiteSpace(args[i + 1]))
                    {
@@ -67,13 +84,12 @@ namespace URLNotesGrabberCORE
                    }
                    else
                    {
-                        Console.WriteLine("--Missing API section after -api/--api. Using default TumblrApi.--");
+                        Console.WriteLine("--Missing API section after --api. Using default TumblrApi.--");
                    }
                    continue;
                }

-                if (string.Equals(args[i], "-start", StringComparison.OrdinalIgnoreCase) ||
-                    string.Equals(args[i], "--start", StringComparison.OrdinalIgnoreCase))
+                if (string.Equals(args[i], "--start", StringComparison.OrdinalIgnoreCase))
                {
                    if (i + 1 < args.Length && !string.IsNullOrWhiteSpace(args[i + 1]))
                    {
@@ -82,7 +98,7 @@ namespace URLNotesGrabberCORE
                    }
                    else
                    {
-                        Console.WriteLine("--Missing blog name after -start/--start. Ignoring.--");
+                        Console.WriteLine("--Missing blog name after --start. Ignoring.--");
                    }
                    continue;
                }
@@ -123,7 +139,10 @@ namespace URLNotesGrabberCORE
                Console.SetOut(dualLogger);
            }

-            List<string> contains = settings.GetValue<string>("ContainsList").Split(',').ToList();
+            string? containsListSetting = settings.GetValue<string>("ContainsList");
+            if (string.IsNullOrEmpty(containsListSetting))
+                throw new InvalidOperationException("ContainsList is not configured in appsettings.json");
+            List<string> contains = containsListSetting.Split(',').ToList();
            bool logTraversalRecordImports = settings.GetValue("LogTraversalRecordImports", false);

            if (args.Length == 0) //Traverse folder structure to add posts and thus blogs to DB
@@ -147,57 +166,23 @@ namespace URLNotesGrabberCORE
                switch (args[0])
                {
                    case "-?":
-                        Console.WriteLine("\t Parse .txt files to find blogs");
-
-                        Console.WriteLine("-?\t Usage help");
-
-                        Console.WriteLine("-parse\t Parse .txt files with specified blogname");
-
-                        Console.WriteLine("-test\t Calls API for given blogname and postID");
-
-                        Console.WriteLine("-posts\t For each Post in DB, write blogname to file");
-
-                        Console.WriteLine("-blogs\t For each Blog in DB, write blogname to file");
-
-                        Console.WriteLine("-collect [0|1] [datetime]\t Collect Notes from API. 1=only posts without notes. 0=full re-check of all posts: a single resumable pass (interrupt & relaunch to resume; stops when complete, retrigger for a new pass). Optional datetime overrides the cutoff and runs as a one-off (bypasses resume tracking).");
-
-                        Console.WriteLine("-blogsR\t For each Note that is a REPLY, write blogname to file ");
-
-                        Console.WriteLine("-blogsO\t For each Blog in DB, write blogname to file, but limit via a passed start and stop range ");
-
-                        Console.WriteLine("-replies\t Fetch and update missing reply text for all replies in database");
-
-                        Console.WriteLine("-likes\t Fetch likes: initial backfill for new blogs, incremental refresh for blogs past cooldown. Optional blog name forces single-blog run.");
-
-                        Console.WriteLine("-force\t (with -likes) Ignore cooldown and refresh every fully-backfilled blog");
-
-                        Console.WriteLine("-urldump\t Scan all posts' text columns and extract suspected URLs to configured file");
-
-                        Console.WriteLine("-api3\t Use TumblrApi3 settings from appsettings.json");
-
-                        Console.WriteLine("-api4\t Use TumblrApi4 settings from appsettings.json");
-
-                        Console.WriteLine("-start [blogname]\t Start traversal alphabetically at this blog name");
-
-                        Console.WriteLine("-api [section]\t Use a specific API settings section from appsettings.json (e.g. TumblrApi3)");
-
-                        Console.WriteLine("-ingest [blogname]\t Ingest Tumblr .txt exports from appSettings:PathTTRoot into TL.db (all blogs, or single blog if name given)");
-
-                        Console.WriteLine("-output\t Export posts from TL.db back to .txt files in each blog's TTFolderPath");
-
-                        Console.WriteLine("-revert [blogname]\t Recursively scan the PathInput tree and restore *.bak back to *.txt (current .txt saved as next-free .bkN); optional blogname filters by path substring");
-
-                        Console.WriteLine("-correct [bakPath]\t Dry-run: report multi-line field updates available from a BAK directory");
-
-                        Console.WriteLine("-correct -apply [bakPath]\t Apply BAK-file corrections to matching posts (prompts yes/no)");
-
-                        Console.WriteLine("-updatepaths [rootPath]\t Read .tumblr/.tmblrpriv metadata from <root>\\Index and set Blogs.TTFolderPath");
-
-                        Console.WriteLine("-importposts [path-to-posts.db]\t One-time migration: copy legacy ThreeTxtFileHelper posts.db rows into TL.db");
-
+                    case "-h":
+                    case "--help":
+                        PrintHelp();
                        break;

-                    case "-parse":
+                    case "-V":
+                    case "--version":
+                        Console.WriteLine(System.Reflection.Assembly.GetExecutingAssembly().GetName().Version?.ToString() ?? "unknown");
+                        break;
+
+                    case "--parse":
+                        if (args.Length < 2)
+                        {
+                            Console.WriteLine("Usage: --parse <blogname>");
+                            exitCode = 2;
+                            break;
+                        }
                        string blogNameToParse = args[1];
                        int postsAdded = 0;
                        try
@@ -212,23 +197,23 @@ namespace URLNotesGrabberCORE
                        Console.WriteLine($"Total posts added: {postsAdded}");
                        break;
                    
-                    case "-test":
+                    case "--test":
                        Console.WriteLine("Test command not implemented");
                        break;

-                    case "-post":
+                    case "--post":
                        TraverseDirectoryForCorruption(settings.GetValue<string>("PathInput"), settings.GetValue<string>("PathOutputBlogs"), contains);
                        break;

-                    case "-posts": //write post's blogs to file
+                    case "--posts": //write post's blogs to file
                        WritePostBlogsToFile(settings.GetValue<string>("PathOutputPosts"));
                        break;

-                    case "-blogs": //write blogs to file
+                    case "--blogs": //write blogs to file
                        WriteBlogsToFile(settings.GetValue<string>("PathOutputBlogs"));
                        break;

-                    case "-collect": //collect notes from all posts
+                    case "--collect": //collect notes from all posts
                        bool withoutNotesOnly = true;
                        DateTime? beforeDate = DateTime.Now;
                        bool explicitDateSupplied = false;
@@ -236,6 +221,7 @@ namespace URLNotesGrabberCORE
                        if (args.Length < 2)
                        {
                            Console.WriteLine("--Expected WITHOUTNOTESONLY (0, 1) [OPTIONAL: BEFOREDATE]--");
+                            exitCode = 2;
                            break;
                        }

@@ -268,6 +254,7 @@ namespace URLNotesGrabberCORE
                            else
                            {
                                Console.WriteLine($"ERROR: Invalid date format '{args[2]}'");
+                                exitCode = 2;
                                break;
                            }
                        }
@@ -297,14 +284,14 @@ namespace URLNotesGrabberCORE
                        CollectNotes(settings.GetValue<string>("PathOutput"), withoutNotesOnly, beforeDate, managedCollectRun).GetAwaiter().GetResult();
                        break;

-                    case "-blogsR": //collect notes from all posts
+                    case "--blogsR": //collect notes from all posts
                        WriteBlogsToFile(settings.GetValue<string>("PathOutputBlogs"), true);
                        break;

-                    case "-blogsO": //collect notes from all posts
+                    case "--blogsO": //collect notes from all posts
                        int from = 1, to = 999999, top = 100;

-                        if (args[1] is not null && args[2] is not null && args[3] is not null)
+                        if (args.Length >= 4 && args[1] is not null && args[2] is not null && args[3] is not null)
                        {
                            from = int.Parse(args[1]);
                            to = int.Parse(args[2]);
@@ -313,14 +300,16 @@ namespace URLNotesGrabberCORE
                        else
                        {
                            Console.WriteLine("--Expected FROM TO--");
+                            exitCode = 2;
+                            break;
                        }
                        WriteBlogsToFile(settings.GetValue<string>("PathOutputBlogs"), false, from, to, top);
                        break;

-                    case "-bop": //collect notes from all posts
+                    case "--bop": //collect notes from all posts
                        from = 1; to = 999999; top = 100;

-                        if (args[1] is not null && args[2] is not null && args[3] is not null)
+                        if (args.Length >= 4 && args[1] is not null && args[2] is not null && args[3] is not null)
                        {
                            from = int.Parse(args[1]);
                            to = int.Parse(args[2]);
@@ -329,66 +318,70 @@ namespace URLNotesGrabberCORE
                        else
                        {
                            Console.WriteLine("--Expected FROM TO--");
+                            exitCode = 2;
+                            break;
                        }
                        WriteBlogsToFileAll(settings.GetValue<string>("PathOutputBlogs"), false, from, to, top);
                        break;

-                    case "-replies": //update reply text
+                    case "--replies": //update reply text
                        CollectMissingReplyText().GetAwaiter().GetResult();
                        break;

-                    case "-likes":
+                    case "--likes":
                        string likeBlog = args.Length > 1 ? args[1] : null;
                        int cooldownDays = settings.GetValue("LikesRefreshCooldownDays", 7);
                        CollectLikes(likeBlog, contains, cooldownDays, forceIgnoreCooldown).GetAwaiter().GetResult();
                        break;

-                    case "-urldump":
+                    case "--urldump":
                        DumpUrls(settings.GetValue<string>("PathOutputUrls"));
                        break;

-                    case "-ingest":
-                        IngestMode.Run(config, args.Skip(1).ToArray());
+                    case "--ingest":
+                        exitCode = IngestMode.Run(config, args.Skip(1).ToArray());
                        break;

-                    case "-output":
-                        OutputMode.Run(config);
+                    case "--output":
+                        exitCode = OutputMode.Run(config);
                        break;

-                    case "-revert":
-                        RevertMode.Run(config, args.Length > 1 ? args[1] : null);
+                    case "--revert":
+                        exitCode = RevertMode.Run(config, args.Length > 1 ? args[1] : null);
                        break;

-                    case "-correct":
+                    case "--correct":
                    {
-                        bool applyChanges = args.Skip(1).Any(a => string.Equals(a, "-apply", StringComparison.OrdinalIgnoreCase) || string.Equals(a, "--apply", StringComparison.OrdinalIgnoreCase));
+                        bool applyChanges = args.Skip(1).Any(a => string.Equals(a, "--apply", StringComparison.OrdinalIgnoreCase));
                        var correctArgs = args.Skip(1)
-                            .Where(a => !string.Equals(a, "-apply", StringComparison.OrdinalIgnoreCase) && !string.Equals(a, "--apply", StringComparison.OrdinalIgnoreCase))
+                            .Where(a => !string.Equals(a, "--apply", StringComparison.OrdinalIgnoreCase))
                            .ToArray();
-                        CorrectMode.Run(config, correctArgs, applyChanges);
+                        exitCode = CorrectMode.Run(config, correctArgs, applyChanges);
                        break;
                    }

-                    case "-updatepaths":
+                    case "--updatepaths":
                    {
                        string rootPath = args.Length > 1 ? args[1] : (settings.GetValue<string>("PathTTRoot") ?? settings.GetValue<string>("PathInput") ?? string.Empty);
-                        UpdateBlogPathsRunner.Run(rootPath);
+                        exitCode = UpdateBlogPathsRunner.Run(rootPath);
                        break;
                    }

-                    case "-importposts":
+                    case "--importposts":
                    {
                        if (args.Length < 2)
                        {
-                            Console.WriteLine("Usage: -importposts <path-to-legacy-posts.db>");
+                            Console.WriteLine("Usage: --importposts <path-to-legacy-posts.db>");
+                            exitCode = 2;
                            break;
                        }
-                        LegacyPostsDbImporter.Run(args[1]);
+                        exitCode = LegacyPostsDbImporter.Run(args[1]);
                        break;
                    }

                    default:
                        Console.WriteLine("** Unknown Command ** " + args[0]);
+                        exitCode = 2;
                        break;
                }
            }
@@ -397,6 +390,69 @@ namespace URLNotesGrabberCORE

            System.Console.WriteLine("<fin>:/");
            //System.Console.ReadKey();
+            return exitCode;
+        }
+
+        static void PrintHelp()
+        {
+            Console.WriteLine("\t Parse .txt files to find blogs");
+
+            Console.WriteLine("-?, -h, --help\t Usage help");
+
+            Console.WriteLine("-V, --version\t Print the application version");
+
+            Console.WriteLine("--\t End of options: treat every following token as a literal operand");
+
+            Console.WriteLine("--parse\t Parse .txt files with specified blogname");
+
+            Console.WriteLine("--test\t Calls API for given blogname and postID");
+
+            Console.WriteLine("--post\t Traverse the input directory tree checking .txt files for corruption");
+
+            Console.WriteLine("--posts\t For each Post in DB, write blogname to file");
+
+            Console.WriteLine("--blogs\t For each Blog in DB, write blogname to file");
+
+            Console.WriteLine("--collect [0|1] [datetime]\t Collect Notes from API. 1=only posts without notes. 0=full re-check of all posts: a single resumable pass (interrupt & relaunch to resume; stops when complete, retrigger for a new pass). Optional datetime overrides the cutoff and runs as a one-off (bypasses resume tracking).");
+
+            Console.WriteLine("--blogsR\t For each Note that is a REPLY, write blogname to file ");
+
+            Console.WriteLine("--blogsO\t For each Blog in DB, write blogname to file, but limit via a passed start and stop range ");
+
+            Console.WriteLine("--bop [from] [to] [top]\t Write ALL blog names to file, limited by FROM TO TOP range arguments");
+
+            Console.WriteLine("--replies\t Fetch and update missing reply text for all replies in database");
+
+            Console.WriteLine("--likes\t Fetch likes: initial backfill for new blogs, incremental refresh for blogs past cooldown. Optional blog name forces single-blog run.");
+
+            Console.WriteLine("--force\t (with --likes) Ignore cooldown and refresh every fully-backfilled blog");
+
+            Console.WriteLine("--urldump\t Scan all posts' text columns and extract suspected URLs to configured file");
+
+            Console.WriteLine("--api3\t Use TumblrApi3 settings from appsettings.json");
+
+            Console.WriteLine("--api4\t Use TumblrApi4 settings from appsettings.json");
+
+            Console.WriteLine("--start [blogname]\t Start traversal alphabetically at this blog name");
+
+            Console.WriteLine("--api [section]\t Use a specific API settings section from appsettings.json (e.g. TumblrApi3)");
+
+            Console.WriteLine("--ingest [blogname]\t Ingest Tumblr .txt exports from appSettings:PathTTRoot into TL.db (all blogs, or single blog if name given)");
+
+            Console.WriteLine("--output\t Export posts from TL.db back to .txt files in each blog's TTFolderPath");
+
+            Console.WriteLine("--revert [blogname]\t Recursively scan the PathInput tree and restore *.bak back to *.txt (current .txt saved as next-free .bkN); optional blogname filters by path substring");
+
+            Console.WriteLine("--correct [bakPath]\t Dry-run: report multi-line field updates available from a BAK directory");
+
+            Console.WriteLine("--correct --apply [bakPath]\t Apply BAK-file corrections to matching posts (prompts yes/no)");
+
+            Console.WriteLine("--updatepaths [rootPath]\t Read .tumblr/.tmblrpriv metadata from <root>\\Index and set Blogs.TTFolderPath");
+
+            Console.WriteLine("--importposts [path-to-posts.db]\t One-time migration: copy legacy ThreeTxtFileHelper posts.db rows into TL.db");
+
+            Console.WriteLine();
+            Console.WriteLine("Exit status: 0 = success; 1 = unexpected error; 2 = usage error (unknown command or bad/missing arguments)");
        }

        static void WritePostBlogsToFile(string outPath)
@@ -2,7 +2,7 @@
  "profiles": {
    "URLNotesGrabberCORE": {
      "commandName": "Project",
-      "commandLineArgs": "-collect 1 -api4"
+      "commandLineArgs": "--collect 1 --api4"
    }
  }
 }
Author	SHA1	Message	Date
jim	f541ec4260	fix: correctly detect rate-limit state for single-key API pools IsAllRateLimited() short-circuited true for any pool with 0 or 1 keys, with minRetrySeconds left at 0 regardless of whether that key was actually rate-limited. SleepUntilAnyAvailable() checks "minRetry <= 0" to decide whether to skip sleeping, so with exactly one key it always skipped the wait and let callers hammer the API again immediately after a 429, even mid-cooldown. The per-key loop already computes this correctly for any key count; the special case only needs to cover the true no-keys edge case, where there's nothing to wait on.	2026-06-30 21:22:12 -05:00
jim	f549f020e1	refactor: standardize SQLiteConnection disposal via using; guard config Replace the try/finally { connection.Close(); } pattern used across most of DataAccess.cs with using declarations, so disposal happens automatically and can't be skipped by a future edit that adds an early return before the finally. Left the shared-connection (ownsConnection) call sites alone since those intentionally outlive a single method call. Also drop a stray unused `using static ... JSType` import, and make a missing ContainsList config setting fail with a clear InvalidOperationException instead of a NullReferenceException from Split(',') on null.	2026-06-30 20:54:18 -05:00
jim	0ff80a0fd3	fix: parameterize AddPost fallback UPDATE, guard args indexing Posts.hasImage/DateModified fallback update built its WHERE clause via raw string concatenation of blogName/postID, unlike every other query in this method — a blog name containing a single quote would break or inject into the query. Switch it to parameters. --parse, --blogsO, and --bop indexed args[1..3] before checking args.Length, so a missing argument threw IndexOutOfRangeException instead of hitting the intended usage message.	2026-06-30 20:41:51 -05:00
jimandClaude Opus 4.8	4df73367fb	BREAKING: switch all multi-char commands to POSIX --double-dash Rename every multi-character option/command from single-dash to double-dash (--likes, --collect, --force, etc.) to follow the POSIX long-option convention. Single-character short options (-h, -V, -?) keep their single dash, as POSIX prescribes. Breaking: existing invocations/scripts using single-dash forms now report Unknown Command and must be updated. Run profile (launchSettings.json) and CLI docs (copilot-instructions.md) updated to match. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-06-09 15:26:38 -05:00
jimandClaude Opus 4.8	a437fa87d3	Document -post and -bop commands in --help These two commands were handled by the switch but never listed in help. --help now covers every command the program accepts. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-06-09 15:19:51 -05:00
jimandClaude Opus 4.8	32a1583efd	Document exit-status codes in --help output The new 0/1/2 exit codes had no footprint in --help; add an Exit status line so the documented behavior matches what the program now returns. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-06-09 15:18:25 -05:00
jimandClaude Opus 4.8	03676432bd	Add POSIX-friendly CLI handling: --, --help/--version, exit codes Keep the existing single-dash switch style and case-insensitive matching, but add the cheap, non-breaking POSIX wins: - `--` end-of-options: tokens after a bare `--` are treated as literal operands - `--help`/`-h` (alongside `-?`) and `-V`/`--version` - Main returns a real exit code: 2 for usage errors, propagates handler return codes, and a top-level catch yields a quiet 1 on unhandled errors Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-06-09 15:12:10 -05:00