fix(posttype): stop Unknown.txt and type posts at their source

PostType becomes an output filename, so an unset or unvalidated value does
not stay a data problem -- it creates a file. OutputMode wrote untyped rows
to `PostType ?? "Unknown"`, IngestMode read that file back and derived the
literal type "Unknown" from its name, and the two would have regenerated
each other indefinitely.

Nothing was setting the type in the first place. AddPost -- the path every
notes/likes harvest goes through -- omitted PostType from its INSERT column
list entirely, so 1790 rows across 469 blogs had none. Ingest could never
repair them: it types a post only when it meets it inside a real export
file, and these posts appear in none.

Type at the source, from what each path actually knows:

- TraverseDirectory takes it from the filename it is already reading
  ("texts.txt" -> "texts"), the same rule ingest uses.
- CollectLikes has no file, so it reads the legacy-format `type` field that
  GrabLikes already requests with npf=false, mapped singular -> plural.
- AddPost/UpdatePost gained the plumbing to carry it. UpdatePost fills a
  missing type but never overwrites one, and its change-detection clause
  had to learn about PostType or the SET would be unreachable for a row
  whose content was already current.

PostTypes is the single source of truth: eight canonical names, and
anything else normalizes to null. Null is safe -- OutputMode skips those
rows -- while a stray value would have become a stray file. IngestMode and
TraverseDirectory now skip non-export .txt files outright, and the legacy
importer no longer passes a pre-column NULL straight back in.

For the rows already stored untyped, content is the only signal left, so
the migration infers from which columns they carry. Verified against a copy
of the live database: 1780 of 1790 typed, 10 left untyped for want of any
content at all, idempotent on a second pass. HasImage is deliberately not
consulted -- it is set on 12,420 of 19,828 known text posts.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jim
2026-08-20 08:00:56 -05:00
co-authored by Claude Opus 5
parent 954ec353a5
commit 34da632e6a
6 changed files with 266 additions and 13 deletions
+19 -1
View File
@@ -85,7 +85,25 @@ namespace URLNotesGrabberCORE
{
string rawBlogName = Path.GetFileName(Path.GetDirectoryName(file) ?? "unknown");
string blogName = Regex.Replace(rawBlogName, @"_\d+$", "");
string postType = Path.GetFileNameWithoutExtension(file);
// The filename becomes the row's PostType, and PostType later becomes an
// output filename -- so an unrecognized name here would mint a new type and
// a new file from any stray .txt that happens to sit in the tree. Only the
// eight real export files are ingestable.
//
// This is also what breaks the Unknown.txt cycle: OutputMode used to write
// untyped rows to Unknown.txt, and this scan would read it straight back
// and stamp those rows with the literal type "Unknown", making the file
// regenerate itself forever.
string? resolvedPostType = PostTypes.FromFileName(file);
if (resolvedPostType == null)
{
filesSkipped++;
continue;
}
// Non-nullable from here so the local Flush() below stays warning-clean:
// nullable flow analysis does not reach into local functions.
string postType = resolvedPostType;
if (targetBlog != null && !string.Equals(blogName, targetBlog, StringComparison.OrdinalIgnoreCase))
{