URLNotesGrabberCore/URLNotesGrabberCORE/TL.db.md

# `TL.db` — schema notes

The SQLite database behind **URLNotesGrabberCORE** and its sibling crawlers, and the one
[Rolodex](https://git.basso.land/jim/Rolodex) reads.

Everything below was read out of the live file, not inferred from code. Counts are as of
**2026-07-29**; re-run the queries at the bottom to refresh them.

- Journal mode: **WAL** — `TL.db-wal` and `TL.db-shm` live beside the file and are part of
  the database. Copying `TL.db` alone gives you whatever was last checkpointed, not the
  current state.
- Page size: 4096.

---

## The three content tables

| Table | Rows | What it is |
|---|--:|---|
| `Blogs` | 144,367 | The crawl registry — one row per known blog, plus crawl-state flags |
| `Posts` | 14,589 | Stored post content. Only 3,602 blogs actually have any |
| `Notes` | 1,189,604 | The engagement graph: `NoteBlogName` acted on `(RootBlogName, PostID)` |

The engagement graph is the interesting part. 31,888 distinct blogs appear as engagers —
far more than the 3,602 that have stored posts — which is what makes this a social graph
rather than a post archive.

### `Blogs`

```sql
CREATE TABLE "Blogs" (
    "BlogName"              TEXT,
    "HasBeenOutput"         INTEGER DEFAULT 0,
    "IsActive"              INTEGER DEFAULT 1,
    "DateAdded"             TEXT NOT NULL DEFAULT '12/24/25',
    "ByLikes"               INTEGER NOT NULL DEFAULT 0,
    "LikesPulled"           INTEGER NOT NULL DEFAULT 0,
    "LikesCursor"           INTEGER DEFAULT 0,
    "DateModified"          TEXT,
    "DateCreated"           TEXT,
    LikesNewestTimestamp    INTEGER DEFAULT 0,
    LikesLastRefreshed      INTEGER DEFAULT 0,
    LikesLastNewCount       INTEGER DEFAULT 0,
    TTFolderPath            TEXT,
    PRIMARY KEY("BlogName")
);
```

`BlogName` is the primary key, so it is the only indexed way in. There is no index on any
flag or date — filtering or sorting on those scans all 144k rows, which is affordable
here and is not on `Notes`.

Flag distribution: `IsActive = 1` on 144,366 of 144,367 rows, `HasBeenOutput = 1` on
5,369, `ByLikes = 1` on 2. `IsActive` carries a second meaning as of Rolodex — see
[`Blogs.IsActive`](#blogsisactive--now-written-by-two-applications) below.

The columns after `DateCreated` were added later by `ALTER TABLE`, which is why they carry
no quoting in the stored DDL. That is the normal way this schema grows.

**`DateAdded` is not written consistently.** 126,423 rows hold ISO `yyyy-MM-dd HH:mm:ss`;
17,944 hold US-format `M/d/yy` from a bulk import. As text those two sort into different
parts of the table, so anything ordering or range-filtering on this column has to
normalise first — see `DateSql` in Rolodex.

### `Posts`

```sql
CREATE TABLE "Posts" (
    "BlogName"              TEXT,
    "PostID"                INTEGER,
    "HasNotesGathered"      INTEGER DEFAULT 0,
    "reblogURL"             TEXT,
    "NotFound"              INTEGER DEFAULT 0,
    "PostDate"              TEXT,
    "NotesGatheredDateTime" INTEGER NOT NULL DEFAULT 1729746000,
    "HasImage"              INTEGER NOT NULL DEFAULT 0,
    "PostURL"               TEXT,
    "Slug"                  TEXT,
    "ReblogKey"             TEXT,
    "ReblogName"            TEXT,
    "Summary"               TEXT,
    "Quote"                 TEXT,
    "Body"                  TEXT,
    "Tags"                  TEXT,
    "Link"                  TEXT,
    "PhotoURL"              TEXT,
    "PhotoCaption"          TEXT,
    "DownloadedFiles"       TEXT,
    "AudioCaption"          TEXT,
    "Question"              TEXT,
    "Answer"                TEXT,
    "Title"                 TEXT,
    "ByLikes"               INTEGER NOT NULL DEFAULT 0,
    "RootBlogName"          TEXT,
    "RootURL"               TEXT,
    "DateModified"          TEXT,
    "DateCreated"           TEXT,
    PostType                TEXT,
    PRIMARY KEY("BlogName","PostID")
);
```

**The key is `(BlogName, PostID)`, not `PostID`.** This matters more than it looks: 325
post IDs exist under more than one blog, so an ID on its own is both ambiguous *and*
unindexed. Any lookup should carry the blog name, and a batch lookup should group by blog
so it stays on the leading column of the key.

Notable:

- **`PostType` is `NULL` on all 14,589 rows.** The column exists but nothing has ever
  populated it. Treat it as unpopulated rather than as a type discriminator.
- `HasImage = 1` on 14,268 rows — nearly all of them. It records that the post *had* a
  picture, not that a usable URL was kept, so it is not a reliable predictor that anything
  will render.
- `PhotoURL` is largely unused; in practice the image markup lives inside `Body`.
- `NotFound = 1` on 4,663 rows — posts that have since been deleted upstream.
- The content columns (`Body`, `Quote`, `Question`, `Answer`, …) are the heavy ones. List
  views should not select them.

### `Notes`

```sql
CREATE TABLE "Notes" (
    "RootBlogName"      TEXT,
    "PostID"            INTEGER,
    "NoteBlogName"      TEXT,
    "TimeStamp"         INTEGER,
    "Type"              TEXT,
    "replyText"         TEXT DEFAULT '.',
    "DatetimeCrawled"   TEXT DEFAULT '2/12/26 12am',
    "DateModified"      TEXT,
    "DateCreated"       TEXT,
    IsActive            INTEGER NOT NULL DEFAULT 1,
    PRIMARY KEY("RootBlogName","PostID","TimeStamp","Type","NoteBlogName")
) WITHOUT ROWID;

CREATE INDEX "ix_NoteBlogName01" ON "Notes" ("NoteBlogName");
```

**`WITHOUT ROWID`, since 2026-08-07.** The rows live in the primary key's b-tree
rather than in a rowid table with a separate key index beside it. Nothing about the
SQL surface changes — same columns, same types, same constraint — but two
consequences are worth knowing before adding an index here:

- There is no `rowid` on this table. `SELECT rowid FROM Notes` is an error, and no
  code in any of the three apps relied on it.
- A secondary index carries the whole five-column primary key as its row reference
  instead of a compact rowid, so indexes on this table are **expensive**.
  `ix_NoteBlogName01` costs 58 MB, up from 25 MB before the conversion. It earns
  that: Rolodex filters on `NoteBlogName` and the crawler joins on it.

**`Notes_idx_06e01ae3` on `TimeStamp DESC` was dropped at the same time.** It cost
14 MB as a rowid index and would have cost 58 MB after the conversion. It was worth
neither: the crawler's only `TimeStamp` filter (`>= 1535778000`) excludes 786 rows
of 1.18M, Rolodex's default Notes sort carries a three-column tiebreaker that forces
a full sort regardless, and the reply-matching `UPDATE` uses `ABS(TimeStamp - ?) <= 5`,
which no index on `TimeStamp` can serve. The one path that got slower is Rolodex's
Notes page with a date-range filter: 60 ms to 164 ms.

See `../shrink-db.sql` for the full rationale and the applied result.

**`DatetimeCrawled` is `NULL` on 1,148,077 rows, and that is the honest value.** Those
rows previously stored the literal string `'2/12/26 12am'` — this column's own DDL
default, written as a bulk backfill placeholder rather than as a crawl time. They were
set to `NULL` on 2026-08-07, which is what consumers already displayed them as: the
string parses as a date in neither format this schema writes.

Note the trap: **the `DEFAULT '2/12/26 12am'` clause is still in the DDL above.** Any
`INSERT` that omits this column writes the placeholder straight back. The crawler names
it explicitly on every insert, so nothing reintroduces it today, but a new writer that
forgets to would — which is why consumers should keep treating an unparseable value here
as "unknown" rather than assuming `NULL` is now the only such marker.

One row per engagement event. `TimeStamp` is **unix seconds** — unlike every date column
elsewhere in the schema, which are text.

| `Type` | Rows | Share |
|---|--:|--:|
| `like` | 947,955 | 79.7% |
| `reblog` | 224,323 | 18.9% |
| `reply` | 15,201 | 1.3% |
| `posted` | 2,106 | 0.2% |
| `post_attribution` | 19 | — |

At 1.19M rows this is the table that dictates how the whole database has to be queried:

- **Nothing should run an unbounded `SELECT` or a bare `COUNT(*)` here.** A count scans
  the lot on every call.
- The only fast access paths are the primary key's leading columns (`RootBlogName`, then
  `PostID`) and `ix_NoteBlogName01` on `NoteBlogName`. "Notes received by a blog" and
  "notes given by a blog" are both cheap; almost nothing else is.
- **Every** ordering here is a full sort of whatever the filters leave, `TimeStamp`
  included. That was already true in practice of the default `TimeStamp` order, whose
  tiebreakers forced a sort even while `Notes_idx_06e01ae3` existed; since that index
  was dropped on 2026-08-07 it is true unconditionally. Filter first, then sort.
- `replyText` is `'.'` on 1,167,464 rows — only `reply` notes carry real text.

### Referential integrity

There are no foreign keys, and the tables do not perfectly agree:

- 4 `Posts` rows name a blog with no `Blogs` row.
- 15 of the 31,888 distinct engagers have no `Blogs` row.

So a name appearing in `Notes` or `Posts` is not a guarantee that the registry knows about
it. Joins from those tables back to `Blogs` should tolerate a miss.

---

## The `'.'` placeholder convention

**The crawler writes a single dot into text columns it has no value for, rather than
`NULL`.** This is the single most surprising thing about the schema and it affects every
consumer.

| Column | `'.'` rows |
|---|--:|
| `Notes.replyText` | 1,167,464 |
| `Posts.Title` | 13,144 |
| `Posts.Body` | 172 |

Any query whose output reaches a human should collapse it:

```sql
NULLIF(NULLIF(SomeColumn, '.'), '') AS SomeColumn
```

Empty string turns up too, hence the double `NULLIF`. Not every column is affected —
`Blogs.TTFolderPath` and `Blogs.DateModified` currently have zero dot rows — but new
columns tend to acquire them, so treat cleaning as the default for any text column
rendered to a user.

---

## Supporting tables

Crawler bookkeeping. Rolodex ignores all of these.

| Table | Rows | What it is |
|---|--:|---|
| `DailyAPICount` | 133 | `(Date TEXT PK, APICount INTEGER)` — per-day API call tally against the rate limit |
| `ApiKeyPoolState` | 2 | `(KeyName TEXT PK, RetryUntil INTEGER)` — per-key backoff; `RetryUntil` is unix seconds |
| `ApiKeyPoolMeta` | 1 | `(Id PK CHECK (Id = 1), LastIndex)` — round-robin cursor. Singleton by check constraint |
| `CollectRunState` | 1 | `(Id PK CHECK (Id = 1), RunCutoff, RunComplete, RunStarted, RunCompletedAt)` — resume state for an interrupted collection run. Also a singleton |

---

## `Blogs.IsActive` — now written by two applications

`IsActive` has always been the crawler's work-selection flag. `GetBlogs` in
`DataAccess.cs` joins on it to decide what to collect:

```sql
SELECT NoteBlogName, count(*) FROM notes
INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName
WHERE blogs.IsActive = @isActive AND ...
```

Nothing inside the crawler *writes* it — it is an input, set from outside.

**Rolodex is now one of the things that sets it.** Removing a blog through the Rolodex UI
runs exactly this:

```sql
UPDATE Blogs SET IsActive = 0 WHERE BlogName = ?;
```

Rolodex adds no column and changes no schema. It reuses this flag because the two meanings
were judged to be one decision: a blog you do not want in the browsing UI is a blog you do
not want to keep crawling. Removal therefore stops collection, and the Rolodex
confirmation screen says so before anyone commits.

- `1` (or absent/NULL) — live. Crawled, and visible in Rolodex.
- `0` — removed. Not crawled, hidden from the Rolodex registry, dashboard counts and
  engagement rollups.

Restoring is the same `UPDATE` with a `1`. Nothing is destroyed either way: the blog's
`Posts` and `Notes` rows are never touched, and Rolodex deliberately keeps showing them
under its Posts and Notes pages. Removing a blog hides the blog, not what it collected.

### What other tools need to know

1. **Setting `IsActive = 0` now also hides the blog from Rolodex**, and setting it back to
   `1` makes it reappear. If another tool deactivates blogs in bulk, it is also removing
   them from the browsing UI — which may be exactly right, but it is no longer a
   crawler-only decision.
2. **Re-crawling a removed blog will not bring it back**, since nothing in the crawler
   writes the flag. An `INSERT OR REPLACE` on the `Blogs` row *would*, by resetting it to
   the column default of `1`. Prefer an `UPDATE` of the specific columns, or
   `INSERT … ON CONFLICT DO UPDATE SET` naming only the columns being refreshed.
3. **NULL is treated as live.** The column is `INTEGER DEFAULT 1` with no `NOT NULL`, so
   Rolodex reads it through `COALESCE(IsActive, 1)`. A NULL therefore leaves the blog
   visible rather than stranding it outside both the registry and the removed list, where
   no screen could reach it. Write `0` or `1`, not NULL.
4. **Backing the feature out is a configuration change, not a migration.** Because there is
   no Rolodex-owned column, setting `Rolodex__EnableBlogDeletion=false` is the whole of it;
   there is nothing to drop. Any blogs already at `IsActive = 0` simply go back to being
   ordinary inactive blogs.

---

## `Posts.IsActive` and `Notes.IsActive` — optional, and not in this database yet

The same flag is being extended to the two content tables, with the same meaning: `0` is
removed, anything else — including `NULL` — is live. **Neither column exists in the live
`TL.db` as of 2026-07-29**; the DDL quoted above for `Posts` and `Notes` is complete. Like
`Blogs.IsActive`, they are written from outside this crawler.

The crawler therefore treats both as optional, and as nothing it owns:

- **It never writes them.** No `INSERT` column list names `IsActive`, no `UPDATE` sets it,
  and `MapPrefixToColumn` — the only place a column name is chosen at runtime — cannot map
  to it. Re-crawling a removed post or note refreshes its content and leaves the flag at
  `0`. There is no `INSERT OR REPLACE` on `Posts` or `Notes` for a default to be reset by.
- **It filters on them only when they exist.** `HasIsActiveColumn` in `DataAccess.cs` asks
  `PRAGMA table_info` once per table per database path and caches the answer; the filter
  is `COALESCE(IsActive, 1) = 1`, and it is dropped entirely when the column is absent.
  Naming a missing column is a hard SQLite error, so this is what lets one build run
  against databases on both sides of the change. The cache lives for the process — adding
  the columns to a live database takes effect on the next run.

Every read that selects posts or notes carries the filter: `GetPosts`, `GetReplies`,
`GetRepliesWithMissingText`, `GetRepliesWithFilledText`, `GetAllPostTextColumns`,
`GetAllPostsForBlog`, `GetPost`, `GetPostByIdAnyBlog`, and the engagement queries that
count or join `Notes` (`GetBlogs`, `GetBlogsAll`, `GetBlogsForLikes`). The one deliberate
omission is the `LEFT JOIN Notes` in `GetPosts`: nothing is selected from it and it can
neither add nor remove a row, so filtering it would buy nothing.

`LegacyPostsDbImporter` is unfiltered too — it reads a foreign legacy database whose
`Posts` table is not this schema.

Two consequences worth stating plainly, both inherited from how `Blogs.IsActive` is
handled:

1. **Removal hides a row; it does not freeze it.** The write paths are keyed on a post the
   caller already selected, so an ingest or a correction run still overwrites the content
   of a removed post. Only selection is filtered.
2. **`NULL` is live.** Write `0` or `1`, not `NULL`, but a `NULL` leaves the row visible
   rather than stranding it.

---

## Reproducing the numbers

```sql
SELECT 'Blogs', COUNT(*) FROM Blogs
UNION ALL SELECT 'Posts', COUNT(*) FROM Posts
UNION ALL SELECT 'Notes', COUNT(*) FROM Notes;

-- note type mix
SELECT Type, COUNT(*) FROM Notes GROUP BY Type ORDER BY 2 DESC;

-- the two date shapes in Blogs.DateAdded
SELECT CASE WHEN DateAdded LIKE '____-__-__%' THEN 'ISO' ELSE 'US' END, COUNT(*)
FROM Blogs GROUP BY 1;

-- post IDs that are ambiguous without a blog name
SELECT COUNT(*) FROM (
    SELECT PostID FROM Posts GROUP BY PostID HAVING COUNT(DISTINCT BlogName) > 1);

-- rows that reference a blog the registry does not have
SELECT COUNT(*) FROM Posts p
WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = p.BlogName);
```

Open the file read-only so an inspection can never disturb a running crawl:

```bash
sqlite3 "file:TL.db?mode=ro" ".schema"
```