URLNotesGrabberCore/URLNotesGrabberCORE/TL.db.md

TL.db — schema notes

The SQLite database behind URLNotesGrabberCORE and its sibling crawlers, and the one Rolodex reads.

Everything below was read out of the live file, not inferred from code. Counts are as of 2026-07-29; re-run the queries at the bottom to refresh them.

• Journal mode: WAL — TL.db-wal and TL.db-shm live beside the file and are part of the database. Copying TL.db alone gives you whatever was last checkpointed, not the current state.
• Page size: 4096.

---

The three content tables

Table	Rows	What it is
Blogs	144,367	The crawl registry — one row per known blog, plus crawl-state flags
Posts	14,589	Stored post content. Only 3,602 blogs actually have any
Notes	1,189,604	The engagement graph: NoteBlogName acted on (RootBlogName, PostID)

The engagement graph is the interesting part. 31,888 distinct blogs appear as engagers — far more than the 3,602 that have stored posts — which is what makes this a social graph rather than a post archive.

Blogs

CREATE TABLE "Blogs" (
    "BlogName"              TEXT,
    "HasBeenOutput"         INTEGER DEFAULT 0,
    "IsActive"              INTEGER DEFAULT 1,
    "DateAdded"             TEXT NOT NULL DEFAULT '12/24/25',
    "ByLikes"               INTEGER NOT NULL DEFAULT 0,
    "LikesPulled"           INTEGER NOT NULL DEFAULT 0,
    "LikesCursor"           INTEGER DEFAULT 0,
    "DateModified"          TEXT,
    "DateCreated"           TEXT,
    LikesNewestTimestamp    INTEGER DEFAULT 0,
    LikesLastRefreshed      INTEGER DEFAULT 0,
    LikesLastNewCount       INTEGER DEFAULT 0,
    TTFolderPath            TEXT,
    PRIMARY KEY("BlogName")
);

BlogName is the primary key, so it is the only indexed way in. There is no index on any flag or date — filtering or sorting on those scans all 144k rows, which is affordable here and is not on Notes.

Flag distribution: IsActive = 1 on 144,366 of 144,367 rows (effectively always true, so the Active filter is close to a no-op today), HasBeenOutput = 1 on 5,369, ByLikes = 1 on 2.

The columns after DateCreated were added later by ALTER TABLE, which is why they carry no quoting in the stored DDL. That is the normal way this schema grows — and it is how IsDeleted gets there too, so expect that column in the DDL of any database Rolodex has opened.

DateAdded is not written consistently. 126,423 rows hold ISO yyyy-MM-dd HH:mm:ss; 17,944 hold US-format M/d/yy from a bulk import. As text those two sort into different parts of the table, so anything ordering or range-filtering on this column has to normalise first — see DateSql in Rolodex.

Posts

CREATE TABLE "Posts" (
    "BlogName"              TEXT,
    "PostID"                INTEGER,
    "HasNotesGathered"      INTEGER DEFAULT 0,
    "reblogURL"             TEXT,
    "NotFound"              INTEGER DEFAULT 0,
    "PostDate"              TEXT,
    "NotesGatheredDateTime" INTEGER NOT NULL DEFAULT 1729746000,
    "HasImage"              INTEGER NOT NULL DEFAULT 0,
    "PostURL"               TEXT,
    "Slug"                  TEXT,
    "ReblogKey"             TEXT,
    "ReblogName"            TEXT,
    "Summary"               TEXT,
    "Quote"                 TEXT,
    "Body"                  TEXT,
    "Tags"                  TEXT,
    "Link"                  TEXT,
    "PhotoURL"              TEXT,
    "PhotoCaption"          TEXT,
    "DownloadedFiles"       TEXT,
    "AudioCaption"          TEXT,
    "Question"              TEXT,
    "Answer"                TEXT,
    "Title"                 TEXT,
    "ByLikes"               INTEGER NOT NULL DEFAULT 0,
    "RootBlogName"          TEXT,
    "RootURL"               TEXT,
    "DateModified"          TEXT,
    "DateCreated"           TEXT,
    PostType                TEXT,
    PRIMARY KEY("BlogName","PostID")
);

The key is (BlogName, PostID), not PostID. This matters more than it looks: 325 post IDs exist under more than one blog, so an ID on its own is both ambiguous and unindexed. Any lookup should carry the blog name, and a batch lookup should group by blog so it stays on the leading column of the key.

Notable:

• PostType is NULL on all 14,589 rows. The column exists but nothing has ever populated it. Treat it as unpopulated rather than as a type discriminator.
• HasImage = 1 on 14,268 rows — nearly all of them. It records that the post had a picture, not that a usable URL was kept, so it is not a reliable predictor that anything will render.
• PhotoURL is largely unused; in practice the image markup lives inside Body.
• NotFound = 1 on 4,663 rows — posts that have since been deleted upstream.
• The content columns (Body, Quote, Question, Answer, …) are the heavy ones. List views should not select them.

Notes

CREATE TABLE "Notes" (
    "RootBlogName"    TEXT,
    "PostID"          INTEGER,
    "NoteBlogName"    TEXT,
    "TimeStamp"       INTEGER,
    "Type"            TEXT,
    "replyText"       TEXT DEFAULT '.',
    "DatetimeCrawled" TEXT DEFAULT '2/12/26 12am',
    "DateModified"    TEXT,
    "DateCreated"     TEXT,
    PRIMARY KEY("RootBlogName","PostID","TimeStamp","Type","NoteBlogName")
);

CREATE INDEX "Notes_idx_06e01ae3" ON "Notes" ("TimeStamp" DESC);
CREATE INDEX "ix_NoteBlogName01"  ON "Notes" ("NoteBlogName");

One row per engagement event. TimeStamp is unix seconds — unlike every date column elsewhere in the schema, which are text.

Type	Rows	Share
like	947,955	79.7%
reblog	224,323	18.9%
reply	15,201	1.3%
posted	2,106	0.2%
post_attribution	19	—

At 1.19M rows this is the table that dictates how the whole database has to be queried:

• Nothing should run an unbounded SELECT or a bare COUNT(*) here. A count scans the lot on every call.
• The only fast access paths are the primary key's leading columns (RootBlogName, then PostID) and ix_NoteBlogName01 on NoteBlogName. "Notes received by a blog" and "notes given by a blog" are both cheap; almost nothing else is.
• Ordering by anything but TimeStamp is a full sort of whatever the filters leave.
• replyText is '.' on 1,174,706 rows — only reply notes carry real text.

Referential integrity

There are no foreign keys, and the tables do not perfectly agree:

• 4 Posts rows name a blog with no Blogs row.
• 15 of the 31,888 distinct engagers have no Blogs row.

So a name appearing in Notes or Posts is not a guarantee that the registry knows about it. Joins from those tables back to Blogs should tolerate a miss.

---

The '.' placeholder convention

The crawler writes a single dot into text columns it has no value for, rather than NULL. This is the single most surprising thing about the schema and it affects every consumer.

Column	'.' rows
Notes.replyText	1,174,706
Posts.Title	13,144
Posts.Body	172

Any query whose output reaches a human should collapse it:

NULLIF(NULLIF(SomeColumn, '.'), '') AS SomeColumn

Empty string turns up too, hence the double NULLIF. Not every column is affected — Blogs.TTFolderPath and Blogs.DateModified currently have zero dot rows — but new columns tend to acquire them, so treat cleaning as the default for any text column rendered to a user.

---

Supporting tables

Crawler bookkeeping. Rolodex ignores all of these.

Table	Rows	What it is
DailyAPICount	133	(Date TEXT PK, APICount INTEGER) — per-day API call tally against the rate limit
ApiKeyPoolState	2	(KeyName TEXT PK, RetryUntil INTEGER) — per-key backoff; RetryUntil is unix seconds
ApiKeyPoolMeta	1	(Id PK CHECK (Id = 1), LastIndex) — round-robin cursor. Singleton by check constraint
CollectRunState	1	(Id PK CHECK (Id = 1), RunCutoff, RunComplete, RunStarted, RunCompletedAt) — resume state for an interrupted collection run. Also a singleton

---

Blogs.IsDeleted — added by Rolodex

ALTER TABLE Blogs ADD COLUMN IsDeleted INTEGER NOT NULL DEFAULT 0;

This column is not written by the crawler. Rolodex adds it automatically at startup if it is missing, and uses it to hide a blog from its own UI without destroying anything:

• 0 — the blog is live. NOT NULL DEFAULT 0 means every existing row, and every row the crawler creates afterwards, starts here.
• 1 — somebody removed the blog through Rolodex.

Removal is a soft delete and nothing else changes: the row keeps every crawl flag it had, and its Posts and Notes rows are untouched. Restoring is the same UPDATE setting the column back to 0.

Do not write NULL here. Rolodex reads the column through COALESCE(IsDeleted, 0) precisely so a NULL cannot strand a row — without it a NULL satisfies neither the live test nor its negation, and the blog would disappear from the registry and from the removed list with no way back through the UI. The NOT NULL declaration is the real guard; the COALESCE is the belt to its braces.

What other tools need to know

1. Adding the column is safe for SELECT, and for INSERT that names its columns. It is not safe for INSERT INTO Blogs VALUES (...) without a column list — that form breaks the moment column count changes. If any crawler does this, give it an explicit column list.
2. The crawler should leave the column alone. Writing to it would silently un-remove or remove blogs behind Rolodex's back.
3. Re-crawling a removed blog will not bring it back. An INSERT OR REPLACE on the Blogs row would, by resetting the column to its default 0. If that matters, prefer an UPDATE of the specific columns, or INSERT … ON CONFLICT DO UPDATE SET naming only the crawl columns.
4. A tool that lists blogs should decide whether it cares. Rolodex filters WHERE COALESCE(IsDeleted, 0) = 0 everywhere a blog surfaces. A crawler probably should not — a blog hidden from a browsing UI is not necessarily one to stop crawling. That is a deliberate choice, not an oversight.

Rolodex degrades if the column is absent — a read-only or immutable=1 deployment cannot take the ALTER — by falling back to its previous behaviour of showing every blog. So removing the column is a supported way to back the feature out:

ALTER TABLE Blogs DROP COLUMN IsDeleted;

---

Reproducing the numbers

SELECT 'Blogs', COUNT(*) FROM Blogs
UNION ALL SELECT 'Posts', COUNT(*) FROM Posts
UNION ALL SELECT 'Notes', COUNT(*) FROM Notes;

-- note type mix
SELECT Type, COUNT(*) FROM Notes GROUP BY Type ORDER BY 2 DESC;

-- the two date shapes in Blogs.DateAdded
SELECT CASE WHEN DateAdded LIKE '____-__-__%' THEN 'ISO' ELSE 'US' END, COUNT(*)
FROM Blogs GROUP BY 1;

-- post IDs that are ambiguous without a blog name
SELECT COUNT(*) FROM (
    SELECT PostID FROM Posts GROUP BY PostID HAVING COUNT(DISTINCT BlogName) > 1);

-- rows that reference a blog the registry does not have
SELECT COUNT(*) FROM Posts p
WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = p.BlogName);

Open the file read-only so an inspection can never disturb a running crawl:

sqlite3 "file:TL.db?mode=ro" ".schema"