jimandClaude Opus 5
eded5271ea
Replaces the Blogs.IsDeleted section. Rolodex adds no column of its own; it reuses the crawler's existing IsActive flag, so removing a blog in the UI also stops it being collected. Co-Authored-By: Claude Opus 5 <[email protected]>
293 lines
11 KiB
Markdown
293 lines
11 KiB
Markdown
# `TL.db` — schema notes
|
|
|
|
The SQLite database behind **URLNotesGrabberCORE** and its sibling crawlers, and the one
|
|
[Rolodex](https://git.basso.land/jim/Rolodex) reads.
|
|
|
|
Everything below was read out of the live file, not inferred from code. Counts are as of
|
|
**2026-07-29**; re-run the queries at the bottom to refresh them.
|
|
|
|
- Journal mode: **WAL** — `TL.db-wal` and `TL.db-shm` live beside the file and are part of
|
|
the database. Copying `TL.db` alone gives you whatever was last checkpointed, not the
|
|
current state.
|
|
- Page size: 4096.
|
|
|
|
---
|
|
|
|
## The three content tables
|
|
|
|
| Table | Rows | What it is |
|
|
|---|--:|---|
|
|
| `Blogs` | 144,367 | The crawl registry — one row per known blog, plus crawl-state flags |
|
|
| `Posts` | 14,589 | Stored post content. Only 3,602 blogs actually have any |
|
|
| `Notes` | 1,189,604 | The engagement graph: `NoteBlogName` acted on `(RootBlogName, PostID)` |
|
|
|
|
The engagement graph is the interesting part. 31,888 distinct blogs appear as engagers —
|
|
far more than the 3,602 that have stored posts — which is what makes this a social graph
|
|
rather than a post archive.
|
|
|
|
### `Blogs`
|
|
|
|
```sql
|
|
CREATE TABLE "Blogs" (
|
|
"BlogName" TEXT,
|
|
"HasBeenOutput" INTEGER DEFAULT 0,
|
|
"IsActive" INTEGER DEFAULT 1,
|
|
"DateAdded" TEXT NOT NULL DEFAULT '12/24/25',
|
|
"ByLikes" INTEGER NOT NULL DEFAULT 0,
|
|
"LikesPulled" INTEGER NOT NULL DEFAULT 0,
|
|
"LikesCursor" INTEGER DEFAULT 0,
|
|
"DateModified" TEXT,
|
|
"DateCreated" TEXT,
|
|
LikesNewestTimestamp INTEGER DEFAULT 0,
|
|
LikesLastRefreshed INTEGER DEFAULT 0,
|
|
LikesLastNewCount INTEGER DEFAULT 0,
|
|
TTFolderPath TEXT,
|
|
PRIMARY KEY("BlogName")
|
|
);
|
|
```
|
|
|
|
`BlogName` is the primary key, so it is the only indexed way in. There is no index on any
|
|
flag or date — filtering or sorting on those scans all 144k rows, which is affordable
|
|
here and is not on `Notes`.
|
|
|
|
Flag distribution: `IsActive = 1` on 144,366 of 144,367 rows, `HasBeenOutput = 1` on
|
|
5,369, `ByLikes = 1` on 2. `IsActive` carries a second meaning as of Rolodex — see
|
|
[`Blogs.IsActive`](#blogsisactive--now-written-by-two-applications) below.
|
|
|
|
The columns after `DateCreated` were added later by `ALTER TABLE`, which is why they carry
|
|
no quoting in the stored DDL. That is the normal way this schema grows.
|
|
|
|
**`DateAdded` is not written consistently.** 126,423 rows hold ISO `yyyy-MM-dd HH:mm:ss`;
|
|
17,944 hold US-format `M/d/yy` from a bulk import. As text those two sort into different
|
|
parts of the table, so anything ordering or range-filtering on this column has to
|
|
normalise first — see `DateSql` in Rolodex.
|
|
|
|
### `Posts`
|
|
|
|
```sql
|
|
CREATE TABLE "Posts" (
|
|
"BlogName" TEXT,
|
|
"PostID" INTEGER,
|
|
"HasNotesGathered" INTEGER DEFAULT 0,
|
|
"reblogURL" TEXT,
|
|
"NotFound" INTEGER DEFAULT 0,
|
|
"PostDate" TEXT,
|
|
"NotesGatheredDateTime" INTEGER NOT NULL DEFAULT 1729746000,
|
|
"HasImage" INTEGER NOT NULL DEFAULT 0,
|
|
"PostURL" TEXT,
|
|
"Slug" TEXT,
|
|
"ReblogKey" TEXT,
|
|
"ReblogName" TEXT,
|
|
"Summary" TEXT,
|
|
"Quote" TEXT,
|
|
"Body" TEXT,
|
|
"Tags" TEXT,
|
|
"Link" TEXT,
|
|
"PhotoURL" TEXT,
|
|
"PhotoCaption" TEXT,
|
|
"DownloadedFiles" TEXT,
|
|
"AudioCaption" TEXT,
|
|
"Question" TEXT,
|
|
"Answer" TEXT,
|
|
"Title" TEXT,
|
|
"ByLikes" INTEGER NOT NULL DEFAULT 0,
|
|
"RootBlogName" TEXT,
|
|
"RootURL" TEXT,
|
|
"DateModified" TEXT,
|
|
"DateCreated" TEXT,
|
|
PostType TEXT,
|
|
PRIMARY KEY("BlogName","PostID")
|
|
);
|
|
```
|
|
|
|
**The key is `(BlogName, PostID)`, not `PostID`.** This matters more than it looks: 325
|
|
post IDs exist under more than one blog, so an ID on its own is both ambiguous *and*
|
|
unindexed. Any lookup should carry the blog name, and a batch lookup should group by blog
|
|
so it stays on the leading column of the key.
|
|
|
|
Notable:
|
|
|
|
- **`PostType` is `NULL` on all 14,589 rows.** The column exists but nothing has ever
|
|
populated it. Treat it as unpopulated rather than as a type discriminator.
|
|
- `HasImage = 1` on 14,268 rows — nearly all of them. It records that the post *had* a
|
|
picture, not that a usable URL was kept, so it is not a reliable predictor that anything
|
|
will render.
|
|
- `PhotoURL` is largely unused; in practice the image markup lives inside `Body`.
|
|
- `NotFound = 1` on 4,663 rows — posts that have since been deleted upstream.
|
|
- The content columns (`Body`, `Quote`, `Question`, `Answer`, …) are the heavy ones. List
|
|
views should not select them.
|
|
|
|
### `Notes`
|
|
|
|
```sql
|
|
CREATE TABLE "Notes" (
|
|
"RootBlogName" TEXT,
|
|
"PostID" INTEGER,
|
|
"NoteBlogName" TEXT,
|
|
"TimeStamp" INTEGER,
|
|
"Type" TEXT,
|
|
"replyText" TEXT DEFAULT '.',
|
|
"DatetimeCrawled" TEXT DEFAULT '2/12/26 12am',
|
|
"DateModified" TEXT,
|
|
"DateCreated" TEXT,
|
|
PRIMARY KEY("RootBlogName","PostID","TimeStamp","Type","NoteBlogName")
|
|
);
|
|
|
|
CREATE INDEX "Notes_idx_06e01ae3" ON "Notes" ("TimeStamp" DESC);
|
|
CREATE INDEX "ix_NoteBlogName01" ON "Notes" ("NoteBlogName");
|
|
```
|
|
|
|
One row per engagement event. `TimeStamp` is **unix seconds** — unlike every date column
|
|
elsewhere in the schema, which are text.
|
|
|
|
| `Type` | Rows | Share |
|
|
|---|--:|--:|
|
|
| `like` | 947,955 | 79.7% |
|
|
| `reblog` | 224,323 | 18.9% |
|
|
| `reply` | 15,201 | 1.3% |
|
|
| `posted` | 2,106 | 0.2% |
|
|
| `post_attribution` | 19 | — |
|
|
|
|
At 1.19M rows this is the table that dictates how the whole database has to be queried:
|
|
|
|
- **Nothing should run an unbounded `SELECT` or a bare `COUNT(*)` here.** A count scans
|
|
the lot on every call.
|
|
- The only fast access paths are the primary key's leading columns (`RootBlogName`, then
|
|
`PostID`) and `ix_NoteBlogName01` on `NoteBlogName`. "Notes received by a blog" and
|
|
"notes given by a blog" are both cheap; almost nothing else is.
|
|
- Ordering by anything but `TimeStamp` is a full sort of whatever the filters leave.
|
|
- `replyText` is `'.'` on 1,174,706 rows — only `reply` notes carry real text.
|
|
|
|
### Referential integrity
|
|
|
|
There are no foreign keys, and the tables do not perfectly agree:
|
|
|
|
- 4 `Posts` rows name a blog with no `Blogs` row.
|
|
- 15 of the 31,888 distinct engagers have no `Blogs` row.
|
|
|
|
So a name appearing in `Notes` or `Posts` is not a guarantee that the registry knows about
|
|
it. Joins from those tables back to `Blogs` should tolerate a miss.
|
|
|
|
---
|
|
|
|
## The `'.'` placeholder convention
|
|
|
|
**The crawler writes a single dot into text columns it has no value for, rather than
|
|
`NULL`.** This is the single most surprising thing about the schema and it affects every
|
|
consumer.
|
|
|
|
| Column | `'.'` rows |
|
|
|---|--:|
|
|
| `Notes.replyText` | 1,174,706 |
|
|
| `Posts.Title` | 13,144 |
|
|
| `Posts.Body` | 172 |
|
|
|
|
Any query whose output reaches a human should collapse it:
|
|
|
|
```sql
|
|
NULLIF(NULLIF(SomeColumn, '.'), '') AS SomeColumn
|
|
```
|
|
|
|
Empty string turns up too, hence the double `NULLIF`. Not every column is affected —
|
|
`Blogs.TTFolderPath` and `Blogs.DateModified` currently have zero dot rows — but new
|
|
columns tend to acquire them, so treat cleaning as the default for any text column
|
|
rendered to a user.
|
|
|
|
---
|
|
|
|
## Supporting tables
|
|
|
|
Crawler bookkeeping. Rolodex ignores all of these.
|
|
|
|
| Table | Rows | What it is |
|
|
|---|--:|---|
|
|
| `DailyAPICount` | 133 | `(Date TEXT PK, APICount INTEGER)` — per-day API call tally against the rate limit |
|
|
| `ApiKeyPoolState` | 2 | `(KeyName TEXT PK, RetryUntil INTEGER)` — per-key backoff; `RetryUntil` is unix seconds |
|
|
| `ApiKeyPoolMeta` | 1 | `(Id PK CHECK (Id = 1), LastIndex)` — round-robin cursor. Singleton by check constraint |
|
|
| `CollectRunState` | 1 | `(Id PK CHECK (Id = 1), RunCutoff, RunComplete, RunStarted, RunCompletedAt)` — resume state for an interrupted collection run. Also a singleton |
|
|
|
|
---
|
|
|
|
## `Blogs.IsActive` — now written by two applications
|
|
|
|
`IsActive` has always been the crawler's work-selection flag. `GetBlogs` in
|
|
`DataAccess.cs` joins on it to decide what to collect:
|
|
|
|
```sql
|
|
SELECT NoteBlogName, count(*) FROM notes
|
|
INNER JOIN blogs ON blogs.BlogName = notes.NoteBlogName
|
|
WHERE blogs.IsActive = @isActive AND ...
|
|
```
|
|
|
|
Nothing inside the crawler *writes* it — it is an input, set from outside.
|
|
|
|
**Rolodex is now one of the things that sets it.** Removing a blog through the Rolodex UI
|
|
runs exactly this:
|
|
|
|
```sql
|
|
UPDATE Blogs SET IsActive = 0 WHERE BlogName = ?;
|
|
```
|
|
|
|
Rolodex adds no column and changes no schema. It reuses this flag because the two meanings
|
|
were judged to be one decision: a blog you do not want in the browsing UI is a blog you do
|
|
not want to keep crawling. Removal therefore stops collection, and the Rolodex
|
|
confirmation screen says so before anyone commits.
|
|
|
|
- `1` (or absent/NULL) — live. Crawled, and visible in Rolodex.
|
|
- `0` — removed. Not crawled, hidden from the Rolodex registry, dashboard counts and
|
|
engagement rollups.
|
|
|
|
Restoring is the same `UPDATE` with a `1`. Nothing is destroyed either way: the blog's
|
|
`Posts` and `Notes` rows are never touched, and Rolodex deliberately keeps showing them
|
|
under its Posts and Notes pages. Removing a blog hides the blog, not what it collected.
|
|
|
|
### What other tools need to know
|
|
|
|
1. **Setting `IsActive = 0` now also hides the blog from Rolodex**, and setting it back to
|
|
`1` makes it reappear. If another tool deactivates blogs in bulk, it is also removing
|
|
them from the browsing UI — which may be exactly right, but it is no longer a
|
|
crawler-only decision.
|
|
2. **Re-crawling a removed blog will not bring it back**, since nothing in the crawler
|
|
writes the flag. An `INSERT OR REPLACE` on the `Blogs` row *would*, by resetting it to
|
|
the column default of `1`. Prefer an `UPDATE` of the specific columns, or
|
|
`INSERT … ON CONFLICT DO UPDATE SET` naming only the columns being refreshed.
|
|
3. **NULL is treated as live.** The column is `INTEGER DEFAULT 1` with no `NOT NULL`, so
|
|
Rolodex reads it through `COALESCE(IsActive, 1)`. A NULL therefore leaves the blog
|
|
visible rather than stranding it outside both the registry and the removed list, where
|
|
no screen could reach it. Write `0` or `1`, not NULL.
|
|
4. **Backing the feature out is a configuration change, not a migration.** Because there is
|
|
no Rolodex-owned column, setting `Rolodex__EnableBlogDeletion=false` is the whole of it;
|
|
there is nothing to drop. Any blogs already at `IsActive = 0` simply go back to being
|
|
ordinary inactive blogs.
|
|
|
|
---
|
|
|
|
## Reproducing the numbers
|
|
|
|
```sql
|
|
SELECT 'Blogs', COUNT(*) FROM Blogs
|
|
UNION ALL SELECT 'Posts', COUNT(*) FROM Posts
|
|
UNION ALL SELECT 'Notes', COUNT(*) FROM Notes;
|
|
|
|
-- note type mix
|
|
SELECT Type, COUNT(*) FROM Notes GROUP BY Type ORDER BY 2 DESC;
|
|
|
|
-- the two date shapes in Blogs.DateAdded
|
|
SELECT CASE WHEN DateAdded LIKE '____-__-__%' THEN 'ISO' ELSE 'US' END, COUNT(*)
|
|
FROM Blogs GROUP BY 1;
|
|
|
|
-- post IDs that are ambiguous without a blog name
|
|
SELECT COUNT(*) FROM (
|
|
SELECT PostID FROM Posts GROUP BY PostID HAVING COUNT(DISTINCT BlogName) > 1);
|
|
|
|
-- rows that reference a blog the registry does not have
|
|
SELECT COUNT(*) FROM Posts p
|
|
WHERE NOT EXISTS (SELECT 1 FROM Blogs b WHERE b.BlogName = p.BlogName);
|
|
```
|
|
|
|
Open the file read-only so an inspection can never disturb a running crawl:
|
|
|
|
```bash
|
|
sqlite3 "file:TL.db?mode=ro" ".schema"
|
|
```
|