Skip to content

field_note · aug 24

Why I put SQLite in the browser: IndexedDB can't filter on one field and sort by another

The one query that made me stop working around IndexedDB and start replacing it. What a cursor can and cannot do, why fetch-everything-and-sort-in-JS stops working at a few thousand rows, and what a WASM SQLite build actually costs you.

Every offline-capable app I have built eventually asks the browser for the same thing:

Give me the open issues, newest first, twenty-five at a time.

Filter on one field, sort by another, page the result. In Postgres that is one statement and an index. In IndexedDB it is not expressible, and the workaround you reach for instead is the reason this post exists.

What a cursor can actually do

IndexedDB gives you object stores and indexes, and you read through a cursor. A cursor walks exactly one index, in that index's order.

So you can have this:

terminal
// ordered by status, because that is the index you opened
store.index('status').openCursor(IDBKeyRange.only('open'))

Or this:

terminal
// ordered by updated, but now you see every status
store.index('updated').openCursor(null, 'prev')

You cannot have both. The index you filter on is the index that determines your order. There is no second index to apply, no planner deciding to seek on one and sort by another, no ORDER BY separate from the WHERE.

One index per query, versus a plannerIndexedDB cursoropen indexstatuswalk 1,199 rowsin status ordersort all of them in JavaScripton the main thread, to show 25SQL plannerseekstatus = openorder by updateda second indexreturn 25 rowsoff the main threadThe difference is not speed. It is how many rows cross into JavaScript.One returns everything that matched. The other returns the page you asked for.

A cursor is not a slow planner. It is a different thing — an ordered walk of one index, with no mechanism for a second ordering.

The workaround, and where it stops working

So you do what everyone does. Fetch the matching rows, sort them in JavaScript, slice the page you wanted:

terminal
const all = await db.issues.where('status').equals('open').toArray();
all.sort((a, b) => b.updated - a.updated);
const page = all.slice(0, 25);

This is correct. It is also fine — genuinely fine — at a few hundred rows.

What it does is trade a bounded cost for an unbounded one. toArray() on 1,199 open issues doesn't return 25 rows, it returns 1,199, each one deserialised across the structured clone boundary into a JavaScript object, on the main thread, so you can throw away 1,174 of them. Add a second filter and it gets worse, because you are now scanning in JS what an index would have skipped.

The tell is that the cost scales with how much data matched, not with how much you are displaying. A list view that is instant on your seed data and janky on a real account is almost always this.

Why not just add a compound index?

Because it only works when you know the filter in advance.

[status+updated] handles "open issues by date" beautifully. Then someone adds a label filter, and a date range, and an assignee — and a compound index answers exactly one combination of them. Every additional filter is another index, and n optional filters means you cannot enumerate the combinations.

This is what a query planner is for. It picks the index at query time from the predicates it was given. There is no planner in IndexedDB, so the picking has to happen when you write the schema, which is to say before you know.

What I built instead

granthdb is SQLite compiled to WebAssembly, running in the browser tab, behind the API Dexie already gave everyone:

terminal
const grownups = await db.friends
  .where('age').above(18)
  .orderBy('name')
  .toArray();

If you have written Dexie, you have written this. That is deliberate — the API is not the interesting part, and asking people to learn a new one to get a query planner is a bad trade. What changed is underneath.

Documents are stored as JSON in a column. Declared indexes become generated columns over json_extract, with real SQLite indexes on them. So the planner does actual index work on what is, from your side, still a document store — and where(...).orderBy(...) compiles to one statement that returns twenty-five rows.

A few things fall out of having SQL down there rather than a cursor:

  • count() without iterating anything
  • sum(), avg(), min(), max() evaluated in SQLite, so one number crosses the worker boundary instead of every row that went into it
  • bulkGet of 500 keys as one IN query rather than 500 round trips
  • deep paging that stays correct, because the ordering is pinned to the bound index rather than to insertion order

What it costs, honestly

This is not free, and the places it is not free are predictable.

A WASM download. SQLite compiled to WebAssembly is several hundred kilobytes. Notion found that loading it synchronously made their first page slower than the network it was replacing, and shipped it fully asynchronously with the first page served from the network. That is the right shape: the local database earns its cost on the second navigation, not the first.

A worker, and therefore async everywhere. The queries run off the main thread, which is the point, but it means there is no synchronous read anywhere. If your current code does localStorage.getItem in a render path, that is a real refactor.

It is a cache, not a source of truth. Browser storage is evictable — Safari clears script-writable storage after seven days without interaction. Anything you put here has to be rebuildable from your server, and if it isn't, this is the wrong tool.

It is not a sync engine. It keeps no server copy and resolves no conflicts between users. If two people editing the same record is your problem, this layer does not solve it and bolting sync on is the larger project.

When the answer is still IndexedDB, or nothing

I would rather say this here than have someone find out later:

  • A handful of key-value reads — a theme, a dismissed banner, a feature flag. Use localStorage. Shipping a WASM SQLite build to store {"theme":"dark"} is worse engineering, not better.
  • A few hundred records you read once. Fetch and sort in JS. It is fine, and it is less code than any of this.
  • Data that changes constantly for everyone. A live ticker has nothing to cache; you would be adding a database to display a websocket.

The line is roughly: are you storing a list you query, or a blob you read? A blob you read wants no database at all.

The measurement that decides it

If you take one thing from this: the question is not "is local faster than the network". It is "faster for whom".

Notion measured roughly a 20% improvement in navigation time overall — and 28% in Australia, 31% in China, 33% in India, because that is where the network was the bottleneck. But on slow devices their p95 got worse before they tuned it, since reading from a cheap disk can lose to a fast connection.

Their fix is the one worth stealing: stop choosing, and race them.

terminal
const rows = await Promise.any([
  db.pages.where('workspace').equals(id).toArray()
    .then((r) => (r.length ? r : Promise.reject())),
  fetch(`/api/pages?workspace=${id}`).then((r) => r.json()),
]);

Promise.any rather than race, so an empty or failed local read cannot win by returning nothing. Local usually wins and costs nothing; when the disk is slow it loses, and the user gets the network answer instead of waiting for the cache to lose slowly.

Measure p95 and p99, not the mean. The mean hides exactly the users you hurt.

Get it

terminal
npm install granthdb @sqlite.org/sqlite-wasm

There is a sandbox if you would rather write a query than install anything.

The two follow-ups to this are how two tabs share one SQLite file without corrupting it, which is the genuinely hard part, and when a browser app needs a real database, which is the honest version of this decision. The full write-up of the project is at /case-studies/granthdb-browser-sqlite.

Working through something like this? I help teams ship AI and cloud systems that hold up, and cost what they should.