The background job that could not finish: OOM, advisory locks, and a cursor on a row

A nightly job in a multi-tenant app had a strange failure signature. The queue dashboard said it succeeded. It said the runtime was 210 milliseconds. It also said the job had spent seventeen minutes in the queue and succeeded on the fourth attempt.

All four of those facts were true, and together they described a job that had never done any work at all.

This is the full path from that dashboard row to a fix, including the two places I was wrong. It is written for anyone who runs a long periodic job over a dataset that grows — a nightly recompute, a scheduled export, a rollup, a sweep. The specific stack is Rails and Postgres with GoodJob, but the failure mode is not Ruby's and the fix is not Rails-specific. Sidekiq, Celery, Sidekiq's sidekiq-cron, Solid Queue, a bare cron entry on a box — the same four questions decide whether your job survives.

The symptom lies, so start at the kernel

The queue told me the job was interrupted. That word covers a lot of ground: a deploy, a graceful shutdown, a lost network partition, a host going away. Guessing between them is how you end up fixing the wrong thing for a week.

On Linux there is one command that settles it:

dmesg -T | grep oom-kill

[Sat Sep 19 14:22:30 2026] worker invoked oom-killer: gfp_mask=0xcc0, order=0
[Sat Sep 19 14:22:30 2026] oom-kill:constraint=CONSTRAINT_MEMCG,...,task=bundle
[Sat Sep 19 14:22:30 2026] Memory cgroup out of memory: Killed process (bundle)
                           anon-rss:815284kB

CONSTRAINT_MEMCG is the important token. This is not the host running out of memory — it is the container's cgroup limit, and the limit was 800 MB against an anon-rss of 815 MB. The process did not crash. It did not raise. The kernel removed it between two instructions.

That distinction matters more than it looks, and it is the hinge of everything below: a process that is killed cannot run its own error handling. No rescue, no ensure, no at_exit, no "mark this run as failed" callback. Any design that depends on the dying process recording its own death is a design that has never been tested against the way processes actually die.

Takeaway: when a job reports "interrupted", "lost", or vanishes with no backtrace, go to dmesg -T on the worker host before you read a single line of application code. It is one command and it eliminates half the hypothesis space.

Why it ran a hundred times instead of failing once

A single out-of-memory kill is a bug. A hundred of them is a loop, and the loop is the expensive part — it is what turned a broken feature into background load heavy enough to slow down unrelated web requests.

Here is the mechanism, and it is worth understanding even if you never touch this particular queue library. GoodJob claims a job by taking a session-level Postgres advisory lock. Session-level means the lock belongs to the database connection, not to a transaction. When the worker process dies, its connection drops, and Postgres releases the lock immediately — that is the correct and desirable behaviour, because it is what stops a crashed worker from wedging a job forever.

But nothing in that cycle counts.

The job is not being retried. A retry implies a counter and a ceiling. This is the job being re-claimed, which is a different thing with no counter anywhere in it. The worker restarts, sees an unclaimed job, takes it, and relearns the same first sixty seconds of work until it is killed again.

So where did "succeeded in 210 ms on the fourth attempt" come from? From a safety valve further up: the application counted abandoned runs and, past a threshold, declined to start another one. Declining is the correct behaviour. Reporting that decline as a success is not — and that is how a completely broken nightly job stayed invisible on a dashboard for days.

Takeaway, and it generalises: if your job can die without raising, then "the job finished" and "the work happened" are two different facts and you need to store them separately. Record progress in your own domain table, not only in the queue's. Queue rows are ephemeral, get pruned on a retention window, and — critically — are written by the very process that is about to be killed.

The first two fixes were real, and neither one was the cause

This is the part I would want to read, so I will not tidy it up.

Fix one: the index. Aggregates over the output table were slow, so I added a covering index. It was a correct index. The query planner still sequential- scanned, and when I forced the index the plan showed the reason:

Index Only Scan using idx_scores_agg  (actual rows=100000)
  Heap Fetches: 28000

An index-only scan is only "only" when Postgres can trust the visibility map to say a page is all-visible. The visibility map is maintained by VACUUM. This table was completely rewritten by an upsert_all every single hour, so a meaningful share of its pages were always dirty and VACUUM could never get far enough ahead. Heap fetches on more than a quarter of the rows is the index doing the scan and the table doing the scan.

The lesson has nothing to do with this app: a table you rewrite faster than you can vacuum it will never sustain index-only scans. If your read pattern needs one, the fix is not a better index — it is to stop deriving the number per request and precompute it into a small table you read by primary key.

Fix two: the obvious memory hog. Two phases of the pipeline loaded whole collections into memory before writing anything. I batched both. Measurable, correct, shipped. The kills continued.

That is the trap worth naming. I had found a cause, fixed it properly, watched the symptom persist, and had to go back. An enumerated fix leaves siblings; a list of three suspects that you fix one at a time will keep looking like a failed fix right up until you find the fourth.

The actual cause was one guard that could never fire

The write-back phase looked bounded. It even had a filter:

def scores_for(type, ids)
  scope = Row.where(type: type)
  scope = scope.where(subject_id: ids) if ids   # <- the guard
  scope.pluck(:subject_id, :score, :evidence, :touched_at).to_h { ... }
end

On an incremental run, ids is a small array and this is fine. On a full run — the only run that was dying — the caller passed nil to mean everything. So if ids was false, the filter never applied, and the method loaded every row in the table into one hash. Each row carried a JSON blob.

Measured inside the running container:

A 291 MB allocation, in one statement, on a process with an 800 MB ceiling that had already spent 346 MB on the rest of the pipeline.

nil meant "no filter" to the query and "everything is fine" to the guard. The fix is to make the parameter required and batch the read, but the design lesson is sharper than the patch: a conditional filter whose "off" state means "unbounded" is not a guard, it is a trapdoor. If some callers legitimately want everything, that should be an explicit, named, visibly dangerous path — scores_for_all_in_batches — not the same method with an argument omitted.

After batching: 169 s → 17 s, peak memory 387 MB → 197 MB.

Bounded is not the same as resumable

Here is where most write-ups stop, and where the interesting design problem actually begins.

The memory fix made the job able to finish. It did nothing about what happens when it is killed for any other reason — a deploy, a node eviction, a spot reclaim, a different memory spike next quarter. A ten-minute job that starts over on every interruption is a job with a hidden deadline: as soon as the mean time between interruptions drops below the runtime, it stops converging. It doesn't get slower. It stops finishing at all, and nothing in your metrics will say so.

I looked hard at Shopify's job-iteration here, because it is the best-known answer in this space and it does support several backends. It is genuinely good, and it did not fit:

job-iteration checkpoints on graceful termination. The worker receives TERM, finishes the current iteration, persists the cursor, and re-enqueues.

Against a deploy, that is exactly right. Against SIGKILL from the OOM killer, there is no signal to receive and no opportunity to persist anything. Same reason as the very first section: a killed process cannot save its own state.

So the checkpoint has to be written as a normal consequence of doing work, not as a reaction to shutting down. Which means the design constraint is:

After every unit of work, the fact that it happened must already be durable before the next unit begins.

That is the whole idea. Everything else is bookkeeping.

The shape: phases, a cursor, and a clock frozen on the row

The pipeline already had ordered phases. I added two columns to the table that already tracked runs — phase and cursor — and turned the job into a driver that performs one bounded slice and writes where it got to.

Four details made it correct rather than merely plausible. They are the ones I would check in a code review of anyone else's version:

1. Keyset paging, not offset. Each slice does WHERE id > ? ORDER BY id LIMIT n. OFFSET re-walks the prefix on every slice, so the job gets quadratically slower as it progresses, and — worse — the underlying set can change during a ten-minute run, which makes OFFSET silently skip and repeat rows. A cursor on the primary key cannot.

2. Every slice must be idempotent. A slice that is interrupted will be run again. Every write is an upsert keyed on the subject, so replaying a slice rewrites its own rows and touches nothing else. If any step in your pipeline appends, increments, or sends, it cannot be a slice until you make it keyed.

3. The clock is frozen on the run row. This one is subtle and it will bite you. The pipeline ends by deleting rows that this run did not write — WHERE computed_at < :now. If each slice used Time.current, a later slice would delete what an earlier slice had just written. So now is read once from the run's started_at and every slice uses it. A twelve-minute run behaves exactly like an instantaneous one.

4. The budget is a wall clock, not a slice count. My first version did one slice per job and re-enqueued. It was elegant and it would have been a disaster: the queue's poll interval in production is 15 seconds, and a large workspace takes about ninety slices. That is twenty-two minutes of waiting to do ninety seconds of work — I would have reintroduced the exact "seventeen minutes in the queue" symptom I started from, with a straight face, in the fix for it.

So the job keeps taking slices until a wall-clock budget is spent, then hands over. The cursor is still written after every slice, so the budget bounds the job without weakening what an interruption costs.

deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + MAX_RUNTIME
loop do
  return finish!(run) if plan.advance! == :done
  break if Process.clock_gettime(Process::CLOCK_MONOTONIC) >= deadline
end
self.class.perform_later(run_id)

Use CLOCK_MONOTONIC, not wall-clock time. NTP can step the system clock backwards and your loop will simply not exit.

One counter-intuitive consequence: I had to remove the concurrency lock. A total_limit: 1 key on this job would see the current job still performing at the moment it enqueues its own successor, and defer the successor to the next poll — paying the queue latency on every hop. Sequencing did not need a lock, because a successor only exists once its predecessor has run.

Result: one job of 266 s at 613 MB became three jobs totalling 75 s at 370 MB, longest single job 31 s. And the first production run of the largest workspace — a workspace that had never once completed — finished in ten minutes with zero retries.

The part nobody warns you about: the cost moved

The run succeeded. Twenty minutes later a database alarm fired.

Bounding the heap did not remove the work. It relocated it. The old code read everything once and hoarded it in Ruby; the new code streams continuously from Postgres, which means the database now holds the working set, the buffer cache churns, and sorts spill to temp files. The job stopped being a memory problem for the worker and became an I/O problem for the database.

This is Tesler's law — complexity is conserved; you only choose who absorbs it — and it is the single most useful thing in this whole post. Every time you fix a resource problem, ask which resource just inherited it. Nothing in my test suite could have told me this. It took one dashboard, on one metric I had no reason to be watching, fifteen minutes after I had already declared victory.

Which made the schedule the last bug

The scheduled sweep had a catch-up rule: run any workspace whose last success is older than 26 hours. Sensible in steady state. But the job had been disabled for a day while I fixed it, so every workspace was stale, and the first tick after re-enabling would have started all of them at once — against the database that had just shown me it could not carry one large run with room to spare.

The fix is a cap, and the slider is the argument:

Two things about that cap are easy to get wrong.

A cap without an order is a starvation bug. Whichever workspaces sort last never run at all, forever, and nothing alerts on it. Candidates are taken oldest-first by last successful run, NULLS FIRST, so a workspace that has never completed one outranks any workspace that has. That ordering was not theoretical: the second-largest workspace in the system had never once had a successful run, and it was exactly what a naive cap would have stranded permanently.

A limited relation is not a limited query. This bit me:

# Wrong: find_each forces its own ORDER BY id batching and silently
# discards both the ordering and the limit.
scope.order(...).limit(3).find_each { |t| enqueue(t) }

find_each exists to iterate large sets in id order, so it overrides yours. I returned ids instead and iterated those.

Measured across the first real tick, capped at three: database free memory floor 324 MB against a 200 MB alarm, versus 129 MB for a single uncapped run. Three concurrent runs cost less memory pressure than one uncapped one, at higher read throughput — the work streams through instead of accumulating.

Two tests that passed whether the code worked or not

Both of these are more valuable than the fix, because they are mistakes you can make in any codebase this week.

The assertion that matched the wrong thing. I wrote a guard that the query was now filtered, by asserting the SQL contained subject_id. It passed. It also passed with the fix reverted — because subject_id appears in the SELECT list, not only the WHERE clause. The companion assertion, that the bind-parameter count was <= batch_size, passed against zero binds, which is precisely the unbounded case. Two assertions, both green, both blind. Fixed by splitting on WHERE and requiring the count to be between 1 and the batch size.

The fixture that made the case unreachable. A test for the fan-out cap created three candidates, capped at two, and asserted which two were chosen. It passed — and it passed with the ordering deleted, and again with the ordering reversed. The cause was travel_to: the fixture rows were stamped with 30.hours.ago outside the block, so they resolved against the real clock, which sat twelve hours away from the traveled one. One candidate landed outside the eligibility window, only two were ever eligible, the cap never bound, and the assertion was measuring nothing.

Which is the actual discipline, and it is one line:

A fix leaves one test behind, and you verify the test by reverting the fix and watching it fail.

Not by reading it. By running it. I ran three reverts against that cap — remove the limit, remove the ordering, reverse the ordering — and the second and third came back green, which is the only reason I found out. If you take one habit from this post, take that one.

The checklist

For any long periodic job over data that grows:

1. Can it be killed without raising? Then progress must be durable in your own table before the next unit starts. The dying process cannot save you.
2. Does every "run everything" path have an explicit name? A filter that is skipped when an argument is nil is unbounded by default.
3. Is the runtime bounded by a wall clock, and is that clock monotonic?
4. Is a single unit idempotent? It will be replayed. Upsert, don't append.
5. If the job deletes what it didn't write, is its now frozen?
6. When the fan-out catches up, is it capped — and is the cap ordered by starvation?
7. Which resource just inherited the cost? Check it on a dashboard, fifteen minutes after you think you are done.
8. Does each new test fail when you revert the fix? Run it. Don't read it.

The first six are architecture. The last two are why the architecture stays true next quarter.