After the write cuts, the migration lane could safely go from two threads to four. "Safely" was a claim about a database I had, at that point, only watched from the app's side. The person who owns the system had one instruction for the whole project — do not take the database down again — so before the config change I measured the environment properly. This is what a small capacity audit looks like when the question is "can this lane be wider", followed by the plan for scaling the engine past one customer.
Numbers are real; the environment is a staging stack that mirrors production's shape.
The connection ceiling
Managed Postgres sets max_connections from instance memory unless you
override it; on a 2 GiB instance that formula gives 181. Against that
ceiling, what the application can demand is arithmetic:
pool per process = web threads (3) + job threads (24) + 2 = 29
processes = 1 jobs container + 2 web workers = 3
theoretical peak = 87
Realistically far less: job threads hold a connection only while performing, web threads only during a request. The seven-day peak was 37. The email phase peaked at 26. Widening the lane adds two job threads per process, so two connections per process, and the fetch threads inside a task hold none by design. Fine, with a factor of four to spare.
The trap in this measurement was the first command I ran: describe-db-instances
lists every instance in the account, and I read the metrics for index zero,
which was a different application's database. The connection counts I
reported for ten minutes were someone else's. Name the identifier.
What the web tier actually runs
The queue library can run jobs inside the web process, and the web container carried the same job-thread setting as the jobs container. If the web tier had been executing jobs too, "a four-thread lane" would have meant twelve threads across three processes, and the fetch-thread arithmetic would have been wrong by a factor of three.
It was not. A small resolver decides the execution mode from the process topology: clustered web (more than one worker) means external, jobs run elsewhere. Reading that resolver, and then confirming the live setting from inside the container, is the difference between knowing the lane's width and assuming it.
def self.good_job_execution_mode(env = ENV)
fallback = clustered?(env) ? :external : :async
requested = env["GOOD_JOB_EXECUTION_MODE"]
return fallback if requested.blank?
GoodJob::Configuration::EXECUTION_MODES.include?(requested.to_sym) ? requested.to_sym : fallback
end
Storage that cannot grow
The instance had 20 GB of storage and no autoscaling. Free space was 9.6 GB, and it had fallen 1.7 GB during the last six hours of the email phase and 4.1 GB over the week. Where the six gigabytes of database went:
| Table | Size | Note |
|---|---|---|
| inbound integration events | 1.28 GB | not migration data |
| audit log | 1.27 GB | not migration data |
| contacts | 822 MB | 517,000 rows |
| email messages | 559 MB | 84,710 rows, average body 9.4 KB |
| activities | 389 MB | |
| deals | 256 MB | |
| queue jobs + executions | 462 MB | 255,000 rows, seven-day retention |
| migration ledger | 210 MB | 671,000 rows |
| write-ahead log | 2.3 GB |
A full migration of this customer's size costs about 2.5 to 3 GB, so roughly three more would have filled the volume — and a storage-full instance stops accepting writes, which is the exact outage the instruction was about. The fix is one setting with no downtime: a maximum allocated storage. It was deferred by decision until more customers exist, and the number that triggers raising it again is written down: under one migration's worth of free space.
The instance and its credits
The application host is a burstable instance. Burstable instances earn CPU credits at a baseline rate and spend them above it; a workload that runs hot for hours can exhaust the balance and be throttled to baseline — which presents as "everything got slow" with nothing in the application logs. This one was in unlimited-credit mode with a full balance all week and zero surplus charged, so a faster import could not throttle it. That is a thing to check once and write down, not a thing to discover during a cutover.
What the lane change measured
With the write cuts in place — budget row off the hot path, one transaction per deal, heartbeat and log throttled, ten deals per task — the lane went to four threads. The first run on it was a write-heavy one, 77,000 deals recognised or imported in seven and a half minutes:
| Baseline | Peak during the run | Ceiling | |
|---|---|---|---|
| Connections | 19–21 | 26 | 181 |
| CPU | 6% | 26% | — |
| Write IOPS | 10 | 133 | 3,000 |
Everything returned to baseline within a minute of the run ending. The lane could have gone wider. It didn't need to, and a wider lane would have crossed the source API's lowest burst tier for a customer on a small plan.
Three processes, one pool per process, one ceiling. The lane's width is a number you can only trust after checking which processes execute jobs.
A plan for scaling
One customer proved the engine. Ten would expose the seams. This is the order I would build in, each item small enough to ship alone.
1. Estimate before starting. Every listing call costs 20 tokens per 500 rows and every email body 2, so a customer's cost in tokens and hours is computable from the preview counts before they press Run: about 13,000 tokens for 330,000 records, and about 770,000 for their 90,000 emails. Show it. A customer on a small plan should learn that their email history takes two days before it takes two days.
2. Headers first, bodies later. The email phase is 95% of the cost and almost all of it is per-message body fetches. Importing headers and snippets from the one listing call per deal — one request instead of one plus N — and fetching bodies as a separate, resumable phase would cut the phase to about a quarter and let the timeline show subjects immediately. It needs a "body not fetched yet" state in the product, which is the only reason it is not already done.
3. Discovery that skips what is done. A rerun re-lists every deal with mail to find the few that are new — 30,000 listing calls to discover ten. A per-deal "mail listed at" marker in the ledger makes a rerun cost only what changed.
4. Batch by measured cost, not by count. Ten deals per task was the right number for this customer's mail; for a customer whose deals average forty messages it is too many, and for one whose deals average one it is too few. The duration-by-payload query from the measurement post is the input; the planner can size batches from a run's own early tasks.
5. Fairness by lane, then by pool. MAX_INFLIGHT per run keeps two
concurrent migrations sharing a lane fairly. Past a handful of concurrent
customers, migrations should leave the shared jobs process for a worker pool
of their own, so a 400,000-record import cannot delay a password reset
email. That is a queue-topology change, not an engine change.
6. Budget-aware scheduling. A 429 already parks a task until the daily reset. The next step is planning around it: an email phase that would cross midnight in the source's time zone should be scheduled to, so it pauses at a phase boundary rather than mid-deal.
7. Storage as a first-class resource. Autoscaling on. Retention on queue rows for migration jobs shorter than the seven-day default, because a migration produces thousands of them and reads none back. The ledger stays; it is the idempotency contract for every future rerun.
8. Observability that answers the next question. The three queries that answered every question in this series — ledger grouped by kind, outcome, reason and run; task duration grouped by payload; the run log grouped by reason — should be a page, not something I type into a console over a tunnel. With the "latest verdict wins" rule, the ledger's reason column is finally current enough to build that on.
9. Back-pressure that is measured, not felt. The pressure breaker already defers work when active queries exceed a threshold. Add the two numbers this audit had to fetch by hand — free storage and connection headroom — as inputs, and a migration slows itself down before an operator has to.
10. A cutover playbook. Dry run for counts and a token estimate; the real run with entities only; email overnight where a pause costs nothing; verify one deal with notes, activities and mail linked; recovery runs by entity for anything the ledger says is skipped; stop and resume at any point. Every step of that exists now. Writing it down is what makes the second customer take an afternoon instead of a week.
What I'd tell someone about to widen a lane
Measure the ceiling and the arithmetic against it. Check which processes actually execute jobs. Look at storage growth per run, not just free space. Know whether the host can be throttled. Then widen, and read the same three graphs during the first run on the new width. The change itself was three numbers in two files; the confidence to make it came from an hour of reading the environment instead of the code.