Migrating a live server across CPU architectures: the seven things that weren't in my plan

I moved a production service from an x86 instance to an ARM one, and replaced its on-disk session store with SQLite, in the same night. Both worked. Neither went the way the plan said.

This is the write-up I wanted to read beforehand — not the happy path, which is four commands, but the seven things that only show up once you're doing it on a box people depend on. Everything here is generic: one host, one database on that host, no replica, no load balancer. The unglamorous shape most small production systems actually have.

The setup, so the constraints make sense

• One VM. Application, background workers, PostgreSQL, and a couple of sidecar containers, all on it.
• Its only inbound path is an outbound-dialled tunnel. The security group has zero inbound rules.
• The root volume is the database and the session state. No managed database, no object store for sessions.
• Deploys are containerised, orchestrated by a tool that SSHes to the host.

If you have a load balancer and a managed database, most of this is easier and you should skip to the parts about measurement. If you don't — and a lot of us don't — read on.

1. You cannot resize a VM across CPU architectures

This is the one that decides the whole shape of the migration, so it's worth being precise about why.

Two facts combine. First, most Terraform setups pin the machine image:

resource "aws_instance" "app" {
  ami           = data.aws_ssm_parameter.ubuntu.value
  instance_type = var.instance_type

  # The SSM parameter tracks "current", which rotates whenever the vendor
  # publishes a new image. Without this, the first apply after any such publish
  # silently plans to destroy and rebuild the box.
  lifecycle {
    ignore_changes = [ami]
  }
}

That ignore_changes is correct and you want it. It also means the image stays whatever architecture it was built from.

Second, Terraform treats an instance_type change as an in-place update — stop, modify attribute, start — not a replacement. So changing t3.medium to t4g.medium plans a modify, and the cloud provider rejects it: an ARM instance type cannot boot an x86 image.

Force a replacement instead and you destroy the root volume, which on this shape of system is the database.

So a second instance is mandatory. Not a nice-to-have for safety — the only mechanism that works. And that reframes the whole job: you are not migrating a server, you are standing up a new one and moving state onto it.

The way to express that without a scary diff:

variable "migration_instance_type" {
  description = "Temporary second host for a cross-architecture migration. Null = no second host."
  type        = string
  default     = null
}

resource "aws_instance" "migration" {
  count = var.migration_instance_type == null ? 0 : 1

  instance_type = var.migration_instance_type
  user_data     = local.user_data   # SHARED with aws_instance.app, not copied
  # ...everything else identical
}

Two details in there matter more than they look.

user_data is shared via a local, not copy-pasted. user_data forces replacement. If the two hosts render even one byte differently, the final step — where you promote the migration host into the original resource address — plans a replacement of production. Sharing the local makes byte-identity structural rather than a hope.

The gate is the plan, not the intention. Before applying anything, the plan must read:

Plan: 1 to add, 0 to change, 0 to destroy.

Anything touching the live instance destructively means stop and re-read. I ran that plan before opening the pull request, not after, and it's what proved the user_data extraction was safe — user_data was absent from the live instance's diff.

2. Grep cannot find every reference to your host

I swept the repository for the old instance ID and found eleven references — deploy config, two workflow files, some comments. Updated all of them. Merged. Quiesced the old box. Started the deploy.

It died at the first SSH:

not authorized to perform: ssm:StartSession on resource:
  arn:aws:ec2:...:instance/i-0xxxxxxxxxxxxxxxx

The deploy role's policy scopes session access to specific instance ARNs. And in Terraform, that reference doesn't look like an instance ID:

Resource = [
  "arn:aws:ec2:${var.region}:${account}:instance/${aws_instance.app.id}",
  # ...
]

Grep found every literal. It could not find the one written as a resource attribute. And this failed late — after the old host was already stopped and users were already in the outage window.

The generalisable lesson: "find everything that references the old host" is not a text search when the host is an infrastructure resource. It's a question about the resource graph. Ask your tool, not your editor:

# What actually depends on this resource?
terraform graph | grep -A5 'aws_instance.app'

# Or, more bluntly: what does the rendered plan mention?
terraform show -json plan.out | jq -r '.. | strings | select(test("aws_instance.app"))'

The fix, once found, is unremarkable — allow both hosts while a migration is in flight, and let the list collapse back when it isn't:

Resource = concat(
  ["arn:aws:ec2:${var.region}:${account}:instance/${aws_instance.app.id}"],
  [for m in aws_instance.migration : "arn:aws:ec2:${var.region}:${account}:instance/${m.id}"],
  ["arn:aws:ssm:${var.region}::document/AWS-StartSSHSession"]
)

Keeping the old host in the list during the migration isn't laziness. It's what keeps rollback possible.

3. Two-pass rsync: the outage is the delta, not the data

The session store was over a gigabyte across hundreds of thousands of tiny files. My first plan was to copy it during the maintenance window. That plan was wrong by an order of magnitude, because with hundreds of thousands of small files you are bound by per-file round trips, not bandwidth.

The fix is old and reliable: copy it twice.

# PASS 1 — while everything is still live and serving.
# Slow, and it does not matter, because nobody is waiting.
rsync -a --delete --info=stats2 \
  user@old:/var/lib/app/sessions/ /var/lib/app/sessions/

# ... now stop the service. The outage window starts HERE ...

# PASS 2 — only what changed during pass 1.
rsync -a --delete --info=stats2 \
  user@old:/var/lib/app/sessions/ /var/lib/app/sessions/

Measured on the real thing:

The window carried five seconds of copying instead of two and a half minutes. The technique costs nothing but a second command.

Where it doesn't apply: anything with an internally consistent on-disk format that's being written during pass 1 — a database's data directory, for instance. For those you want the engine's own export, which brings me to the next one.

4. Use a logical dump across architectures, not a file copy

It is tempting to copy the database directory along with everything else. Don't.

PostgreSQL does not guarantee on-disk binary compatibility across platforms. In practice the same major version on two 64-bit little-endian architectures will very often work — which is exactly what makes it dangerous, because "usually works" failure modes surface later, under load, as corruption rather than a clean error.

# On the old host, after writes have stopped
docker exec pg pg_dump -U app -d appdb -Fc -f /tmp/app.dump

# On the new host
docker exec pg pg_restore -U app -d appdb --clean --if-exists --no-owner /tmp/app.dump

The database here was under 100 MB. pg_dump took 2.6 seconds and produced a file a tenth that size. There was never a reason to take the risk.

The general rule: for the filesystem, copy bytes; for a database, ask the engine.

5. Verify with checksums, not row counts

After restoring, comparing row counts feels like verification. It isn't — it proves you have the same number of things, not the same things.

Because the old host was still running (stopped containers, database up), I could compare the two databases directly:

-- Counts: necessary, not sufficient
SELECT count(*) FROM records WHERE id <= :cutoff;

-- Content, which is the actual check
SELECT md5(string_agg(
  id || '|' || coalesce(body,'') || '|' || direction || '|' || status,
  ',' ORDER BY id
)) FROM records WHERE id <= :cutoff;

Counts matched. The checksum did not.

That's the moment worth describing, because the instinct is to panic. Instead, isolate — checksum one column at a time:

SELECT md5(string_agg(id || '|' || coalesce(body,''),     ',' ORDER BY id)) FROM records WHERE id <= :cutoff;
SELECT md5(string_agg(id || '|' || direction,             ',' ORDER BY id)) FROM records WHERE id <= :cutoff;
SELECT md5(string_agg(id || '|' || created_at,            ',' ORDER BY id)) FROM records WHERE id <= :cutoff;
SELECT md5(string_agg(id || '|' || status,                ',' ORDER BY id)) FROM records WHERE id <= :cutoff;

Result: body, timestamps and direction were byte-identical. Only status differed. And the distribution explained itself:

delivered   -5
sent        -4
read        +9

Nine rows moved forward into read. Those were delivery receipts arriving on the live system after the dump was taken. The old copy was frozen; the new one was ahead. Nothing was lost — the mutable column had simply kept moving, as it should.

Checksum immutable columns to prove integrity. Expect mutable ones to differ, and check the direction of the difference. A count would have told me nothing; a whole-row checksum would have scared me into a rollback I didn't need.

6. Logs describe events. Ask the kernel for state.

After the cutover the service's log looked alarming — page after page of:

[gateway] <id> closed: 408 (connectionLost)
[gateway] <id> closed: 403 (forbidden)

and not a single line saying anything had connected. I nearly rolled back.

Then I asked the operating system instead:

PID=$(docker inspect -f '{{.State.Pid}}' gateway)
nsenter -t $PID -n ss -tn state established | grep -c ':443'

A count of live connections that matched, exactly, what the database said should be connected. The service was completely healthy.

The log only recorded closes. It had no line for a successful connection, so a busy, healthy service that constantly cycles connections produces a log that reads like nothing but failure. The log was not lying; it was answering a different question from the one I was asking.

That check then paid for itself twice. Before the storage migration there were eight more sockets than working accounts. Those eight turned out to be sessions whose credentials file was zero bytes — a torn write from some past crash. The old code checked only that the file existed, so it opened a connection for each, failed, and retried forever. They had been burning sockets for who knows how long, and the only visible symptom was log noise everyone had learned to ignore.

When a log and a metric disagree, believe the thing you can count. Sockets, file descriptors, row counts, RSS. Logs tell you what someone chose to write down.

7. Unenforced preconditions fail late, silently, and completely

The storage rewrite replaced "one JSON file per key" with one SQLite database per session. The reason was arithmetic: a 15-byte record occupies a 4 KB filesystem block. At that ratio you are paying roughly 270 bytes of disk for every byte of data, and the bill is charged in inodes as much as in gigabytes.

The absolute figures don't matter. The ratio does, and it is entirely a property of block size versus record size — which means you can predict it before writing any code: divide your filesystem's block size by your median record size.

The rewrite came with a migration script and a documented instruction: run this before deploying the new image. And the new code resumed sessions like this:

const ids = fs.readdirSync(SESSIONS_DIR, { withFileTypes: true })
  .filter((d) => d.isFile() && d.name.endsWith('.db'))   // <-- only .db
  .map((d) => d.name.slice(0, -3));

Read that filter against the failure case. Deploy the image without having run the migration and it finds zero .db files. It resumes zero sessions. Every account goes offline — and silently, because the old directories are untouched, nothing throws, and the process looks perfectly healthy.

One forgotten command, a full outage. And any restart triggers it: a bootstrap deploy, an out-of-memory kill, someone clicking a "reboot this container" button.

The fix is to make it a property of the code rather than of whoever read the runbook:

export async function resumeSessions() {
  // Fold any legacy folders into .db files FIRST. This must happen here and not
  // in a runbook step: the scan below only sees .db files, so a boot that
  // skipped it would resume nothing. Idempotent — a no-op on every later boot.
  const m = migrateAll(SESSIONS_DIR);
  for (const f of m.failed) {
    console.error(`MIGRATION FAILED for ${f.id} (${f.reason}) — not resuming; folder left intact`);
  }
  // ...then the scan
}

Two rules make that safe to run unattended, and both are worth stealing:

Only migrate when the destination does not exist. Once a session is live on the new store, its old folder is frozen and immediately stale. Re-migrating from it would overwrite live state with a snapshot. Absence of the destination is the only trustworthy signal that the source is still authoritative.

Delete a destination that fails verification. Leaving a half-written database in place means the next boot happily resumes from it. Deleting it means that one account stays down and loud, with its source folder intact and retryable. Down and obvious beats up and subtly wrong.

const r = migrateSession(dir, dbFile);
const v = verifySession(dir, dbFile);   // re-read every source record, compare bytes
if (v.mismatches.length) {
  fs.rmSync(dbFile, { force: true });   // unresumable on purpose
  failed.push({ id, reason: `${v.mismatches.length}/${v.checked} mismatched` });
}

The measurement mistakes, which were mine

Two numbers I quoted confidently and got wrong. Both are easy to make.

I read totals as rates. I pulled message volume grouped by hour-of-day over fourteen days to pick a low-traffic window. The quietest hour showed 30. I called it "30 per hour" and estimated the outage would delay 7–8 messages.

It was 30 messages across fourteen days — about two per hour. The real cost was one or two delayed messages. I overstated it by 14×.

-- What I ran: a total
SELECT extract(hour FROM created_at AT TIME ZONE 'Etc/UTC') AS h, count(*)
FROM messages WHERE created_at > now() - interval '14 days'
GROUP BY 1 ORDER BY 1;

-- What I should have run: a rate
SELECT extract(hour FROM created_at AT TIME ZONE 'Etc/UTC') AS h,
       round(count(*) / 14.0, 1) AS per_hour
FROM messages WHERE created_at > now() - interval '14 days'
GROUP BY 1 ORDER BY 2;

Put the unit in the column name and this stops happening.

The rehearsal was six times optimistic. A synthetic benchmark of the storage migration — same file count, generated data — ran in 37.7 seconds. On real data it took 3 minutes 40 seconds. Same count, same shapes, six times slower. Synthetic files are written sequentially and land contiguously; files accumulated over months by a live process do not.

So I dry-ran it against the actual production directory, in a throwaway container with no network, while the service kept running:

docker run --rm --network none \
  -v /var/lib/app/sessions:/sessions \
  -v /tmp/migration-scripts:/app:ro \
  node:24-slim node /app/migrate.js /sessions

That answered "does every session convert cleanly?" — they did — before anything was committed. Then I deleted the databases it produced, because they were already minutes stale and the boot-time migration skips any session that already has one. Proving it works and leaving the artefacts behind would have cut over onto stale state.

That dry run is only possible because the migration code had no third-party dependencies — stdlib only. Worth designing for deliberately: a migration you can run from a stock base image is a migration you can rehearse anywhere.

The tests that had never run

While tidying up I checked what continuous integration actually executed. Static analysis, a linter, some config guards. Not the test suite.

Hundreds of examples. Green. Executed by nothing.

A permissions change had shipped that week covered by specs that no machine had ever run. They happened to pass when I finally wired them up, which is luck wearing the costume of competence.

test:
  runs-on: ubuntu-latest
  services:
    postgres:
      image: postgres:16
      env: { POSTGRES_USER: postgres, POSTGRES_PASSWORD: postgres, POSTGRES_DB: app_test }
      ports: ['5432:5432']
      # Without this the steps race the database and fail on timing, not on code
      options: >-
        --health-cmd pg_isready --health-interval 10s
        --health-timeout 5s --health-retries 5
  env:
    RAILS_ENV: test
    DATABASE_URL: postgres://postgres:postgres@localhost:5432/app_test
  steps:
    - uses: actions/checkout@v7
    - uses: ruby/setup-ruby@v1
      with: { bundler-cache: true }
    - run: bin/rails db:test:prepare
    - run: bundle exec rspec

If you take one thing from this post, make it this: go and check, right now, whether your CI actually runs your tests. Not whether the tests exist. Whether a machine runs them without being asked.

And once they run — make them a required status check. A job that reports red but doesn't block a merge is documentation, not a gate.

What I'd do the same, and what I'd change

Keep:

• The plan gate. 1 to add, 0 to change, 0 to destroy as a hard precondition, read before every apply.
• The old host stopped, not terminated, for a week. Costs a few dollars, buys the ability to compare two databases directly — which is how I proved nothing was lost.
• Two-pass rsync. Free, and it took the window from minutes to seconds.
• Setting delete_on_termination = false on the volume that holds the data. It's a one-line change that converts a catastrophic mistake into a recoverable one.

Change:

• Ask the resource graph, not grep, for everything that references the host.
• Dry-run the data migration against production before the window, not as part of it. It's usually safe — read the source, write beside it — and it converts the biggest unknown into a measurement.
• Put units in column names.
• Distrust synthetic benchmarks of anything I/O-bound on aged data.

The migration itself — new machine, copy state, repoint, verify — is genuinely four steps. Everything above is what surrounds those four steps when the system is real, has one of everything, and people are using it while you work.

The outage ran about forty-five minutes against the fifteen I'd predicted. Two of the three overruns were my own errors, both of the same kind: something I assumed rather than measured. That's the whole lesson, really. The plan is the cheap part.