Debugging a box you can only reach through SSM

A partner platform's replay loop delivered 487,000 webhooks in eleven hours and pinned a box at 99% CPU. I picked the investigation up a week later, and the box had:

• no SSH — port 22 closed, no key, no bastion
• no public database port
• no log aggregation for that environment
• and an IAM role that could open a Session Manager shell but could not ssm:SendCommand

That last constraint is the one that makes this post worth writing. Every AWS debugging guide reaches for aws ssm send-command, which is the sane tool. If your role doesn't have it, you are down to an interactive terminal you must drive from a script — and that terminal will mislead you in at least four specific ways.

Everything below is read-only. Nothing here mutates application state.

Getting into a second account

The engagement had two AWS accounts behind IAM Identity Center. You do not need to log out of one to use the other — SSO issues named profiles, and every command below takes --profile.

aws configure sso --profile client-b

The wizard asks for four things and two of them are non-obvious:

Then, per shell:

aws sso login --profile client-b
export AWS_PROFILE=client-b
aws sts get-caller-identity

get-caller-identity is the habit worth keeping. It prints the assumed role, so you know exactly which permission set you have before you start blaming a misconfiguration for what is actually a denial. Mine came back as assumed-role/PowerScoped/<user> — which is how I knew SendCommand was going to be a problem.

Find the box without a console tab

aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" \
  --query "Reservations[].Instances[].[InstanceId,Tags[?Key=='Name']|[0].Value]" \
  --output text

i-0aaa...  app-staging
i-0bbb...  app-dev
i-0ccc...  app-production

Confirm the instance is actually reachable through Session Manager before you try — a missing agent or a missing instance profile looks identical to a network problem from the client side:

aws ssm describe-instance-information \
  --query "InstanceInformationList[].[InstanceId,PingStatus,PlatformName]" \
  --output text

Ask CloudWatch before you ask the box

The cheapest step, and the one people skip. You can confirm or kill the entire "the box was pegged" hypothesis without a shell at all:

aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0bbb... \
  --start-time 2026-07-21T00:00:00Z \
  --end-time   2026-07-22T00:00:00Z \
  --period 3600 --statistics Average Maximum \
  --output json > cpu.json

Redirect to a file. Do not pipe CloudWatch JSON straight into jq through a shell wrapper you did not write — mine mangled it, and I spent minutes debugging a parser instead of an incident. Write it down, then read it.

This gave me an eleven-hour plateau above 90% with clean shoulders either side, which immediately rules out "gradual leak" and points at "something started, ran flat out, and stopped".

The shell you can't script

Normally:

aws ssm send-command --document-name AWS-RunShellScript \
  --parameters 'commands=["docker ps"]' --instance-ids i-0bbb...

Denied. So was AWS-StartNonInteractiveCommand. What was allowed was ssm:StartSession against the default shell document — an interactive PTY.

The obvious move fails:

echo "docker ps" | aws ssm start-session --target i-0bbb...
# Cannot perform start session: EOF

The pipe closes stdin the instant echo finishes, and the session dies during handshake. You have to hold stdin open and feed it on a human-ish schedule:

#!/bin/zsh
# usage: ssm.sh <instance-id> 'cmd1' ['cmd2' ...]
TARGET="$1"; shift
{
  sleep 3                                    # let the session establish
  for c in "$@"; do
    printf '%s\n' "$c"
    sleep "${SSM_GAP:-6}"                    # let the command finish
  done
  printf 'exit\n'
  sleep 3                                    # let output flush
} | aws ssm start-session --profile "${AWS_PROFILE}" --target "$TARGET" 2>&1

It is crude and it works. The gaps are the whole trick: there is no request/ response framing, so your only synchronisation primitive is waiting long enough. Which brings us to the failure mode that cost me the most time.

Lie #1: slow commands get truncated, not reported

Send a command that takes longer than SSM_GAP and the next command interleaves into a half-finished stream. You get partial output that looks like a complete answer. No error, no marker.

An application-framework boot is the classic offender — 20+ seconds before your query even starts. The fix is to stop trying to read output live. Write results to a file inside the container, then cat it as a separate command:

./ssm.sh i-0bbb... \
  'sudo docker exec app bin/rails runner "..." > /tmp/out.txt 2>&1' \
  'cat /tmp/out.txt'

With SSM_GAP set generously on the first command. Two round trips, but the second one is instant and honest.

Lie #2: backgrounding doesn't survive

The obvious escape — background the slow thing, poll for the file — does not work:

./ssm.sh i-0bbb... 'nohup long_job > /tmp/out.txt 2>&1 &'

The process is killed when the session ends. Both & and nohup failed for me; the session teardown takes the process group with it. If you need genuinely long-running work, you need SendCommand or a job you can enqueue and inspect later. Adjust the plan rather than fighting it.

Docker: names carry commit SHAs

Container names in a SHA-tagged deployment scheme are not stable across releases, so never hardcode one:

sudo docker ps --format '{{.Names}}\t{{.Status}}\t{{.Image}}'

app-web-9f3c1a2   Up 6 days   ghcr.io/org/app:9f3c1a2
app-job-9f3c1a2   Up 6 days   ghcr.io/org/app:9f3c1a2

Resolve it in the same command that uses it:

C=$(sudo docker ps --format '{{.Names}}' | grep -m1 job)
sudo docker exec "$C" bin/rails runner "puts Rails.env"

Useful read-only one-liners:

# how long has this actually been up — bounds any "since when" question
sudo docker inspect -f '{{.State.StartedAt}}' "$C"

# resource pressure right now, one shot rather than streaming
sudo docker stats --no-stream

# what the container logged around the incident window
sudo docker logs --since 2026-07-21T05:00:00 --until 2026-07-21T18:00:00 "$C" 2>&1 | tail -200

--since/--until are the ones to remember. docker logs | grep on a multi-gigabyte log will happily eat your session's time budget and then get truncated by Lie #1.

Lie #3: the database you found is not the database it uses

This is the mistake I actually made, and it invalidated my first full set of numbers.

There was a Postgres running on the box. I queried it. I got a clean, coherent, entirely wrong story — because the application did not use it. It was a stale local copy, frozen weeks earlier, and every figure I derived from it was low by about 12%.

What saved it was a sanity check that costs one query: the newest row was older than the container's uptime. A live system's most recent record cannot predate the process serving it. That contradiction is worth checking every time before you believe a number.

Ask the application what it is connected to. Do not go rummaging for credentials:

sudo docker exec "$C" bin/rails runner \
  'c = ActiveRecord::Base.connection_db_config.configuration_hash
   puts [c[:host], c[:database], c[:username]].inspect'

Then run every query through the app's own connection, which is both correct and avoids handling credentials at all:

sudo docker exec "$C" bin/rails runner \
  'puts InboundEvent.group("date(created_at)").count.sort.last(5).inspect'

Two rules I would now treat as non-negotiable on someone else's infrastructure:

• Never dump environment variables or secrets to your terminal to find a connection string. printenv | grep DATABASE puts a live production credential into your scrollback, your shell history, and any transcript of your session. Ask the framework for its resolved config instead — it answers the question without materialising the password.
• Never write diagnostic markers into real rows. If you cannot answer a question without mutating data, the answer is not worth it.

Lie #4: "no errors" is not "it's working"

The drain phase produced this, and it is a good note to end on:

env=production
queues="critical:3;default:3;bulk:2;bulk_email:2"
unfinished: {"bulk" => 950, "critical" => 5, "default" => 1}
errors: []

950 queued jobs, an empty error list, a pool configured to serve that queue — and the backlog was not moving. An empty error list means nothing recorded a failure, which covers "everything is fine" and "nothing is running" equally well.

The distinguishing question is never "are there errors", it is "is the number changing". Sample the same counter twice, a minute apart. A stalled worker and a healthy one look identical in a single snapshot, and the whole discipline of debugging through a keyhole is refusing to conclude anything from one.

The checklist

1. sts get-caller-identity — know your permissions before you blame the system
2. CloudWatch before shell — shape of the graph narrows the hypothesis for free
3. Redirect JSON to a file before parsing it
4. Pace your stdin; assume any slow command was truncated
5. Resolve container names at use time
6. Verify the datastore is the one the app actually uses — check newest row against process uptime
7. Ask the framework for config; never print secrets
8. Sample twice before concluding anything is stuck or healthy