AWS alerts into a chat channel, without a monitoring vendor

I picked up an AWS account running a real product and went looking for the monitoring. There wasn't any. Zero CloudWatch alarms. Zero SNS topics. The only CloudWatch-shaped thing in the account was an unused IAM policy attachment.

Detection was customer-reports-first. Two separate outages the week before had both been noticed by a human clicking around, not by a machine.

The obvious answer is Datadog. The obvious answer costs more per month than the infrastructure it watches, when the infrastructure is one EC2 instance and one small RDS. So the constraint I set was: native AWS only, under $20/month, and every threshold has to be defensible. This post is the build.

Identifiers are changed throughout — this was a client engagement — but the architecture, the code, and the failure modes are exactly as shipped.

The shape

terminal

CloudWatch alarm ─┐
RDS event ────────┼──→ SNS topic ──→ Lambda ──→ Discord channel
Budget notice ────┘     app-<env>-alerts   cw-discord

Four things worth noticing before the code:

One topic per environment, not one per severity. Severity travels inside the message. Splitting topics means every new severity is a new subscription to wire and forget.
RDS event subscriptions publish to the same topic. These are not metrics. They are AWS telling you in English what it is doing to your database, and they are the single highest-signal thing in this diagram.
Budgets publish there too, so cost surprises land in the same place as outages. Same audience, same urgency ladder.
The Lambda is stdlib only. No requests, no layer, no build step, nothing to keep patched. urllib.request posts JSON perfectly well.

The notifier

The whole delivery path is one function. Its job is to turn three unrelated AWS payload shapes into one readable message, and to fail loudly when it can't.

terminal

"""Fan CloudWatch alarms, RDS events and Budgets notices out to a chat webhook."""

import json, os, re, urllib.error, urllib.parse, urllib.request
import boto3

ssm = boto3.client("ssm")

REGION = os.environ.get("AWS_REGION", "us-east-1")
WEBHOOK_PARAM_TEMPLATE = os.environ.get(
    "WEBHOOK_PARAM_TEMPLATE", "/app/{env}/ops/webhook_url"
)
MENTION_ON_CRITICAL = os.environ.get("MENTION_ON_CRITICAL", "1") == "1"

COLOR_ALARM, COLOR_OK, COLOR_WARN, COLOR_INFO = 0xE01E5A, 0x2EB67D, 0xECB22E, 0x5865F2
STATE_COLOR = {"ALARM": COLOR_ALARM, "OK": COLOR_OK, "INSUFFICIENT_DATA": COLOR_WARN}

# Cache resolved webhooks for the life of the execution environment.
# Parameter Store is cheap, not free, and alarm storms arrive in bursts.
_webhook_cache = {}

# "[critical] Human sentence | runbook#anchor"
DESCRIPTION_RE = re.compile(r"^\[(?P<severity>\w+)\]\s*(?P<body>.*)$", re.DOTALL)


def _env_from_topic_arn(topic_arn):
    """arn:...:app-staging-alerts -> staging"""
    topic_name = topic_arn.rsplit(":", 1)[-1] if topic_arn else ""
    match = re.match(r"^app-(?P<env>[a-z0-9-]+)-alerts$", topic_name)
    if not match:
        raise ValueError(f"cannot derive env from topic name {topic_name!r}")
    return match.group("env")


def _webhook_for_env(env):
    if env not in _webhook_cache:
        param = WEBHOOK_PARAM_TEMPLATE.format(env=env)
        resp = ssm.get_parameter(Name=param, WithDecryption=True)
        _webhook_cache[env] = resp["Parameter"]["Value"].strip()
    return _webhook_cache[env]

Two decisions are already visible there.

The webhook URL lives in SSM Parameter Store as a SecureString, not in a Lambda environment variable. A webhook URL is a credential — anyone holding it can post into your incident channel. In an environment variable it is readable by anyone with lambda:GetFunctionConfiguration, which is a much wider blast radius than ssm:GetParameter on one path. Rotating it is then a one-line put-parameter with no redeploy, because the value is read at invoke time.

One function serves every environment, deriving the target channel from the SNS topic the record arrived on. Two functions means two copies of this logic drifting apart.

Parsing three payload shapes

CloudWatch sends structured JSON. RDS event subscriptions send differently structured JSON with space-separated key names. Budgets sends prose. The dispatch is a type sniff, not a schema:

terminal

def handler(event, context):
    records = event.get("Records") or []
    if not records:
        raise ValueError("no SNS Records in event")

    results = []
    for record in records:
        sns = record.get("Sns") or {}
        env = _env_from_topic_arn(sns.get("TopicArn", ""))
        raw = sns.get("Message", "") or ""
        subject = sns.get("Subject", "") or ""

        try:
            msg = json.loads(raw)
        except (json.JSONDecodeError, TypeError):
            msg = None

        if isinstance(msg, dict) and "AlarmName" in msg and "NewStateValue" in msg:
            embed, mention = _alarm_embed(msg)
        elif isinstance(msg, dict) and ("Event Source" in msg or "Source ID" in msg):
            embed, mention = _rds_event_embed(msg)
        else:
            embed, mention = _generic_embed(subject, raw)

        payload = {
            "username": f"AWS {env}",
            "embeds": [embed],
            # Never let a message body @-mention anyone by accident;
            # only the explicit @here below is allowed through.
            "allowed_mentions": {"parse": ["everyone"] if mention else []},
        }
        if mention:
            payload["content"] = "@here"

        results.append(_post(_webhook_for_env(env), payload))

    return {"delivered": len(results), "statuses": results}

That allowed_mentions line is not paranoia. Alarm descriptions and RDS event messages are text you did not write being pasted into a channel. Pin the mention policy at the payload level and a message body can never wake the team by accident.

The severity smuggling trick

CloudWatch gives you exactly one free-text field that survives the trip through SNS: AlarmDescription. So encode structure into it and parse it back out:

terminal

def _split_description(description):
    """Pull the severity tag and runbook anchor back out of an AlarmDescription."""
    severity, body, runbook = "warning", description or "", ""
    match = DESCRIPTION_RE.match(body.strip()) if body else None
    if match:
        severity = match.group("severity").lower()
        body = match.group("body").strip()
    if "|" in body:
        body, _, tail = body.rpartition("|")
        body, runbook = body.strip(), tail.strip()
    return severity, body, runbook

The provisioner writes descriptions in the shape [critical] RDS burst credits under 75 and draining. | runbook: docs/alerts-runbook.md#rds-credits-low. The notifier splits it into a severity colour, a human sentence, and a runbook anchor that lands in the message. The person reading the alert at 2am gets a link to the exact section that tells them what to do. That single field is worth more than any dashboard.

The timestamp bug you will hit

Both sources send timestamps. Neither sends a valid one.

terminal

def _iso(value):
    """Normalise AWS timestamps to RFC3339, or drop them.

    CloudWatch sends 2026-07-29T06:28:00.123+0000 (offset without a colon).
    RDS event subscriptions send 2026-07-29 06:50:12.113 (space separator, no
    timezone at all). Discord rejects the whole webhook with a 400 on a malformed
    timestamp, so an un-normalised value silently loses the alert.
    """
    if not value or not isinstance(value, str):
        return None
    value = value.strip().replace(" ", "T", 1)
    match = re.search(r"([+-]\d{2})(\d{2})$", value)          # +0000 -> +00:00
    if match:
        value = value[: match.start()] + f"{match.group(1)}:{match.group(2)}"
    elif not value.endswith("Z") and not re.search(r"[+-]\d{2}:\d{2}$", value):
        value += "Z"                                          # RDS times are UTC, unlabelled
    try:
        from datetime import datetime
        datetime.fromisoformat(value.replace("Z", "+00:00"))
    except ValueError:
        return None
    return value

Note the failure mode: an alert that cannot be formatted is an alert that is never delivered, and the only trace is a 400 in a Lambda log nobody reads. Anything that cannot be confidently fixed up is dropped — an alert without a timestamp still delivers, which is the whole point.

The source of truth is a JSON file

Every alarm is a row in a committed file, not a click in a console:

terminal

{
  "suffix": "rds-credits-low",
  "severity": "critical",
  "description": "RDS burst credits under 75 and draining.",
  "runbook": "rds-credits-low",
  "namespace": "AWS/RDS",
  "metric_name": "CPUCreditBalance",
  "dimensions": [{ "Name": "DBInstanceIdentifier", "Value": "db-prod" }],
  "statistic": "Average",
  "unit": "Count",
  "period": 300,
  "evaluation_periods": 3,
  "datapoints_to_alarm": 3,
  "comparison_operator": "LessThanThreshold",
  "threshold": 75,
  "treat_missing_data": "missing"
}

A shell script reads that file and applies it. The interesting constraint is what the script is not allowed to contain:

provision.sh contains no delete or remove verb, at all. Tearing something down has to be a deliberate manual act, never something a re-run can do by accident.

Every call is either natively idempotent or guarded by a describe-first check. The handful that are not — lambda add-permission 409s on a duplicate statement id — get an explicit guard:

terminal

# add-permission is not idempotent -- it 409s on a duplicate statement id.
if aws lambda get-policy --function-name "$LAMBDA_NAME" \
     --query Policy --output text 2>/dev/null | grep -q "sns-${TOPIC_NAME}"; then
  skip "invoke permission for $TOPIC_NAME exists"
else
  aws lambda add-permission --function-name "$LAMBDA_NAME" \
    --statement-id "sns-${TOPIC_NAME}" --action lambda:InvokeFunction \
    --principal sns.amazonaws.com --source-arn "$TOPIC_ARN"
fi

And a freshly created IAM role is not immediately assumable, which will bite you on exactly the first run and never again:

terminal

ATTEMPT=0
until aws lambda create-function --function-name "$LAMBDA_NAME" ... ; do
  ATTEMPT=$((ATTEMPT + 1))
  [[ $ATTEMPT -ge 6 ]] && { echo "create-function failed" >&2; exit 1; }
  warn "waiting for IAM role propagation (attempt $ATTEMPT)"
  sleep 5
done

Why not Terraform

This is the decision I expect the most argument about, so here is the honest version.

Terraform in that account already owned EC2, RDS, S3, IAM and the security groups. The priority was that adding monitoring could not possibly propose a change to any of them. A new module invites a plan that offers to do something unwanted next to a production database, at exactly the moment you are least equipped to review it carefully — you came to add an alarm, not to read a 200-line diff.

Scripts guarantee that. The cost is real: these resources are invisible to terraform plan and can drift when someone edits a threshold in the console.

That cost is mitigated, not solved. The mitigation is a second script:

terminal

./verify.sh --env production          # drift report: declared vs live
./verify.sh --env production --test   # push a test message through to chat

It reads the same JSON and compares every field against what is actually in AWS. One detail there took a debugging round to find:

terminal

# Numeric fields must be compared as numbers, not as text: CloudWatch echoes
# thresholds back as floats, so a declared 40 comes back as 40.0 and a string
# comparison would report drift on every single integer threshold.
cmp_num() {
  awk -v a="$1" -v b="$2" 'BEGIN { exit !(a + 0 == b + 0) }' \
    || add_problem "$3 declared=$1 live=$2"
}

A drift checker that cries drift on everything is worse than no drift checker, because you stop reading it in week two.

It also checks the reverse direction — alarms that exist in AWS but are not in the file, which is how you catch a console hand-edit or a leftover from a renamed alarm:

terminal

DECLARED="$(jq -r --arg p "$PREFIX" '[.alarms[].suffix | $p + .] | .[]' "$SPEC" | sort)"
ACTUAL="$(echo "$LIVE" | jq -r '.[].n' | sort)"
comm -13 <(echo "$DECLARED") <(echo "$ACTUAL")

If the alarm set ever stops changing shape, transcribing the JSON into Terraform is mechanical. Until then, this trades a real risk for a manageable one.

Test the pipe, not the thing the pipe watches

The mistake I have watched teams make is testing an alerting path by breaking something. Don't. Add a canary alarm on a metric nothing ever publishes:

terminal

aws cloudwatch put-metric-alarm \
  --alarm-name "app-${ENV}-canary" \
  --alarm-description "[warning] Synthetic alarm used to test delivery. Never fires on its own." \
  --namespace "App/Synthetic" --metric-name "AlertPipelineCanary" \
  --dimensions "Name=Env,Value=${ENV}" \
  --statistic Maximum --period 300 --evaluation-periods 1 \
  --comparison-operator GreaterThanThreshold --threshold 0 \
  --treat-missing-data notBreaching \
  --alarm-actions "$TOPIC_ARN" --ok-actions "$TOPIC_ARN"

It can only ever change state via set-alarm-state, which makes it the safe way to prove the path end to end, on demand, at 3pm on a Tuesday:

terminal

aws cloudwatch set-alarm-state --alarm-name "app-prod-canary" --state-value ALARM \
  --state-reason "delivery test -- if you can read this in chat, the path works"
sleep 15
aws cloudwatch set-alarm-state --alarm-name "app-prod-canary" --state-value OK \
  --state-reason "delivery test complete"

It sits in INSUFFICIENT_DATA forever. That is correct, and the drift checker knows to expect it.

One rollup, no extra pages

A composite alarm gives you a single "is anything wrong" status without doubling the noise:

terminal

aws cloudwatch put-composite-alarm \
  --alarm-name "app-${ENV}-degraded" \
  --alarm-rule 'ALARM("app-prod-rds-cpu") OR ALARM("app-prod-ec2-status") OR ...'

Deliberately no --alarm-actions. The children already page. Pointing the rollup at the same topic doubles every alert, which is the fastest way to teach people to mute a channel.

What it cost

~$15/month for two environments. Dominated not by the alarms ($0.10 each) but by custom metrics at $0.30 each — the host disk/memory metrics and the log-derived counters. The Lambda invocations round to zero. That is a rounding error against one engineer-hour, and it is roughly a fiftieth of the entry price of the SaaS alternative for this size of system.

The honest caveat: this buys you alerting, not observability. There is no distributed tracing here, no APM, no log search. For a one-box product with a handful of engineers, alerting is the 90% — you need to know when it is broken far more than you need a flame graph.

Before you wire any of this, work out what "broken enough to page" actually means for your service. The SLO & Error Budget Calculator turns an uptime target into a budget you can spend, and the Production Readiness Scorecard will tell you which of these gaps to close first.

The takeaway

Monitoring you can afford and understand beats monitoring you can't. The pieces that made this work are not AWS-specific: a committed file as the source of truth, a script with no delete verb, a drift report that only speaks when it has something to say, and a canary so you can test the pipe without breaking the thing.

The part that surprised me was how much of the value came from one text field. Putting the runbook anchor in the alert body turned every page from "something is wrong" into "here is the page that tells you what to do". Do that first.

AWS alerts into a chat channel, without a monitoring vendor

The shape

The notifier

Parsing three payload shapes

The severity smuggling trick

The timestamp bug you will hit

The source of truth is a JSON file

Why not Terraform

Test the pipe, not the thing the pipe watches

One rollup, no extra pages

What it cost

The takeaway

related_reading