feat(infra): alarm on ERROR-level server log lines via CloudWatch Logs metric filters

docs/PRODUCTION.md

@@ -197,8 +197,10 @@ Before treating a deployment as production-ready:
 - On the AWS reference deployment, metrics are on by default: the endpoint
   binds loopback inside the ECS task and an ADOT sidecar exports selected
   metrics to CloudWatch dashboards and alarms — no external exposure, no
-  bearer token needed. See
-  [`SERVER_AWS_DEPLOY.md`](./SERVER_AWS_DEPLOY.md#metrics-dashboard-and-alarms).
+  bearer token needed. ERROR-level server log lines additionally feed a
+  metric-filter alarm that does not depend on the metrics pipeline. See
+  [`SERVER_AWS_DEPLOY.md`](./SERVER_AWS_DEPLOY.md#metrics-dashboard-and-alarms)
+  and [its log-level alarms section](./SERVER_AWS_DEPLOY.md#log-level-alarms).
 - If you scrape Prometheus yourself in a **self-managed deployment**, set
   `GUARDIAN_METRICS_ENABLED=true`, bind an explicitly routable
   `GUARDIAN_METRICS_ADDR` only if the scraper lives outside the host or task,

docs/SERVER_AWS_DEPLOY.md

@@ -290,6 +290,8 @@ aws_region = "us-east-1"
 # alarm_latency_threshold_seconds = 1
 # alarm_cpu_threshold_percent = 85
 # alarm_memory_threshold_percent = 90
+# cloudwatch_log_alarms_enabled = true # ERROR log metric filter + log-errors alarm, WARN filter with the dashboard (needs guardian_log_format = "json")
+# alarm_log_error_threshold = 0
 
 # Optional: Route 53 hosted zone ID
 # route53_zone_id = "Z1234567890ABC"
@@ -560,9 +562,10 @@ grpcurl -import-path crates/server/proto -proto guardian.proto -d '{}' guardian.
 Application metrics ship to CloudWatch by default. Two switches control
 this: `guardian_metrics_enabled` turns on the server's Prometheus endpoint,
 and `cloudwatch_metrics_enabled` deploys the ADOT sidecar, EMF log group,
-IAM policy, dashboard, and alarms on top of it. The export pipeline
-cascades off with the endpoint, so `guardian_metrics_enabled = false` alone
-turns everything off. Disabling only `cloudwatch_metrics_enabled` keeps the
+IAM policy, dashboard, and metric-based alarms on top of it. The export
+pipeline cascades off with the endpoint, so `guardian_metrics_enabled =
+false` alone turns all of that off (the [log-level alarm](#log-level-alarms)
+is gated separately). Disabling only `cloudwatch_metrics_enabled` keeps the
 endpoint without publishing CloudWatch custom metrics — but note the
 endpoint stays **loopback-only**, so that mode is useful only for an
 alternative in-task collector you add by customizing the module; the stack
@@ -621,6 +624,7 @@ exposes no knobs for a routable bind address.
 | `<stack>-metrics-refresh-failures` | Slow-aggregate refresher attempts are failing; delta/proposal/account gauges are stale |
 | `<stack>-metrics-refresh-stale` | The refresh timestamp stopped advancing for ≥ 10 min (hung or dead refresher — catches what the failures counter cannot) |
 | `<stack>-ecs-cpu-high` / `<stack>-ecs-memory-high` | ECS service average CPU/memory exceeds `alarm_cpu_threshold_percent` (85%) / `alarm_memory_threshold_percent` (90%); must sit above the autoscaling targets (enforced at plan time) |
+| `<stack>-server-log-errors` | More than `alarm_log_error_threshold` (default 0) ERROR-level server log lines per 5-minute period for two consecutive periods. Absolute count from a log metric filter, so it catches low-volume faults the rate alarms cannot; independent of the metrics pipeline. See [Log-level alarms](#log-level-alarms) |
 
 Every alarm description ends with the stack name and console links to the
 `<stack>-server` dashboard and the server log group (streams `ecs/*` for
@@ -733,10 +737,72 @@ not being invited to the channel, or a workspace or channel ID typed as a
 name.
 
 Set `guardian_metrics_enabled = false` to turn everything off (no metrics env
-vars, no sidecar, no dashboard, no alarms — the CloudWatch flag cascades off
-with it), or only `cloudwatch_metrics_enabled = false` to keep the
-loopback-only endpoint without any CloudWatch export (see the caveat above
-about what that mode is useful for).
+vars, no sidecar, no dashboard, no metric-based alarms — the CloudWatch flag
+cascades off with it), or only `cloudwatch_metrics_enabled = false` to keep
+the loopback-only endpoint without any CloudWatch export (see the caveat
+above about what that mode is useful for). The log-level alarm below is
+independent of both flags and stays on.
+
+### Log-level alarms
+
+Independently of the metrics pipeline, CloudWatch Logs **metric filters** on
+the server log group (`infra/log_alarms.tf`) count the server's own log lines
+by level and publish them as custom metrics under `<metrics_namespace>/Logs`
+(the `log_metrics_namespace` output; kept apart from the scraped metrics so
+"metrics arriving in `metrics_namespace`" stays a pipeline health check):
+`log_error_events` (lines with `level = "ERROR"`) and, when the dashboard is
+deployed, `log_warn_events` (`level = "WARN"`). They read the container's log
+output directly, so they need neither the metrics endpoint nor the ADOT
+sidecar; the ERROR filter and its alarm keep working with
+`guardian_metrics_enabled = false`, and in that mode they are the only alarm
+left. They cover *logged* faults, not process liveness: a task that panics
+or crash-loops at startup prints plain-text panic output, not JSON, which
+the filters never see (the metrics-missing and ECS alarms cover that when
+the pipeline is on). The alarm notifies the same effective action list as
+every other alarm (operator `alarm_actions` plus the managed SNS topic);
+note that the managed topic and its Slack channel cascade off with the
+metrics pipeline, so with `guardian_metrics_enabled = false` it reaches only
+the ARNs you pass in `alarm_actions`.
+
+- The filters match the JSON `level` field the server emits with
+  `guardian_log_format = "json"` (the default). Terraform rejects the plan if
+  `cloudwatch_log_alarms_enabled` is true with a `text` or `compact` format;
+  set `cloudwatch_log_alarms_enabled = false` to run those formats.
+- `<stack>-server-log-errors` alarms on `log_error_events`: more than
+  `alarm_log_error_threshold` (default 0) ERROR lines in *each* of two
+  consecutive 5-minute periods. A persistent fault, however slow, pages
+  within 10 minutes; one isolated line does not; a burst confined to a single
+  period does not page on its own either (it shows on the dashboard and, if
+  large enough, through the rate alarms). It overlaps with the rate alarms on
+  purpose: on a low-traffic stack a few failures never move a percentage that
+  ALB health checks dominate, but every one is an ERROR line.
+- What counts as `ERROR`: the centralized HTTP 5xx / gRPC-internal log line,
+  background-job and canonicalization failures, **and** some client-caused
+  rejections the server logs at `ERROR` before mapping them to 4xx (rejected
+  signatures or unauthorized cosigner keys in `metadata/auth`, invalid
+  credentials in `configure_account`). A persistently misconfigured client
+  retrying every minute therefore trips the alarm at the default threshold;
+  if that is expected on a stack, raise `alarm_log_error_threshold` so a
+  known trickle is tolerated while a burst still pages. Tightening the
+  server's log levels is the longer-term fix.
+- `WARN` is dashboard-only (the *Server log lines by level* widget) and its
+  filter is created only alongside the dashboard, since nothing else reads
+  it. No alarm: the server logs `WARN` for client-caused and self-healing
+  conditions (retried RPC, rate limiting), so one would page for normal
+  operation.
+- A metric filter applies to the whole log group, which also carries the
+  `adot` and `ca-init` streams. The ADOT Collector writes console-encoded
+  lines (not JSON, lowercase level), which a JSON pattern never matches —
+  collector faults surface through `<stack>-metrics-missing` — and the
+  one-shot CA initializer prints nothing on success.
+- To see what fired, query the log group in CloudWatch Logs Insights:
+
+```sql
+fields @timestamp, message, code, detail, target, span.account_id
+| filter level = "ERROR"
+| sort @timestamp desc
+| limit 50
+```
 
 ### Verify metrics after a deploy
 
@@ -746,6 +812,8 @@ NS=$(terraform -chdir=infra output -raw metrics_namespace)
 DASH=$(terraform -chdir=infra output -raw metrics_dashboard_name)
 LOG_GROUP=$(terraform -chdir=infra output -raw server_log_group)
 ALARM=$(terraform -chdir=infra output -raw metrics_missing_alarm_name)
+LOG_ALARM=$(terraform -chdir=infra output -raw server_log_errors_alarm_name)
+LOG_NS=$(terraform -chdir=infra output -raw log_metrics_namespace)
 
 # 1. Metrics arriving in the namespace (allow ~2 minutes after task start)
 aws cloudwatch list-metrics --namespace "$NS" --output table | head -40
@@ -765,6 +833,7 @@ aws logs tail "$LOG_GROUP" --log-stream-name-prefix adot --since 15m
 terraform -chdir=infra output alarm_actions
 aws cloudwatch set-alarm-state --alarm-name "$ALARM" \
   --state-value ALARM --state-reason "notification path test"
+
 # Nothing in Slack? Confirm the topic is on the alarm and that the publish
 # succeeded (a denied publish is recorded here and never reaches Amazon Q):
 aws cloudwatch describe-alarms --alarm-names "$ALARM" \
@@ -775,6 +844,24 @@ aws cloudwatch describe-alarm-history --alarm-name "$ALARM" \
 SLACK_CONFIG=$(terraform -chdir=infra output -raw alarm_slack_configuration_name)
 [ -n "$SLACK_CONFIG" ] && aws logs tail "/aws/chatbot/$SLACK_CONFIG" \
   --region us-east-1 --since 15m
+
+# 5. Log metric filters are attached and counting (skipped when
+#    cloudwatch_log_alarms_enabled = false: LOG_ALARM is empty then).
+#    default_value = "0" publishes zeros only while log lines are being
+#    ingested without matching ERROR events; a healthy but quiet service
+#    (ALB health checks log nothing at the default filter) ingests no
+#    lines, so an empty Datapoints list is expected and is handled by
+#    treat_missing_data = "notBreaching", not a sign the filter is missing.
+if [ -n "$LOG_ALARM" ]; then
+  aws logs describe-metric-filters --log-group-name "$LOG_GROUP" \
+    --query 'metricFilters[].{name:filterName,pattern:filterPattern}'
+  aws cloudwatch get-metric-statistics --namespace "$LOG_NS" \
+    --metric-name log_error_events --statistics Sum --period 300 \
+    --start-time "$(( $(date +%s) - 1800 ))" --end-time "$(date +%s)"
+  aws cloudwatch set-alarm-state --alarm-name "$LOG_ALARM" \
+    --state-value ALARM --state-reason "notification path test"
+  # returns to OK on the next evaluation (real zeros or missing data)
+fi
 ```
 
 Run step 4 on each environment's stack: a `guardian` (devnet) transition must
@@ -837,10 +924,11 @@ aws ecr delete-repository --repository-name guardian-server --force --region us-
 | Secrets Manager | Secrets containing the Falcon and ECDSA ack private keys used to seed the server keystore in prod |
 | Security Groups | ALB, server, and database security groups |
 | CloudWatch Log Groups | Cluster execute-command logs, server logs, and the EMF metrics log group |
+| CloudWatch Log Metric Filters | ERROR (and, with the dashboard, WARN) line counts from the server log group, published as custom metrics |
 | IAM Role | ECS task execution and runtime roles |
 | ADOT Sidecar | OpenTelemetry Collector container in the server task exporting Prometheus metrics to CloudWatch |
 | CloudWatch Dashboard | `<stack>-server` application and ECS overview |
-| CloudWatch Alarms | Error rate, latency, canonicalization, metrics pipeline, and ECS saturation alarms |
+| CloudWatch Alarms | Error rate, latency, canonicalization, metrics pipeline, ECS saturation, and server log-error alarms |
 | SNS Topic | Optional `<stack>-alarms` topic every alarm notifies on ALARM/OK (`alarm_notifications_enabled`) |
 | Amazon Q chat configuration | Optional Slack channel configuration `<stack>-alarms-slack` subscribed to the alarm topic, plus its notifications-only IAM role |
 
@@ -877,6 +965,9 @@ aws ecr delete-repository --repository-name guardian-server --force --region us-
 | `alarm_sns_topic_arn` | ARN of the managed alarm SNS topic, empty when not enabled |
 | `alarm_slack_configuration_name` | Name of the Amazon Q Slack channel configuration (error log group `/aws/chatbot/<name>`, us-east-1), empty when not configured |
 | `alarm_slack_configuration_arn` | ARN of the Amazon Q Slack channel configuration, empty when not configured |
+| `cloudwatch_log_alarms_enabled` | Whether the server log group's ERROR metric filter and the log-errors alarm are deployed |
+| `log_metrics_namespace` | CloudWatch namespace receiving the log-level metric-filter counts (`<metrics_namespace>/Logs`) |
+| `server_log_errors_alarm_name` | Name of the alarm on ERROR-level server log lines for this stack |
 
 ## Stage Profiles
 

docs/architecture/infra.md

@@ -155,6 +155,7 @@ Mapping AWS resources to the Terraform files that own them:
 | ADOT metrics sidecar + collector config | [`ecs.tf`](../../infra/ecs.tf), [`observability.tf`](../../infra/observability.tf) | Non-essential container in the server task; config injected via `AOT_CONFIG_CONTENT`. |
 | CloudWatch dashboard + alarms | [`observability.tf`](../../infra/observability.tf) | `<stack>-server` dashboard; error-rate, latency, canonicalization, metrics-pipeline, and ECS saturation alarms. |
 | Alarm SNS topic + Slack channel | [`alerting.tf`](../../infra/alerting.tf) | Opt-in `<stack>-alarms` topic appended to every alarm's ALARM/OK actions; optional Amazon Q Developer in chat applications Slack channel configuration subscribed to it, with a notifications-only channel role. |
+| CloudWatch log metric filters + log-errors alarm | [`log_alarms.tf`](../../infra/log_alarms.tf) | ERROR/WARN line counts from the server log group as custom metrics; alarm on sustained ERROR output. Gated by `cloudwatch_log_alarms_enabled`, requires JSON logs (plan-time precondition). |
 | ADOT EMF log-write policy | [`iam.tf`](../../infra/iam.tf) | Task-role, stream-level writes on the EMF group only. |
 | Route 53 alias | [`dns.tf:12`](../../infra/dns.tf#L12) | Created when `route53_zone_id` is set; hostname migrations may temporarily add a second record. |
 | Cloudflare CNAME | [`dns.tf:27`](../../infra/dns.tf#L27) | Created when `cloudflare_zone_id` is set; can be proxied, with the same temporary migration support. |
@@ -280,6 +281,12 @@ Developer in chat applications, alongside any operator-supplied
 and `cloudwatch_metrics_enabled` (the export pipeline), both on by
 default; enablement and verification live in
 [`SERVER_AWS_DEPLOY.md`](../SERVER_AWS_DEPLOY.md#metrics-dashboard-and-alarms).
+Log-level monitoring is separate from that pipeline: metric filters on the
+server log group ([`log_alarms.tf`](../../infra/log_alarms.tf)) count
+ERROR/WARN lines from the JSON log output and an alarm fires on sustained
+ERROR output, so alerting on logged faults survives
+`guardian_metrics_enabled = false` (process liveness does not: a
+crash-looping task prints no JSON).
 Tracing exporters remain an open gap.
 
 ## Things that are deliberately not here

infra/README.md

@@ -28,6 +28,7 @@ Resources created:
 - ADOT Collector sidecar in the server task exporting Guardian Prometheus metrics to CloudWatch (EMF)
 - CloudWatch dashboard (`<stack>-server`) and alarms (error rate, latency, canonicalization, metrics pipeline, ECS saturation)
 - Optional SNS topic (`<stack>-alarms`) receiving every alarm's ALARM/OK transitions, and an optional Amazon Q Developer in chat applications (formerly AWS Chatbot) Slack channel configuration subscribed to it (`alerting.tf`)
+- CloudWatch Logs metric filters counting the server's ERROR (and, with the dashboard, WARN) log lines, with an alarm on sustained ERROR output (independent of the metrics pipeline; requires JSON logs)
 
 The Guardian metrics endpoint binds loopback inside the task's shared network
 namespace; only the sidecar can reach it — it is never exposed via the ALB or
@@ -289,7 +290,7 @@ aws ecr delete-repository --repository-name "$ECR_REPO_NAME" --force --region "$
 | `guardian_log_format` | `json` | Log format for `GUARDIAN_LOG_FORMAT` (`text`, `json`, `compact`) |
 | `log_retention_days` | `7` | CloudWatch log retention in days for the cluster and server groups (prod pins them to 365) and for the EMF metrics group |
 | `guardian_metrics_enabled` | `true` | Guardian Prometheus metrics endpoint (loopback-only inside the task) |
-| `cloudwatch_metrics_enabled` | `true` | ADOT sidecar + EMF export + CloudWatch dashboard/alarms (cascades off when the endpoint is disabled) |
+| `cloudwatch_metrics_enabled` | `true` | ADOT sidecar + EMF export + CloudWatch dashboard/metric-based alarms (cascades off when the endpoint is disabled; the log-based alarm is gated separately) |
 | `adot_image` | pinned ADOT Collector release | Digest-pinned sidecar image |
 | `metrics_namespace` | `<Title(stack_name)>/Server` | CloudWatch namespace for application metrics |
 | `alarm_actions` | `[]` | ARNs (e.g. SNS topics) notified on alarm/ok transitions, in addition to the managed topic |
@@ -300,6 +301,8 @@ aws ecr delete-repository --repository-name "$ECR_REPO_NAME" --force --region "$
 | `alarm_latency_threshold_seconds` | `1` | Average HTTP latency alarm threshold |
 | `alarm_cpu_threshold_percent` | `85` | ECS CPU saturation alarm threshold |
 | `alarm_memory_threshold_percent` | `90` | ECS memory saturation alarm threshold |
+| `cloudwatch_log_alarms_enabled` | `true` | ERROR log metric filter on the server log group + log-errors alarm (plus a WARN filter when the dashboard exists); requires `guardian_log_format = "json"` (plan-time check) |
+| `alarm_log_error_threshold` | `0` | ERROR log lines per 5-minute period tolerated before a period counts as breaching (two consecutive periods alarm) |
 
 ## Outputs
 
@@ -334,7 +337,7 @@ aws ecr delete-repository --repository-name "$ECR_REPO_NAME" --force --region "$
 | `guardian_dashboard_commitment_rate_burst_per_sec` | Effective fleet-wide dashboard per-commitment burst budget |
 | `guardian_dashboard_commitment_rate_per_min` | Effective fleet-wide dashboard per-commitment sustained budget |
 | `guardian_metrics_enabled` | Whether the Guardian Prometheus metrics endpoint is enabled |
-| `cloudwatch_metrics_enabled` | Whether the metrics sidecar, dashboard, and alarms are deployed |
+| `cloudwatch_metrics_enabled` | Whether the metrics sidecar, dashboard, and metric-based alarms are deployed |
 | `metrics_missing_alarm_name` | Name of the metrics-pipeline heartbeat alarm |
 | `metrics_namespace` | CloudWatch namespace receiving Guardian application metrics |
 | `metrics_dashboard_name` | CloudWatch dashboard name |
@@ -343,6 +346,9 @@ aws ecr delete-repository --repository-name "$ECR_REPO_NAME" --force --region "$
 | `alarm_sns_topic_arn` | Managed alarm SNS topic ARN, empty when not enabled |
 | `alarm_slack_configuration_name` | Amazon Q Slack channel configuration name (error log group `/aws/chatbot/<name>`), empty when not configured |
 | `alarm_slack_configuration_arn` | Amazon Q Slack channel configuration ARN, empty when not configured |
+| `cloudwatch_log_alarms_enabled` | Whether the ERROR log metric filter and log-errors alarm are deployed |
+| `log_metrics_namespace` | CloudWatch namespace receiving the log-level metric-filter counts (`<metrics_namespace>/Logs`) |
+| `server_log_errors_alarm_name` | Name of the alarm on ERROR-level server log lines |
 
 ## Stage Profiles
 

infra/data.tf

@@ -200,6 +200,14 @@ locals {
   # Effective alarm/ok action list consumed by every alarm: operator ARNs
   # first, managed topic appended.
   effective_alarm_actions = concat(var.alarm_actions, local.alarm_notifications_enabled ? [aws_sns_topic.alarms[0].arn] : [])
+  # Log-level monitoring (log_alarms.tf): metric-filter counts of the
+  # server's ERROR / WARN lines. Own sub-namespace so metrics_namespace
+  # keeps meaning "what the ADOT pipeline exported" (its presence is the
+  # documented pipeline health check).
+  effective_guardian_log_format = lower(trimspace(var.guardian_log_format))
+  log_metrics_namespace         = "${local.metrics_namespace}/Logs"
+  log_error_metric_name         = "log_error_events"
+  log_warn_metric_name          = "log_warn_events"
 
   # Custom domain configuration
   domain_enabled      = var.domain_name != ""