AWS Infrastructure Operations Rules

Rules for AWS infrastructure incident response and operations.

Infrastructure Debugging Order#

When an incident occurs, check in the following order (RDS is the most common cause):

1. Check RDS status (storage-full is the most frequent)
2. Check EC2/ASG instance status
3. Check Target Group health checks
4. Check ALB access logs
5. Check CodeDeploy deployment status
6. Check CloudWatch logs

Quick Status Check Commands#

# RDS status
aws rds describe-db-instances \
  --query 'DBInstances[*].{ID:DBInstanceIdentifier,Status:DBInstanceStatus,Storage:AllocatedStorage}' \
  --output table

# ASG instance status
aws autoscaling describe-auto-scaling-groups \
  --query 'AutoScalingGroups[?contains(AutoScalingGroupName,`kobic`)].{Name:AutoScalingGroupName,Desired:DesiredCapacity,Instances:Instances[*].{ID:InstanceId,Health:HealthStatus}}' \
  --output json

# Target Group health
aws elbv2 describe-target-health \
  --target-group-arn <target-group-arn> \
  --output table

# Recent deployment status
aws deploy list-deployments \
  --application-name kobic-app \
  --deployment-group-name kobic-<env>-dg \
  --max-items 3 \
  --query 'deployments'

RDS Storage-Full Response#

Symptoms#

• Server 502/503 errors (app shows "server maintenance")
• RDS instance status: storage-full
• CloudWatch FreeStorageSpace metric reaches 0

Emergency Storage Expansion#

# Applied immediately, no downtime
aws rds modify-db-instance \
  --db-instance-identifier kobic-<env> \
  --allocated-storage <new_size_gb> \
  --max-allocated-storage <new_max_gb> \
  --apply-immediately

Cautions:

• RDS storage expansion is irreversible (cannot shrink)
• 6-hour cooldown after expansion
• DB operates normally even in storage-optimization state
• Update Terraform config.auto.tfvars after emergency expansion (to prevent drift)

Replication Slot Management#

CDC replication slots accumulate WAL and consume storage.

-- Check slot status
SELECT slot_name, slot_type, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) as retained_wal
FROM pg_replication_slots;

-- Delete inactive slots
SELECT pg_drop_replication_slot('slot_name_here');

CodeDeploy ASG Accumulation Issue#

Symptoms#

• CodeDeploy attempts to deploy to deleted ASGs during deployment → failure

Resolution#

# Check ASG list in CodeDeploy Deployment Group
aws deploy get-deployment-group \
  --application-name kobic-app \
  --deployment-group-name kobic-<env>-dg \
  --query 'deploymentGroupInfo.autoScalingGroups[].name'

# Update to include only the correct ASG
aws deploy update-deployment-group \
  --application-name kobic-app \
  --current-deployment-group-name kobic-<env>-dg \
  --auto-scaling-groups "kobic-<env>-asg"

Docker-Based Deployment Workflow#

GitHub Actions → Docker Build → ECR Push → CodeDeploy → ASG Instance (Docker Compose)

CloudWatch Alarm Status#

ElastiCache Redis Cautions#

• AUTH cannot be enabled when Transit Encryption is disabled
• Serverpod requires SERVERPOD_PASSWORD_redis environment variable when Redis is enabled
• Set SERVERPOD_PASSWORD_redis="" to start server (operates without AUTH)
• Authentication works normally on DB-based auth without Redis

Checklist#

Incident Response#

• Check RDS instance status (describe-db-instances)
• Check CloudWatch FreeStorageSpace metric
• Check replication slot retained WAL
• Check ASG/Target Group health checks
• Check CodeDeploy deployment status

Terraform Drift Prevention#

• Update config.auto.tfvars after emergency AWS CLI changes
• Verify drift with terraform plan
• Update code via separate PR