RDS Incident Response and Operations#

한마디로 (비개발자용)#

이 스킬은 서비스의 데이터가 저장되는 "창고(데이터베이스)"에 문제가 생겼을 때 대응하는 응급 매뉴얼입니다. 창고가 꽉 차거나, 갑자기 느려지거나, 손님(사용자)이 "공사 중" 화면을 보게 되는 상황을 빠르게 진단하고 해결하도록 돕습니다. 마치 건물 관리인이 정전이나 누수에 대비한 대응 절차를 미리 정리해 둔 것과 비슷합니다.

무엇을·언제 (비개발자용)#

• 무엇을 해 주나요
  • 데이터 창고(RDS)의 상태를 점검하고, 저장 공간이 부족하면 즉시 늘려 줍니다.
  • 데이터베이스가 느려지거나 멈췄을 때 원인을 찾아냅니다.
  • 쌓여서 공간을 잡아먹는 불필요한 기록을 정리합니다.
  • 응급 조치 후 설정 문서(인프라 코드)와 실제 상태가 어긋나지 않도록 맞춰 줍니다.
• 언제 작동하나요
  • 화면에 502/503 오류나 "서버 점검 중" 안내가 뜰 때
  • 데이터 창고가 거의 다 찼거나 상태가 이상할 때(storage-full 등)
  • 서비스가 전반적으로 느려졌을 때
  • 데이터 복제 과정에서 기록(WAL)이 계속 쌓여 공간을 차지할 때

핵심 용어 (비개발자용)#

Covers RDS PostgreSQL fault diagnosis, storage expansion, and replication slot management.

Triggers#

• Server 502/503 errors (app shows "server maintenance")
• RDS instance status anomalies (storage-full, modifying, etc.)
• Database performance degradation
• Replication slot WAL accumulation

See rules/aws-operations.md for infrastructure debugging order

RDS Storage-Full Response#

Diagnosis#

# 1. Check RDS instance status
aws rds describe-db-instances \
  --db-instance-identifier kobic- < env >   \
  --query  ' DBInstances[0].{Status:DBInstanceStatus,Storage:AllocatedStorage,MaxStorage:MaxAllocatedStorage} ' 

 # 2. FreeStorageSpace metric (last 1 hour)
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name FreeStorageSpace \
  --dimensions Name=DBInstanceIdentifier,Value=kobic- < env >   \
  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 --statistics Minimum \
  --output table

# 3. Check table sizes after DB connection
SELECT schemaname, tablename,
       pg_size_pretty(pg_total_relation_size(schemaname|| ' . ' ||tablename)) as total_size
FROM pg_tables
WHERE schemaname =  ' public ' 
 ORDER BY pg_total_relation_size(schemaname|| ' . ' ||tablename) DESC
LIMIT 20;

Emergency Storage Expansion#

# Applied immediately, no downtime
aws rds modify-db-instance \
  --db-instance-identifier kobic- < env >   \
  --allocated-storage  < new_size_gb >   \
  --max-allocated-storage  < new_max_gb >   \
  --apply-immediately

Cautions:

• RDS storage expansion is irreversible (once increased, cannot decrease)
• 6-hour cooldown after expansion (no further expansion possible)
• DB operates normally even in storage-optimization state

Terraform Drift Handling#

After emergency expansion, Terraform code and actual infrastructure are out of sync:

# config.auto.tfvars update required
db_configs = {
   " staging "   = {
    allocated_storage       = 200  # Reflect emergency expansion
    max_allocated_storage   = 500  # Reflect emergency expansion
    # ...
  }
}

Replication Slot Management#

CDC (Change Data Capture) replication slots accumulate WAL and consume storage.

-- Check replication slot status
SELECT slot_name, slot_type, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) as retained_wal
FROM pg_replication_slots;

-- Delete inactive slots (ClickHouse ClickPipe, etc.)
SELECT pg_drop_replication_slot( ' slot_name_here ' );

RDS Instance Specifications#

Current Configuration (Graviton2 ARM64)#

See rules/aws-operations.md for CloudWatch alarm status

Checklist#

Storage-Full Response#

• Check status with aws rds describe-db-instances
• Check CloudWatch FreeStorageSpace metric
• Analyze table sizes (pg_total_relation_size)
• Check replication slot retained WAL
• Execute emergency storage expansion
• Update Terraform config.auto.tfvars (separate PR)

Regular Monitoring#

• Monthly storage utilization check
• Replication slot WAL size check
• CloudWatch alarm alarm_actions connection verification