Day 12: 灾难恢复 + 生产事故注入

1. 止血 (stop the bleeding):
   - 隔离故障组件,防止扩散
   - 流量切走 / 副本下线
   
2. 诊断 (diagnose):
   - 看 events / logs / metrics
   - 不要急着改,先理解 "为什么"
   
3. 修复 (recover):
   - 优先用 "已知好状态" 回滚(snapshot / 上一版本)
   - 复杂事故记录 timeline,事后复盘

错误流程 (新手):
  cp-2 etcd data 丢 → 在 cp-2 上 etcdctl snapshot restore
  → 启动 etcd 用恢复的 data → 加入集群?
  ❌ 错!restore 出来的 etcd 是"独立 cluster",member-id 跟现有 cluster 不一致
  → etcd 拒绝 join, "member ID mismatch"

正确流程:
  1. etcdctl member remove <cp-2 member id>  (从集群 active member 列表移除)
  2. 在 cp-2 上清空 etcd data dir
  3. 修改 etcd static pod manifest 加 --initial-cluster-state=existing
  4. etcd member add <cp-2 url> 让 cluster 接受新 member
  5. cp-2 etcd 启动,作为"新 member"加入(数据从 leader 拉)

3 个 cp etcd 全死 (机房断电 + 数据损毁):
  1. 在某个还有 snapshot 的节点跑 restore:
     etcdctl snapshot restore snapshot.db \
       --name k8s-cp-1 \
       --initial-cluster k8s-cp-1=https://10.0.24.31:2380 \
       --initial-cluster-token etcd-cluster-1 \
       --initial-advertise-peer-urls https://10.0.24.31:2380 \
       --data-dir /var/lib/etcd-new
  
  2. mv /var/lib/etcd-new /var/lib/etcd
  
  3. 修改 etcd manifest 启动单节点 cluster (initial-cluster-state=new)
  
  4. 其他 2 个 cp 重新 join: kubeadm reset + kubeadm join with new token

dmesg:
  oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=cri-containerd-...scope
  Out of memory: Killed process 1530928 (stress) total-vm:6292240kB, anon-rss:6188660kB
  Tasks in /kubepods.slice/kubepods-besteffort.slice/... are going to be killed due to memory.oom.group set
  oom_reaper: reaped process 1530928

kubectl get pod mem-hog:
  Status: OOMKilled, exitCode 137

mem-hog-v2:
  Limits:    cpu 500m, memory 512Mi  ← LimitRange 自动注入
  Requests:  cpu 100m, memory 128Mi
  Status:    OOMKilled (15s 内)
  ExitCode:  137

cilium-agent Pods: All deleted (5/5 gone)
Node taints:
  k8s-cp-1: node.cilium.io/agent-not-ready (set by cilium operator)
  k8s-cp-2: node.cilium.io/agent-not-ready
  ... 所有节点

[1分钟] 1. 确认影响范围:
   ETCD_POD=$(kubectl get pod -n kube-system -l component=etcd \
     --field-selector=spec.nodeName=k8s-cp-1 -o name | head -1)
   kubectl exec -n kube-system $ETCD_POD -- etcdctl endpoint health \
     --endpoints=<3-cp-2379-list>

[3分钟] 2. 拉最新 snapshot (灾难前一次的):
   ls -la /backup/etcd-snapshots/
   # 找最近的 *.db

[5分钟] 3. member remove 死掉的 cp:
   etcdctl member remove <cp-id>

[10分钟] 4. 修复磁盘 / 重建节点 / 清 /var/lib/etcd

[15分钟] 5. member add + 改 manifest --initial-cluster-state=existing
   etcdctl member add <new-cp-name> --peer-urls=https://<cp-ip>:2380
   # 改 /etc/kubernetes/manifests/etcd.yaml 加 existing + 新 cluster list

[20分钟] 6. mv etcd.yaml 回 manifests/
   等 30s, kubelet 自启 etcd

[25分钟] 7. 验证 3/3 healthy
   etcdctl endpoint health
   kubectl get pods -n kube-system -l component=etcd

1. 别慌。worker 节点上 Pod 一直在跑(kubelet local mode)
2. 找最近 snapshot(异地 backup,不能丢在本机房!)
3. 选 1 个 cp 节点:
   etcdctl snapshot restore snapshot.db \
     --name <cp1-hostname> \
     --initial-cluster <cp1>=https://<cp1-ip>:2380 \
     --initial-cluster-token <new-token> \
     --initial-advertise-peer-urls https://<cp1-ip>:2380 \
     --data-dir /var/lib/etcd-new
4. mv /var/lib/etcd-new → /var/lib/etcd
5. 改 etcd manifest --initial-cluster-state=new + initial-cluster 仅含 cp1
6. cp1 起来后,其他 cp 走 kubeadm reset + kubeadm join 重新加入
7. 时间点丢失 = 距上次 snapshot 时间(所以高频 snapshot 必要!)

[即刻] 1. 隔离: kubectl drain <node> --ignore-daemonsets
        → 把 workload 调走

[5分钟] 2. 找元凶 Pod:
        kubectl top pods -A --sort-by=memory | head -10
        # 看哪个 Pod mem 异常高

[10分钟] 3. 临时杀掉元凶: kubectl delete pod <elite-name>

[长期修复]:
4. namespace 加 LimitRange 默认 limit
5. namespace 加 ResourceQuota 总量
6. Kyverno policy require-resources enforce (Day 5.F)
7. Grafana 加 alert: 当 mem_util > 85% 持续 5min → 钉钉

[立刻] 1. 不要 panic delete 任何 Pod (避免破坏现有 work)
       2. 检查: kubectl get pods -n kube-system -l k8s-app=cilium
        a. 如果空 (DS 被误删) → helm upgrade cilium 重装
        b. 如果 CrashLoopBackOff → 看 log + 重启 Pod
        c. 如果 Pending → 看 events (image pull / RBAC)

[5分钟] 3. 业务流量切走 (DNS 切到备 cluster 或 fallback page)

[10分钟] 4. helm upgrade cilium cilium/cilium --reuse-values
        sleep 60
        kubectl get pods -n kube-system -l k8s-app=cilium

[15分钟] 5. 验证: 起一个 test Pod 看 IP 分配 + ping 跨节点 Pod

[预防为主!]
1. 监控 cert 过期: prometheus alert
   apiserver_client_certificate_expiration_seconds{quantile="0.01"} < 30*24*3600

[一年 -1天] 2. 提前续期:
   sudo kubeadm certs renew all
   # 重启 cp 上的 apiserver / controller / scheduler

[已过期但还能 ssh]:
3. ssh 进 cp,kubeadm certs renew all
4. 直接重启 etcd / kubelet / restart kube-apiserver pod

[集群完全失联]:
5. 进单个 cp 节点:
   kubeadm certs renew all
   systemctl restart kubelet
6. 用 /etc/kubernetes/admin.conf 在该节点本地 kubectl 救集群