企业官网建设流程全解析

故障告警

• 告警发生时间：2026-10-09 15:11:00
• 告警标题： ceph 文件系统降级

故障现象

• ceph mds 故障，进程自动重启
• mds 自动恢复
• 故障时间约 2 分钟

分析

• [用户操作 readdir 导致 （通过日志分析 发现大量用户 readdir 行为发生)
• [对应 用户信息 (参考日志，定位用户 ip 地址)
• 主机 gd18-k8s-noah-ai-llm-gpu-prod002-node-s0109

故障定位方法

1. 从日志检测用户操作，找到操作最多，最可以的操作
2. 根据操作定位到主机
3. 登录到主机，利用 epbf 找到执行对应操作类型的 PID
4. 根据 PID 获取到 Pod 信息 进程信息

故障根因

• 进程启动命令时候直接用 CephFS 根目录作为 logdir

排查方法

重点 ！！ 必须要当前用户还在操作才可以发现 !!!

参考日志，先获取 readdir 用户 id caller_uid=0, caller_gid=0{})

2026-10-09T15:11:16.711+0800 7f19354e4700 1 -- [v2:10.42.134.21:6929/3265411376,v1:10.42.134.21:6930/3265411376] <== client.222085723 v1:10.42.144.23:0/3207384292 1485045558 ==== client_request(client.222085723:26024257309 readdir #0x10003b67ef9 2026-10-09T15:11:16.708921+0800 caller_uid=0, caller_gid=0{}) v2 ==== 166+0+0 (unknown 3475745249 0 0) 0x55f3aead78c0 con 0x55f261402400 |

需要登录到 10.42.134.21 节点上进行后续处理

在节点中进行排查

执行下面脚本
输出格式为 用户ID - 进程PID - 进程名称 - 重复执行次数
ceph_readdir 是 linux 内核中的调用函数， 后续有说明

yum install -y bpftrace bpftrace -e ' kprobe:ceph_readdir { @[uid, pid, comm] = count(); } interval:s:10 { print(@); clear(@); }' Attaching 2 probes... @[1000, 1326516, jupyterhub-sing]: 2 @[1000, 3516740, /opt/conda/bin/]: 4 @[1000, 976884, node]: 10 @[0, 2920201, tensorboard]: 113682 @[1000, 1326516, jupyterhub-sing]: 2 @[0, 2920201, tensorboard]: 87347 @[1000, 1326516, jupyterhub-sing]: 2 @[0, 2920201, tensorboard]: 140600 @[1000, 1326516, jupyterhub-sing]: 2 @[0, 2920201, tensorboard]: 119331 <--- uid = 0 pid = 2920201 @[1000, 3516743, /opt/conda/bin/]: 2 @[1000, 3516740, /opt/conda/bin/]: 2 @[1000, 1326516, jupyterhub-sing]: 2 @[1000, 3516742, /opt/conda/bin/]: 2

PID 为 2920201

根据 PID 获取 POD 信息

# cat /proc/2920201/cgroup 13:rdma:/ 12:hugetlb:/kubepods/besteffort/podf99049b4-1ddb-4af3-bbd2-1405d918b961/5fd83966896c0aaaf8de0124217ef6062f462640d4bde50bf475fa6ba92ec7fd 11:cpu,cpuacct:/kubepods/besteffort/podf99049b4-1ddb-4af3-bbd2-1405d918b961/5fd83966896c0aaaf8de0124217ef6062f462640d4bde50bf475fa6ba92ec7fd 10:memory:/kubepods/besteffort/podf99049b4-1ddb-4af3-bbd2-1405d918b961/5fd83966896c0aaaf8de0124217ef6062f462640d4bde50bf475fa6ba92ec7fd 9:ioasids:/ 8:cpuset:/kubepods/besteffort/podf99049b4-1ddb-4af3-bbd2-1405d918b961/5fd83966896c0aaaf8de0124217ef6062f462640d4bde50bf475fa6ba92ec7fd 7:perf_event:/kubepods/besteffort/podf99049b4-1ddb-4af3-bbd2-1405d918b961/5fd83966896c0aaaf8de0124217ef6062f462640d4bde50bf475fa6ba92ec7fd 6:net_cls,net_prio:/kubepods/besteffort/podf99049b4-1ddb-4af3-bbd2-1405d918b961/5fd83966896c0aaaf8de0124217ef6062f462640d4bde50bf475fa6ba92ec7fd 5:blkio:/kubepods/besteffort/podf99049b4-1ddb-4af3-bbd2-1405d918b961/5fd83966896c0aaaf8de0124217ef6062f462640d4bde50bf475fa6ba92ec7fd 4:devices:/kubepods/besteffort/podf99049b4-1ddb-4af3-bbd2-1405d918b961/5fd83966896c0aaaf8de0124217ef6062f462640d4bde50bf475fa6ba92ec7fd 3:pids:/kubepods/besteffort/podf99049b4-1ddb-4af3-bbd2-1405d918b961/5fd83966896c0aaaf8de0124217ef6062f462640d4bde50bf475fa6ba92ec7fd 2:freezer:/kubepods/besteffort/podf99049b4-1ddb-4af3-bbd2-1405d918b961/5fd83966896c0aaaf8de0124217ef6062f462640d4bde50bf475fa6ba92ec7fd 1:name=systemd:/kubepods/besteffort/podf99049b4-1ddb-4af3-bbd2-1405d918b961/5fd83966896c0aaaf8de0124217ef6062f462640d4bde50bf475fa6ba92ec7fd

POD 信息为 5fd83966896c0aaaf8de0124217ef6062f462640d4bde50bf475fa6ba92ec7fd

获取 POD name 方法

yum install -y crictl # crictl inspect 5fd83966896c0aaaf8de0124217ef6062f462640d4bde50bf475fa6ba92ec7fd | \ > jq -r '.status.labels | "namespace=\(.["io.kubernetes.pod.namespace"]) pod=\(.["io.kubernetes.pod.name"])"' WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. namespace=mlp pod=tf-board-default-5bdf4c5df5-sxzwg

获取 docker 信息

# docker ps | grep tf-board-default-5bdf4c5df5-sxzwg 5fd83966896c 4b333ee54d27 "/apps/sh/start-tfbo…" 9 days ago Up 9 days k8s_tf-board-default_tf-board-default-5bdf4c5df5-sxzwg_mlp_f99049b4-1ddb-4af3-bbd2-1405d918b961_20 b351b6be1c12 google_containers/pause:0.8.0 "/pause" 4 months ago Up 4 months k8s_POD_tf-board-default-5bdf4c5df5-sxzwg_mlp_f99049b4-1ddb-4af3-bbd2-1405d918b961_0

获取进程方法

# ps -fp 2920201 UID PID PPID C STIME TTY TIME CMD root 2920201 2920149 56 Oct01 ? 5-07:11:00 /usr/local/bin/python /usr/local/bin/tensorboard --logdir /apps/dat/training/ --host 0.0.0.0 --port 185

获取 IP 方法

# nsenter -t 2920201 -n ip -4 addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 59: eth0@if58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link-netnsid 0 inet 10.43.144.212/24 scope global eth0 valid_lft forever preferred_lft forever

信息汇总

• pod name namespace=mlp pod=tf-board-default-5bdf4c5df5-sxzwg
• pod ip 10.43.144.212
• 进程 /usr/local/bin/python /usr/local/bin/tensorboard --logdir /apps/dat/training/ --host 0.0.0.0 --port 185
• 进程会重复遍历日志目录导致该故障

解决方法

• logdir 不要使用 CephFS 文件系统或指定 CephFS 中某一个目录(不是最优选择) 不要使用 CephFS 根目录

CephFS MDS 实时监控用户操作方式

• 在 CephFS MDS 中可以执行命令， 可以实时检测用户操作的统计
• 需要利用 ceph daemon mds.xxx.asok perf dump 命令定期获取 mds 性能状态
• 对 ceph 性能状态进行比对就可以得到用户操作

参考表格中，代表了 perf dump 中常见的用户操作类型

• 利用下面命令就可以对用户操作进行监控
• 只针对某一种类型进行监控方法

ASOK=/var/run/ceph/vip-ceph-mds.$(hostname -s ).asok prev=$(ceph daemon $ASOK perf dump | jq '.mds_server.req_setfilelock_latency.avgcount') while true; do sleep 10 cur=$(ceph daemon $ASOK perf dump | jq '.mds_server.req_setfilelock_latency.avgcount') echo "$(date +%H:%M:%S) setfilelock_rate=$(( (cur-prev)*6/10 ))/s" prev=$cur done

• 函数方式的监控方法

ASOK=/var/run/ceph/vip-ceph-mds.$(hostname -s).asok watch_rate() { local c=$1 i=${2:-10} prev cur prev=$(ceph -c /etc/ceph/vip-ceph.conf daemon $ASOK perf dump | jq ".$c.avgcount // .$c") while true; do sleep $i cur=$(ceph daemon $ASOK perf dump | jq ".$c.avgcount // .$c") echo "$(date +%H:%M:%S) $c rate=$(( (cur-prev)/i ))/s" prev=$cur done }

• 执行持续监控方法 watch_rate mds_server.req_create_latency 10

20:44:13 mds_server.req_create_latency rate=18/s 20:44:23 mds_server.req_create_latency rate=15/s 20:44:33 mds_server.req_create_latency rate=16/s 20:44:44 mds_server.req_create_latency rate=20/s 20:44:54 mds_server.req_create_latency rate=16/s

• 执行通用监控方式

ASOK=/var/run/ceph/vip-ceph-mds.$(hostname -s).asok INTERVAL=10 prev=$(ceph -c /etc/ceph/vip-ceph.conf daemon $ASOK perf dump | jq '.mds_server') while true; do sleep $INTERVAL cur=$(ceph daemon $ASOK perf dump | jq '.mds_server') echo "===== $(date +%H:%M:%S) MDS user-behavior rates (ops/s) =====" jq -rn --argjson p "$prev" --argjson c "$cur" --argjson i $INTERVAL ' [ $c | to_entries[] | select(.key | test("^req_.*_latency$")) | { op: (.key | sub("^req_";"") | sub("_latency$";"")), rate: (((.value.avgcount - ($p[.key].avgcount // 0)) / $i) * 10000 | round / 10000) } ] | sort_by(-.rate) | .[] | select(.rate > 0) | "\(.op)\t\(.rate)"' prev=$cur done ===== 20:46:32 MDS user-behavior rates (ops/s) ===== readdir 147.2 lookup 26.3 setfilelock 13.5 create 9.6 rename 9.3 getattr 2.7 open 0.3 unlink 0.3 setattr 0.1 ===== 20:46:42 MDS user-behavior rates (ops/s) ===== readdir 125.9 lookup 25.1 create 10 rename 9.6 setfilelock 2 open 0.7 setattr 0.4 getattr 0.2 lookupino 0.1 unlink 0.1

在 K8S 主机上执行日常排查

• 先找到操作量最大服务器]
• 查询当前服务器操作量最大的操作

k8s 下利用 EBPF 进行实时 CephFS 操作监控方法

kport 函数对应信息

函数确认

• 不同版本可能函数名称不一样
• 利用下面方法确认函数是否可用

# bpftrace -l 'kprobe:ceph_*' kprobe:ceph_add_cap kprobe:ceph_adjust_caps_max_min kprobe:ceph_adjust_quota_realms_count kprobe:ceph_aes_crypt kprobe:ceph_aio_complete_req 。。。。。。。 这里太多了， 忽略, 每个内存版本都可能使用了不一样的函数，因此没必要详细输出 。。。。。。。 kprobe:ceph_zero_page_vector_range kprobe:ceph_zero_partial_object

利用 ebpf 进行监控

• 利用该脚本就可以很方便地获取到当前正在执行疯狂操作的进程

bpftrace -e ' // 打开: ceph_open(struct inode *inode, struct file *file) kprobe:ceph_open { $f = (struct file *)arg1; @open[str($f->f_path.dentry->d_name.name), uid, comm] = count(); } // 创建文件: ceph_create(struct inode *dir, struct dentry *dentry, umode_t mode) kprobe:ceph_create { $d = (struct dentry *)arg1; printf("%s CREATE uid=%d comm=%s dir=%s/%s\n", strftime("%H:%M:%S", nsecs), uid, comm, str($d->d_parent->d_name.name), str($d->d_name.name)); } // 建目录: ceph_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode) kprobe:ceph_mkdir { $d = (struct dentry *)arg1; printf("%s MKDIR uid=%d comm=%s %s/%s\n", strftime("%H:%M:%S", nsecs), uid, comm, str($d->d_parent->d_name.name), str($d->d_name.name)); } // 删除文件: ceph_unlink(struct inode *dir, struct dentry *dentry) kprobe:ceph_unlink { $d = (struct dentry *)arg1; printf("%s UNLINK uid=%d comm=%s %s/%s\n", strftime("%H:%M:%S", nsecs), uid, comm, str($d->d_parent->d_name.name), str($d->d_name.name)); } // 改名: ceph_rename(struct inode *old_dir, struct dentry *old, struct inode *new_dir, struct dentry *new) kprobe:ceph_rename { $o = (struct dentry *)arg1; $n = (struct dentry *)arg3; printf("%s RENAME uid=%d comm=%s %s -> %s\n", strftime("%H:%M:%S", nsecs), uid, comm, str($o->d_name.name), str($n->d_name.name)); } // 写数据: ceph_write_iter(struct kiocb *iocb, struct iov_iter *to) kprobe:ceph_write_iter { $k = (struct kiocb *)arg0; @write[str($k->ki_filp->f_path.dentry->d_name.name), uid, comm] = count(); } interval:s:10 { print(@open); print(@write); clear(@open); clear(@write); }'

返回执行结果

@open[vscode.lock, 1000, libuv-worker]: 10 @write[events.out.tfevents.1791603961.jupyter-stan-2eluo.1428341.1, 1000, /opt/conda/bin/]: 1 @write[events.out.tfevents.1791601538.jupyter-stan-2eluo.1427656.0, 1000, /opt/conda/bin/]: 1 @write[events.out.tfevents.1791603947.jupyter-stan-2eluo.1428341.0, 1000, /opt/conda/bin/]: 3 @open[vscode.lock, 1000, libuv-worker]: 10 @write[events.out.tfevents.1791603947.jupyter-stan-2eluo.1428341.0, 1000, /opt/conda/bin/]: 1 @open[vscode.lock, 1000, libuv-worker]: 10 @write[events.out.tfevents.1791601543.jupyter-stan-2eluo.1427656.1, 1000, /opt/conda/bin/]: 1 @write[events.out.tfevents.1791601538.jupyter-stan-2eluo.1427656.0, 1000, /opt/conda/bin/]: 2

说明

• open | write 代表操作
• [vscode.lock, 1000, libuv-worker] 格式为 [被操作的对象, uid, 进程]
• 最后代表每 10 秒发生的次数