[ 屁之博客 ]
[ 屁之博客 ]
$ 登录
发布于: -/最后更新: -/2 分钟

K8s无法挂载PVC Filesystem issues has been detected

摘要

服务无法访问，怀疑 RKE2 配置导致 CSI 挂载失败。Immich-db-1 无法自动挂载，提示缺少 fsRepair 参数。检查发现 XFS 文件系统日志未完成，尝试 xfs_repair 失败。根据提示，将磁盘挂载到临时目录重放日志，确认无误后卸载，最后删除并重建 Pod 恢复服务。

晚上发现服务无法访问，打开一看应该是捣鼓RKE2配置的时候把CSI连接炸了

immich-db-1的Event显示无法自动挂载磁盘

纯文本
MountVolume.MountDevice failed for volume "pvc-656ad344-6ef5-4333-a38f-9d1e2217342b" : rpc error: code = Internal desc = Failed to stage volume ChipyChipy_csi-volumes_pvc-656ad344-6ef5-4333-a38f-9d1e2217342b, err: Filesystem issues has been detected and will not be repaired for the volume ChipyChipy_csi-volumes_pvc-656ad344-6ef5-4333-a38f-9d1e2217342b as the fsRepair parameter is not set in the StorageClass

由于创建StorageClass之初没有加上fsRepair的参数，现在想加也来不及，就尝试手动修复

在目标节点上检查挂载情况

shell
lsblk

NAME     MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda        8:0    0     2G  0 disk
└─mpathd 253:4    0     2G  0 mpath /var/lib/kubelet/pods/36848f1e-e509-49d7-92a2-281c6a1e0871/volumes/kubernetes.io~csi/pvc-599898b0-dc50-4255-a17a-85e3fe939c99/mount
                                    /var/lib/kubelet/plugins/hpe.com/mounts/ChipyChipy_csi-volumes_pvc-599898b0-dc50-4255-a17a-85e3fe939c99
sdb        8:16   0    15G  0 disk
└─mpathc 253:19   0    15G  0 mpath /var/lib/kubelet/plugins/hpe.com/mounts/ChipyChipy_csi-volumes_pvc-656ad344-6ef5-4333-a38f-9d1e2217342b

可以看到sdb的挂载路径比其他少了一条

确定FileSystem

shell
sudo blkid /dev/mapper/mpathc

[sudo] password for mako:
/dev/mapper/mpathc: UUID="a9516c11-be83-44a7-ae1e-7e6589998596" BLOCK_SIZE="512" TYPE="xfs"

把已有的连接断开

shell
sudo umount -l /var/lib/kubelet/plugins/hpe.com/mounts/ChipyChipy_csi-volumes_pvc-656ad344-6ef5-4333-a38f-9d1e2217342b

尝试修复

shell
sudo xfs_repair /dev/mapper/mpathc

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

目标地址存在没写完的日志文件，根据提示我们直接简单挂载一下

shell
sudo mkdir -p /tmp/repair_mount
sudo mount /dev/mapper/mpathc /tmp/repair_mount

如果没有问题的话就直接断开就行了 大功告成

shell
sudo umount /tmp/repair_mount

之后回到K8s中把db pod删了重建就可以了

正文结束