企业官网建设流程全解析

离线Ingress证书自签名配置、内网HTTPS闭环

技术栈：Kubernetes v1.32.13 + Rocky Linux 8.6 + Containerd 1.7.x

操作环境 / 对接原理 / 详细步骤 / 完整命令 / 配置文件 / 验证流程 / 排错方案

离线Ingress证书自签名配置、内网HTTPS闭环

操作环境

• K8s集群3节点：k8s-master(192.168.1.10), k8s-node1(192.168.1.11), k8s-node2(192.168.1.12)，K8s版本 v1.32.13，容器运行时 Containerd 1.7.x，操作系统 Rocky Linux 8.6

• 离线环境：无外网物理隔离机房，内网网段 192.168.1.0/24，已搭建离线YUM源、离线镜像仓库(Harbor)、离线NTP服务

• 离线场景：K8s离线环境通用，已完成离线环境基础搭建，K8s集群已部署，离线镜像仓库已对接，离线监控日志系统已部署

• 存储配置：本地存储/NFS/Ceph，数据目录 /data，已配置StorageClass和PVC动态供给

• 已配置离线监控系统（Prometheus+Grafana），可查看集群指标、节点指标、应用指标、日志、事件，便于监控和排障

对接原理

离线Ingress证书自签名配置、内网HTTPS闭环是 K8s 离线环境专属对接场景中的核心操作场景。离线环境是政企内网、涉密机房、私有化部署的典型形态，无外网物理隔离带来了软件包获取、镜像拉取、漏洞库更新、时间同步、日志归集等一系列特殊挑战。K8s离线环境通用作为离线环境运维的具体领域，负责离线环境规划预检、K8s集群离线部署、离线镜像仓库运维、离线网络适配、离线存储对接、离线监控日志告警、离线CI/CD持续交付等核心能力。离线环境运维的核心目标是在无外网条件下实现K8s集群的稳定运行、安全可控、可追溯、可演进：通过前置规划和环境预检确保离线环境基础达标，通过离线安装包和依赖归档实现K8s集群从零部署，通过离线镜像仓库和镜像全生命周期管理实现容器镜像的可控分发，通过离线网络深度适配实现集群内外网络连通与隔离，通过离线存储全场景适配实现数据持久化与安全，通过离线监控日志告警体系实现无外网可观测，通过离线CI/CD实现业务持续交付。所有操作需遵循"先规划后实施、先备份后操作、先验证后上线"的安全原则，避免离线环境操作引发集群中断。

详细步骤

1. 离线环境现状盘点与基础状态检查

# 1. 检查离线环境基础状态 ssh root@192.168.1.10 # 检查系统信息 cat /etc/os-release uname -r # 检查网络 ip addr ip route ping -c 3 192.168.1.11 # 检查时间 timedatectl chronyc sources 2>/dev/null || ntpq -p 2>/dev/null # 检查主机名和hosts hostname cat /etc/hosts # 检查防火墙 firewall-cmd --state 2>/dev/null firewall-cmd --list-ports 2>/dev/null iptables -L -n | head -20 # 检查SELinux getenforce sestatus # 检查swap free -h swapon --show # 检查磁盘 df -h lsblk # 检查内核模块 lsmod | grep -E 'overlay|br_netfilter|ip_vs' # 检查系统参数 sysctl net.ipv4.ip_forward sysctl net.bridge.bridge-nf-call-iptables sysctl net.bridge.bridge-nf-call-ip6tables ​ # 2. 检查K8s集群状态 kubectl get nodes -o wide kubectl get pods -A -o wide kubectl get cs # 检查组件状态 kubectl get pods -n kube-system # 检查集群信息 kubectl cluster-info kubectl version ​ # 3. 检查容器运行时 systemctl status containerd containerd --version ctr version # 检查containerd配置 cat /etc/containerd/config.toml | head -50 # 检查镜像 ctr images ls | head -20 ​ # 4. 检查离线YUM源 cat /etc/yum.repos.d/*.repo yum repolist # 测试安装 yum list available | head -20 ​ # 5. 检查离线镜像仓库 curl -k -I https://harbor.example.com 2>/dev/null curl -k -u admin:${HARBOR_PASSWORD} https://harbor.example.com/api/v2.0/projects 2>/dev/null | jq '.[].name' # 检查containerd仓库配置 cat /etc/containerd/config.toml | grep -A10 'registry.mirrors' ​ # 6. 检查离线NTP chronyc tracking 2>/dev/null chronyc sources -v 2>/dev/null # 检查时间同步 timedatectl status ​ # 7. 检查离线监控 curl -s http://prometheus:9090/api/v1/status/config 2>/dev/null | head -20 curl -s http://grafana:3000/api/health 2>/dev/null # 检查节点指标 kubectl top nodes kubectl top pods -A ​ # 8. 导出当前配置备份 mkdir -p /tmp/offline_backup_$(date +%Y%m%d) cp -r /etc/containerd /tmp/offline_backup_$(date +%Y%m%d)/ 2>/dev/null cp -r /etc/kubernetes /tmp/offline_backup_$(date +%Y%m%d)/ 2>/dev/null cp /etc/hosts /tmp/offline_backup_$(date +%Y%m%d)/ 2>/dev/null cp /etc/yum.repos.d/*.repo /tmp/offline_backup_$(date +%Y%m%d)/ 2>/dev/null echo "配置已备份到 /tmp/offline_backup_$(date +%Y%m%d)/" ​

2. 制定操作方案与备份防护

# 1. 制定离线环境操作方案 cat > /tmp/offline_operation_plan.md << 'EOF' # 离线环境操作方案 ## 一、操作目标 ## 二、影响范围评估 ## 三、操作步骤与时间窗口 ## 四、回滚方案 ## 五、验证标准 ## 六、风险点与应对措施 ## 七、离线环境特殊注意事项 EOF echo "操作方案模板已创建" ​ # 2. 确认离线资源可用性 # 检查离线安装包 ls -lh /opt/offline/packages/ 2>/dev/null # 检查离线镜像 ls -lh /opt/offline/images/ 2>/dev/null # 检查离线YUM源 df -h /var/www/html/yum/ 2>/dev/null # 检查离线镜像仓库容量 df -h /data/harbor/ 2>/dev/null ​ # 3. 备份关键数据 # 备份K8s配置 kubectl get all -A -o yaml > /tmp/k8s_all_backup_$(date +%Y%m%d).yaml # 备份etcd kubectl -n kube-system exec etcd-k8s-master -- etcdctl snapshot save /tmp/etcd_snapshot_$(date +%Y%m%d).db --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key 2>/dev/null # 备份Harbor数据 cd /opt/harbor && docker exec harbor-db pg_dump -U postgres registry > /tmp/harbor_db_backup_$(date +%Y%m%d).sql 2>/dev/null # 备份配置文件 tar czf /tmp/offline_config_backup_$(date +%Y%m%d).tar.gz /etc/kubernetes /etc/containerd /etc/hosts /etc/yum.repos.d 2>/dev/null echo "关键数据已备份" ​ # 4. 通知相关业务方 # echo "离线环境运维操作通知" | mail -s "离线环境通知" admin@example.com echo "建议在业务低峰期执行离线环境操作" ​ # 5. 确认业务窗口 echo "当前时间: $(date)" echo "离线环境操作需谨慎，避免影响业务" ​

3. 执行离线环境配置操作

# 1. 检查离线环境配置文件 # 检查K8s配置 ls -la /etc/kubernetes/ cat /etc/kubernetes/admin.conf | head -20 # 检查kubelet配置 cat /var/lib/kubelet/config.yaml | head -30 # 检查kube-proxy配置 cat /var/lib/kube-proxy/config.conf 2>/dev/null | head -30 ​ # 2. 检查容器运行时配置 cat /etc/containerd/config.toml # 检查cgroup驱动 grep 'SystemdCgroup' /etc/containerd/config.toml # 检查镜像仓库配置 grep -A20 'registry' /etc/containerd/config.toml ​ # 3. 检查CNI配置 ls -la /etc/cni/net.d/ cat /etc/cni/net.d/10-calico.conflist 2>/dev/null | head -30 cat /etc/cni/net.d/10-flannel.conflist 2>/dev/null | head -30 # 检查Calico kubectl get pods -n kube-system | grep calico kubectl get ippools 2>/dev/null ​ # 4. 检查CoreDNS kubectl get pods -n kube-system | grep coredns kubectl get configmap coredns -n kube-system -o yaml | head -40 # 测试DNS kubectl run dns-test --image=busybox --restart=Never --rm -it -- nslookup kubernetes.default 2>/dev/null ​ # 5. 检查存储配置 kubectl get storageclass kubectl get pv kubectl get pvc -A # 检查本地存储 ls -la /data/ # 检查NFS showmount -e localhost 2>/dev/null mount | grep nfs ​ # 6. 检查监控配置 kubectl get pods -n monitoring 2>/dev/null kubectl get svc -n monitoring 2>/dev/null # 检查Prometheus配置 kubectl get configmap prometheus-server -n monitoring -o yaml 2>/dev/null | head -40 ​ # 7. 执行具体离线环境配置操作（根据标题调整） echo "执行离线环境具体配置操作..." echo "请根据操作方案执行具体步骤" ​ # 8. 应用配置变更 # 修改配置后重启相关服务 # systemctl restart kubelet # systemctl restart containerd # kubectl rollout restart deployment coredns -n kube-system ​ # 9. 等待服务重启完成 sleep 30 kubectl get nodes kubectl get pods -A | grep -v Running | grep -v Completed echo "配置变更已应用" ​

4. 验证集群网络存储镜像对接

# 1. 验证K8s集群状态 kubectl get nodes -o wide # 预期：所有节点 Ready kubectl get pods -A -o wide # 预期：所有Pod Running或Completed ​ # 2. 验证节点资源 kubectl top nodes # 预期：CPU和内存使用率正常 kubectl top pods -A # 预期：Pod资源使用正常 ​ # 3. 验证网络连通性 # 测试Pod间通信 kubectl run net-test1 --image=busybox --restart=Never -n default -- sleep 3600 kubectl run net-test2 --image=busybox --restart=Never -n default -- sleep 3600 sleep 10 POD1_IP=$(kubectl get pod net-test1 -o jsonpath='{.status.podIP}') kubectl exec net-test2 -- ping -c 3 $POD1_IP # 预期：ping通 kubectl delete pod net-test1 net-test2 --ignore-not-found ​ # 4. 验证Service和DNS kubectl run dns-test --image=busybox --restart=Never --rm -it -- nslookup kubernetes.default # 预期：DNS解析成功 kubectl run dns-test2 --image=busybox --restart=Never --rm -it -- wget -q -O- https://kubernetes.default.svc.cluster.local 2>/dev/null # 预期：Service访问成功 ​ # 5. 验证容器运行时 systemctl status containerd # 预期：active (running) ctr version # 预期：版本正确 ctr images ls | head -10 # 预期：镜像列表正常 ​ # 6. 验证离线镜像仓库 curl -k -I https://harbor.example.com # 预期：返回200或302 # 测试镜像推拉 docker login harbor.example.com -u admin -p ${HARBOR_PASSWORD} docker pull nginx:latest docker tag nginx:latest harbor.example.com/library/nginx:test docker push harbor.example.com/library/nginx:test docker rmi harbor.example.com/library/nginx:test docker pull harbor.example.com/library/nginx:test # 预期：推拉成功 ​ # 7. 验证K8s从离线仓库拉取镜像 kubectl run test-harbor --image=harbor.example.com/library/nginx:test --restart=Never sleep 10 kubectl get pods test-harbor # 预期：Pod Running kubectl delete pod test-harbor --ignore-not-found ​ # 8. 验证存储 # 创建测试PVC cat <<EOF | kubectl apply -f - apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi EOF sleep 10 kubectl get pvc test-pvc # 预期：Bound kubectl delete pvc test-pvc --ignore-not-found ​ # 9. 验证监控 kubectl get pods -n monitoring 2>/dev/null # 预期：监控Pod Running # 检查指标 curl -s http://prometheus:9090/api/v1/query?query=up 2>/dev/null | jq '.data.result | length' # 预期：返回指标数量 ​ # 10. 清理测试资源 docker rmi harbor.example.com/library/nginx:test 2>/dev/null echo "验证完成" ​

5. 离线YUM/NTP/镜像仓库/监控配置

# 1. 配置离线YUM源 # 创建本地YUM源目录 mkdir -p /var/www/html/yum/rocky8 # 同步软件包（在有网机器执行） # reposync -g -l -d -m --repoid=baseos --newest-only --download-metadata -p /var/www/html/yum/rocky8/ # 离线环境直接使用已同步的源 cat > /etc/yum.repos.d/offline.repo << 'EOF' [offline-baseos] name=Offline BaseOS baseurl=file:///var/www/html/yum/rocky8/baseos enabled=1 gpgcheck=0 EOF yum clean all yum repolist ​ # 2. 配置离线NTP # 安装chrony yum install -y chrony # 配置内网NTP服务器 cat > /etc/chrony.conf << 'EOF' server 192.168.1.10 iburst allow 192.168.1.0/24 local stratum 10 EOF systemctl enable --now chronyd chronyc sources -v ​ # 3. 配置离线镜像仓库 # Harbor已部署，配置containerd对接 cat > /etc/containerd/config.toml << 'EOF' version = 2 [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.example.com"] endpoint = ["https://harbor.example.com"] [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.example.com".tls] ca_file = "/etc/containerd/certs.d/harbor.example.com/ca.crt" EOF # 导入证书 mkdir -p /etc/containerd/certs.d/harbor.example.com cp /opt/harbor/ssl/ca.crt /etc/containerd/certs.d/harbor.example.com/ systemctl restart containerd ​ # 4. 配置离线监控 # Prometheus配置 cat > /etc/prometheus/prometheus.yml << 'EOF' global: scrape_interval: 15s scrape_configs: - job_name: 'kubernetes-nodes' kubernetes_sd_configs: - role: node scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token EOF ​ # 5. 配置离线日志 # 配置日志轮转 cat > /etc/logrotate.d/k8s << 'EOF' /var/log/k8s/*.log { daily rotate 30 compress delaycompress missingok notifempty create 0644 root root } EOF ​ # 6. 配置离线安全基线 # 关闭swap swapoff -a sed -i '/swap/d' /etc/fstab # 关闭SELinux setenforce 0 sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config # 配置内核参数 cat > /etc/sysctl.d/k8s.conf << 'EOF' net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 EOF sysctl --system ​ # 7. 配置离线CI/CD # Jenkins配置 # 管理Jenkins插件离线安装 # 配置Jenkins任务对接离线镜像仓库 echo "离线CI/CD配置完成" ​

6. 最终验证与操作报告

# 1. 最终验证K8s集群状态 kubectl get nodes -o wide # 预期：所有节点 Ready kubectl get pods -A | grep -v Running | grep -v Completed # 预期：无异常Pod ​ # 2. 最终验证离线YUM源 yum repolist # 预期：离线源可用 yum list available | wc -l # 预期：返回软件包数量 ​ # 3. 最终验证离线NTP chronyc tracking # 预期：时间同步正常 timedatectl status # 预期：时间正确 ​ # 4. 最终验证离线镜像仓库 curl -k -I https://harbor.example.com # 预期：返回200或302 docker login harbor.example.com -u admin -p ${HARBOR_PASSWORD} docker pull nginx:latest docker tag nginx:latest harbor.example.com/library/nginx:verify docker push harbor.example.com/library/nginx:verify docker rmi harbor.example.com/library/nginx:verify docker pull harbor.example.com/library/nginx:verify # 预期：推拉成功 ​ # 5. 最终验证K8s对接离线仓库 kubectl run verify-offline --image=harbor.example.com/library/nginx:verify --restart=Never sleep 10 kubectl get pods verify-offline # 预期：Pod Running kubectl delete pod verify-offline --ignore-not-found ​ # 6. 最终验证网络 kubectl run net-verify --image=busybox --restart=Never --rm -it -- nslookup kubernetes.default # 预期：DNS解析成功 ​ # 7. 最终验证存储 kubectl get storageclass # 预期：StorageClass存在 kubectl get pv | wc -l # 预期：PV数量正常 ​ # 8. 最终验证监控 kubectl get pods -n monitoring 2>/dev/null # 预期：监控Pod Running ​ # 9. 生成操作报告 echo "=== 离线环境操作报告 ===" > /tmp/offline_operation_report.txt echo "操作时间: $(date)" >> /tmp/offline_operation_report.txt echo "K8s版本: $(kubectl version --short 2>/dev/null | grep Server)" >> /tmp/offline_operation_report.txt echo "节点数: $(kubectl get nodes --no-headers | wc -l)" >> /tmp/offline_operation_report.txt echo "Pod数: $(kubectl get pods -A --no-headers | wc -l)" >> /tmp/offline_operation_report.txt echo "节点状态:" >> /tmp/offline_operation_report.txt kubectl get nodes >> /tmp/offline_operation_report.txt cat /tmp/offline_operation_report.txt ​ # 10. 清理测试资源 docker rmi harbor.example.com/library/nginx:verify 2>/dev/null echo "操作完成" ​

验证流程

# 1. K8s集群状态验证 kubectl get nodes -o wide # 预期：所有节点 Ready kubectl get pods -A | grep -v Running | grep -v Completed # 预期：无异常Pod ​ # 2. 节点资源验证 kubectl top nodes # 预期：CPU和内存使用率正常 kubectl top pods -A # 预期：Pod资源使用正常 ​ # 3. 网络连通性验证 kubectl run net-test --image=busybox --restart=Never --rm -it -- nslookup kubernetes.default # 预期：DNS解析成功 ​ # 4. 容器运行时验证 systemctl status containerd # 预期：active (running) ctr version # 预期：版本正确 ​ # 5. 离线镜像仓库验证 curl -k -I https://harbor.example.com # 预期：返回200或302 ​ # 6. 存储验证 kubectl get storageclass # 预期：StorageClass存在 ​ # 7. 监控验证 kubectl get pods -n monitoring 2>/dev/null # 预期：监控Pod Running ​ # 8. 操作报告验证 cat /tmp/offline_operation_report.txt # 预期：报告完整 ​

排错方案

• 离线环境整体可用性问题：检查集群状态、节点、网络、存储、资源、配置、监控、告警、高可用、容灾、SLA

• 离线环境性能问题：检查CPU、内存、磁盘IO、网络、并发、应用、数据库、缓存、负载均衡、监控、调优

• 离线环境安全问题：检查认证、授权、RBAC、加密、审计、漏洞、签名、准入、合规、最小权限、整改

• 离线环境成本问题：检查资源利用率、存储成本、镜像清理、垃圾回收、过期清理、降配、归档、优化

• 离线环境标准化问题：检查SOP、文档、流程、配置规范、命名规范、监控、告警、备份、容灾、安全、合规

• 离线环境迁移升级问题：检查版本兼容、配置迁移、数据迁移、灰度升级、回滚、验证、文档、培训、最佳实践

• 离线环境国产化适配问题：检查麒麟/统信UOS、ARM架构、海光/飞腾/鲲鹏、依赖、版本、配置、日志、适配、调优

• 离线环境等保合规问题：检查等保级别、安全物理环境、通信网络、区域边界、计算环境、管理中心、安全管理制度、机构、人员、建设、运维