Kafka 云原生化：在 Kubernetes 部署 Kafka 集群（3 broker）与 ZooKeeper（3 节点）

1. 架构设计要点

• ZooKeeper 集群：作为 Kafka 的元数据管理器和协调服务
• Kafka Broker 集群：3 个 broker 实现消息分区和复制
• 存储方案：使用 PersistentVolume 保障数据持久化
• 网络配置：Headless Service 提供稳定的 DNS 发现机制

2. ZooKeeper 部署（3 节点）

配置文件 (zookeeper-statefulset.yaml)：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zookeeper
spec:
  serviceName: zk-hs
  replicas: 3
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
      - name: zk
        image: zookeeper:3.8
        env:
        - name: ZOO_MY_ID
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        ports:
        - containerPort: 2181
        volumeMounts:
        - name: datadir
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: datadir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi

服务配置 (zookeeper-service.yaml)：

apiVersion: v1
kind: Service
metadata:
  name: zk-hs
spec:
  clusterIP: None
  selector:
    app: zookeeper
  ports:
  - port: 2181

3. Kafka Broker 部署（3 节点）

配置文件 (kafka-statefulset.yaml)：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka-broker
spec:
  serviceName: kafka-hs
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
      - name: kafka
        image: confluentinc/cp-kafka:7.4.0
        env:
        - name: KAFKA_BROKER_ID
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: "zk-hs:2181"
        - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
          value: "INTERNAL:PLAINTEXT"
        - name: KAFKA_ADVERTISED_LISTENERS
          value: "INTERNAL://$(HOSTNAME).kafka-hs:9092"
        ports:
        - containerPort: 9092
        volumeMounts:
        - name: datadir
          mountPath: /var/lib/kafka
  volumeClaimTemplates:
  - metadata:
      name: datadir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 20Gi

服务配置 (kafka-service.yaml)：

apiVersion: v1
kind: Service
metadata:
  name: kafka-hs
spec:
  clusterIP: None
  selector:
    app: kafka
  ports:
  - port: 9092

4. 部署验证

# 创建资源
kubectl apply -f zookeeper-service.yaml
kubectl apply -f zookeeper-statefulset.yaml
kubectl apply -f kafka-service.yaml
kubectl apply -f kafka-statefulset.yaml

# 验证Pod状态
kubectl get pods -l app=zookeeper
kubectl get pods -l app=kafka

# 测试Kafka连接
kubectl exec -it kafka-broker-0 -- \
  kafka-topics --bootstrap-server kafka-hs:9092 --list

5. 关键配置说明

6. 生产环境优化建议

1. 资源配额：配置 CPU/Memory limits 防止资源争抢

   resources:
     limits:
       cpu: "2"
       memory: 4Gi
   

2. 监控集成：添加 Prometheus exporter 监控指标
3. 安全加固：
   • 启用 TLS 加密通信
   • 配置 SASL 身份验证
4. 跨区部署：使用 topologySpreadConstraints 实现 AZ 分布

注：完整部署需包含 RBAC 权限配置和健康检查（livenessProbe/readinessProbe），此处简化示例聚焦核心架构。