基于kubernetes部署ceph
·
基于kubernetes部署ceph
前提条件
-
已经安装了 Kubernetes 集群，且集群版本不低于 v1.17.0
-
Kubernetes 集群有至少 3 个工作节点，且每个工作节点都有一块初系统盘以外的 未格式化 的裸盘（工作节点是虚拟机时，未格式化的裸盘可以是虚拟磁盘），用于创建 3 个 Ceph OSD
-
也可以只有 1 个工作节点，挂载了一块 未格式化 的裸盘
-
在节点机器上执行
lsblk -f指令可以查看磁盘是否需被格式化，输出结果如下lsblk -f NAME FSTYPE LABEL UUID MOUNTPOINT vda └─vda1 LVM2_member eSO50t-GkUV-YKTH-WsGq-hNJY-eKNf-3i07IB ├─ubuntu--vg-root ext4 c2366f76-6e21-4f10-a8f3-6776212e2fe4 / └─ubuntu--vg-swap_1 swap 9492a3dc-ad75-47cd-9596-678e8cf17ff9 [SWAP] vdb -
如果
FSTYPE字段不为空，则表示该磁盘上已经被格式化。在上面的例子中，可以将磁盘vdb用于 Ceph 的 OSD，而磁盘vda及其分区则不能用做 Ceph 的 OSD
安装 Rook - Ceph
- 部署ceph集群
kubectl create -f https://kuboard.cn/statics/learning/ceph/rook-1.5.4/crds.yaml
kubectl create -f https://kuboard.cn/statics/learning/ceph/rook-1.5.4/common.yaml
kubectl create -f https://kuboard.cn/statics/learning/ceph/rook-1.5.4/operator.yaml
kubectl create -f https://kuboard.cn/statics/learning/ceph/rook-1.5.4/cluster.yaml
- 执行
watch kubectl get pods -n rook-ceph指令，直到所有的 Pod 处于Running或者Completed状态，如下所示：
说明： 国内源：将
k8s.gcr.io
的镜像地址替换成了
swr.cn-east-2.myhuaweicloud.com/kuboard-dependency
------------------------------------------------------------------------------------------------
watch kubectl get pods -n rook-ceph
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-5hfb7 3/3 Running 0 39m
csi-cephfsplugin-5xdz4 3/3 Running 0 39m
csi-cephfsplugin-provisioner-5c65b94c8d-9txpv 6/6 Running 0 39m
csi-cephfsplugin-provisioner-5c65b94c8d-rt4fp 6/6 Running 0 39m
csi-cephfsplugin-pstw9 3/3 Running 0 39m
csi-rbdplugin-ft4dk 3/3 Running 0 39m
csi-rbdplugin-fxj9n 3/3 Running 0 39m
csi-rbdplugin-provisioner-569c75558-h2jv7 6/6 Running 0 39m
csi-rbdplugin-provisioner-569c75558-q4fkt 6/6 Running 0 39m
csi-rbdplugin-rw7jn 3/3 Running 0 39m
rook-ceph-crashcollector-k8s-node-01-6fbb5cb4b8-nwvhj 1/1 Running 0 35m
rook-ceph-crashcollector-k8s-node-02-5c67f6f9f5-qm47c 1/1 Running 0 37m
rook-ceph-crashcollector-k8s-node-03-7f6cfc655b-b8cv2 1/1 Running 0 40m
rook-ceph-mgr-a-5844874f9c-rqggg 1/1 Running 0 35m
rook-ceph-mon-a-67b6865644-4bkm9 1/1 Running 0 40m
rook-ceph-mon-b-59f855c47d-tg44q 1/1 Running 0 40m
rook-ceph-mon-d-7576586cc9-nn94w 1/1 Running 0 37m
rook-ceph-operator-6db6f67cd4-smhz8 1/1 Running 0 41m
rook-ceph-osd-prepare-k8s-node-01-x2p74 0/1 Completed 0 35m
rook-ceph-osd-prepare-k8s-node-02-7j4s7 0/1 Completed 0 35m
rook-ceph-osd-prepare-k8s-node-03-w2mgf 0/1 Completed 0 35m
rook-discover-9hzds 1/1 Running 0 41m
rook-discover-gz7xv 1/1 Running 0 41m
rook-discover-hljrn 1/1 Running 0 41m
- Ceph 集群部署好以后，可以通过 Ceph 提供 块存储、文件存储和对象存储。此处，我们通过如下指令来创建文件存储服务
cat > myfs.yaml <<EOF
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
name: myfs
namespace: rook-ceph
spec:
metadataPool:
replicated:
size: 3
dataPools:
- replicated:
size: 3
preservePoolsOnDelete: true
metadataServer:
activeCount: 1
activeStandby: true
EOF
--------------------------------------------
kubectl create -f myfs.yaml
登录ceph仪表盘
可以使用 CephCluster CRD 中的设置启用仪表板。CephCluster CRD 必须将仪表板设置设置为 。 这是示例清单中的默认设置。enabled``true 默认端口8443
spec:
dashboard:
enabled: true
nodeport访问
vi dashboard-external-https.yaml
--------------------------------
apiVersion: v1
kind: Service
metadata:
name: rook-ceph-mgr-dashboard-external-https
namespace: rook-ceph
labels:
app: rook-ceph-mgr
rook_cluster: rook-ceph
spec:
ports:
- name: dashboard
port: 8443
protocol: TCP
targetPort: 8443
selector:
app: rook-ceph-mgr
rook_cluster: rook-ceph
sessionAffinity: None
type: NodePort
------------------------------
kubectl create -f dashboard-external-https.yaml
------------------------------
$ kubectl -n rook-ceph get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-mgr ClusterIP 10.108.111.192 <none> 9283/TCP 4h
rook-ceph-mgr-dashboard ClusterIP 10.110.113.240 <none> 8443/TCP 4h
rook-ceph-mgr-dashboard-external-https NodePort 10.101.209.6 <none> 8443:31176/TCP 4h
---------------------------获取nodeport端口------------------------
kubectl get pod -n rook-ceph -o wide| grep rook-ceph-mgr
登录：已配置端口 8443
------------------------------------------------------------------
https://10.110.113.240:8443
获取账号： admin
获取密码：
kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
使用命令
使用tool
vi toolbox.yaml
-----------------
apiVersion: apps/v1
kind: Deployment
metadata:
name: rook-ceph-tools
namespace: rook-ceph
labels:
app: rook-ceph-tools
spec:
replicas: 1
selector:
matchLabels:
app: rook-ceph-tools
template:
metadata:
labels:
app: rook-ceph-tools
spec:
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: rook-ceph-tools
image: rook/ceph:v1.5.12
command: ["/tini"]
args: ["-g", "--", "/usr/local/bin/toolbox.sh"]
imagePullPolicy: IfNotPresent
env:
- name: ROOK_CEPH_USERNAME
valueFrom:
secretKeyRef:
name: rook-ceph-mon
key: ceph-username
- name: ROOK_CEPH_SECRET
valueFrom:
secretKeyRef:
name: rook-ceph-mon
key: ceph-secret
volumeMounts:
- mountPath: /etc/ceph
name: ceph-config
- name: mon-endpoint-volume
mountPath: /etc/rook
volumes:
- name: mon-endpoint-volume
configMap:
name: rook-ceph-mon-endpoints
items:
- key: data
path: mon-endpoints
- name: ceph-config
emptyDir: {}
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 5
--------------------------------------------------------
启动 rook-ceph-tools pod
kubectl create -f toolbox.yaml
--------------------------------------------------------
等待工具箱 pod 下载其容器并进入状态：running
kubectl -n rook-ceph rollout status deploy/rook-ceph-tools
--------------------------------------------------------
一旦 rook-ceph-tools pod 运行，你可以通过以下方式连接到它
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
示例：
ceph status
ceph osd status
ceph df
rados df
腾讯云面向开发者汇聚海量精品云计算使用和开发经验，营造开放的云计算技术生态圈。
更多推荐
0
0- 0
已为社区贡献1条内容
所有评论(0)