CentOS Stream 8
Sponsored Link

Ceph Pacific : Cephadm #1 クラスターの設定2021/07/08
 
Ceph クラスター デプロイ ツール [Cephadm] を使用した Ceph クラスターの新規構築です。
当例では 3台 のノードでクラスターを構成します。
3台 のノードにはそれぞれ空きブロックデバイスがあることが前提です。
(当例では [/dev/sdb] を使用) 
                                         |
            +----------------------------+----------------------------+
            |                            |                            |
            |10.0.0.51                   |10.0.0.52                   |10.0.0.53 
+-----------+-----------+    +-----------+-----------+    +-----------+-----------+
|   [node01.srv.world]  |    |   [node02.srv.world]  |    |   [node03.srv.world]  |
|     Object Storage    +----+     Object Storage    +----+     Object Storage    |
|     Monitor Daemon    |    |                       |    |                       |
|     Manager Daemon    |    |                       |    |                       |
+-----------------------+    +-----------------------+    +-----------------------+
[1]
[Cephadm] から各ノードの制御に Python 3 が必要となります。
よって、全ノードに、こちらの [1] を参考に Python 3 をインストールしておきます。
[2] [Cephadm] はコンテナーベースで Ceph クラスターを構築します。
よって、全ノードに Podman 等々をインストールしておきます。
[root@node01 ~]#
dnf module -y reset container-tools

[root@node01 ~]#
dnf module -y enable -y container-tools:3.0

[root@node01 ~]#
dnf module -y install container-tools:3.0/common

[3] [Monitor Daemon], [Manager Daemon] を設定するノードで [Cephadm] をインストールします。
当例では [node01] で進めます。
[root@node01 ~]#
dnf -y install centos-release-ceph-pacific epel-release
[root@node01 ~]#
dnf -y install cephadm
[4] Ceph クラスターを新規構築します。
[root@node01 ~]#
mkdir -p /etc/ceph

[root@node01 ~]#
cephadm bootstrap --mon-ip 10.0.0.51 

Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman|docker (/usr/bin/podman) is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: f492755c-dfa7-11eb-be3f-525400238cef
Verifying IP 10.0.0.51 port 3300 ...
Verifying IP 10.0.0.51 port 6789 ...
Mon IP 10.0.0.51 is in CIDR network 10.0.0.0/24
- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
Pulling container image docker.io/ceph/ceph:v16...

.....
.....

Enabling firewalld port 8443/tcp in current zone...
Ceph Dashboard is now available at:

             URL: https://node01.srv.world:8443/
            User: admin
        Password: 0feuzppddy

You can access the Ceph CLI with:

        sudo /usr/sbin/cephadm shell --fsid 54952d44-dfaa-11eb-8457-525400238cef -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

Please consider enabling telemetry to help improve Ceph:

        ceph telemetry on

For more information see:

        https://docs.ceph.com/docs/pacific/mgr/telemetry/

Bootstrap complete.
# Ceph Cli を有効にする

[root@node01 ~]#
alias ceph='cephadm shell -- ceph'

[root@node01 ~]#
echo "alias ceph='cephadm shell -- ceph'" >> ~/.bashrc
[root@node01 ~]#
ceph -v

ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
# OSD 未設定のため 現時点では [HEALTH_WARN] で OK

[root@node01 ~]#
ceph -s 

  cluster:
    id:     54952d44-dfaa-11eb-8457-525400238cef
    health: HEALTH_WARN
            OSD count 0 < osd_pool_default_size 3

  services:
    mon: 1 daemons, quorum node01 (age 6m)
    mgr: node01.baopmi(active, since 5m)
    osd: 0 osds: 0 up, 0 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:
# 各役割サービスごとにコンテナーが起動

[root@node01 ~]#
podman ps 

CONTAINER ID  IMAGE                                                                                        COMMAND               CREATED        STATUS            PORTS   NAMES
d04ad8657c23  docker.io/ceph/ceph:v16                                                                      -n mon.node01 -f ...  7 minutes ago  Up 7 minutes ago          ceph-54952d44-dfaa-11eb-8457-525400238cef-mon.node01
f48c67036745  docker.io/ceph/ceph:v16                                                                      -n mgr.node01.bao...  7 minutes ago  Up 7 minutes ago          ceph-54952d44-dfaa-11eb-8457-525400238cef-mgr.node01.baopmi
f56a54640c93  docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949  -n client.crash.n...  5 minutes ago  Up 5 minutes ago          ceph-54952d44-dfaa-11eb-8457-525400238cef-crash.node01
062660aa0cce  docker.io/prom/node-exporter:v0.18.1                                                         --no-collector.ti...  5 minutes ago  Up 5 minutes ago          ceph-54952d44-dfaa-11eb-8457-525400238cef-node-exporter.node01
b65e5d534027  docker.io/prom/prometheus:v2.18.1                                                            --config.file=/et...  4 minutes ago  Up 4 minutes ago          ceph-54952d44-dfaa-11eb-8457-525400238cef-prometheus.node01
550e2259e584  docker.io/prom/alertmanager:v0.20.0                                                          --web.listen-addr...  4 minutes ago  Up 4 minutes ago          ceph-54952d44-dfaa-11eb-8457-525400238cef-alertmanager.node01
72aac10547c3  docker.io/ceph/ceph-grafana:6.7.4                                                            /bin/bash             4 minutes ago  Up 4 minutes ago          ceph-54952d44-dfaa-11eb-8457-525400238cef-grafana.node01
# 各コンテナー起動用の systemd サービスが起動

[root@node01 ~]#
systemctl status ceph-* --no-pager 

*  ceph-54952d44-dfaa-11eb-8457-525400238cef@alertmanager.node01.service - Ceph alertmanager.node01 for 54952d44-dfaa-11eb-8457-525400238cef
   Loaded: loaded (/etc/systemd/system/ceph-54952d44-dfaa-11eb-8457-525400238cef@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-08 14:10:00 JST; 4min 54s ago
 Main PID: 16560 (conmon)
    Tasks: 12 (limit: 23674)
   Memory: 17.9M
   CGroup: /system.slice/system-ceph\x2d54952d44\x2ddfaa\x2d11eb\x2d8457\x2d525400238cef.slice/ceph-54952d44-dfaa-11eb-8457-525400238cef@alertmanager.node01.service
.....
.....

*  ceph-54952d44-dfaa-11eb-8457-525400238cef@prometheus.node01.service - Ceph prometheus.node01 for 54952d44-dfaa-11eb-8457-525400238cef
   Loaded: loaded (/etc/systemd/system/ceph-54952d44-dfaa-11eb-8457-525400238cef@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-08 14:09:43 JST; 5min ago
 Main PID: 15087 (conmon)
    Tasks: 10 (limit: 23674)
   Memory: 28.4M
   CGroup: /system.slice/system-ceph\x2d54952d44\x2ddfaa\x2d11eb\x2d8457\x2d525400238cef.slice/ceph-54952d44-dfaa-11eb-8457-525400238cef@prometheus.node01.service
.....
.....

*  ceph-54952d44-dfaa-11eb-8457-525400238cef@crash.node01.service - Ceph crash.node01 for 54952d44-dfaa-11eb-8457-525400238cef
   Loaded: loaded (/etc/systemd/system/ceph-54952d44-dfaa-11eb-8457-525400238cef@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-08 14:08:46 JST; 6min ago
 Main PID: 13890 (conmon)
    Tasks: 4 (limit: 23674)
   Memory: 7.7M
   CGroup: /system.slice/system-ceph\x2d54952d44\x2ddfaa\x2d11eb\x2d8457\x2d525400238cef.slice/ceph-54952d44-dfaa-11eb-8457-525400238cef@crash.node01.service
.....
.....

*  ceph-54952d44-dfaa-11eb-8457-525400238cef@mon.node01.service - Ceph mon.node01 for 54952d44-dfaa-11eb-8457-525400238cef
   Loaded: loaded (/etc/systemd/system/ceph-54952d44-dfaa-11eb-8457-525400238cef@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-08 14:07:19 JST; 7min ago
 Main PID: 8248 (conmon)
    Tasks: 2 (limit: 23674)
   Memory: 1.2M
   CGroup: /system.slice/system-ceph\x2d54952d44\x2ddfaa\x2d11eb\x2d8457\x2d525400238cef.slice/ceph-54952d44-dfaa-11eb-8457-525400238cef@mon.node01.service
.....
.....

*  ceph-54952d44-dfaa-11eb-8457-525400238cef@node-exporter.node01.service - Ceph node-exporter.node01 for 54952d44-dfaa-11eb-8457-525400238cef
   Loaded: loaded (/etc/systemd/system/ceph-54952d44-dfaa-11eb-8457-525400238cef@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-08 14:09:25 JST; 5min ago
 Main PID: 14626 (conmon)
    Tasks: 7 (limit: 23674)
   Memory: 34.5M
   CGroup: /system.slice/system-ceph\x2d54952d44\x2ddfaa\x2d11eb\x2d8457\x2d525400238cef.slice/ceph-54952d44-dfaa-11eb-8457-525400238cef@node-exporter.node01.service
.....
.....

*  ceph-54952d44-dfaa-11eb-8457-525400238cef@grafana.node01.service - Ceph grafana.node01 for 54952d44-dfaa-11eb-8457-525400238cef
   Loaded: loaded (/etc/systemd/system/ceph-54952d44-dfaa-11eb-8457-525400238cef@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-08 14:10:05 JST; 4min 50s ago
 Main PID: 16934 (conmon)
    Tasks: 13 (limit: 23674)
   Memory: 32.2M
   CGroup: /system.slice/system-ceph\x2d54952d44\x2ddfaa\x2d11eb\x2d8457\x2d525400238cef.slice/ceph-54952d44-dfaa-11eb-8457-525400238cef@grafana.node01.service
.....
.....

+ ceph-54952d44-dfaa-11eb-8457-525400238cef.target - Ceph cluster 54952d44-dfaa-11eb-8457-525400238cef
   Loaded: loaded (/etc/systemd/system/ceph-54952d44-dfaa-11eb-8457-525400238cef.target; enabled; vendor preset: disabled)
   Active: active since Thu 2021-07-08 14:07:12 JST; 7min ago

Jul 08 14:07:12 node01 systemd[1]: Reached target Ceph cluster 54952d44-dfa…cef.

*  ceph-54952d44-dfaa-11eb-8457-525400238cef@mgr.node01.baopmi.service - Ceph mgr.node01.baopmi for 54952d44-dfaa-11eb-8457-525400238cef
   Loaded: loaded (/etc/systemd/system/ceph-54952d44-dfaa-11eb-8457-525400238cef@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-08 14:07:21 JST; 7min ago
 Main PID: 8528 (conmon)
    Tasks: 2 (limit: 23674)
   Memory: 1.2M
   CGroup: /system.slice/system-ceph\x2d54952d44\x2ddfaa\x2d11eb\x2d8457\x2d525400238cef.slice/ceph-54952d44-dfaa-11eb-8457-525400238cef@mgr.node01.baopmi.service
.....
.....
関連コンテンツ