CentOS Stream 8
Sponsored Link

Ceph Quincy : Cephadm #2 Configure Cluster
2022/06/14
 
Configure Ceph Cluster with [Cephadm] that is a Ceph Cluster Deploy tool.
For example on here, Configure Ceph Cluster with 3 Nodes like follows.
Furthermore, each Storage Node has a free block device to use on Ceph Nodes.
(use [/dev/sdb] on this example)
                                         |
            +----------------------------+----------------------------+
            |                            |                            |
            |10.0.0.51                   |10.0.0.52                   |10.0.0.53 
+-----------+-----------+    +-----------+-----------+    +-----------+-----------+
|   [node01.srv.world]  |    |   [node02.srv.world]  |    |   [node03.srv.world]  |
|     Object Storage    +----+     Object Storage    +----+     Object Storage    |
|     Monitor Daemon    |    |                       |    |                       |
|     Manager Daemon    |    |                       |    |                       |
+-----------------------+    +-----------------------+    +-----------------------+

[1]
[2] To add Nodes to Cluster, run like follows.
For example, add [node02], [node03].
# transfer SSH public key

[root@node01 ~]#
ssh-copy-id -f -i /etc/ceph/ceph.pub root@node02


Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@node02'"
and check to make sure that only the key(s) you wanted were added.

[root@node01 ~]#
ssh-copy-id -f -i /etc/ceph/ceph.pub root@node03


Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@node03'"
and check to make sure that only the key(s) you wanted were added.

# add target Nodes to Cluster

[root@node01 ~]#
ceph orch host add node02.srv.world

Inferring fsid 1d53a99e-eba9-11ec-96f5-525400ba92a3
Using recent ceph image quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a
Added host 'node02.srv.world' with addr '10.0.0.52'
[root@node01 ~]#
ceph orch host add node03.srv.world

Inferring fsid 1d53a99e-eba9-11ec-96f5-525400ba92a3
Using recent ceph image quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a
Added host 'node03.srv.world' with addr '10.0.0.53'

[root@node01 ~]#
ceph orch host ls

Inferring fsid 1d53a99e-eba9-11ec-96f5-525400ba92a3
Using recent ceph image quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a
HOST              ADDR       LABELS  STATUS
node01.srv.world  10.0.0.51  _admin
node02.srv.world  10.0.0.52
node03.srv.world  10.0.0.53
3 hosts in cluster
[3] To configure OSD, run like follows.
For example, configure [node01], [node02], [node03].
# list available devices
# possible to set if [AVAIL = True]

[root@node01 ~]#
ceph orch device ls

Inferring fsid 1d53a99e-eba9-11ec-96f5-525400ba92a3
Using recent ceph image quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a
HOST              PATH      TYPE  DEVICE ID   SIZE  AVAILABLE  REJECT REASONS
node01.srv.world  /dev/sdb  hdd               171G  Yes
node02.srv.world  /dev/sdb  hdd               171G  Yes
node02.srv.world  /dev/sdc  hdd               171G  Yes
node03.srv.world  /dev/sdb  hdd               171G  Yes

# configure OSD

[root@node01 ~]#
ceph orch daemon add osd node01.srv.world:/dev/sdb

Inferring fsid 1d53a99e-eba9-11ec-96f5-525400ba92a3
Using recent ceph image quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a
Created osd(s) 0 on host 'node01.srv.world'
[root@node01 ~]#
ceph orch daemon add osd node02.srv.world:/dev/sdb

Inferring fsid 1d53a99e-eba9-11ec-96f5-525400ba92a3
Using recent ceph image quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a
Created osd(s) 1 on host 'node02.srv.world'
[root@node01 ~]#
ceph orch daemon add osd node03.srv.world:/dev/sdb

Inferring fsid 1d53a99e-eba9-11ec-96f5-525400ba92a3
Using recent ceph image quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a
Created osd(s) 2 on host 'node03.srv.world'

# after few minutes, status turns to [HEALTH_OK]

[root@node01 ~]#
ceph -s

Using recent ceph image quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a
  cluster:
    id:     1d53a99e-eba9-11ec-96f5-525400ba92a3
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum node01.srv.world,node02,node03 (age 7m)
    mgr: node01.srv.world.iqrcwq(active, since 18m), standbys: node02.obtbij
    osd: 3 osds: 3 up (since 26s), 3 in (since 88s)

  data:
    pools:   1 pools, 1 pgs
    objects: 2 objects, 449 KiB
    usage:   61 MiB used, 480 GiB / 480 GiB avail
    pgs:     1 active+clean

  io:
    client:   767 B/s rd, 22 KiB/s wr, 0 op/s rd, 1 op/s wr
[4] To remove OSD, run like follows.
[root@node01 ~]#
ceph osd tree

Inferring fsid 1d53a99e-eba9-11ec-96f5-525400ba92a3
Using recent ceph image quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a
ID  CLASS  WEIGHT   TYPE NAME        STATUS  REWEIGHT  PRI-AFF
-1         0.46857  root default
-3         0.15619      host node01
 0    hdd  0.15619          osd.0        up   1.00000  1.00000
-5         0.15619      host node02
 1    hdd  0.15619          osd.1        up   1.00000  1.00000
-7         0.15619      host node03
 2    hdd  0.15619          osd.2        up   1.00000  1.00000

# for example, remove OSD ID [2]

[root@node01 ~]#
ceph orch osd rm 2

Inferring fsid 1d53a99e-eba9-11ec-96f5-525400ba92a3
Using recent ceph image quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a
Scheduled OSD(s) for removal

# show removal status
# completes if no entries are shown
# takes many times

[root@node01 ~]#
ceph orch osd rm status

Inferring fsid 1d53a99e-eba9-11ec-96f5-525400ba92a3
Using recent ceph image quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a
OSD  HOST              STATE     PGS  REPLACE  FORCE  ZAP    DRAIN STARTED AT
2    node03.srv.world  draining    1  False    False  False  2022-06-14 06:36:17.420850
Matched Content