Debian 12 bookworm
Sponsored Link

OpenStack Zed : Magnum 利用方法2023/06/30

 
OpenStack Container Infrastructure Management Service(Magnum)の利用方法です。
当例では以下のような環境を例に Magnum をインストールしています。
------------+--------------------------+--------------------------+------------
            |                          |                          |
        eth0|10.0.0.30             eth0|10.0.0.50             eth0|10.0.0.51
+-----------+-----------+  +-----------+-----------+  +-----------+-----------+
|   [ dlp.srv.world ]   |  | [ network.srv.world ] |  |  [ node01.srv.world ] |
|     (Control Node)    |  |     (Network Node)    |  |     (Compute Node)    |
|                       |  |                       |  |                       |
|  MariaDB    RabbitMQ  |  |  Neutron L2/L3 Agent  |  |        Libvirt        |
|  Memcached  Nginx     |  |   Neutron Metadata    |  |      Nova Compute     |
|  Keystone   httpd     |  |     Open vSwitch      |  |    Neutron L2 Agent   |
|  Glance     Nova API  |  |     iSCSI Target      |  |      Open vSwitch     |
|  Neutron Server       |  |     Cinder Volume     |  |                       |
|  Neutron Metadata     |  |    Heat API/Engine    |  |                       |
|  Cinder API           |  |         Nginx         |  |                       |
|  Barbican API         |  |    Magnum Services    |  |                       |
+-----------------------+  +-----------------------+  +-----------------------+

[1] Control ノードで、コンテナー用の VM イメージ (Fedora CoreOS) をダウンロードして Glance に登録しておきます。
root@dlp ~(keystone)#
wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/37.20230303.3.0/x86_64/fedora-coreos-37.20230303.3.0-openstack.x86_64.qcow2.xz

root@dlp ~(keystone)#
xz -dv fedora-coreos-37.20230303.3.0-openstack.x86_64.qcow2.xz

root@dlp ~(keystone)#
openstack image create Fedora-CoreOS --file=fedora-coreos-37.20230303.3.0-openstack.x86_64.qcow2 --disk-format=qcow2 --container-format=bare --property os_distro='fedora-coreos' --public

[2] Magnum の利用方法です。例として、[admin] ユーザーで Kubernetes クラスターを作成します。
root@dlp ~(keystone)#
openstack flavor list

+----+-----------+------+------+-----------+-------+-----------+
| ID | Name      |  RAM | Disk | Ephemeral | VCPUs | Is Public |
+----+-----------+------+------+-----------+-------+-----------+
| 1  | m1.small  | 2048 |   10 |         0 |     1 | True      |
| 2  | m1.medium | 4096 |   10 |         0 |     2 | True      |
| 3  | m1.large  | 8192 |   10 |         0 |     4 | True      |
| 4  | m2.large  | 8192 |   10 |        10 |     4 | True      |
| 5  | m3.large  | 8192 |   20 |         0 |     4 | True      |
+----+-----------+------+------+-----------+-------+-----------+

root@dlp ~(keystone)#
openstack keypair list

+--------+-------------------------------------------------+------+
| Name   | Fingerprint                                     | Type |
+--------+-------------------------------------------------+------+
| my-key | 24:33:d1:3d:ad:3c:13:92:7b:78:04:77:4e:a0:f6:70 | ssh  |
+--------+-------------------------------------------------+------+

root@dlp ~(keystone)#
openstack network list

+--------------------------------------+---------+--------------------------------------+
| ID                                   | Name    | Subnets                              |
+--------------------------------------+---------+--------------------------------------+
| ae5fdb1f-efb9-412b-9053-b81106c90336 | private | b3234f3f-b3e3-41d2-b3a4-ca3d4ec55d24 |
| b9b8536b-0696-4a5b-af05-602587435c0a | public  | 0129c597-e411-450e-a4f9-b387c68eddb3 |
+--------------------------------------+---------+--------------------------------------+

root@dlp ~(keystone)#
openstack subnet list

+--------------------------------------+----------------+--------------------------------------+------------------+
| ID                                   | Name           | Network                              | Subnet           |
+--------------------------------------+----------------+--------------------------------------+------------------+
| 0129c597-e411-450e-a4f9-b387c68eddb3 | public-subnet  | b9b8536b-0696-4a5b-af05-602587435c0a | 10.0.0.0/24      |
| b3234f3f-b3e3-41d2-b3a4-ca3d4ec55d24 | private-subnet | ae5fdb1f-efb9-412b-9053-b81106c90336 | 192.168.100.0/24 |
+--------------------------------------+----------------+--------------------------------------+------------------+

# Kubernetes クラスター テンプレート作成
root@dlp ~(keystone)# openstack coe cluster template create k8s-cluster-template \
--image Fedora-CoreOS \
--external-network public \
--fixed-network private \
--fixed-subnet private-subnet \
--dns-nameserver 10.0.0.10 \
--network-driver calico \
--docker-storage-driver overlay2 \
--docker-volume-size 10 \
--master-flavor m3.large \
--flavor m3.large \
--coe kubernetes 
Request to create cluster template k8s-cluster-template accepted
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| insecure_registry     | -                                    |
| labels                | {}                                   |
| updated_at            | -                                    |
| floating_ip_enabled   | True                                 |
| fixed_subnet          | private-subnet                       |
| master_flavor_id      | m3.large                             |
| uuid                  | 19959f08-5962-4359-afeb-2267ec67804b |
| no_proxy              | -                                    |
| https_proxy           | -                                    |
| tls_disabled          | False                                |
| keypair_id            | -                                    |
| public                | False                                |
| http_proxy            | -                                    |
| docker_volume_size    | 10                                   |
| server_type           | vm                                   |
| external_network_id   | public                               |
| cluster_distro        | fedora-coreos                        |
| image_id              | Fedora-CoreOS                        |
| volume_driver         | -                                    |
| registry_enabled      | False                                |
| docker_storage_driver | overlay2                             |
| apiserver_port        | -                                    |
| name                  | k8s-cluster-template                 |
| created_at            | 2023-06-30T06:56:07.534093+00:00     |
| network_driver        | calico                               |
| fixed_network         | private                              |
| coe                   | kubernetes                           |
| flavor_id             | m3.large                             |
| master_lb_enabled     | False                                |
| dns_nameserver        | 10.0.0.10                            |
| hidden                | False                                |
| tags                  | -                                    |
+-----------------------+--------------------------------------+

# 2 ノード構成で Kubernetes クラスター作成
root@dlp ~(keystone)# openstack coe cluster create k8s-cluster \
--cluster-template k8s-cluster-template \
--master-count 1 \
--node-count 1 \
--keypair my-key 
Request to create cluster 7790f694-91b4-4f3b-bd7c-98276b75330f accepted

# ステータス確認
# [CREATE_IN_PROGRESS] 中は作成が進んでいる状態

root@dlp ~(keystone)#
openstack coe cluster list

+--------------------------------------+-------------+---------+------------+--------------+--------------------+---------------+
| uuid                                 | name        | keypair | node_count | master_count | status             | health_status |
+--------------------------------------+-------------+---------+------------+--------------+--------------------+---------------+
| 7790f694-91b4-4f3b-bd7c-98276b75330f | k8s-cluster | my-key  |          1 |            1 | CREATE_IN_PROGRESS | None          |
+--------------------------------------+-------------+---------+------------+--------------+--------------------+---------------+

# Heat のオーケストレーションシステムを利用して作成される

root@dlp ~(keystone)#
openstack stack list

+--------------------------------------+--------------------------+----------------------------------+--------------------+----------------------+--------------+
| ID                                   | Stack Name               | Project                          | Stack Status       | Creation Time        | Updated Time |
+--------------------------------------+--------------------------+----------------------------------+--------------------+----------------------+--------------+
| 44ff9533-48c9-4b9c-a945-5cc4d7d17ada | k8s-cluster-5d747yxstr3q | 757625ae78404e38a8cfdd7c6d262860 | CREATE_IN_PROGRESS | 2023-06-30T06:56:59Z | None         |
+--------------------------------------+--------------------------+----------------------------------+--------------------+----------------------+--------------+

# 各チェックポイントの進行状況を確認

root@dlp ~(keystone)#
openstack stack list --nested | grep k8s-cluster

| 22a31b8c-04a3-480f-b43e-be52bd8f8859 | k8s-cluster-5d747yxstr3q-kube_minions-7k7yznhzwfkb-0-b4ollnbov4r4                                 | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T07:00:07Z | None         | 321cf254-97c5-4d0b-866c-04bd529e06ea |
| 9053ee77-ac52-4d56-9aaa-5f4c4bc9acc5 | k8s-cluster-5d747yxstr3q-api_address_floating_switch-oviashfdkuvx                                 | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T07:00:06Z | None         | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada |
| 321cf254-97c5-4d0b-866c-04bd529e06ea | k8s-cluster-5d747yxstr3q-kube_minions-7k7yznhzwfkb                                                | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T07:00:06Z | None         | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada |
| 7c42be78-59c6-4fcb-82fa-bd11f8bf868d | k8s-cluster-5d747yxstr3q-api_address_lb_switch-k77qarn7dobf                                       | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T07:00:05Z | None         | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada |
| b98a5ce4-eedd-4e55-b57c-79995681bc05 | k8s-cluster-5d747yxstr3q-etcd_address_lb_switch-wbd5t6rgun3a                                      | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T07:00:04Z | None         | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada |
| ec132afc-e8c0-4647-9fb1-bc0b85aace77 | k8s-cluster-5d747yxstr3q-kube_masters-hrd7vlt6xvos-0-dewo6txtuo54-api_address_switch-oi4fxlbsewmv | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:57:29Z | None         | 88244e99-1df8-4a99-b5fe-4ddca9f70f66 |
| 88244e99-1df8-4a99-b5fe-4ddca9f70f66 | k8s-cluster-5d747yxstr3q-kube_masters-hrd7vlt6xvos-0-dewo6txtuo54                                 | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:57:07Z | None         | e7074d57-631e-43e4-aa39-95d4d8c2ea7c |
| e7074d57-631e-43e4-aa39-95d4d8c2ea7c | k8s-cluster-5d747yxstr3q-kube_masters-hrd7vlt6xvos                                                | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:57:06Z | None         | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada |
| f85de343-388a-48ee-877f-b3bcc19f0a44 | k8s-cluster-5d747yxstr3q-etcd_lb-4ja6buynrvyp                                                     | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:57:05Z | None         | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada |
| 61e764ad-3e2f-4adf-a9f5-6016aa9af351 | k8s-cluster-5d747yxstr3q-api_lb-lsk5qra5uxce                                                      | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:57:05Z | None         | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada |
| 965f9ea8-5830-45b4-862e-5b45300c0415 | k8s-cluster-5d747yxstr3q-network-xbrtftx2g7dd-network_switch-2colkowcycmx                         | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:57:03Z | None         | 3c6ea82c-c54c-4ac6-8d32-57700ab88585 |
| 3c6ea82c-c54c-4ac6-8d32-57700ab88585 | k8s-cluster-5d747yxstr3q-network-xbrtftx2g7dd                                                     | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:57:03Z | None         | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada |
| 44ff9533-48c9-4b9c-a945-5cc4d7d17ada | k8s-cluster-5d747yxstr3q                                                                          | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:56:59Z | None         | None                                 |

# 正常に完了した場合は [CREATE_COMPLETE] + [HEALTHY] 状態

root@dlp ~(keystone)#
openstack coe cluster list

+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
| uuid                                 | name        | keypair | node_count | master_count | status          | health_status |
+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
| 7790f694-91b4-4f3b-bd7c-98276b75330f | k8s-cluster | my-key  |          1 |            1 | CREATE_COMPLETE | HEALTHY       |
+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+

# インスタンスが作成され稼働中

root@dlp ~(keystone)#
openstack server list

+--------------------------------------+-----------------------------------+--------+-------------------------------------+---------------+----------+
| ID                                   | Name                              | Status | Networks                            | Image         | Flavor   |
+--------------------------------------+-----------------------------------+--------+-------------------------------------+---------------+----------+
| 4ad48718-5fd7-462d-913e-d2384d5eb010 | k8s-cluster-5d747yxstr3q-node-0   | ACTIVE | private=192.168.100.115             | Fedora-CoreOS | m3.large |
| 1e21221d-fda8-41d1-8e50-c8ebd7cd26a6 | k8s-cluster-5d747yxstr3q-master-0 | ACTIVE | private=10.0.0.205, 192.168.100.192 | Fedora-CoreOS | m3.large |
+--------------------------------------+-----------------------------------+--------+-------------------------------------+---------------+----------+
[3] Kubernetes クラスターにアクセスして利用するには以下のように実行します。
# [kubectl] インストール

root@dlp ~(keystone)#
apt -y install kubernetes-client
root@dlp ~(keystone)#
openstack coe cluster list

+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
| uuid                                 | name        | keypair | node_count | master_count | status          | health_status |
+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
| 7790f694-91b4-4f3b-bd7c-98276b75330f | k8s-cluster | my-key  |          1 |            1 | CREATE_COMPLETE | HEALTHY       |
+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+

root@dlp ~(keystone)#
openstack coe cluster config k8s-cluster

export KUBECONFIG=/root/config
root@dlp ~(keystone)#
export KUBECONFIG=/root/config
root@dlp ~(keystone)#
kubectl get nodes

NAME                                STATUS   ROLES    AGE    VERSION
k8s-cluster-5d747yxstr3q-master-0   Ready    master   7m8s   v1.23.3
k8s-cluster-5d747yxstr3q-node-0     Ready    <none>   5m8s   v1.23.3

# pods を作成して動作確認

root@dlp ~(keystone)#
kubectl create deployment test-nginx --image=nginx --replicas=2

deployment.apps/test-nginx created
root@dlp ~(keystone)#
kubectl get pods -o wide

NAME                         READY   STATUS    RESTARTS   AGE   IP               NODE                              NOMINATED NODE   READINESS GATES
test-nginx-976fbbd77-wbp5j   1/1     Running   0          12s   10.100.176.196   k8s-cluster-5d747yxstr3q-node-0   <none>           <none>
test-nginx-976fbbd77-x8xrg   1/1     Running   0          12s   10.100.176.195   k8s-cluster-5d747yxstr3q-node-0   <none>           <none>

root@dlp ~(keystone)#
kubectl expose deployment test-nginx --type="NodePort" --port 80

service/test-nginx exposed
root@dlp ~(keystone)#
kubectl get services test-nginx

NAME         TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
test-nginx   NodePort   10.254.124.127   <none>        80:31615/TCP   6s

root@dlp ~(keystone)#
kubectl port-forward service/test-nginx --address 0.0.0.0 10443:80 &

Forwarding from 0.0.0.0:10443 -> 80
root@dlp ~(keystone)#
curl localhost:10443

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
.....
.....
[4] 一般ユーザーでも Magnum を利用したい場合は、設定変更が必要です。
root@dlp ~(keystone)#
openstack role list

+----------------------------------+------------------+
| ID                               | Name             |
+----------------------------------+------------------+
| 109c6488ef4347d681ba691f8ac97a6b | member           |
| 6055174790cd4797ae3671b38d935c53 | service          |
| 81f2687109a245b580690b6e8f9095db | heat_stack_owner |
| 82699442db4e4edea2fb7986340b5ab4 | admin            |
| db9927603e564f94b2f0a4312d1a63da | heat_stack_user  |
| eb3d0d5b10cd456093463a2c12e41b82 | reader           |
+----------------------------------+------------------+

root@dlp ~(keystone)#
openstack project list

+----------------------------------+------------------------------------------------------------------+
| ID                               | Name                                                             |
+----------------------------------+------------------------------------------------------------------+
| 1c2f5dc8c4f9494fbaaa4217c8e3585a | hiroshima                                                        |
| 757625ae78404e38a8cfdd7c6d262860 | admin                                                            |
| d8b09d86ed7743039f92b2e542ea26c1 | service                                                          |
| e614281e49f54c51abdc90b92e938846 | 757625ae78404e38a8cfdd7c6d262860-44ff9533-48c9-4b9c-a945-5cc4d7d |
+----------------------------------+------------------------------------------------------------------+

root@dlp ~(keystone)#
openstack user list

+----------------------------------+---------------------+
| ID                               | Name                |
+----------------------------------+---------------------+
| a45313c73c4842d7b5ba90f56be73de2 | admin               |
| b166b462bd214a34b99e079c806bef55 | glance              |
| 9fb0ad5b33f9492594f4139f5b428dab | nova                |
| 5db174d072dd4bcc90d34d7b0c4f0e36 | placement           |
| 1b4f2549a8ee462d909185b6a321e265 | neutron             |
| de51d5f0ee2c485885877d21f5b424e0 | serverworld         |
| 5b1712badfdb4824927488f7882cc7ca | cinder              |
| f485fdd327064f56ac0182c3b084525b | heat                |
| 65993a5e1fe0476486e82091276c2827 | heat_domain_admin   |
| b0c091c98dbe452db61fdb9d8d44d719 | barbican            |
| c5916ff29c094da88560ba100c6f0340 | magnum              |
| 846e638c9f394766bf193e09782e1b4e | magnum_domain_admin |
+----------------------------------+---------------------+

# 例として [hiroshima] プロジェクトの [serverworld] ユーザーを [heat_stack_owner] ロールに追加

root@dlp ~(keystone)#
openstack role add --project hiroshima --user serverworld heat_stack_owner
# Neutron サーバー稼働ノードで設定変更

root@dlp ~(keystone)#
vi /etc/neutron/policy.d/00_default_policy.yaml
# 最終行に追記
# 管理者のみに許可されている権限をユーザーにも付与する

"create_port:fixed_ips:subnet_id": "(rule:context_is_advsvc) or (rule:network_owner) or (rule:admin_only) or (rule:shared) or (role:member and project_id:%(project_id)s)"
"create_port:allowed_address_pairs": "(rule:admin_only) or (rule:network_owner) or (role:member and project_id:%(project_id)s)"
"create_port:allowed_address_pairs:ip_address": "(rule:admin_only) or (rule:network_owner) or (role:member and project_id:%(project_id)s)"

root@dlp ~(keystone)#
systemctl restart neutron-api

# 以上で、一般ユーザーでも以下のようにクラスターが作成できる

debian@dlp ~(keystone)$
openstack coe cluster list

+--------------------------------------+--------------+---------+------------+--------------+-----------------+---------------+
| uuid                                 | name         | keypair | node_count | master_count | status          | health_status |
+--------------------------------------+--------------+---------+------------+--------------+-----------------+---------------+
| 81e42408-4ec6-4d1d-8bec-41017494e7dc | k8s-cluster2 | mykey   |          1 |            1 | CREATE_COMPLETE | None          |
+--------------------------------------+--------------+---------+------------+--------------+-----------------+---------------+
関連コンテンツ