Ubuntu 22.04
Sponsored Link

OpenStack Zed : Magnum 利用方法2022/10/11

 
OpenStack Container Infrastructure Management Service(Magnum)の利用方法です。
当例では以下のような環境を例に Magnum をインストールしています。
------------+-----------------------------+-----------------------------+------------
            |                             |                             |
        eth0|10.0.0.30                eth0|10.0.0.50                eth0|10.0.0.51
+-----------+-----------+     +-----------+-----------+     +-----------+-----------+
|   [ dlp.srv.world ]   |     | [ network.srv.world ] |     |  [ node01.srv.world ] |
|     (Control Node)    |     |     (Network Node)    |     |     (Compute Node)    |
|                       |     |                       |     |                       |
|  MariaDB    RabbitMQ  |     |      Open vSwitch     |     |        Libvirt        |
|  Memcached  Nginx     |     |     Neutron Server    |     |      Nova Compute     |
|  Keystone   httpd     |     |      OVN-Northd       |     |      Open vSwitch     |
|  Glance     Nova API  |     |  Nginx  iSCSI Target  |     |   OVN Metadata Agent  |
|  Cinder API           |     |     Cinder Volume     |     |     OVN-Controller    |
|  Barbican API         |     |    Heat API/Engine    |     |                       |
|                       |     |    Magnum Services    |     |                       |
+-----------------------+     +-----------------------+     +-----------------------+

[1] Control ノードで、コンテナー用の VM イメージ (Fedora CoreOS) をダウンロードして Glance に登録しておきます。
root@dlp ~(keystone)#
wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20220424.3.0/x86_64/fedora-coreos-35.20220424.3.0-openstack.x86_64.qcow2.xz

root@dlp ~(keystone)#
xz -dv fedora-coreos-35.20220424.3.0-openstack.x86_64.qcow2.xz

root@dlp ~(keystone)#
openstack image create Fedora-CoreOS --file=fedora-coreos-35.20220424.3.0-openstack.x86_64.qcow2 --disk-format=qcow2 --container-format=bare --property os_distro='fedora-coreos' --public

[2] Magnum の利用方法です。例として、[admin] ユーザーで Kubernetes クラスターを作成します。
root@dlp ~(keystone)#
openstack flavor list

+----+----------+------+------+-----------+-------+-----------+
| ID | Name     |  RAM | Disk | Ephemeral | VCPUs | Is Public |
+----+----------+------+------+-----------+-------+-----------+
| 0  | m1.small | 2048 |   10 |         0 |     1 | True      |
| 1  | m1.large | 8192 |   20 |         0 |     4 | True      |
+----+----------+------+------+-----------+-------+-----------+

root@dlp ~(keystone)#
openstack keypair list

+--------+-------------------------------------------------+------+
| Name   | Fingerprint                                     | Type |
+--------+-------------------------------------------------+------+
| my-key | 82:ec:cc:1a:8d:07:73:17:4c:82:3a:40:cd:dd:2c:a7 | ssh  |
+--------+-------------------------------------------------+------+

root@dlp ~(keystone)#
openstack network list

+--------------------------------------+---------+--------------------------------------+
| ID                                   | Name    | Subnets                              |
+--------------------------------------+---------+--------------------------------------+
| 394ec727-7e9a-473e-8068-45aa841a9ac2 | public  | ea6cb6cb-74ed-4007-bd25-b3fb84111e84 |
| 7336a271-bec9-4d33-8dc5-e9f3a9892ed7 | private | a454cf3e-fef5-46f6-8468-4a26cbd14983 |
+--------------------------------------+---------+--------------------------------------+

root@dlp ~(keystone)#
openstack subnet list

+--------------------------------------+----------------+--------------------------------------+------------------+
| ID                                   | Name           | Network                              | Subnet           |
+--------------------------------------+----------------+--------------------------------------+------------------+
| a454cf3e-fef5-46f6-8468-4a26cbd14983 | private-subnet | 7336a271-bec9-4d33-8dc5-e9f3a9892ed7 | 192.168.100.0/24 |
| ea6cb6cb-74ed-4007-bd25-b3fb84111e84 | public-subnet  | 394ec727-7e9a-473e-8068-45aa841a9ac2 | 10.0.0.0/24      |
+--------------------------------------+----------------+--------------------------------------+------------------+

# Kubernetes クラスター テンプレート作成
root@dlp ~(keystone)# openstack coe cluster template create k8s-cluster-template \
--image Fedora-CoreOS \
--external-network public \
--fixed-network private \
--fixed-subnet private-subnet \
--dns-nameserver 10.0.0.10 \
--network-driver calico \
--docker-storage-driver overlay2 \
--docker-volume-size 5 \
--master-flavor m1.large \
--flavor m1.large \
--coe kubernetes 
Request to create cluster template k8s-cluster-template accepted
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| insecure_registry     | -                                    |
| labels                | {}                                   |
| updated_at            | -                                    |
| floating_ip_enabled   | True                                 |
| fixed_subnet          | private-subnet                       |
| master_flavor_id      | m1.large                             |
| uuid                  | 5bc379ef-e462-46d1-bc58-ab125754a6d6 |
| no_proxy              | -                                    |
| https_proxy           | -                                    |
| tls_disabled          | False                                |
| keypair_id            | -                                    |
| public                | False                                |
| http_proxy            | -                                    |
| docker_volume_size    | 5                                    |
| server_type           | vm                                   |
| external_network_id   | public                               |
| cluster_distro        | fedora-coreos                        |
| image_id              | Fedora-CoreOS                        |
| volume_driver         | -                                    |
| registry_enabled      | False                                |
| docker_storage_driver | overlay2                             |
| apiserver_port        | -                                    |
| name                  | k8s-cluster-template                 |
| created_at            | 2022-10-11T09:33:19+00:00            |
| network_driver        | calico                               |
| fixed_network         | private                              |
| coe                   | kubernetes                           |
| flavor_id             | m1.large                             |
| master_lb_enabled     | False                                |
| dns_nameserver        | 10.0.0.10                            |
| hidden                | False                                |
| tags                  | -                                    |
+-----------------------+--------------------------------------+

# 2 ノード構成で Kubernetes クラスター作成
root@dlp ~(keystone)# openstack coe cluster create k8s-cluster \
--cluster-template k8s-cluster-template \
--master-count 1 \
--node-count 1 \
--keypair my-key 
Request to create cluster 9a548a60-0645-4226-abb9-e0bfc2e5a635 accepted

# ステータス確認
# [CREATE_IN_PROGRESS] 中は作成が進んでいる状態

root@dlp ~(keystone)#
openstack coe cluster list

+--------------------------------------+-------------+---------+------------+--------------+--------------------+---------------+
| uuid                                 | name        | keypair | node_count | master_count | status             | health_status |
+--------------------------------------+-------------+---------+------------+--------------+--------------------+---------------+
| 6de09900-2130-41fc-80f2-6b52b9ec6e5b | k8s-cluster | my-key  |          1 |            1 | CREATE_IN_PROGRESS | None          |
+--------------------------------------+-------------+---------+------------+--------------+--------------------+---------------+

# Heat のオーケストレーションシステムを利用して作成される

root@dlp ~(keystone)#
openstack stack list

+--------------------------------------+--------------------------+----------------------------------+--------------------+----------------------+--------------+
| ID                                   | Stack Name               | Project                          | Stack Status       | Creation Time        | Updated Time |
+--------------------------------------+--------------------------+----------------------------------+--------------------+----------------------+--------------+
| 50f8e343-98f6-4453-a538-aa11061bc10c | k8s-cluster-kwixo7jj667g | 231a529f51394426b07c56f2c5ec580e | CREATE_IN_PROGRESS | 2022-10-11T09:33:44Z | None         |
+--------------------------------------+--------------------------+----------------------------------+------------   -----+----------------------+--------------+

# 各チェックポイントの進行状況を確認

root@dlp ~(keystone)#
openstack stack list --nested | grep k8s-cluster

| ccff612b-f6cb-4025-a101-0f2361b92da3 | k8s-cluster-kwixo7jj667g-api_address_floating_switch-phx23dpfl3ss                                 | 231a529f51394426b07c56f2c5ec580e | CREATE_COMPLETE | 2022-10-11T09:36:33Z | None         | 50f8e343-98f6-4453-a538-aa11061bc10c |
| 9e47dd8f-afec-4834-9275-a84425bc8e81 | k8s-cluster-kwixo7jj667g-kube_minions-yrbxy7swqphg-0-rvyhkenhy654                                 | 231a529f51394426b07c56f2c5ec580e | CREATE_COMPLETE | 2022-10-11T09:36:33Z | None         | 3b10e3f8-271b-41fb-a18c-96d076fdd615 |
| 3b10e3f8-271b-41fb-a18c-96d076fdd615 | k8s-cluster-kwixo7jj667g-kube_minions-yrbxy7swqphg                                                | 231a529f51394426b07c56f2c5ec580e | CREATE_COMPLETE | 2022-10-11T09:36:33Z | None         | 50f8e343-98f6-4453-a538-aa11061bc10c |
| d26c6578-95a7-416a-ac09-a2f607ce0800 | k8s-cluster-kwixo7jj667g-api_address_lb_switch-wbillafc5jut                                       | 231a529f51394426b07c56f2c5ec580e | CREATE_COMPLETE | 2022-10-11T09:36:32Z | None         | 50f8e343-98f6-4453-a538-aa11061bc10c |
| 1686ec93-a20c-4ddc-b051-be63f2a0234a | k8s-cluster-kwixo7jj667g-etcd_address_lb_switch-melqp53yixlk                                      | 231a529f51394426b07c56f2c5ec580e | CREATE_COMPLETE | 2022-10-11T09:36:32Z | None         | 50f8e343-98f6-4453-a538-aa11061bc10c |
| 3e335984-9683-43a6-9991-b3af254c59f8 | k8s-cluster-kwixo7jj667g-kube_masters-bbhprjjb7jh5-0-635ycot4ewb7-api_address_switch-wys26ua5fwvh | 231a529f51394426b07c56f2c5ec580e | CREATE_COMPLETE | 2022-10-11T09:34:07Z | None         | 74c358a6-2250-4368-9c39-f6cb2d366ba5 |
| 74c358a6-2250-4368-9c39-f6cb2d366ba5 | k8s-cluster-kwixo7jj667g-kube_masters-bbhprjjb7jh5-0-635ycot4ewb7                                 | 231a529f51394426b07c56f2c5ec580e | CREATE_COMPLETE | 2022-10-11T09:33:52Z | None         | 50304a43-8dfd-47c3-88fd-43294a86b767 |
| 50304a43-8dfd-47c3-88fd-43294a86b767 | k8s-cluster-kwixo7jj667g-kube_masters-bbhprjjb7jh5                                                | 231a529f51394426b07c56f2c5ec580e | CREATE_COMPLETE | 2022-10-11T09:33:51Z | None         | 50f8e343-98f6-4453-a538-aa11061bc10c |
| f1ff2bf9-9737-4fcc-a2dc-7743426b1c5d | k8s-cluster-kwixo7jj667g-etcd_lb-43t5sohvumdp                                                     | 231a529f51394426b07c56f2c5ec580e | CREATE_COMPLETE | 2022-10-11T09:33:50Z | None         | 50f8e343-98f6-4453-a538-aa11061bc10c |
| 5e611c63-6d74-427b-b960-052d620bc76a | k8s-cluster-kwixo7jj667g-api_lb-aogajfne4pzy                                                      | 231a529f51394426b07c56f2c5ec580e | CREATE_COMPLETE | 2022-10-11T09:33:50Z | None         | 50f8e343-98f6-4453-a538-aa11061bc10c |
| d01d1ab1-f959-4791-a629-6366d53b9214 | k8s-cluster-kwixo7jj667g-network-qv5w4e7xrsr2-network_switch-7znct5ov42ef                         | 231a529f51394426b07c56f2c5ec580e | CREATE_COMPLETE | 2022-10-11T09:33:48Z | None         | 58af7e37-21be-432c-bd6c-002bac956d40 |
| 58af7e37-21be-432c-bd6c-002bac956d40 | k8s-cluster-kwixo7jj667g-network-qv5w4e7xrsr2                                                     | 231a529f51394426b07c56f2c5ec580e | CREATE_COMPLETE | 2022-10-11T09:33:48Z | None         | 50f8e343-98f6-4453-a538-aa11061bc10c |
| 50f8e343-98f6-4453-a538-aa11061bc10c | k8s-cluster-kwixo7jj667g                                                                          | 231a529f51394426b07c56f2c5ec580e | CREATE_COMPLETE | 2022-10-11T09:33:44Z | None         | None                                 |

# 正常に完了した場合は [CREATE_COMPLETE] + [HEALTHY] 状態

root@dlp ~(keystone)#
openstack coe cluster list

+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
| uuid                                 | name        | keypair | node_count | master_count | status          | health_status |
+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
| 6de09900-2130-41fc-80f2-6b52b9ec6e5b | k8s-cluster | my-key  |          1 |            1 | CREATE_COMPLETE | HEALTHY       |
+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+

# インスタンスが作成され稼働中

root@dlp ~(keystone)#
openstack server list

+--------------------------------------+-----------------------------------+--------+-------------------------------------+---------------+----------+
| ID                                   | Name                              | Status | Networks                            | Image         | Flavor   |
+--------------------------------------+-----------------------------------+--------+-------------------------------------+---------------+----------+
| 3ca7c1ad-9f91-42de-aa18-36c059cf72ed | k8s-cluster-kwixo7jj667g-node-0   | ACTIVE | private=10.0.0.233, 192.168.100.47  | Fedora-CoreOS | m1.large |
| e6a6ccc0-c12c-4266-82cf-cc68d2d76967 | k8s-cluster-kwixo7jj667g-master-0 | ACTIVE | private=10.0.0.218, 192.168.100.167 | Fedora-CoreOS | m1.large |
+--------------------------------------+-----------------------------------+--------+-------------------------------------+---------------+----------+
[3] Kubernetes クラスターにアクセスして利用するには以下のように実行します。
# [kubectl] インストール

root@dlp ~(keystone)#
snap install kubectl --classic

kubectl 1.25.2 from Canonical✓ installed
root@dlp ~(keystone)#
openstack coe cluster list

+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
| uuid                                 | name        | keypair | node_count | master_count | status          | health_status |
+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
| 6de09900-2130-41fc-80f2-6b52b9ec6e5b | k8s-cluster | my-key  |          1 |            1 | CREATE_COMPLETE | HEALTHY       |
+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+

root@dlp ~(keystone)#
openstack coe cluster config k8s-cluster

export KUBECONFIG=/root/config
root@dlp ~(keystone)#
export KUBECONFIG=/root/config
root@dlp ~(keystone)#
kubectl get nodes

NAME                                STATUS   ROLES    AGE    VERSION
k8s-cluster-kwixo7jj667g-master-0   Ready    master   11m    v1.23.3
k8s-cluster-kwixo7jj667g-node-0     Ready    <none>   9m5s   v1.23.3

root@dlp ~(keystone)#
kubectl get pods -n kube-system

NAME                                         READY   STATUS    RESTARTS   AGE
calico-kube-controllers-54799d958b-dfd8z     1/1     Running   0          11m
calico-node-2ddgr                            1/1     Running   0          11m
calico-node-vnpr5                            1/1     Running   0          9m58s
coredns-56448757b9-9fgpj                     1/1     Running   0          11m
coredns-56448757b9-brfk4                     1/1     Running   0          11m
dashboard-metrics-scraper-67f57ff746-r47l9   1/1     Running   0          11m
k8s-keystone-auth-jhhjh                      1/1     Running   0          11m
kube-dns-autoscaler-6d5b5dc777-dpgfq         1/1     Running   0          11m
kubernetes-dashboard-7b88d986b4-8vkkf        1/1     Running   0          11m
magnum-metrics-server-6c4c77844b-965fn       1/1     Running   0          11m
npd-7b7pt                                    1/1     Running   0          9m37s

# pods を作成して動作確認

root@dlp ~(keystone)#
kubectl create deployment test-nginx --image=nginx --replicas=2

deployment.apps/test-nginx created
root@dlp ~(keystone)#
kubectl get pods -o wide

NAME                         READY   STATUS    RESTARTS   AGE   IP              NODE                              NOMINATED NODE   READINESS GATES
test-nginx-976fbbd77-f2nsn   1/1     Running   0          29s   10.100.154.68   k8s-cluster-kwixo7jj667g-node-0   <none>           <none>
test-nginx-976fbbd77-x4m45   1/1     Running   0          29s   10.100.154.67   k8s-cluster-kwixo7jj667g-node-0   <none>           <none>

root@dlp ~(keystone)#
kubectl expose deployment test-nginx --type="NodePort" --port 80

service/test-nginx exposed
root@dlp ~(keystone)#
kubectl get services test-nginx

NAME         TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
test-nginx   NodePort   10.254.113.41   <none>        80:32743/TCP   5s

root@dlp ~(keystone)#
kubectl port-forward service/test-nginx --address 0.0.0.0 10443:80 &

Forwarding from 0.0.0.0:10443 -> 80
root@dlp ~(keystone)#
curl localhost:10443

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
.....
.....
[4] 一般ユーザーでも Magnum を利用したい場合は、設定変更が必要です。
root@dlp ~(keystone)#
openstack role list

+----------------------------------+------------------+
| ID                               | Name             |
+----------------------------------+------------------+
| 1df48dad71a745849c2847a3b5ec1785 | reader           |
| 384ed6af259c4fa88083f811393f0567 | heat_stack_user  |
| 452c8bdf83534aba852b732c7a149096 | member           |
| 4dc68cf727eb44f0b69e905ff64f3a29 | heat_stack_owner |
| 5b696b9d402f4545b4592aa3f7733c68 | CloudUser        |
| 5c0c4dc038b5441f9b64345ea113f3cb | admin            |
+----------------------------------+------------------+

root@dlp ~(keystone)#
openstack project list

+----------------------------------+-----------+
| ID                               | Name      |
+----------------------------------+-----------+
| 231a529f51394426b07c56f2c5ec580e | admin     |
| 4dd1e1f6bac441ff9c77002c3ab4c58a | hiroshima |
| ee51c09c765e4a4d958c2bcc4dbb5665 | service   |
+----------------------------------+-----------+

root@dlp ~(keystone)#
openstack user list

+----------------------------------+---------------------+
| ID                               | Name                |
+----------------------------------+---------------------+
| 7c9dbb0cdc284113964d618b24cc722e | admin               |
| 0aef0c555a4f4cc885c80dd0eb4f7c72 | glance              |
| 4d912be26a4149d590dbfb7bc8b015fa | nova                |
| b3ad6ca00649400a8eae75d64a1c2181 | placement           |
| bcf45964c78c40fba4c125e31dee3721 | neutron             |
| 9345b7f4de21427abf31ad308cf952f8 | serverworld         |
| 3042ebda5502448097dd0bbbe3058812 | cinder              |
| 1bfd1646d0e04371a3c6199bc98d152f | heat                |
| b477de75cfbe4a5e863bc4d7ea1d2d5f | heat_domain_admin   |
| 55ad9ce9899d415ea9fba4d31a12597d | barbican            |
| aa26495baad24a9fac173217ab2cfc16 | magnum              |
| 8726616d293e45b992e976513b25fadd | magnum_domain_admin |
+----------------------------------+---------------------+

# 例として [hiroshima] プロジェクトの [serverworld] ユーザーを [heat_stack_owner] ロールに追加

root@dlp ~(keystone)#
openstack role add --project hiroshima --user serverworld heat_stack_owner
# Neutron サーバー稼働ノードで設定変更

root@network:~#
vi /etc/neutron/policy.json
# 以下の内容で新規作成
# 管理者のみに許可されている権限を設定なしで上書きする

{
  "create_port:fixed_ips:subnet_id": "",
  "create_port:allowed_address_pairs": "",
  "create_port:allowed_address_pairs:ip_address": "",
}

root@network:~#
systemctl restart neutron-server

# 以上で、一般ユーザーでも以下のようにクラスターが作成できる

ubuntu@dlp ~(keystone)$
openstack coe cluster list

+--------------------------------------+---------------+---------+------------+--------------+-----------------+---------------+
| uuid                                 | name          | keypair | node_count | master_count | status          | health_status |
+--------------------------------------+---------------+---------+------------+--------------+-----------------+---------------+
| d5b982a3-fa21-4760-803b-a93ccc5096c8 | k8s-cluster-1 | mykey   |          1 |            1 | CREATE_COMPLETE | None          |
+--------------------------------------+---------------+---------+------------+--------------+-----------------+---------------+
関連コンテンツ