CentOS Stream 9
Sponsored Link

OpenStack Zed : How to use Magnum
2022/11/11
 
Install OpenStack Container Infrastructure Management Service (Magnum).
This example is based on the environment like follows.
------------+-----------------------------+-----------------------------+------------
            |                             |                             |
        eth0|10.0.0.30                eth0|10.0.0.50                eth0|10.0.0.51
+-----------+-----------+     +-----------+-----------+     +-----------+-----------+
|   [ dlp.srv.world ]   |     | [ network.srv.world ] |     |  [ node01.srv.world ] |
|     (Control Node)    |     |     (Network Node)    |     |     (Compute Node)    |
|                       |     |                       |     |                       |
|  MariaDB    RabbitMQ  |     |      Open vSwitch     |     |        Libvirt        |
|  Memcached  Nginx     |     |     Neutron Server    |     |      Nova Compute     |
|  Keystone   httpd     |     |      OVN-Northd       |     |      Open vSwitch     |
|  Glance     Nova API  |     |  Nginx  iSCSI Target  |     |   OVN Metadata Agent  |
|  Cinder API           |     |     Cinder Volume     |     |     OVN-Controller    |
|  Barbican API         |     |    Heat API/Engine    |     |                       |
|                       |     |    Magnum Services    |     |                       |
+-----------------------+     +-----------------------+     +-----------------------+

[1] On Control Node, Download a VM image for containers (Fedora CoreOS) and add it to Glance.
[root@dlp ~(keystone)]#
wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20220424.3.0/x86_64/fedora-coreos-35.20220424.3.0-openstack.x86_64.qcow2.xz

[root@dlp ~(keystone)]#
xz -dv fedora-coreos-35.20220424.3.0-openstack.x86_64.qcow2.xz

[root@dlp ~(keystone)]#
openstack image create Fedora-CoreOS --file=fedora-coreos-35.20220424.3.0-openstack.x86_64.qcow2 --disk-format=qcow2 --container-format=bare --property os_distro='fedora-coreos' --public

[2] How to use Magnum. For example, Create Kubernetes Cluster with [admin] user.
[root@dlp ~(keystone)]#
openstack flavor list

+----+----------+------+------+-----------+-------+-----------+
| ID | Name     |  RAM | Disk | Ephemeral | VCPUs | Is Public |
+----+----------+------+------+-----------+-------+-----------+
| 0  | m1.small | 2048 |   10 |         0 |     1 | True      |
| 1  | m1.large | 8192 |   20 |         0 |     4 | True      |
+----+----------+------+------+-----------+-------+-----------+

[root@dlp ~(keystone)]#
openstack keypair list

+--------+-------------------------------------------------+------+
| Name   | Fingerprint                                     | Type |
+--------+-------------------------------------------------+------+
| my-key | 97:b9:5e:34:55:0b:59:d8:c9:e2:19:52:b3:78:28:db | ssh  |
+--------+-------------------------------------------------+------+

[root@dlp ~(keystone)]#
openstack network list

+--------------------------------------+---------+--------------------------------------+
| ID                                   | Name    | Subnets                              |
+--------------------------------------+---------+--------------------------------------+
| 004ca400-3d79-4c1b-b8fb-bba1e273000e | public  | 80213093-ac95-469a-b4d9-8c6adc93cfe1 |
| c9c79a1d-136e-4036-8fe0-eaac79f05f5d | private | 85e66c6a-66eb-490f-a4fa-e21f91678c89 |
+--------------------------------------+---------+--------------------------------------+

[root@dlp ~(keystone)]#
openstack subnet list

+--------------------------------------+----------------+--------------------------------------+------------------+
| ID                                   | Name           | Network                              | Subnet           |
+--------------------------------------+----------------+--------------------------------------+------------------+
| 80213093-ac95-469a-b4d9-8c6adc93cfe1 | public-subnet  | 004ca400-3d79-4c1b-b8fb-bba1e273000e | 10.0.0.0/24      |
| 85e66c6a-66eb-490f-a4fa-e21f91678c89 | private-subnet | c9c79a1d-136e-4036-8fe0-eaac79f05f5d | 192.168.100.0/24 |
+--------------------------------------+----------------+--------------------------------------+------------------+

# create Kubernetes Cluster template
[root@dlp ~(keystone)]# openstack coe cluster template create k8s-cluster-template \
--image Fedora-CoreOS \
--external-network public \
--fixed-network private \
--fixed-subnet private-subnet \
--dns-nameserver 10.0.0.10 \
--network-driver calico \
--docker-storage-driver overlay2 \
--docker-volume-size 5 \
--master-flavor m1.large \
--flavor m1.large \
--coe kubernetes 
Request to create cluster template k8s-cluster-template accepted
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| insecure_registry     | -                                    |
| labels                | {}                                   |
| updated_at            | -                                    |
| floating_ip_enabled   | True                                 |
| fixed_subnet          | private-subnet                       |
| master_flavor_id      | m1.large                             |
| uuid                  | fe4f0d22-d03a-45c8-8f19-1d57c7345fb6 |
| no_proxy              | -                                    |
| https_proxy           | -                                    |
| tls_disabled          | False                                |
| keypair_id            | -                                    |
| public                | False                                |
| http_proxy            | -                                    |
| docker_volume_size    | 5                                    |
| server_type           | vm                                   |
| external_network_id   | public                               |
| cluster_distro        | fedora-coreos                        |
| image_id              | Fedora-CoreOS                        |
| volume_driver         | -                                    |
| registry_enabled      | False                                |
| docker_storage_driver | overlay2                             |
| apiserver_port        | -                                    |
| name                  | k8s-cluster-template                 |
| created_at            | 2022-11-11T06:27:35+00:00            |
| network_driver        | calico                               |
| fixed_network         | private                              |
| coe                   | kubernetes                           |
| flavor_id             | m1.large                             |
| master_lb_enabled     | False                                |
| dns_nameserver        | 10.0.0.10                            |
| hidden                | False                                |
| tags                  | -                                    |
+-----------------------+--------------------------------------+

# create Kubernetes Cluster with 2 nodes
[root@dlp ~(keystone)]# openstack coe cluster create k8s-cluster \
--cluster-template k8s-cluster-template \
--master-count 1 \
--node-count 1 \
--keypair my-key 
Request to create cluster a93d1aaf-bc17-499d-aac6-43dcc4f43563 accepted

# verify status
# proceed to create cluster during [CREATE_IN_PROGRESS] state

[root@dlp ~(keystone)]#
openstack coe cluster list

+--------------------------------------+-------------+---------+------------+--------------+--------------------+---------------+
| uuid                                 | name        | keypair | node_count | master_count | status             | health_status |
+--------------------------------------+-------------+---------+------------+--------------+--------------------+---------------+
| a93d1aaf-bc17-499d-aac6-43dcc4f43563 | k8s-cluster | my-key  |          1 |            1 | CREATE_IN_PROGRESS | None          |
+--------------------------------------+-------------+---------+------------+--------------+--------------------+---------------+

# Heat orchestration System is used for creating

[root@dlp ~(keystone)]#
openstack stack list

+--------------------------------------+--------------------------+----------------------------------+--------------------+----------------------+--------------+
| ID                                   | Stack Name               | Project                          | Stack Status       | Creation Time        | Updated Time |
+--------------------------------------+--------------------------+----------------------------------+--------------------+----------------------+--------------+
| 70368ed7-a877-45dd-a183-d3cbc78ce7ac | k8s-cluster-gp2lk52qjexm | 939974d6fc8e400eb01f7dd749935ea3 | CREATE_IN_PROGRESS | 2022-11-11T06:28:41Z | None         |
+--------------------------------------+--------------------------+----------------------------------+--------------------+----------------------+--------------+

# confirm checkpoints for creation

[root@dlp ~(keystone)]#
openstack stack list --nested | grep k8s-cluster

| 63bbd57c-9a9f-48c9-b025-7ca3b1885055 | k8s-cluster-gp2lk52qjexm-kube_minions-ilpjay5sq7jy-0-mp357vdhzihz                                 | 939974d6fc8e400eb01f7dd749935ea3 | CREATE_COMPLETE | 2022-11-11T06:31:52Z | None         | cbe2f480-22b9-46f0-9ad0-f96202794139 |
| cbe2f480-22b9-46f0-9ad0-f96202794139 | k8s-cluster-gp2lk52qjexm-kube_minions-ilpjay5sq7jy                                                | 939974d6fc8e400eb01f7dd749935ea3 | CREATE_COMPLETE | 2022-11-11T06:31:51Z | None         | 70368ed7-a877-45dd-a183-d3cbc78ce7ac |
| 4cb8a1db-e9fd-406a-adb4-43974bfd8275 | k8s-cluster-gp2lk52qjexm-api_address_floating_switch-5ioiquyxrgjm                                 | 939974d6fc8e400eb01f7dd749935ea3 | CREATE_COMPLETE | 2022-11-11T06:31:51Z | None         | 70368ed7-a877-45dd-a183-d3cbc78ce7ac |
| 81953818-92ea-4745-aefa-8457e282e088 | k8s-cluster-gp2lk52qjexm-api_address_lb_switch-5mcli22dsrvw                                       | 939974d6fc8e400eb01f7dd749935ea3 | CREATE_COMPLETE | 2022-11-11T06:31:49Z | None         | 70368ed7-a877-45dd-a183-d3cbc78ce7ac |
| 7e9415ac-fc25-4087-b81a-f1bf6e4d77e1 | k8s-cluster-gp2lk52qjexm-etcd_address_lb_switch-vbojcqzoehl5                                      | 939974d6fc8e400eb01f7dd749935ea3 | CREATE_COMPLETE | 2022-11-11T06:31:49Z | None         | 70368ed7-a877-45dd-a183-d3cbc78ce7ac |
| 23c372d7-ef69-4cfa-bb7f-bafbd6d24927 | k8s-cluster-gp2lk52qjexm-kube_masters-4itoqbkbeztc-0-3ivhjocnornd-api_address_switch-3igkdpvypbus | 939974d6fc8e400eb01f7dd749935ea3 | CREATE_COMPLETE | 2022-11-11T06:29:03Z | None         | cced0d2c-d9ed-40dd-91e3-ce03b25237c4 |
| cced0d2c-d9ed-40dd-91e3-ce03b25237c4 | k8s-cluster-gp2lk52qjexm-kube_masters-4itoqbkbeztc-0-3ivhjocnornd                                 | 939974d6fc8e400eb01f7dd749935ea3 | CREATE_COMPLETE | 2022-11-11T06:28:47Z | None         | 81e6151c-1924-461d-98be-93d667608461 |
| 81e6151c-1924-461d-98be-93d667608461 | k8s-cluster-gp2lk52qjexm-kube_masters-4itoqbkbeztc                                                | 939974d6fc8e400eb01f7dd749935ea3 | CREATE_COMPLETE | 2022-11-11T06:28:46Z | None         | 70368ed7-a877-45dd-a183-d3cbc78ce7ac |
| 33b48200-f4fc-4da7-8310-cc5788198f91 | k8s-cluster-gp2lk52qjexm-api_lb-kx6mof3i43y6                                                      | 939974d6fc8e400eb01f7dd749935ea3 | CREATE_COMPLETE | 2022-11-11T06:28:45Z | None         | 70368ed7-a877-45dd-a183-d3cbc78ce7ac |
| fcb5b6d2-9c30-42f3-82f8-caf4ed4ff19c | k8s-cluster-gp2lk52qjexm-etcd_lb-dt57ua253q6w                                                     | 939974d6fc8e400eb01f7dd749935ea3 | CREATE_COMPLETE | 2022-11-11T06:28:44Z | None         | 70368ed7-a877-45dd-a183-d3cbc78ce7ac |
| 5728894e-a468-4c3a-b4aa-5384a3c848cb | k8s-cluster-gp2lk52qjexm-network-nhbm3pjgi4wk-network_switch-p7dzze45aywb                         | 939974d6fc8e400eb01f7dd749935ea3 | CREATE_COMPLETE | 2022-11-11T06:28:43Z | None         | 3eb24927-c04f-49d2-ad95-b25cbc91023d |
| 3eb24927-c04f-49d2-ad95-b25cbc91023d | k8s-cluster-gp2lk52qjexm-network-nhbm3pjgi4wk                                                     | 939974d6fc8e400eb01f7dd749935ea3 | CREATE_COMPLETE | 2022-11-11T06:28:42Z | None         | 70368ed7-a877-45dd-a183-d3cbc78ce7ac |
| 70368ed7-a877-45dd-a183-d3cbc78ce7ac | k8s-cluster-gp2lk52qjexm                                                                          | 939974d6fc8e400eb01f7dd749935ea3 | CREATE_COMPLETE | 2022-11-11T06:28:41Z | None         | None                                 |

# if sucessfully finished, state is [CREATE_COMPLETE] + [HEALTHY]

[root@dlp ~(keystone)]#
openstack coe cluster list

+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
| uuid                                 | name        | keypair | node_count | master_count | status          | health_status |
+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
| a93d1aaf-bc17-499d-aac6-43dcc4f43563 | k8s-cluster | my-key  |          1 |            1 | CREATE_COMPLETE | HEALTHY       |
+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+

# instances are running

[root@dlp ~(keystone)]#
openstack server list

+--------------------------------------+-----------------------------------+--------+------------------------------------+---------------+----------+
| ID                                   | Name                              | Status | Networks                           | Image         | Flavor   |
+--------------------------------------+-----------------------------------+--------+------------------------------------+---------------+----------+
| 49f37337-5776-4b08-962e-5374e104dfb3 | k8s-cluster-gp2lk52qjexm-node-0   | ACTIVE | private=10.0.0.238, 192.168.100.81 | Fedora-CoreOS | m1.large |
| 887f2958-bd94-4554-8474-254eec36889e | k8s-cluster-gp2lk52qjexm-master-0 | ACTIVE | private=10.0.0.242, 192.168.100.41 | Fedora-CoreOS | m1.large |
+--------------------------------------+-----------------------------------+--------+------------------------------------+---------------+----------+
[3] To access to use Kubernetes Cluster, Set like follows.
# install [kubectl] from Snap

[root@dlp ~(keystone)]#
snap install kubectl --classic

kubectl 1.25.3 from Canonical✓ installed
[root@dlp ~(keystone)]#
openstack coe cluster list

+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
| uuid                                 | name        | keypair | node_count | master_count | status          | health_status |
+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
| a93d1aaf-bc17-499d-aac6-43dcc4f43563 | k8s-cluster | my-key  |          1 |            1 | CREATE_COMPLETE | HEALTHY       |
+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+

[root@dlp ~(keystone)]#
openstack coe cluster config k8s-cluster

export KUBECONFIG=/root/config
[root@dlp ~(keystone)]#
export KUBECONFIG=/root/config
[root@dlp ~(keystone)]#
kubectl get nodes

NAME                                STATUS   ROLES    AGE     VERSION
k8s-cluster-gp2lk52qjexm-master-0   Ready    master   7m45s   v1.23.3
k8s-cluster-gp2lk52qjexm-node-0     Ready    <none>   5m52s   v1.23.3

[root@dlp ~(keystone)]#
kubectl get pods -n kube-system

NAME                                         READY   STATUS    RESTARTS   AGE
calico-kube-controllers-54799d958b-bh4g9     1/1     Running   0          8m21s
calico-node-h2v4d                            1/1     Running   0          6m42s
calico-node-tgnrc                            1/1     Running   0          8m21s
coredns-56448757b9-hbbp7                     1/1     Running   0          8m21s
coredns-56448757b9-wzr5m                     1/1     Running   0          8m21s
dashboard-metrics-scraper-67f57ff746-675qx   1/1     Running   0          8m20s
k8s-keystone-auth-tq8mr                      1/1     Running   0          8m19s
kube-dns-autoscaler-6d5b5dc777-xqtlm         1/1     Running   0          8m21s
kubernetes-dashboard-7b88d986b4-5g5pg        1/1     Running   0          8m20s
magnum-metrics-server-6c4c77844b-2fzhp       1/1     Running   0          8m13s
npd-4gqmx                                    1/1     Running   0          6m22s

# verify cluster to create test pods

[root@dlp ~(keystone)]#
kubectl create deployment test-nginx --image=nginx --replicas=2

deployment.apps/test-nginx created
[root@dlp ~(keystone)]#
kubectl get pods -o wide

NAME                         READY   STATUS    RESTARTS   AGE   IP             NODE                              NOMINATED NODE   READINESS GATES
test-nginx-976fbbd77-fp79q   1/1     Running   0          20s   10.100.81.67   k8s-cluster-gp2lk52qjexm-node-0   <none>           <none>
test-nginx-976fbbd77-xdrcr   1/1     Running   0          20s   10.100.81.68   k8s-cluster-gp2lk52qjexm-node-0   <none>           <none>

[root@dlp ~(keystone)]#
kubectl expose deployment test-nginx --type="NodePort" --port 80

service/test-nginx exposed
[root@dlp ~(keystone)]#
kubectl get services test-nginx

NAME         TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
test-nginx   NodePort   10.254.39.123   <none>        80:30956/TCP   5s

[root@dlp ~(keystone)]#
kubectl port-forward service/test-nginx --address 0.0.0.0 10443:80 &

Forwarding from 0.0.0.0:10443 -> 80
[root@dlp ~(keystone)]#
curl localhost:10443

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
.....
.....
[4] If you'd like to use Magnum with common users, it needs to change some settings.
[root@dlp ~(keystone)]#
openstack role list

+----------------------------------+------------------+
| ID                               | Name             |
+----------------------------------+------------------+
| 075291ce7c66499ea099a012a92f6695 | admin            |
| 1ace023940044b34b5a442eb38e97635 | heat_stack_owner |
| 1c38efda26774c358fabbdfed25547ae | reader           |
| 245b582c47d744c1a6f706898534900a | CloudUser        |
| ae93621edfd4441096d4eb1b1e4be035 | heat_stack_user  |
| d0f53ed114fc4ed1b56679ba0b0ef33d | member           |
+----------------------------------+------------------+

[root@dlp ~(keystone)]#
openstack project list

+----------------------------------+-----------+
| ID                               | Name      |
+----------------------------------+-----------+
| 28b6e37c787240e4a975d3614821cc71 | hiroshima |
| 62f531f4d2934e75b8d7f11cd7d53be3 | service   |
| 939974d6fc8e400eb01f7dd749935ea3 | admin     |
+----------------------------------+-----------+

[root@dlp ~(keystone)]#
openstack user list

+----------------------------------+---------------------+
| ID                               | Name                |
+----------------------------------+---------------------+
| f00e5072d7d6488f935ea680256af89c | admin               |
| 78009f5643d240da9996dff0aee5cc37 | glance              |
| 7ed52b257a47436389a60aef689d20f0 | nova                |
| 2080b84b7e304486b33365a447963bc1 | placement           |
| 6f4af63a84f944f3a7a0c018ef64a4a1 | neutron             |
| 9e8824a151c949ad9105535ead452501 | serverworld         |
| 42de8b3de0c74f3e9f1622c89fc1a178 | cinder              |
| 03b59cb43c8547d4bf0a055dd9edd7a8 | heat                |
| e8c47aafd9324805b90b52f963dc6cad | heat_domain_admin   |
| 17b493375a874f899cb40da602e1bec7 | barbican            |
| 81a4793c2dbd4839bcb5ecedf70d1916 | magnum              |
| 40b1f36c06434e249f1828d4cfbd290a | magnum_domain_admin |
+----------------------------------+---------------------+

# for example, add [serverworld] user in [hiroshima] project to [heat_stack_owner] role

[root@dlp ~(keystone)]#
openstack role add --project hiroshima --user serverworld heat_stack_owner
# on the Node Neutron server is running, change settings like follows

[root@network ~]#
vi /etc/neutron/policy.json
# create new
# overwrite some settings

{
  "create_port:fixed_ips:subnet_id": "",
  "create_port:allowed_address_pairs": "",
  "create_port:allowed_address_pairs:ip_address": "",
}

[root@network ~]#
systemctl restart neutron-server

# that's OK, common users can create clusters

[cent@dlp ~(keystone)]$
openstack coe cluster list

+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
| uuid                                 | name        | keypair | node_count | master_count | status          | health_status |
+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
| fe6d43ca-c2b6-412a-be87-c3eb6dbd515a | k8s-cluster | mykey   |          1 |            1 | CREATE_COMPLETE | HEALTHY       |
+--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
Matched Content