OpenStack Zed : How to use Magnum2023/06/30 |
|
Install OpenStack Container Infrastructure Management Service (Magnum).
This example is based on the environment like follows.
------------+--------------------------+--------------------------+------------
| | |
eth0|10.0.0.30 eth0|10.0.0.50 eth0|10.0.0.51
+-----------+-----------+ +-----------+-----------+ +-----------+-----------+
| [ dlp.srv.world ] | | [ network.srv.world ] | | [ node01.srv.world ] |
| (Control Node) | | (Network Node) | | (Compute Node) |
| | | | | |
| MariaDB RabbitMQ | | Neutron L2/L3 Agent | | Libvirt |
| Memcached Nginx | | Neutron Metadata | | Nova Compute |
| Keystone httpd | | Open vSwitch | | Neutron L2 Agent |
| Glance Nova API | | iSCSI Target | | Open vSwitch |
| Neutron Server | | Cinder Volume | | |
| Neutron Metadata | | Heat API/Engine | | |
| Cinder API | | Nginx | | |
| Barbican API | | Magnum Services | | |
+-----------------------+ +-----------------------+ +-----------------------+
|
| [1] | On Control Node, Download a VM image for containers (Fedora CoreOS) and add it to Glance. |
|
root@dlp ~(keystone)# wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/37.20230303.3.0/x86_64/fedora-coreos-37.20230303.3.0-openstack.x86_64.qcow2.xz root@dlp ~(keystone)# xz -dv fedora-coreos-37.20230303.3.0-openstack.x86_64.qcow2.xz root@dlp ~(keystone)# openstack image create Fedora-CoreOS --file=fedora-coreos-37.20230303.3.0-openstack.x86_64.qcow2 --disk-format=qcow2 --container-format=bare --property os_distro='fedora-coreos' --public |
| [2] | How to use Magnum. For example, Create Kubernetes Cluster with [admin] user. |
|
root@dlp ~(keystone)# openstack flavor list +----+-----------+------+------+-----------+-------+-----------+ | ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public | +----+-----------+------+------+-----------+-------+-----------+ | 1 | m1.small | 2048 | 10 | 0 | 1 | True | | 2 | m1.medium | 4096 | 10 | 0 | 2 | True | | 3 | m1.large | 8192 | 10 | 0 | 4 | True | | 4 | m2.large | 8192 | 10 | 10 | 4 | True | | 5 | m3.large | 8192 | 20 | 0 | 4 | True | +----+-----------+------+------+-----------+-------+-----------+root@dlp ~(keystone)# openstack keypair list +--------+-------------------------------------------------+------+ | Name | Fingerprint | Type | +--------+-------------------------------------------------+------+ | my-key | 24:33:d1:3d:ad:3c:13:92:7b:78:04:77:4e:a0:f6:70 | ssh | +--------+-------------------------------------------------+------+root@dlp ~(keystone)# openstack network list +--------------------------------------+---------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+---------+--------------------------------------+ | ae5fdb1f-efb9-412b-9053-b81106c90336 | private | b3234f3f-b3e3-41d2-b3a4-ca3d4ec55d24 | | b9b8536b-0696-4a5b-af05-602587435c0a | public | 0129c597-e411-450e-a4f9-b387c68eddb3 | +--------------------------------------+---------+--------------------------------------+root@dlp ~(keystone)# openstack subnet list +--------------------------------------+----------------+--------------------------------------+------------------+ | ID | Name | Network | Subnet | +--------------------------------------+----------------+--------------------------------------+------------------+ | 0129c597-e411-450e-a4f9-b387c68eddb3 | public-subnet | b9b8536b-0696-4a5b-af05-602587435c0a | 10.0.0.0/24 | | b3234f3f-b3e3-41d2-b3a4-ca3d4ec55d24 | private-subnet | ae5fdb1f-efb9-412b-9053-b81106c90336 | 192.168.100.0/24 | +--------------------------------------+----------------+--------------------------------------+------------------+ # create Kubernetes Cluster template root@dlp ~(keystone)# openstack coe cluster template create k8s-cluster-template \ --image Fedora-CoreOS \ --external-network public \ --fixed-network private \ --fixed-subnet private-subnet \ --dns-nameserver 10.0.0.10 \ --network-driver calico \ --docker-storage-driver overlay2 \ --docker-volume-size 10 \ --master-flavor m3.large \ --flavor m3.large \ --coe kubernetes Request to create cluster template k8s-cluster-template accepted +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | insecure_registry | - | | labels | {} | | updated_at | - | | floating_ip_enabled | True | | fixed_subnet | private-subnet | | master_flavor_id | m3.large | | uuid | 19959f08-5962-4359-afeb-2267ec67804b | | no_proxy | - | | https_proxy | - | | tls_disabled | False | | keypair_id | - | | public | False | | http_proxy | - | | docker_volume_size | 10 | | server_type | vm | | external_network_id | public | | cluster_distro | fedora-coreos | | image_id | Fedora-CoreOS | | volume_driver | - | | registry_enabled | False | | docker_storage_driver | overlay2 | | apiserver_port | - | | name | k8s-cluster-template | | created_at | 2023-06-30T06:56:07.534093+00:00 | | network_driver | calico | | fixed_network | private | | coe | kubernetes | | flavor_id | m3.large | | master_lb_enabled | False | | dns_nameserver | 10.0.0.10 | | hidden | False | | tags | - | +-----------------------+--------------------------------------+ # create Kubernetes Cluster with 2 nodes root@dlp ~(keystone)# openstack coe cluster create k8s-cluster \ --cluster-template k8s-cluster-template \ --master-count 1 \ --node-count 1 \ --keypair my-key Request to create cluster 7790f694-91b4-4f3b-bd7c-98276b75330f accepted # verify status # proceed to create cluster during [CREATE_IN_PROGRESS] state root@dlp ~(keystone)# openstack coe cluster list +--------------------------------------+-------------+---------+------------+--------------+--------------------+---------------+ | uuid | name | keypair | node_count | master_count | status | health_status | +--------------------------------------+-------------+---------+------------+--------------+--------------------+---------------+ | 7790f694-91b4-4f3b-bd7c-98276b75330f | k8s-cluster | my-key | 1 | 1 | CREATE_IN_PROGRESS | None | +--------------------------------------+-------------+---------+------------+--------------+--------------------+---------------+ # Heat orchestration System is used for creating root@dlp ~(keystone)# openstack stack list +--------------------------------------+--------------------------+----------------------------------+--------------------+----------------------+--------------+ | ID | Stack Name | Project | Stack Status | Creation Time | Updated Time | +--------------------------------------+--------------------------+----------------------------------+--------------------+----------------------+--------------+ | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada | k8s-cluster-5d747yxstr3q | 757625ae78404e38a8cfdd7c6d262860 | CREATE_IN_PROGRESS | 2023-06-30T06:56:59Z | None | +--------------------------------------+--------------------------+----------------------------------+--------------------+----------------------+--------------+ # confirm checkpoints for creation root@dlp ~(keystone)# openstack stack list --nested | grep k8s-cluster | 22a31b8c-04a3-480f-b43e-be52bd8f8859 | k8s-cluster-5d747yxstr3q-kube_minions-7k7yznhzwfkb-0-b4ollnbov4r4 | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T07:00:07Z | None | 321cf254-97c5-4d0b-866c-04bd529e06ea | | 9053ee77-ac52-4d56-9aaa-5f4c4bc9acc5 | k8s-cluster-5d747yxstr3q-api_address_floating_switch-oviashfdkuvx | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T07:00:06Z | None | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada | | 321cf254-97c5-4d0b-866c-04bd529e06ea | k8s-cluster-5d747yxstr3q-kube_minions-7k7yznhzwfkb | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T07:00:06Z | None | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada | | 7c42be78-59c6-4fcb-82fa-bd11f8bf868d | k8s-cluster-5d747yxstr3q-api_address_lb_switch-k77qarn7dobf | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T07:00:05Z | None | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada | | b98a5ce4-eedd-4e55-b57c-79995681bc05 | k8s-cluster-5d747yxstr3q-etcd_address_lb_switch-wbd5t6rgun3a | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T07:00:04Z | None | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada | | ec132afc-e8c0-4647-9fb1-bc0b85aace77 | k8s-cluster-5d747yxstr3q-kube_masters-hrd7vlt6xvos-0-dewo6txtuo54-api_address_switch-oi4fxlbsewmv | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:57:29Z | None | 88244e99-1df8-4a99-b5fe-4ddca9f70f66 | | 88244e99-1df8-4a99-b5fe-4ddca9f70f66 | k8s-cluster-5d747yxstr3q-kube_masters-hrd7vlt6xvos-0-dewo6txtuo54 | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:57:07Z | None | e7074d57-631e-43e4-aa39-95d4d8c2ea7c | | e7074d57-631e-43e4-aa39-95d4d8c2ea7c | k8s-cluster-5d747yxstr3q-kube_masters-hrd7vlt6xvos | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:57:06Z | None | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada | | f85de343-388a-48ee-877f-b3bcc19f0a44 | k8s-cluster-5d747yxstr3q-etcd_lb-4ja6buynrvyp | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:57:05Z | None | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada | | 61e764ad-3e2f-4adf-a9f5-6016aa9af351 | k8s-cluster-5d747yxstr3q-api_lb-lsk5qra5uxce | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:57:05Z | None | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada | | 965f9ea8-5830-45b4-862e-5b45300c0415 | k8s-cluster-5d747yxstr3q-network-xbrtftx2g7dd-network_switch-2colkowcycmx | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:57:03Z | None | 3c6ea82c-c54c-4ac6-8d32-57700ab88585 | | 3c6ea82c-c54c-4ac6-8d32-57700ab88585 | k8s-cluster-5d747yxstr3q-network-xbrtftx2g7dd | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:57:03Z | None | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada | | 44ff9533-48c9-4b9c-a945-5cc4d7d17ada | k8s-cluster-5d747yxstr3q | 757625ae78404e38a8cfdd7c6d262860 | CREATE_COMPLETE | 2023-06-30T06:56:59Z | None | None | # if sucessfully finished, state is [CREATE_COMPLETE] + [HEALTHY] root@dlp ~(keystone)# openstack coe cluster list +--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+ | uuid | name | keypair | node_count | master_count | status | health_status | +--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+ | 7790f694-91b4-4f3b-bd7c-98276b75330f | k8s-cluster | my-key | 1 | 1 | CREATE_COMPLETE | HEALTHY | +--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+ # instances are running root@dlp ~(keystone)# openstack server list +--------------------------------------+-----------------------------------+--------+-------------------------------------+---------------+----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------------------------+--------+-------------------------------------+---------------+----------+ | 4ad48718-5fd7-462d-913e-d2384d5eb010 | k8s-cluster-5d747yxstr3q-node-0 | ACTIVE | private=192.168.100.115 | Fedora-CoreOS | m3.large | | 1e21221d-fda8-41d1-8e50-c8ebd7cd26a6 | k8s-cluster-5d747yxstr3q-master-0 | ACTIVE | private=10.0.0.205, 192.168.100.192 | Fedora-CoreOS | m3.large | +--------------------------------------+-----------------------------------+--------+-------------------------------------+---------------+----------+ |
| [3] | To access to use Kubernetes Cluster, Set like follows. |
|
root@dlp ~(keystone)# openstack coe cluster list +--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+ | uuid | name | keypair | node_count | master_count | status | health_status | +--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+ | 7790f694-91b4-4f3b-bd7c-98276b75330f | k8s-cluster | my-key | 1 | 1 | CREATE_COMPLETE | HEALTHY | +--------------------------------------+-------------+---------+------------+--------------+-----------------+---------------+
root@dlp ~(keystone)#
root@dlp ~(keystone)# openstack coe cluster config k8s-cluster export KUBECONFIG=/root/config root@dlp ~(keystone)# export KUBECONFIG=/root/config
kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-cluster-5d747yxstr3q-master-0 Ready master 7m8s v1.23.3 k8s-cluster-5d747yxstr3q-node-0 Ready <none> 5m8s v1.23.3 # verify cluster to create test pods root@dlp ~(keystone)# kubectl create deployment test-nginx --image=nginx --replicas=2 deployment.apps/test-nginx created kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES test-nginx-976fbbd77-wbp5j 1/1 Running 0 12s 10.100.176.196 k8s-cluster-5d747yxstr3q-node-0 <none> <none> test-nginx-976fbbd77-x8xrg 1/1 Running 0 12s 10.100.176.195 k8s-cluster-5d747yxstr3q-node-0 <none> <none>
root@dlp ~(keystone)#
root@dlp ~(keystone)# kubectl expose deployment test-nginx --type="NodePort" --port 80 service/test-nginx exposed kubectl get services test-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE test-nginx NodePort 10.254.124.127 <none> 80:31615/TCP 6s
root@dlp ~(keystone)#
root@dlp ~(keystone)# kubectl port-forward service/test-nginx --address 0.0.0.0 10443:80 & Forwarding from 0.0.0.0:10443 -> 80 curl localhost:10443
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
.....
.....
|
| [4] | If you'd like to use Magnum with common users, it needs to change some settings. |
|
root@dlp ~(keystone)# openstack role list +----------------------------------+------------------+ | ID | Name | +----------------------------------+------------------+ | 109c6488ef4347d681ba691f8ac97a6b | member | | 6055174790cd4797ae3671b38d935c53 | service | | 81f2687109a245b580690b6e8f9095db | heat_stack_owner | | 82699442db4e4edea2fb7986340b5ab4 | admin | | db9927603e564f94b2f0a4312d1a63da | heat_stack_user | | eb3d0d5b10cd456093463a2c12e41b82 | reader | +----------------------------------+------------------+root@dlp ~(keystone)# openstack project list +----------------------------------+------------------------------------------------------------------+ | ID | Name | +----------------------------------+------------------------------------------------------------------+ | 1c2f5dc8c4f9494fbaaa4217c8e3585a | hiroshima | | 757625ae78404e38a8cfdd7c6d262860 | admin | | d8b09d86ed7743039f92b2e542ea26c1 | service | | e614281e49f54c51abdc90b92e938846 | 757625ae78404e38a8cfdd7c6d262860-44ff9533-48c9-4b9c-a945-5cc4d7d | +----------------------------------+------------------------------------------------------------------+root@dlp ~(keystone)# openstack user list +----------------------------------+---------------------+ | ID | Name | +----------------------------------+---------------------+ | a45313c73c4842d7b5ba90f56be73de2 | admin | | b166b462bd214a34b99e079c806bef55 | glance | | 9fb0ad5b33f9492594f4139f5b428dab | nova | | 5db174d072dd4bcc90d34d7b0c4f0e36 | placement | | 1b4f2549a8ee462d909185b6a321e265 | neutron | | de51d5f0ee2c485885877d21f5b424e0 | serverworld | | 5b1712badfdb4824927488f7882cc7ca | cinder | | f485fdd327064f56ac0182c3b084525b | heat | | 65993a5e1fe0476486e82091276c2827 | heat_domain_admin | | b0c091c98dbe452db61fdb9d8d44d719 | barbican | | c5916ff29c094da88560ba100c6f0340 | magnum | | 846e638c9f394766bf193e09782e1b4e | magnum_domain_admin | +----------------------------------+---------------------+ # for example, add [serverworld] user in [hiroshima] project to [heat_stack_owner] role root@dlp ~(keystone)# openstack role add --project hiroshima --user serverworld heat_stack_owner
# on the Node Neutron server is running, change settings like follows root@dlp ~(keystone)# vi /etc/neutron/policy.d/00_default_policy.yaml # add to the end # overwrite some settings "create_port:fixed_ips:subnet_id": "(rule:context_is_advsvc) or (rule:network_owner) or (rule:admin_only) or (rule:shared) or (role:member and project_id:%(project_id)s)" "create_port:allowed_address_pairs": "(rule:admin_only) or (rule:network_owner) or (role:member and project_id:%(project_id)s)" "create_port:allowed_address_pairs:ip_address": "(rule:admin_only) or (rule:network_owner) or (role:member and project_id:%(project_id)s)"
root@dlp ~(keystone)# systemctl restart neutron-api
# that's OK, common users can create clusters debian@dlp ~(keystone)$ openstack coe cluster list +--------------------------------------+--------------+---------+------------+--------------+-----------------+---------------+ | uuid | name | keypair | node_count | master_count | status | health_status | +--------------------------------------+--------------+---------+------------+--------------+-----------------+---------------+ | 81e42408-4ec6-4d1d-8bec-41017494e7dc | k8s-cluster2 | mykey | 1 | 1 | CREATE_COMPLETE | None | +--------------------------------------+--------------+---------+------------+--------------+-----------------+---------------+ |
| Sponsored Link |