+------------------------------------------------------------------------------------------+ | WARNING: The tool described in this repository is not supported in any way by Red Hat!!! | +==========================================================================================+ +------------------------------------------------------------------------------------------+ About ===== This is a sample python client on top of the generated assisted-installer python library to ease working with assisted installer API Available features: - create/delete cluster (autoinjecting pull secret file and ssh public key, static networking) - create/delete manifests - download discovery iso - wait for hosts - list cluster/hosts - update hostnames and host roles - update cluster attributes such as api vip and ingress vip - launch cluster install - wait for cluster - create day2 cluster for adding hosts Deploying ========= Offline token ------------- When using SAAS mode, an offline token is needed in order to interact with the api. This token can be retrieved at https://cloud.redhat.com/openshift/token using rpm --------- :: sudo dnf -y copr enable karmab/aicli ; sudo dnf -y install aicli using pip --------- Install the package with :: pip3 install aicli To upgrade it later on, use the following :: pip3 install -U aicli assisted-service-client Container mode -------------- :: alias aicli='podman run --security-opt label=disable --net host -it --rm -e AI_OFFLINETOKEN=$AI_OFFLINETOKEN -v $HOME/.aicli:/root/.aicli -v $PWD:/workdir quay.io/karmab/aicli' Where AI_OFFLINETOKEN is an environment variable pointing to your saas offline token With onprem mode, you can instead use ``-e AI_URL=$AI_URL`` The container engine can also be docker. How to use ========== Configuration ------------- Setting target url ~~~~~~~~~~~~~~~~~~ by default, the tool targets the SAAS (which means providing an offline token) Alternatively, You can indicate a target AI_URL, using the flag *–url* The url needs to be provided in this format ``http://$AI_IP:8090`` The target url can also be specified through environment variable AI_URL the flag ``--staging`` can be set to target the internal staging environment Alternatively, set the env variable STAGING to true In container mode, that would be ``-e STAGING=true`` TLS settings ~~~~~~~~~~~~ You can use the flag ca, cert and key to provide ssl ca content, an ssl cert and key path to use for tls purpose Alternatively, set the env variable AI_CA, AI_CERT and AI_KEY Creating an onprem environment ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can run the following command to deploy AI locally using podman: :: aicli create onprem The ip to use for the configuration is detected automatically using a socket connection to 8.8.8.8 If this is blocked, you can use pass ``-P ip=$your_ip`` instead You can also provide ``-P keep=true`` to keep both configmap.yml and pod.yml so you can tweak them yourself When no longer needed, the onprem deployment can be deleted using: :: aicli delete onprem Note that deploying AI this way is not the recommended approach for production/supported usage. Basic usage ----------- the main objects you can interact with are cluster and hosts. For all of them, you can use create/delete/info/list/update subcommand For most of the objects, you can provide either an id or a name. Parameters ~~~~~~~~~~ For most of the commands, you can pass parameters either on the command line by repeating ``-P key=value`` or by putting them in a parameter file, in yaml format. If present, the file ``aicli_parameters.yml`` is also parsed. Typical workflow ---------------- Interacting with the api should be straightforward but let’s walk through the typical steps you can use to deploy a cluster to completion Note: consider using ``aicli create deployment`` to deploy with a single step Create cluster ~~~~~~~~~~~~~~ The basic way to create a cluster is to run: :: aicli create cluster mycluster When creating a cluster, two types of parameters can be provided, *cluster keywords* and *extra keywords*. The cluster keywords are parameters exposed by AI api, while the extra keywords are goodies that provide shortcuts to set things such as sno, network_type,… Those keywords can be listed with ``aicli list cluster-keywords`` and ``aicli list extra-keywords`` For instance, we can use the following command to create a cluster with a specific version and forcing the domain :: aicli create cluster -P openshift_version=4.9 -P base_dns_domain=karmatron.local -P pull_secret=openshift_pull.json mycluster This command will also use your default ssh public key so you don’t need to specify any (using ssh_public_key variable for instance) To create a sno cluster :: aicli create cluster -P sno=true mycluster Within the extra parameters, the most common ones to use are listed below: ===================== ============================================================= Parameter Meaning ===================== ============================================================= api_ip Api ip ingress_ip Ingress ip domain Ingress ip minimal Whether to use minimal iso static_network_config the nmstate data to inject manifests a directory from where pick manifests to inject network_type Which sdn to use sno Whether to deploy a SNO sno_disk Which disk to use for SNO install hosts An array of hosts to automatically update data from pull_secret The path to your pull_secret (openshift_pull.json by default) ignore_validations Whether to ignore all validations ===================== ============================================================= Note: there are DNS requirements associated to the name of the cluster and the domain for an install to be available without /etc/hosts hacks When a cluster gets created, an underlying infraenv named *$cluster_infraenv* also gets created under the hood. In general, you shouldn’t have to care about this object, but notice it is actually where the iso information lives. The purpose of this object is to be able to boot nodes with a discovery iso without deciding initially on which cluster they belong (this is called late binding). The nomenclature we use for the infraenv is consistent with what happens in AI UI, which means you can create a cluster and follow in the UI or use aicli to interact with a cluster created through the UI. You can set the parameter *infraenv* to false to prevent an infraenv to get created for the cluster. You will then have to use the bind subcommand to associate hosts discovered through a given infraenv to some specific cluster. This same parameter can be provided when updating a cluster so that no action on infraenvs happen. Custom ignition ~~~~~~~~~~~~~~~ You can inject custom ignition in the discovery iso by either: - specifying ``ignition_config_override`` parameter with an ignition string For instance, :: ignition_config_override: "{\"ignition\":{\"version\":\"3.1.0\"},\"systemd\":{\"units\":[{\"name\":\"ca-patch.service\",\"enabled\":true,\"contents\":\"[Service]\\nType=oneshot\\nExecStart=/usr/local/bin/ca-patch.sh\\n\\n[Install]\\nWantedBy=multi-user.target\"}]},\"storage\":{\"files\":[{\"path\":\"/usr/local/bin/ca-patch.sh\",\"mode\":720,\"contents\":{\"source\":\"data:text/plain;charset=utf-8;base64,IyEvYmluL2Jhc2gKc3VjY2Vzcz0wCnVudGlsIFsgJHN1Y2Nlc3MgLWd0IDEgXTsgZG8KICB0bXA9JChta3RlbXApCiAgY2F0IDw8RU9GPiR7dG1wfSB8fCB0cnVlCmRhdGE6CiAgcmVxdWVzdGhlYWRlci1jbGllbnQtY2EtZmlsZTogfAokKHdoaWxlIElGUz0gcmVhZCAtYSBsaW5lOyBkbyBlY2hvICIgICAgJGxpbmUiOyBkb25lIDwgPChjYXQgL2V0Yy9rdWJlcm5ldGVzL2Jvb3RzdHJhcC1zZWNyZXRzL2FnZ3JlZ2F0b3ItY2EuY3J0KSkKRU9GCiAgS1VCRUNPTkZJRz0vZXRjL2t1YmVybmV0ZXMvYm9vdHN0cmFwLXNlY3JldHMva3ViZWNvbmZpZyBrdWJlY3RsIC1uIGt1YmUtc3lzdGVtIHBhdGNoIGNvbmZpZ21hcCBleHRlbnNpb24tYXBpc2VydmVyLWF1dGhlbnRpY2F0aW9uIC0tcGF0Y2gtZmlsZSAke3RtcH0KICBpZiBbWyAkPyAtZXEgMCBdXTsgdGhlbgoJcm0gJHt0bXB9CglzdWNjZXNzPTIKICBmaQogIHJtICR7dG1wfQogIHNsZWVwIDYwCmRvbmUK\"}}]},\"kernelArguments\":{\"shouldExist\":[\"ipv6.disable=1\"]}}" - using the extra keyword ``discovery_ignition_file`` and providing a path where to store the ignition directly (that is, without specifying ignition_config_override) Note that for such ignition to be applied to the nodes once they really get installed, a machineconfig is needed (which can be injected as custom manifest) Custom kernel arguments ~~~~~~~~~~~~~~~~~~~~~~~ You can specify kernel arguments that only apply to the discovery ISO by using the keyword ``kernel_arguments``. The data can be provided either as a string (say ``a=xxx b=yyy``) in which case the arguments are considered to be appended or as a dict such as :: kernel_arguments: - operation: append value: 'a=xxx' - operation: append value: 'b=yyy' Custom install config ~~~~~~~~~~~~~~~~~~~~~ You can tweak the install config of a cluster by providing a dictionary such as :: aicli update cluster mycluster -P installconfig="{'CPUPartitioningMode': 'AllNodes'}" Custom networking ~~~~~~~~~~~~~~~~~ Network type ^^^^^^^^^^^^ Default CNI when creating a cluster is ``OVNKubernetes`` the network_type extra keyword can be used to specify another one such as OpenshiftSDN, Calico, Contrail Note that non standard CNIs typically have extra requirements of injecting custom manifests. Tuning network of the nodes ^^^^^^^^^^^^^^^^^^^^^^^^^^^ In order to use custom/static networking for your hosts, you need to provide nmstate information in the parameter file using the field *static_network_config* You can also customize things such as cluster_networks, machine_networks and service_networks, for instance when trying to do a dual stack installation You can find different samples `here `__ covering how to do: - static networking - bonding - dual stack Registry url ^^^^^^^^^^^^ A non standard registry can be specified using the keyword ``registry_url`` (or ``disconnected_url``) which is useful for disconnected install or testing okd. Specify the corresponding ca content using the ``ca`` variable or let it undefined to let aicli fetch this cert using openssl. This approach requires that there exists connectivity between aicli and your custom registry. Adding extra manifests ~~~~~~~~~~~~~~~~~~~~~~ You can inject extra manifests (for instance if you are using a non standard CNI), for instance from the mydir directory, using the following commands :: aicli create manifests --dir mydir mycluster A flag allows you to have them stored in the openshift folder. You can then use ``aicli list manifests mycluster`` to confirm they were properly uploaded, or use ``aicli delete manifests`` for deletion Manifests from a given directory can be specified directly in your parameter file so that they get injected at cluster creation. For this, include the keyword ``manifests`` and point it to the directory where your manifests are stored. You can alternatively provide a list of entries below manifests such as :: manifests: - 10-xxx-ip.yaml: | apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: name: xxx-master labels: machineconfiguration.openshift.io/role: master spec: config: ignition: version: 2.2.0 storage: files: - contents: source: data:text/plain;charset=utf-8;base64,IyEvYmluL3NoCgpOSUM9YnItZXgKSVA9JChhd2sgLUYgJ3JlbG9jYXRlaXA9JyAne3N1YigvIC4qJC8sICIiLCAkMik7IHByaW50ICQyfScgL3Byb2MvY21kbGluZSkKTkVUTUFTSz0kKGF3ayAtRiAncmVsb2NhdGVuZXRtYXNrPScgJ3tzdWIoLyAuKiQvLCAiIiwgJDIpOyBwcmludCAkMn0nIC9wcm9jL2NtZGxpbmUpCgpncmVwIC1xICRJUCAvZXRjL05ldHdvcmtNYW5hZ2VyL3N5c3RlbS1jb25uZWN0aW9ucy8qICYmIGV4aXQgMAoKY29ubmVjdGlvbj0kKG5tY2xpIC10IC1mIE5BTUUsREVWSUNFIGMgcyAtYSB8IGdyZXAgJE5JQyB8IGdyZXAgLXYgb3ZzLXBvcnQgfCBncmVwIC12IG92cy1pZiB8IGN1dCAtZDogLWYxKQpubWNsaSBjb25uZWN0aW9uIG1vZGlmeSAiJGNvbm5lY3Rpb24iICtpcHY0LmFkZHJlc3NlcyAkSVAvJE5FVE1BU0sgaXB2NC5tZXRob2QgYXV0bwppcCBhZGRyIGFkZCAkSVAvJE5FVE1BU0sgZGV2ICROSUMK verification: {} filesystem: root mode: 448 path: /usr/local/bin/xxx.sh systemd: units: - name: xxx.service enabled: true contents: | [Unit] After=network.target ovs-configuration.service Before=network-online.target kubelet.service crio.service [Service] Type=oneshot ExecStart=/usr/local/bin/xxx.sh [Install] WantedBy=multi-user.target Ignoring validations ~~~~~~~~~~~~~~~~~~~~ For testing purposes, validations can be disabled when creating cluster by setting parameter ``ignore_validations`` to ``true`` Gather iso ~~~~~~~~~~ Once the cluster (and the corresponding infraenv) are created, we can get the discovery iso url using the following command :: aicli info iso mycluster or download it locally with :: aicli download iso mycluster Note that when AI api was in v1, a specific call ``create iso`` was needed to trigger the creation of the iso, but it’s no longer needed (the command is maintained for retrocompatibility but does the same as info iso\` When using this call, the expiration time of the token associated to the iso is checked and if necessary, it gets refreshed (and as such so does the url) Use a debugging user in the iso ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For troubleshooting purposes, typically when the host cant be accessed through ssh, you can add a user named aicli with password aicli in the discovery iso by running :: aicli update cluster -P password=true mycluster password boolean can be specified at cluster’s creation Wait for hosts ~~~~~~~~~~~~~~ After booting some nodes with the iso, we normally wait for them to show up in the UI or in ``aicli list hosts`` output. Alternatively, we can use the following command to wait for 3 hosts to appear in mycluster :: aicli wait hosts mycluster -n 3 Optionally Update hosts ~~~~~~~~~~~~~~~~~~~~~~~ Once we have enough nodes, we need them show as ``known`` in list hosts output in order to start the cluster deployment. It might be necessary to update some specific information of the nodes, such as the requested hostname (localhost name is forbidden) or to assign a specific role to the nodes Updating hostnames ^^^^^^^^^^^^^^^^^^ To change a specific host name, we can use the following :: aicli update host $host -P requested_hostname=new_name or simply :: aicli update host $host -P name=new_name If there are several matching hosts belonging to a same cluster, then the name is instead used as a prefix and the host names are sequentially assigned to name-0, name-1, …. That makes it easy to change all the localhost fqdns of your cluster with a single call Updating roles ^^^^^^^^^^^^^^ To change the role of a given host to worker, you can run :: aicli update $host -P role=worker Updating extra args ^^^^^^^^^^^^^^^^^^^ To specify extra args for a given host, you can run :: aicli update $host -P extra_args="xxxx" For instance, you can run the following to append kargs :: aicli update host $host -P extra_args="--append-karg=rd.multipath=default --append-karg=root=/dev/disk/by-label/dm-mpath-root" Specifying installation disk ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To specify installation disk for a given host, you can run :: aicli update $host -P disk="xxxx" For instance, To force installation disk to /dev/sdb, use :: aicli update host $host -P disk=sdb" Skipping formatting specific disks ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To avoid formatting of given disks, you can run :: aicli update $host -P skip_disks=[disk1,disk2] For instance, To skip formatting of disks /dev/sdb and /dev/sdc, use :: aicli update host $host -P skip_disk=[sdb,sdc]" Note: the parameter ``skip_formatting_disks`` can be used instead but you will need to provide a dict with disk_id and skip_formatting Specifying node labels ^^^^^^^^^^^^^^^^^^^^^^ To declare specific labels for a host, you can run :: aicli update $host -P labels=[label1,label2,key3=value3] For instance, To specify the label ``productionready`` for a host, use :: aicli update host $host -P labels=[productionready]" Note: the parameter ``node_labels`` can be used instead Applying custom ignition on first boot of a host ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We can use the extra keyword ``ignition_file`` and provide a path where to store the ignition directly and call :: aicli update host $host -P ignition_file=your_ignition_path" Updating from a parameter file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ You can specify in your parameter file a hosts array so that the information for updating hosts is gathered from there. For instance, if you have the following information in your parameter file :: hosts: - name: xxx.fantastic.com role: master - name: yyy.fantastic.com role: worker extra_args: "ip=dhcp6" Running ``aicli update hosts --paramfile my_params.yml`` will change the roles of the hosts with the corresponding name, if found, and add the specified extra_args for the second host. You can either use the name to match a host, or specify id (to match system uuid) or mac (to match any mac address from the host inventory) so that the name of the host can be forced. When using the create deployment workflow, update hosts is run as one of the steps. Updating cluster ~~~~~~~~~~~~~~~~ At this step, you might need to update cluster data so that the cluster is ready to install. For instance, you might want to specify api vip and ingress vip now that hosts cidrs have been discovered. For this, you can run :: aicli update cluster -P api_ip=$api_ip -P ingress_ip=$ingress_ip mycluster Launch cluster deployment ~~~~~~~~~~~~~~~~~~~~~~~~~ Once your hosts all show as known, the cluster status should appear as ready in ``aicli info cluster mycluster`` At this point, you can trigger the deployment using the following command :: aicli start mycluster Monitor deployment ~~~~~~~~~~~~~~~~~~ Wait for cluster ^^^^^^^^^^^^^^^^ When the cluster is installing, you can wait for it to complete using the following command :: aicli wait mycluster Monitor events ^^^^^^^^^^^^^^ You can also see all events associated to your cluster using :: aicli get events cluster Gather assets ~~~~~~~~~~~~~ Once installation has started, you can gather relevant assets for your cluster such as - kubeconfig - kubeadmin-password - installconfig For instance, to gather the kubeconfig, you can use the following to get it downloaded to your current directory as ``kubeconfig.mycluster`` :: aicli download kubeconfig mycluster Add extra workers ~~~~~~~~~~~~~~~~~ For this purpose, we assume we already have the cluster installed. If we create a new cluster with the same original name and the ‘-day2’ extension, the api code will create a dedicated cluster for adding host purposes. :: aicli create cluster mycluster-day2 Alternatively, we can convert the original cluster into one to use for adding hosts by using the following call but note this is not the recommended approach :: aicli update cluster mycluster -P day2=true Note that when creating the day2 cluster, a DNS check on api_vip_dnsname is done. If it doesn’t succeed and the base cluster is HA, then api vip is used instead of fqdn to guarantee functionality You can also update manually this data using the following command :: aicli update cluster mycluster-day2 -P api_vip_dnsname=$api_ip In both cases, once we have the day2 cluster in the proper state, the same workflow is to be used: - gather the iso associated to this cluster/infraenv with ``aicli info iso mycluster-day2``. - boot nodes with this iso. - wait for them to show in ``aicli list hosts`` output as *known* - launch ``aicli start cluster mycluster-day2`` Individual hosts installation can also be triggered by calling: :: aicli start hosts $host1 $host2 ... Deployment workflow ------------------- Instead of deploying the cluster step by step, you can put all the relevant information in your parameter file and then have all the steps run for you You can use a command such as the following one :: aicli create deployment --paramfile my_params.yml myclu The parameter file could be similar to the following one :: base_dns_domain: karmalabs.com api_vip: 192.168.122.253 ingress_vip: 192.168.122.252 download_iso_path: /var/www/html download_iso_cmd: "chown apache.apache /var/www/html/ci-ai.iso" iso_url: http://192.168.122.1/ci-ai.iso bmc_user: admin bmc_password: password hosts: - name: ci-ai-ctlplane-0 bmc_url: http://192.168.122.1:8000/redfish/v1/Systems/11111111-1111-1111-1111-111111111181 - name: ci-ai-ctlplane-1 bmc_url: http://192.168.122.1:8000/redfish/v1/Systems/11111111-1111-1111-1111-111111111182 - name: ci-ai-ctlplane-2 bmc_url: http://192.168.122.1:8000/redfish/v1/Systems/11111111-1111-1111-1111-111111111183 Note that in this case, we are providing bmc information for our hosts so that they get booted with the discovery iso automatically. We also have the iso downloaded automatically to a path corresponding to a web server If you omit this kind of information, you can still have the deployment done semi automatically by just waiting for the iso url to be displayed and plug it manually to your target nodes. Ansible Integration ------------------- aicli modules are available at https://github.com/karmab/ansible-aicli-modules and provide the following primitives: - ai_cluster - ai_cluster_info - ai_infraenv - ai_infraenv_info - ai_host_info