> For the complete documentation index, see [llms.txt](https://experience-digital-transform.gitbook.io/experience-digital-transformation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://experience-digital-transform.gitbook.io/experience-digital-transformation/sovereign-cloud-stack/2.-scs-cluster-stacks-course/2.3-configuration-customization-and-building-your-own-cluster-stacks.md). # 2.3 Configuration, Customization and Building Your Own Cluster Stacks ## Configuration and Customization Cluster stacks can be configured with env variables. [Envsubst](https://github.com/drone/envsubst) is required to expand environment variables specified in CSO: ```bash GOBIN=/tmp go install github.com/drone/envsubst/v2/cmd/envsubst@latest ``` A user can configure their own repository as a clusterstack source, e.g.: ```bash export GIT_PROVIDER_B64=Z2l0aHVi # github export GIT_ORG_NAME_B64=U292ZXJlaWduQ2xvdWRTdGFjaw== # SovereignCloudStack export GIT_REPOSITORY_NAME_B64=Y2x1c3Rlci1zdGFja3M= # cluster-stacks export GIT_ACCESS_TOKEN_B64=$(echo -n '' | base64 -w0) ``` To use a custom OCI registry, e.g.: ```bash # echo -n "registry.scs.community" | base64 -w0 export OCI_REGISTRY_B64=cmVnaXN0cnkuc2NzLmNvbW11bml0eQ== # registry.scs.community/kaas/cluster-stacks export OCI_REPOSITORY_B64=cmVnaXN0cnkuc2NzLmNvbW11bml0eS9rYWFzL2NsdXN0ZXItc3RhY2tzCg== ``` Deploy CSO — the variables will be replaced in the file: ```bash # Get the latest CSO release version and apply CSO manifests curl -sSL https://github.com/SovereignCloudStack/cluster-stack-operator/releases/latest/\ download/cso-infrastructure-components.yaml | /tmp/envsubst | kubectl apply -f - ``` A cluster can be further configured by the `spec.topology.variables` field, e.g.: ```yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: namespace: labels: managed-secret: cloud-config spec: clusterNetwork: pods: cidrBlocks: - 192.168.0.0/16 serviceDomain: cluster.local services: cidrBlocks: - 10.96.0.0/12 topology: variables: # <-- variables can be set here - name: controlPlaneFlavor value: "SCS-2V-4-20s" - name: controlPlaneRootDisk value: 0 - name: workerFlavor value: "SCS-4V-16" - name: networkExternalId value: "ebfe5546-f09f-4f42-ab54-094e457d42ec" class: openstack-scs2-1-35-v2 controlPlane: replicas: 3 version: v1.35.9 workers: machineDeployments: - class: openstack-scs2-1-35-v2 failureDomain: nova name: openstack-scs2-1-35-v2 replicas: 4 ``` For current `openstack-scs2` variables, use the [Cluster Stacks source template](https://github.com/SovereignCloudStack/cluster-stacks/blob/main/providers/openstack/scs2/cluster-class/templates/cluster-class.yaml). The configuration table in the SCS docs currently covers the older `openstack-scs` cluster class. ### Finding out about Cluster Stack releases from SCS Published versions are stored anonymously in the [SCS registry](https://registry.scs.community/) under `kaas/cluster-stacks`. ```bash oras repo tags registry.scs.community/kaas/cluster-stacks ``` You can inspect a release bundle with, for example: ```bash mkdir openstack-scs2-1-35-v1 oras pull registry.scs.community/kaas/cluster-stacks:openstack-scs2-1-35-v1 \ -o openstack-scs2-1-35-v1 ``` Inspect `metadata.yaml` and the other files in the bundle to see what constitutes the Cluster Stack. **Assignments:** (1) use environment variables to set your own registry from the Quickstart guide assignments as source; (2) push needed images; (3) create a workload cluster using your own registry as a source. ## GPU Support in Cluster Stacks This assumes the underlying OpenStack cloud exposes GPU-capable flavors such as `SCS-8V-32_GNa-72-24` (8 vCPUs, 32 GiB RAM and an NVIDIA Ampere GPU passed through to the VM). GPU worker nodes need suitable drivers. You can inject additional software into generic node images during deployment or build specialized images. Kubernetes also needs a device plugin—for NVIDIA GPUs, the [NVIDIA Device Plugin](https://github.com/NVIDIA/k8s-device-plugin)—so GPU resources appear on the node and can be requested by workloads: ```yaml apiVersion: v1 kind: Pod metadata: name: gpu-workload spec: containers: - name: cuda-container image: nvidia/cuda:12.0-base resources: limits: nvidia.com/gpu: 1 ``` ## Building your own Cluster Stacks This section describes how to develop your own Cluster Stack release from scratch. ### Directory structure Create a new directory, e.g. `my-clusterstack`. Inside this directory create the following sub-directories: * `cluster-class` — containing the helm chart for Cluster Class and CAPI related resources. * `cluster-addon` — containing helm charts to be installed as cluster addons, e.g. a CNI of your choice or metrics-server. * `node-image` (optional) — containing files to build your cluster's node-image (not used in this example). ### Cluster Class Create the directory: ```bash mkdir -p cluster-class/templates ``` Create a `Chart.yaml`, e.g.: ```yaml apiVersion: v2 description: "This is my Cluster Stacks Cluster Class" name: docker-my-cluster-class-1-37-cluster-class type: application version: v1 ``` Generate resources using the clusterctl generator, e.g.: ```bash clusterctl init --infrastructure docker cd templates clusterctl generate cluster my-cluster --infrastructure docker --flavor development \ --kubernetes-version v1.37.0 --control-plane-machine-count=3 --worker-machine-count=3 \ > my-cluster.yaml ``` * Split the file into different yaml files, e.g. `Kind: Cluster` should be in `cluster.yaml`. * Move `kind: Cluster` outside of the `cluster-class` directory, e.g. `mv cluster.yaml ../../cluster.yaml`. * Remove `my-cluster.yaml`: `rm my-cluster.yaml`. * Escape all `{{ }}` so that they are not interpreted by Helm: `sed -i 's/{{/{{ \`{{/g;s/}}/}}\` }}/g' \*.yaml\`. * Verify with `helm template .`. ### Cluster Addon Initialize the cluster addon directory: ```bash cd .. mkdir cluster-addon cd cluster-addon ``` Create an umbrella helm chart for every cluster addon, e.g. cilium: ```bash helm create cni cd cni rm -rf templates ``` Add Cilium as a dependency to the `Chart.yaml` file: ```yaml dependencies: - alias: cilium name: cilium repository: https://helm.cilium.io/ version: 1.20.3 ``` Build dependencies: ```bash helm dep build ``` When all dependency helm charts are created, go to the project main directory and create `clusteraddon.yaml`, e.g.: ```yaml apiVersion: clusteraddonconfig.x-k8s.io/v1alpha1 clusterAddonVersion: clusteraddons.clusterstack.x-k8s.io/v1alpha1 addonStages: AfterControlPlaneInitialized: - name: cni action: apply BeforeClusterUpgrade: - name: cni action: apply ``` ### Build a clusterstack release using csctl Download CSCTL and unpack: ```bash wget https://github.com/SovereignCloudStack/csctl/releases/download/v0.0.5/\ csctl_0.0.5_linux_amd64.tar.gz tar -xzf ~/Downloads/csctl_0.0.5_linux_amd64.tar.gz chmod u+x csctl sudo mv csctl /usr/local/bin/csctl ``` Configure csctl — the configuration has to be specified in `csctl.yaml`, following this structure: ```yaml apiVersion: csctl.clusterstack.x-k8s.io/v1alpha1 config: kubernetesVersion: v1.37.0 clusterStackName: my-clusterstack provider: type: docker apiVersion: docker.csctl.clusterstack.x-k8s.io/v1alpha1 ``` Verify the file/directory structure: ``` my-clusterstack/ cluster-class/ templates/ clusterclass.yaml ... Chart.yaml cluster-addons/ cni/ charts/ cilium-1.20.3.tgz Chart.yaml clusteraddons.yaml csctl.yaml cluster.yaml ``` Setup your repository to publish your cluster stack, e.g. for a git repository: ```bash export GIT_PROVIDER=github #Only github supported export GIT_ORG_NAME= export GIT_REPOSITORY_NAME= export GIT_ACCESS_TOKEN= ``` For an OCI repository, e.g.: ```bash export OCI_REGISTRY= export OCI_REPOSITORY= export OCI_ACCESS_TOKEN= export OCI_USERNAME= export OCI_PASSWORD= ``` Create your cluster stack, e.g.: ```bash csctl create . --output ../my-clusterstack-build --mode hash ``` If you want to publish, you can do so by e.g.: ```bash csctl create . --output ../my-clusterstack-build --mode hash --publish --remote oci ``` **Assignments:** (1) create your own github repository to be used as a source for your own cluster stack implementations; (2) use environment variables to set the cluster stack source to this repository; (3) create your own cluster stack implementation with alternative node images; (4) push the implementation to your own github repository; (5) create a workload cluster using this cluster stack. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://experience-digital-transform.gitbook.io/experience-digital-transformation/sovereign-cloud-stack/2.-scs-cluster-stacks-course/2.3-configuration-customization-and-building-your-own-cluster-stacks.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.