> For the complete documentation index, see [llms.txt](https://experience-digital-transform.gitbook.io/experience-digital-transformation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://experience-digital-transform.gitbook.io/experience-digital-transformation/sovereign-cloud-stack/1.-virtualization-layer-operating-scs-iaas/1.10-supporting-users.md).

# 1.10 Supporting Users

## Responsibilities and Data protection considerations

* In a public cloud model, the cloud operator has **no** business looking into the virtual machines and what customers do there. In many jurisdictions, data protection considerations would prevent the provider's operators from doing so. This creates a very clear delineation of responsibilities: the cloud provider makes sure the virtual infrastructure and its automation via APIs work; what happens inside VMs is the customer's responsibility (assuming the infra works).
* This delineation makes sense even in a private cloud model where the customers may be in the same company.
* In support situations, customers may ask the provider to support within the VM — this is not a standard service and may require special arrangements, including around data protection (commissioned data processing), fees, etc.
* It does make sense for a cloud provider to unilaterally provide information to help customers within their VMs — this may include public images that work well and are regularly updated (the SCS Standard requires a few of them even), and documentation. The following information is written from a user perspective.

## Typical user challenges

### Floating IPs

* Making a VM reachable from the outside typically requires several steps on OpenStack: connecting the network port to a network that is connected to a router that is connected to the outside network; allocating a floating IP from the outside network and attaching it to the network port of that VM; ensuring security groups allow (incoming) traffic to that network port; the VM itself may have firewalling functionality that needs to be adjusted (or disabled).
* Floating IPs are **not** visible from within the VM — you just see the fixed IP address of the network port. Traffic from the outside is DNAT'ted and appears to be directed to the network port from within the VM.

### Security Groups

* A security group is a list of ALLOW rules for a group of network ports. Most security groups allow all outgoing traffic (related packets — responses to outgoing packets — are allowed back in). Many security groups allow all incoming traffic *from the same groups*, meaning all ports inside the group can freely communicate with each other (thus the name group).
* Behind the scenes, when referring to a remote security group, the IP addresses belonging to that group are whitelisted. This is secure due to port security, which means network ports cannot spoof their source IP address (port security can be disabled per port if you want to e.g. create VMs that do advanced networking things such as NAT — be aware such ports can then circumvent security group rules by spoofing their source IP address).
* Adding a port to a security group referenced by this or another security group can mean updating a lot of network filtering rules — this can limit scalability of network operations for setups with lots of VMs; consider assigning IP ranges and filtering for them instead in such setups.

### cloud-init

* While clouds (OpenStack is no different from AWS here) typically support users creating and using their own private images, they provide mechanisms to use generic images and customize them on (first) boot. Customization is done via metadata: `user_data` by the user (and `vendor_data` by the cloud provider, seldom used), which the generic image then picks up and uses to adjust its behavior.
* This can be used to inject public ssh keys (as `authorized_keys`), update/install software, run scripts, create users, etc. This allows doing everything with generic images; by doing all customization on (first) boot, the customer avoids the infamous golden images that get more and more outdated over time.
* Two well-known mechanisms to provide metadata to the VM: attaching a virtual drive (CD-Rom typically, used to be a Floppy Disk) and providing an http network source at the standard OpenStack metadata service address. The software that processes the metadata and does the customization is typically **cloud-init** on Linux — read the documentation to get an idea of all the possibilities. On Windows, **cloudbase-init** provides similar functionality.

### Keypairs

* One thing almost always required on first boot of Linux VMs is the injection of an authorized public ssh key, so the owner of the VM can login and investigate/configure things (for production VMs, it may be best practice to not allow any login).
* The key injection is done by OpenStack providing the keypair selected in horizon or via the API in the metadata, which cloud-init then picks up.
* It's best to just register your public ssh key with OpenStack to make it available. You can have OpenStack create key pairs for you — this is deprecated. OpenStack does **NOT** store the secret key for you; it is displayed once upon generation, grab and store it and keep it safe. There is no way to recover a lost secret key, so you might lose ssh access to your VMs. Most cloud images do not allow a console login by default — OpenStack offers a novnc console for VMs for emergency access if you configured your VM to allow local password-authenticated logins.

### Troubleshooting boot issues (self-support)

* When a created VM does not become accessible, this can be debugged in a number of ways:
  * Most Linux images write the boot log to the serial console — captured by OpenStack, accessible via horizon or `openstack console log show`.
  * Most Linux and Windows images produce output to a virtual VGA graphics card — accessible via horizon or by pointing a web browser to the URL given by `openstack console url show`. Many images also allow input this way, so you could log in if local console logins are allowed.
* If you created your own image, you may be able to observe trouble from the boot-loader trying to locate/load the operating system — this does not typically happen on generic images provided by the provider.
* You may see trouble with your root file-system that suffered corruption on a crash — recoverable by attaching the volume to another VM and using file-system recovery tools. For local storage, you can boot with a rescue image (`openstack server rescue`).
* Your volume may be read-only and reject WRITE requests — this is due to locks held when Ceph went down and needs support from the cloud provider (see the Maintenance section above).
* Your VM might have booted perfectly and you just fail to reach it via the network — double check network settings: do you have a floating IP assigned or another VM attached to the same router to access it? Is the service (ssh) running and the port allowed in firewall and security group rules?
* You may reach it via ssh but it rejects your login — did you inject the right public ssh key? If the metadata service was broken on first VM boot, the ssh key might not have been injected — throw away the VM and recreate it, assuming no valuable data is held on a local root volume. Did you specify the right private ssh key when connecting? Use `ssh -v` for more debug output; beware that ssh-agents may offer added keys ahead of the one you just specified.
* Note that username/password authentication via ssh is not normally enabled in cloud VMs due to security considerations.

### Snapshots and backups

* **Definitions:** a snapshot is a point-in-time state of a storage device; a backup is a copy of a storage device.
* **Snapshots** are typically created within the same storage backend — this allows for quick and cheap creation using copy-on-write (beware that the state of a read-write mounted volume is not necessarily consistent — `sync` helps — due to dirty caches). Useful to protect against user errors or failed application changes, etc. The cost of snapshots increases as they age (as they diverge more).
* **Backups** should be stored on a different storage backend, thus also providing protection against storage backend breakage. Typically slow to create as full copies are created. Snapshots can be backed up (by creating a temporary volume) — for read-write mounted volumes this is the only way to have a backup that at least comes close to a consistent file-system. OpenStack even offers incremental backups, only saving the block-level changes since the last backup.

### IaC tooling

* Cloud-natives do not create VMs via the (horizon or skyline) web interface and then login to manually configure them — instead they automate the process to make it repeatable and less prone to errors. All creation of virtual infrastructure can be automated via the OpenStack set of APIs; all configuration inside the VM can be bootstrapped by cloud-init, using a configuration management system (such as ansible) to take over.
* The OpenStack APIs can be used by a number of Infra-as-Code tools: `openstackclient` command line interface; python OpenStackSDK; Ansible; opentofu (a free fork of the now-proprietary terraform); etc. Of these, the latter ones are the most popular (some people prefer pulumi over the declarative style of opentofu).
* Inside the VMs, small configuration tasks are typically done directly via cloud-init; larger things typically via ansible (other config-mgmt tools such as chef, puppet, saltstack, etc. are possible). Fully automating the deployment makes testing, upgrading, recovering from incidents, etc. a lot easier and in general allows development speed and test coverage to increase by factors — this is probably *the* main benefit of cloud computing.

**Using the OpenStack API**

* Most tools assume a `clouds.yaml` style configuration — you keep two config files, `~/.config/openstack/clouds.yaml` and `~/.config/openstack/secure.yaml`, where you can have several entries, one per OpenStack project. (Reminder: an OpenStack project is a workspace for virtual resources with associated roles/user authorizations in a cloud.)
* The `secure.yaml` file is for your passwords and secrets and must be kept safe; by doing so, you can freely show the `clouds.yaml` to others. Tools do not enforce this separation — it is possible (but not recommended) to put all into `clouds.yaml`. Some people (aggrieved by Windows?) prefer `clouds.yml` (without the `a`), which works as well. The openstack CLI looks for `clouds.yaml` first in the current directory, then in `~/.config/openstack`, then `/etc/openstack/`.
* You can `export OS_CLOUD=$NAME` to make the openstack CLI use the cloud project with that name, avoiding passing `--os-cloud=$NAME` all the time. The old-style `OS_` environment variables (`.openrc` config style) are not recommended, though still fully supported by the openstack-sdk and the openstackclient CLI. Horizon offers users the ability to download openrc and clouds.yaml style config files.

**`clouds.yaml`/`secure.yaml` example with username/password auth** — `clouds.yaml` sets, per named cloud entry (e.g. `gxscs2-community`), the `region_name`, `interface: public`, `identity_api_version: 3`, and under `auth`: the `auth_url`, `project_name`, `user_domain_name` and `project_domain_name` (or, alternatively, a single `project_id` in place of the name/domain pair). The matching `secure.yaml` holds only the sensitive `auth.username` and `auth.password` values for that same cloud entry, kept out of the shareable `clouds.yaml`. In most clouds the user-management and project domains are the same, and if a user has access to several projects, a specific one can be selected with `--os-project-id=` on the CLI without adding a new `clouds.yaml` entry.

**`clouds.yaml`/`secure.yaml` example with application credentials** — `clouds.yaml` sets `region_name`, `interface: public`, `identity_api_version: 3`, `auth_type: v3applicationcredential`, and `auth.auth_url`; `secure.yaml` holds the matching `auth.application_credential_id` and `auth.application_credential_secret`. Remember that indentation is relevant for YAML files. The application credential is valid for one project (selected during creation) and can be limited in validity. By default, appcreds are `restricted`, meaning appcreds cannot be used to authenticate to create more appcreds.

### Stateful vs Stateless design

* Cloud native deployments seek to make most or better all VMs stateless. Stateless means you can throw them away without losing any valuable data. Think of a web application where all configuration comes from the outside (git and vault) and all data is stored in a database — none of the VMs here, neither frontend web servers nor business logic, hold any persistent data.
* Stateless VMs — if automated well — have many advantages: you can typically horizontally scale them on demand (with the help of a loadbalancer); you can throw them away in case of a fault or security incident and create new ones (in practice you might want to save artifacts for analysis/forensics before really throwing away); much less trouble with upgrade, rollback, validation or A/B testing; you don't need to think about backups.
* Stateless VMs are sometimes called **cattle** — in contrast to **pets** (or snowflakes) that are manually maintained VMs that need operator care, backups, etc. If some VMs (backend, database) hold persistent data, treat them differently.

### Scaling

* Horizontal scaling (using more VMs to work in parallel) is typically easy to do with stateless VMs.
* Vertical scaling (creating larger VMs) is supported by OpenStack — you can hot(un)plug disks and network ports into running VMs; changing the number of vCPUs or the amount of memory requires a VM resize operation (causes a reboot of the VM — don't forget to confirm the resize after the rebooted VM has been checked).
* Autoscaling (horizontal scaling) in OpenStack can be done via heat orchestration templates — not very popular, and SCS standards do not require this functionality to be present. Many workloads that would benefit from autoscaling are containerized and use the Kubernetes mechanisms for autoscaling instead.

### Security incidents (provider perspective)

* As a public cloud provider, beware that you will have people sign up for your cloud with stolen credit card data — during the heydays of crypto mining, these could be more than 50% of new customers. You might want to use payment providers that do some checking for you; you might want to limit quota that you assign initially and increase them upon confirming your customer is legitimate.
* Your customers may fail to secure their virtual environments accurately — if they are hacked into, this does *not* in any way breach your integrity as a public cloud provider. By design, you can have evil customers next to legit ones, protected from each other. While a high degree of security isolation is a condition sine qua non for any public cloud, performance isolation may not be as perfect, especially if you allow over-subscription (SCS Standards allow for moderate CPU over-subscription for V-type vCPUs). Without any throttling, evil users might also cause a large load on your control plane (API), so API response times for legit customers may deteriorate.
* In practice you still want to get rid of evil customers and help your legit customers secure their environment, to avoid reputational damage (and of course you want your bills to be paid). Your public images and configuration advice should obviously be suitable to keep your customers secure.
* Providing transparency to your customers in case of security incidents may help (re)establish trust — this is best practice and the SCS community may support you in the right way to publish root cause analysis.
* You will want to have some DoS/DDoS protection in front of the OpenStack API and the public network used by VMs, to protect yourself and your customers — large DDoS attacks are hard to defend against; you will only win if you have more bandwidth than the attacking machines combined, typically only achievable with the help of your internet network provider.

## Practical assignments (user perspective)

* Download a storage volume from a stopped VM. Hint: there are several ways. Hint: OpenStack does not offer downloading volumes, but images. Hint: you can dynamically attach (and detach) volumes (and ports) to running VMs.
* Clone a VM.
* Boot a rescue image.
* Create snapshots and backups from volumes and restore them.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://experience-digital-transform.gitbook.io/experience-digital-transformation/sovereign-cloud-stack/1.-virtualization-layer-operating-scs-iaas/1.10-supporting-users.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.