> For the complete documentation index, see [llms.txt](https://experience-digital-transform.gitbook.io/experience-digital-transformation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://experience-digital-transform.gitbook.io/experience-digital-transformation/sovereign-cloud-stack/1.-virtualization-layer-operating-scs-iaas/1.8-1.9-maintenance-dealing-with-errors-updates-and-upgrades.md).

# 1.8–1.9 Maintenance, Dealing with Errors, Updates and Upgrades

## 1.7 Maintenance and Dealing with errors

There are some maintenance tasks needed for an environment to keep it working well long-term, and some error scenarios that need operator attention.

### Database cleanup

**Nova**

* Deleted Servers (VMs) and Volumes are *not* removed from the database tables by OpenStack — they are just marked as deleted, with the deletion time stored (as was the creation time). You could use those records for billing purposes (a second source of truth, allowing you to double-check records created from ceilometer/gnocchi; you would automatically clean them up in your billing runs).
* Once you've confirmed you don't need them any longer, you can remove them from the database. As admin, you can ask openstack about deleted servers — beware, the list may be huge. There isn't a good way to tell openstack to sort by deletion time, so you might need to retrieve the complete list and start by looking at the bottom.

```bash
# Getting 100 old deleted servers (the `tr -d` is for CiaB container extra CR)
DELETED=$(openstack server list --all --deleted -c ID -f value | tr -d '\r')
```

* It is recommended to use `nova-manage` to move out old entries to shadow tables (for later deletion) — see [nova/2024.2/cli/nova-manage.html](https://docs.openstack.org/nova/2024.2/cli/nova-manage.html).

**Example: Delete deleted servers directly in the database (up till Apr 30 23:59:59 UTC)**

Seasoned SQL admins will surely find more efficient ways to do this with joins and the like. More tables might have foreign key references to the `instances` table, so you might add more removal statements. The source docs show removal statements against the `instance_actions_events`, `instance_actions`, `instance_system_metadata`, `instance_metadata`, `block_device_mappings`, `instance_info_caches`, `instance_extra`, `virtual_interfaces` and finally `instances` tables in the `nova` database, filtered by `deleted_at IS NOT null AND deleted_at < "2025-05-01 00:00:00"` (joined to the `instances` table where needed to find the right rows).

**Cinder Volumes**

* Like Nova, cinder only marks volumes as deleted in the database and records the deletion time, but does not remove the table entries. The openstack CLI does not report them at all — no `--deleted` option. It is recommended to use the `cinder-manage` tool, see [cinder/2024.2/cli/cinder-manage.html](https://docs.openstack.org/cinder/2024.2/cli/cinder-manage.html).
* Database example (emergency): analogous removal statements against the `volume_attachment`, `volume_glance_metadata` and `volumes` tables in the `cinder` database, filtered the same way by `deleted = 1 and deleted_at < "2025-05-01 00:00:00"`. Similar for snapshots and backups. Glance behaves similarly, although deleted images do not tend to emerge in huge numbers.

**Learning** — you need to do regular DB maintenance work with `nova-manage` and `cinder-manage`; these can be part of your regular billing runs. You don't normally tweak database tables directly — consider the above examples as an exception, where you will need to very carefully document and review what you do.

### Stuck volumes

* Occasionally you will find cinder volumes that users can neither use nor clean up — they could have a `reserved` or `deleting` status for extended amounts of time (many minutes), or be reported as attached to a VM that has long gone (and of which you'll only see the UUID, not the name). The cinder service seems to not be very robust if processing of volume changes falls behind in high (control-plane) load situations — a longstanding OpenStack issue.
* Approach: set status to error as admin:

```bash
openstack --os-cloud=admin volume set --status error UUID
```

Now you can clean them up. If this does not work: check in Ceph whether there is an object in the `images` or `volumes` pool that needs deleting (be very careful!) — if it's not gone yet, you may need to remove it from the database as well. Charging customers for an unusable volume may not create enthusiastic responses.

### Stuck LoadBalancers

* Loadbalancers can get stuck in `PENDING_CREATE` or `PENDING_DELETE` — stuck means they remain in this state for more than a minute (and then never make any progress). This happens relatively often with the amphora provider (driver); fortunately the ovn provider does not expose this bad behavior much.
* These cannot be cleaned up by the user; there is no CLI command for the admin to set them to error or delete them either. The solution is to go into the database and set them to error for deletion. Charging customers for stuck loadbalancers may not create enthusiastic responses, so avoid it.
* Known bug in octavia Loadbalancer (affecting the ovn provider): the `octavia_api` container leaks file descriptors — see [github.com/osism/issues/issues/959](https://github.com/osism/issues/issues/959). Automated occasional (nightly) restarts of the `octavia_api` container will help avoid customer impact (alternatively monitor the FD count or the octavia\_api availability and restart when the problem approaches/arises).

### RabbitMQ issues

* If your rabbitMQ process is starved of resources, it might fail to deliver all messages; subscribers can lose connections to rabbitMQ. The result is that backend actions are not taken, and while the API services may happily accept requests, the requested actions never make any progress. See the Cinder volume create failure guide to see how to detect cinder-rabbit issues.

### Power loss on storage

* If your complete (Ceph) storage subsystem goes down while VMs are running and writing to storage, this results in bad behavior — a typical situation after a power outage, but also when a Cloud-in-a-Box gets shut down without stopping all VMs first. The result is ugly: VMs will typically no longer boot; checking the console log (`openstack console log show` or noVNC from horizon) shows their root disks are read-only (cinder/Ceph refusing to let them write).
* Recovery: drop locks from rbd devices that correspond to your volume.

### Recovering from locks held after surprise Ceph shutdown

* This affects volumes in the `volumes` pool with name `volume-${VOLUUID}` and "local" root disks in pool `vms` named `${VMUUID}_disk` (if the instance's root disks are stored on Ceph). Cinder locks these disk files (unless you use `multiattach: True`) in Ceph RBD so they can't be accessed read-write by more than one VM (thus the read-only character of your root disk).

**Example with a running VM booted from volume `82b99323-3b69-4b05-9cb9-770324235099`:**

```bash
dragon@cumulus(test):~ [0]$ rbd lock list volumes/volume-82b99323-3b69-4b05-9cb9-770324235099
There is 1 exclusive lock on this image.
Locker         ID                               Address
client.4105767 auto 139984392802144  192.168.16.10:0/3228397893
dragon@cumulus(test):~ [0]$ rbd lock rm volumes/volume-82b99323-3b69-4b05-9cb9-770324235099 \
    "auto 139984392802144" client.4105767
dragon@cumulus(test):~ [0]$ rbd lock list volumes/volume-82b99323-3b69-4b05-9cb9-770324235099
dragon@cumulus(test):~ [0]$
```

* Don't do this on a volume backing a running server unless you want to see its volume turn read-only. Shut it down (`openstack server stop`) and wait for it to be in `SHUTDOWN` state. On a Cloud-in-a-Box, you can use `/usr/local/bin/shutdown-instances.sh` to force-stop all VMs.

**Troubleshooting Guide** — there is a Troubleshooting Guide available in the SCS IaaS docs with information on trouble with the Manager, OpenStack database, Ceph connection, cinder rabbit trouble, and Ceph medium errors.

**Practical assignment** — trainer will create volumes that are attached to already-gone VMs or stuck in `reserved`. Recover them; watch out to not leak storage space.

## 1.8 Updates and Upgrades

### Security Updates

* Be prepared to install security updates short-term. Typically this involves replacing one or a few service containers with fixed versions — will typically *not* affect running workloads, but might stop control plane (API) access for a number of seconds. It's best practice to notify customers, even if most will never notice.
* Security updates affecting the SCS reference implementation OSISM are published as Blog articles; known trusted partners can get a pre-warning (for coordinated releases).

**Security update example** — see the security advisory on CVE-2024-40767: it documents the vulnerability and under what circumstances providers may be affected; provides information to assess potential attacks and provides mitigation and fixes.

Fixing approach:

```bash
# Download new containers (after manually changing the wanted tag for the nova compute
# container in environments/kolla/images.yml in the configuration repository)
osism apply -a pull nova -l compute

# Restart the containers
osism apply -a refresh-containers nova -l compute
```

### Rebooting systems

* Sometimes hardware needs to be shut down and rebooted — when doing hardware maintenance (e.g. exchanging RAM), deploying new firmware, or installing fixes to the host kernel or hypervisor.
* To avoid disruption of user workloads, OpenStack supports live-migration (host evacuation). Limitations: does not normally work if you expose PCI-pass-through to special hardware (e.g. GPUs); the scheduler (placement) service selects a new host according to the normal scheduling rules (e.g. affinity groups) — ensure you have enough hosts available.
* Performance: works by copying RAM content over in several passes (delta migration) — workloads doing lots of writes to memory will take a while. VMs with only networked volumes (or "local" storage backed by Ceph) don't have trouble moving disk contents; really local storage needs a block-migration service that can be slow to copy disks over. The VM will run a bit slower during the copy process and may be down for a few seconds during the final copy/restart. Gratuitous ARP packets should inform switches of the new location of VM ports, so network disruption should be short as well.
* Rebooting a large set of compute hosts can still be tedious — ensure enough capacity when parallelizing the reboot process; ensure proper error handling when host evacuation fails; announcing this to customers beforehand (as a time of higher risk/lower performance) will be appreciated. See the hints on how to take out and evacuate a compute node in the operations guide.

### Version upgrades

* The SCS reference implementation has a new release twice a year (to follow upstream OpenStack releases) — you need to be prepared to upgrade within the weeks after the release to get full community support. The community has worked with providers successfully ahead of the release to ensure the upgrade works smoothly.
* Highly advisable to test this within test and reference environments. Downloading all new container images (`osism apply -a pull`) prior to deploying them will reduce the downtime. The process will typically involve live-migrations and a reboot for each node. There is an Upgrade Guide you should read and study — the below is just an overview; if upgrading to a new OSISM release, also read the version-specific release notes.

**Typical sequence:**

```bash
# Update the OSISM manager node
osism update manager

# Update docker, traefik, netbox and ensure the facts are in sync
osism update ...
osism reconciler sync
osism apply facts

# Download container images
osism apply -e custom pull-container-images

# Update infrastructure services
set -e
for svc in common loadbalancer opensearch openvswitch ovn memcached redis \
           mariadb rabbitmq; do
  osism apply -a upgrade $svc
done

# Update OpenStack services (the exact list depends on your deployment)
set -e
for svc in keystone horizon placement glance neutron nova cinder designate \
           octavia skyline barbican heat magnum; do
  osism apply -a upgrade $svc
done

# Update prometheus and grafana
osism apply -a upgrade prometheus
osism apply -a upgrade grafana

# Ensure other containers are in sync
set -e
for svc in phpmyadmin cgit homer netdata openstackclient kubernetes clusterapi; do
  osism apply $svc
done

# Clean up unneeded containers
osism apply cleanup-docker-images -e ireallymeanit=yes
```

* It is highly recommended to script the upgrade and test this extensively in your reference/preproduction environment — you may use the testbed scripts as a template.

### Adding and removing compute nodes

* **Removing:** evacuate compute hosts.
* Adjust the inventory.
* Apply all services with `osism apply` to roll out the change.

**Practical assignment** — upgrade a service.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://experience-digital-transform.gitbook.io/experience-digital-transformation/sovereign-cloud-stack/1.-virtualization-layer-operating-scs-iaas/1.8-1.9-maintenance-dealing-with-errors-updates-and-upgrades.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.