Database Management

This page explains how to provision, migrate, and maintain Abacá's dedicated relational database. Abacá uses a MySQL/MariaDB (Galera) instance exclusively for bookkeeping metadata — targets, policies, jobs, and worker fleet state — while all backup data lives in S3. Schema migrations are applied by abaca-manage db_sync, which runs as a Kubernetes Job (abaca-db-sync) before the API and conductor pods start, ensuring the schema is always in the correct state before services accept traffic. Understanding this page is essential before deploying the control plane and after any upgrade that carries schema changes.

Before working with Abacá's database, ensure you have:

• RHOSO ≥ 18 with the abaca OpenShift namespace created
• OpenShift ≥ 4.14 with oc CLI authenticated to the cluster
• A dedicated MySQL/MariaDB (Galera) instance for Abacá — never share the OpenStack control plane's Galera cluster
• The abaca-config-data Kubernetes Secret already rendered (produced by the identity and config steps that precede deployment), containing a valid abaca.conf with the [database] connection string
• The abaca-ca-bundle Kubernetes Secret present in the abaca namespace (created by deploy/rhoso/01-identity.sh)
• The Abacá container image built and pushed to the internal OpenShift registry at image-registry.openshift-image-registry.svc:5000/abaca/abaca-api:latest (produced by deploy/rhoso/05-build.sh)
• For Kolla-Ansible (development/reference only): SSH access to the controller node with sudo rights and docker available in the mariadb container

Provision the database (RHOSO)

Abacá must have its own Galera instance. It must never share the OpenStack control plane's database cluster. The provisioning script deploy/rhoso/03-database.sh handles this automatically.

Step 1 — Run the database provisioning script

bash deploy/rhoso/03-database.sh

The script performs the following actions:

1. Checks whether an existing Galera instance has been configured for Abacá (via ABACA_GALERA_INSTANCE/ABACA_GALERA_NAMESPACE). If none is found, it creates Abacá's own Galera in the abaca namespace.
2. Applies a MariaDBDatabase custom resource to create the abaca schema with utf8mb4 character set.
3. Applies a MariaDBAccount custom resource to create the abaca database user, reading its password from the abaca-db-secret Kubernetes Secret.
4. Waits up to 240 seconds for both resources to reach Ready=True.

Step 2 — Verify the database and account are ready

oc -n abaca get mariadbdatabase abaca -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
# Expected: True

oc -n abaca get mariadbaccount abaca -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
# Expected: True

Step 3 — Run the schema migration Job

Before starting or upgrading the API and conductor, apply the schema migration. The deploy script (deploy/rhoso/07-deploy.sh) does this automatically, but you can run it manually:

# Delete any previous Job (Job specs are immutable; re-apply would fail)
oc -n abaca delete job abaca-db-sync --ignore-not-found

# Apply the migration Job
oc -n abaca apply -f deploy/rhoso/kustomize/base/dbsync-job.yaml

# Wait for the Job to complete
oc -n abaca wait job/abaca-db-sync --for=condition=complete --timeout=120s

Step 4 — Confirm migration success

oc -n abaca logs job/abaca-db-sync
# Expected: Alembic output ending with no errors

Provision the database (Kolla-Ansible — development/reference only)

For Kolla-Ansible reference environments, use the Kolla-specific script. This creates the abaca database and user directly inside the mariadb container and configures ProxySQL if present.

Step 1 — Run the Kolla database provisioning script

bash deploy/kolla/03-database.sh

The script:

1. Reads the MariaDB root password from /etc/kolla/passwords.yml on the target host.
2. Creates the abaca database and user with a fresh random password.
3. Grants all privileges on abaca.* to the abaca user.
4. If ProxySQL is present (Kolla 2024.2+), writes /etc/kolla/proxysql/users/abaca.yaml and loads the user into the ProxySQL runtime without a container restart.
5. Writes ABACA_DB_PASSWORD to cluster1-abaca-secrets.env for use by the config-rendering step.

Step 2 — Verify the credentials file was written

grep ABACA_DB_PASSWORD cluster1-abaca-secrets.env
# Expected: export ABACA_DB_PASSWORD=<random token>

The database connection is configured in the [database] section of abaca.conf. This file is mounted into the abaca-db-sync Job, abaca-api, and abaca-conductor pods from the abaca-config-data Kubernetes Secret.

[database] section

Example [database] block:

[database]
# Use the Kubernetes internal DNS name of Abacá's own Galera service.
# Replace <password> with the value from the abaca-db-secret Secret.
connection = mysql+pymysql://abaca:<password>@abaca-galera.abaca.svc.cluster.local/abaca?charset=utf8mb4

Important: Abacá stores only bookkeeping metadata in this database (targets, policies, jobs, worker fleet state). All backup data lives in S3 Kopia repositories. The database is not the source of truth for backup contents — the Kopia repositories are. This means a lost database can be reconstructed from S3 using the abaca-manage rebuild-from-repository disaster-recovery operation.

Galera sizing guidance

The default deploy/rhoso/03-database.sh provisions a single-replica Galera (ABACA_GALERA_REPLICAS=1). This is appropriate for development and QA environments. For production, raise the replica count to three:

export ABACA_GALERA_REPLICAS=3
bash deploy/rhoso/03-database.sh

Default storage is 10 GiB (ABACA_DB_STORAGE=10G) on the ocs-storagecluster-ceph-rbd storage class (ABACA_STORAGE_CLASS). Adjust these environment variables before running the provisioning script if your cluster uses a different storage class or requires more capacity.

Running schema migrations

Schema migrations are managed by Alembic via the abaca-manage db_sync command. Migrations are additive-only — they never drop columns or tables — so they can be applied safely against a running database before the new service pods start.

Run db_sync directly (inside the cluster):

oc -n abaca exec deployment/abaca-api -- \
  abaca-manage --config-file /etc/abaca/abaca.conf db_sync

In normal deployments, db_sync runs as the abaca-db-sync Kubernetes Job rather than interactively. The Job is defined with backoffLimit: 0 — it makes exactly one attempt. This is intentional: a migration that fails partway leaves tables in an indeterminate state, and retries would produce misleading "table already exists" errors that obscure the real failure. If the Job fails, inspect the logs immediately (see Troubleshooting) before taking any corrective action.

Upgrade workflow

When upgrading Abacá to a new version that includes schema changes:

1. Build and push the new container image (via deploy/rhoso/05-build.sh).
2. Delete the previous abaca-db-sync Job and apply the new one. The deploy script handles this, or do it manually:

oc -n abaca delete job abaca-db-sync --ignore-not-found
oc -n abaca apply -f deploy/rhoso/kustomize/base/dbsync-job.yaml
oc -n abaca wait job/abaca-db-sync --for=condition=complete --timeout=120s

3. Only after the Job completes successfully, roll out the new API and conductor Deployments.

Disaster recovery — rebuild catalog from S3

If the database is lost but the Kopia repositories in S3 are intact, you can reconstruct the Abacá metadata catalog using:

oc -n abaca exec deployment/abaca-api -- \
  abaca-manage --config-file /etc/abaca/abaca.conf rebuild-from-repository

This reads the Kopia repositories in each registered S3 bucket and reconstructs the job and backup records in the database. Run db_sync first to ensure the schema exists before running rebuild-from-repository.

Example 1 — Apply the migration Job and confirm success

Delete any stale Job, apply the manifest, wait for completion, and check the logs:

oc -n abaca delete job abaca-db-sync --ignore-not-found
oc -n abaca apply -f deploy/rhoso/kustomize/base/dbsync-job.yaml
oc -n abaca wait job/abaca-db-sync --for=condition=complete --timeout=120s
oc -n abaca logs job/abaca-db-sync

Expected output (last few lines):

INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 0001_initial, initial schema
INFO  [alembic.runtime.migration] Running upgrade 0001_initial -> 0002_add_policy_cron, add policy cron expression

Example 2 — Verify the Galera and database resources are healthy

# Check Galera cluster readiness
oc -n abaca get galera abaca-galera \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
# Expected: True

# Check the MariaDBDatabase resource
oc -n abaca get mariadbdatabase abaca \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
# Expected: True

# Check the MariaDBAccount resource
oc -n abaca get mariadbaccount abaca \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
# Expected: True

Example 3 — Run db_sync interactively from the API pod

Useful during troubleshooting or after a manual database restore:

oc -n abaca exec deployment/abaca-api -- \
  abaca-manage --config-file /etc/abaca/abaca.conf db_sync

Expected output:

INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] No new upgrade operations to perform.

Example 4 — Provision Abacá's own Galera with production replica count

export ABACA_GALERA_REPLICAS=3
export ABACA_DB_STORAGE=50G
export ABACA_STORAGE_CLASS=ocs-storagecluster-ceph-rbd
bash deploy/rhoso/03-database.sh

Expected final line:

database 'abaca' ready for 'abaca' on abaca-galera.abaca.svc
OK.

Migration Job fails with "table already exists"

Symptom: The abaca-db-sync Job fails immediately, and the log contains a MySQL error 1050 (Table 'X' already exists).

Likely cause: A previous failed migration attempt left partial schema state without an Alembic version stamp. Subsequent Job runs hit the same CREATE TABLE statement but cannot proceed because the table already exists.

Fix:

1. Collect the full log before doing anything else — the first attempt's log contains the real error:
   Copy

   oc -n abaca logs job/abaca-db-sync
   

2. Resolve the underlying cause (see other entries below).
3. If the schema is genuinely partially applied and you need to stamp the current revision manually, exec into the API pod and use abaca-manage to apply only the missing migrations after manually resolving the inconsistency. Do not delete tables without understanding the full migration graph.

Migration Job fails with "Access denied"

Symptom: The abaca-db-sync Job log contains Access denied for user 'abaca'@....

Likely cause: The abaca-config-data Secret contains a stale or incorrect database password, or the MariaDBAccount resource has not yet reached Ready=True.

Fix:

# 1. Check that the account is ready
oc -n abaca get mariadbaccount abaca \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

# 2. Confirm the connection string in the config Secret is correct
oc -n abaca get secret abaca-config-data -o jsonpath='{.data.abaca-api\.conf}' \
  | base64 -d | grep connection

# 3. Re-run the provisioning script if the account is not ready
bash deploy/rhoso/03-database.sh

Migration Job fails with "Can't connect to MySQL server"

Symptom: The Job log shows a connection-refused or timeout error against the Galera hostname.

Likely cause: Abacá's Galera instance is not yet ready, or the connection URL in abaca.conf points to the wrong hostname.

Fix:

# 1. Check Galera readiness
oc -n abaca get galera \
  -o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}'
# Expected: True

# 2. Check the connection URL
oc -n abaca get secret abaca-config-data -o jsonpath='{.data.abaca-api\.conf}' \
  | base64 -d | grep connection

# 3. If Galera is still bootstrapping, wait and retry
oc -n abaca wait galera/abaca-galera --for=jsonpath='{.status.conditions[?(@.type=="Ready")].status}'=True --timeout=900s

abaca-db-sync Job cannot be re-applied after failure

Symptom: Running oc apply -f dbsync-job.yaml after a failure produces: The Job "abaca-db-sync" is invalid: spec.template: Invalid value.

Likely cause: Kubernetes Job specs are immutable after creation. The previous (failed) Job object must be deleted before re-applying.

Fix:

oc -n abaca delete job abaca-db-sync --ignore-not-found
oc -n abaca apply -f deploy/rhoso/kustomize/base/dbsync-job.yaml

Abacá is accidentally pointed at the OpenStack control plane's Galera

Symptom: The connection URL in abaca.conf resolves to the platform's shared Galera. Abacá may appear to function but risks filling or wedging the database that all core OpenStack services depend on.

Likely cause: The ABACA_GALERA_INSTANCE or ABACA_GALERA_NAMESPACE variables were set to the platform's Galera during provisioning. The provisioning script explicitly refuses to auto-adopt the OpenStack control plane's Galera, but a manually edited config can bypass this guard.

Fix: Update the [database] connection in abaca.conf to point to Abacá's own Galera (in the abaca namespace), rebuild the abaca-config-data Secret, and re-run the provisioning script with the correct environment variables:

unset ABACA_GALERA_INSTANCE
unset ABACA_GALERA_NAMESPACE
bash deploy/rhoso/03-database.sh

Kolla: ProxySQL does not pick up the new abaca user

Symptom: On a Kolla 2024.2+ environment, connections from Abacá to the database are refused even though the MariaDB user was created successfully.

Likely cause: The ProxySQL runtime was not updated, or the abaca.yaml file was not written to /etc/kolla/proxysql/users/.

Fix: Re-run the Kolla database provisioning script, which is idempotent and will rewrite the user file and reload the ProxySQL runtime:

bash deploy/kolla/03-database.sh

Confirm the file exists on the controller:

ssh <controller> sudo cat /etc/kolla/proxysql/users/abaca.yaml