Disaster Recovery and Rollback
Recovery is a code change. The portal has no restore button. Settings come back when you revert the commit that changed them. Deleted data does not come back with the revert. You restore data from a backup with gcloud, and then bring the restored database under management with a pull request.
Roll back a deployment or a setting
-
Find the commit that introduced the change.
git log application/fmv-customer-app.yaml -
Revert it. This makes a new commit and keeps the history.
git revert <commit-hash> -
Open a pull request and merge it. The engine applies the previous version or setting.
The same steps undo an infrastructure change, such as a cpuCount increase on a database.
Restore a database
Never restore over a live database. Restore into a new cluster, check it, and then switch the application to it. The old cluster stays for analysis.
-
Restore with
gcloud. Use a backup:gcloud alloydb backups list --region=us-central1 --project=customer-portal-prod \
--sort-by=~createTime --limit=5
gcloud alloydb clusters restore main-recovered \
--backup=BACKUP_ID \
--region=us-central1 \
--project=customer-portal-prodOr use a point in time.
--source-clusterand--point-in-timego together:gcloud alloydb clusters restore main-recovered \
--source-cluster=main \
--region=us-central1 \
--point-in-time=2025-02-03T14:29:00Z \
--project=customer-portal-prodThe command waits until the restore ends. To follow the restore from a second terminal, list the operations of the new cluster:
gcloud alloydb operations list --cluster=main-recovered \
--region=us-central1 --project=customer-portal-prod -
Add a
Databasemanifest with the name of the restored cluster. Merge it so that Infrastream manages the cluster.apiVersion: lowops.manifests.v1
kind: Database
metadata:
name: main-recovered
project: customer-portal
environment: production
organizational-unit: your-ou
organization: your-org
spec:
description: Emergency recovery database.
alloydb:
cpuCount: 4
clusterSize: 3 -
Check the data. Then change the database name in the
DeploymentConfigof the application and merge.
Isolate a project during an incident
Empty the egress allow-list of the Project. Google's services, the Debian and Windows update servers and private addresses inside your network stay allowed. The platform denies all other traffic.
spec:
allowedEgress: []
Add back only the hosts that you have verified.
What is backed up
AlloyDB. The platform turns on automated backups for each cluster. It sets no schedule or retention, so the defaults of Google apply.
- Google documents those defaults as one automated backup a day, kept 14 days, with continuous backup on and a 14-day window.
- The engine reads the
backupConfigfields of aDatabasemanifest but does not apply them. - The platform does not configure point-in-time recovery.
- When the engine updates a cluster, it writes the backup policy as "enabled" only. That can reset a schedule that you set by hand.
Confirm the backup and point-in-time settings in Google Cloud before you rely on them.
VM disks. The platform attaches a snapshot policy: one snapshot a day from 04:00 UTC, kept 14 days. VM manifests cannot change it.
- If the disk has a backup project, the platform takes a snapshot of the disk before it deletes the disk. The name of the snapshot starts with
final-snap-and the name of the disk. If the snapshot fails, the platform does not delete the disk. - To seed a disk from a snapshot, set
sourcebesidediskConfigundervolumeMounts. - If
sourceis empty and the disk has a backup project, the platform searches that project. It uses the newest snapshot whose name contains the name of the disk.
spec:
configuration:
volumeMounts:
/mnt/data:
source: <snapshot self-link> # optional: the snapshot to seed the disk from
diskConfig:
sizeGb: 200
type: pd-ssd
Recovery time
Infrastream publishes no recovery times. Run each procedure as a drill in a test project, time it and record your figures. The time depends on the size of the database, the time to create a cluster, and your cutover.
Keep evidence
Do not delete a compromised resource at once. Rename it and keep it for analysis. Use the pull request as the incident timeline.