Skip to main content

Disaster Recovery and Rollback

Recovery is a code change. The portal has no restore button. Settings come back when you revert the commit that changed them. Deleted data does not come back with the revert. You restore data from a backup with gcloud, and then bring the restored database under management with a pull request.

Roll back a deployment or a setting​

  1. Find the commit that introduced the change.

    git log application/fmv-customer-app.yaml
  2. Revert it. This makes a new commit and keeps the history.

    git revert <commit-hash>
  3. Open a pull request and merge it. The engine applies the previous version or setting.

The same steps undo an infrastructure change, such as a cpuCount increase on a database.

Restore a database​

Never restore over a live database. Restore into a new cluster, check it, and then switch the application to it. The old cluster stays for analysis.

  1. Restore with gcloud. Use a backup:

    gcloud alloydb backups list --region=us-central1 --project=customer-portal-prod \
    --sort-by=~createTime --limit=5

    gcloud alloydb clusters restore main-recovered \
    --backup=BACKUP_ID \
    --region=us-central1 \
    --project=customer-portal-prod

    Or use a point in time. --source-cluster and --point-in-time go together:

    gcloud alloydb clusters restore main-recovered \
    --source-cluster=main \
    --region=us-central1 \
    --point-in-time=2025-02-03T14:29:00Z \
    --project=customer-portal-prod

    The command waits until the restore ends. To follow the restore from a second terminal, list the operations of the new cluster:

    gcloud alloydb operations list --cluster=main-recovered \
    --region=us-central1 --project=customer-portal-prod
  2. Add a Database manifest with the name of the restored cluster. Merge it so that Infrastream manages the cluster.

    apiVersion: lowops.manifests.v1
    kind: Database
    metadata:
    name: main-recovered
    project: customer-portal
    environment: production
    organizational-unit: your-ou
    organization: your-org
    spec:
    description: Emergency recovery database.
    alloydb:
    cpuCount: 4
    clusterSize: 3
  3. Check the data. Then change the database name in the DeploymentConfig of the application and merge.

Isolate a project during an incident​

Empty the egress allow-list of the Project. Google's services, the Debian and Windows update servers and private addresses inside your network stay allowed. The platform denies all other traffic.

spec:
allowedEgress: []

Add back only the hosts that you have verified.

What is backed up​

AlloyDB. The platform turns on automated backups for each cluster. It sets no schedule or retention, so the defaults of Google apply.

  • Google documents those defaults as one automated backup a day, kept 14 days, with continuous backup on and a 14-day window.
  • The engine reads the backupConfig fields of a Database manifest but does not apply them.
  • The platform does not configure point-in-time recovery.
  • When the engine updates a cluster, it writes the backup policy as "enabled" only. That can reset a schedule that you set by hand.

Confirm the backup and point-in-time settings in Google Cloud before you rely on them.

VM disks. The platform attaches a snapshot policy: one snapshot a day from 04:00 UTC, kept 14 days. VM manifests cannot change it.

  • If the disk has a backup project, the platform takes a snapshot of the disk before it deletes the disk. The name of the snapshot starts with final-snap- and the name of the disk. If the snapshot fails, the platform does not delete the disk.
  • To seed a disk from a snapshot, set source beside diskConfig under volumeMounts.
  • If source is empty and the disk has a backup project, the platform searches that project. It uses the newest snapshot whose name contains the name of the disk.
spec:
configuration:
volumeMounts:
/mnt/data:
source: <snapshot self-link> # optional: the snapshot to seed the disk from
diskConfig:
sizeGb: 200
type: pd-ssd

Recovery time​

Infrastream publishes no recovery times. Run each procedure as a drill in a test project, time it and record your figures. The time depends on the size of the database, the time to create a cluster, and your cutover.

Keep evidence​

Do not delete a compromised resource at once. Rename it and keep it for analysis. Use the pull request as the incident timeline.

Next​