Database Operations

Stateless deployments can often roll back by running an older artifact. Databases are harder because data changes persist. DevOps for stateful systems requires migration discipline, backup verification, access control, performance monitoring, and careful rollout sequencing.

Migration Safety

Safe schema changes are usually backward compatible across at least one deployment window. A common pattern is expand, migrate, contract: add new structures, deploy code that writes both or reads both, backfill data, then remove old structures after confidence is high.

Destructive migrations need explicit review, backups, and rollback plans. A rollback that restores yesterday’s backup may lose today’s valid writes, so recovery objectives must be defined before failure.

Backups and Recovery

A backup strategy is incomplete until restoration is tested. Teams need recovery point objectives, recovery time objectives, encryption, access controls, and regular restore drills. The question is not “do backups exist?” but “can the team restore the right data quickly under pressure?”

Online Change Patterns

Databases often serve live traffic while schemas evolve. Safe online change avoids long locks, full-table rewrites during peak traffic, and incompatible application versions. Large backfills should be batched, observable, restartable, and throttled. Migration tooling should record which migrations ran, when, by whom, and against which database.

Application compatibility is crucial. If version A expects column old_name and version B expects new_name, a rollback can fail after the migration. Expand-and-contract avoids this by supporting both versions temporarily.

Data Quality and Idempotency

Operational data work should be idempotent where possible. A backfill that can resume safely after failure is far less risky than one that must run perfectly once. Data validation queries should check counts, null rates, referential integrity, and business invariants. Logs should report progress without exposing sensitive data.

Access and Change Control

Direct production database access is powerful and dangerous. It should be audited, time-limited, and used through reviewed change paths where possible. Emergency manual queries should be copied into an incident timeline and converted into repeatable migrations or runbooks afterward.

Study Task

Plan a zero-downtime rename of a database field used by two service versions. Include schema expansion, dual write or compatibility read, backfill, validation, deploy order, flag removal, and final contraction.

Exercise

For a planned schema change, write the deploy sequence, compatibility window, backfill plan, validation query, rollback option, and data-loss risk.

Operational Review

Database operations should be reviewed with the assumption that mistakes are expensive. Which changes can lock tables? Which changes are irreversible? Which migrations are safe to retry? Which backups have been restored recently? Which queries dominate latency and cost?

The review should include application behavior. Connection pools, retry policies, transaction boundaries, and query patterns can damage the database even when schema design is sound. A traffic spike plus aggressive retries can exhaust connections and turn a slow query into a full outage.

Stateful systems demand humility. Before changing data, define how correctness will be verified. After changing data, validate with queries, metrics, and application behavior. The rollback plan should acknowledge whether data loss is possible.

References & Further Reading

• Refactoring Databases by Scott W. Ambler and Pramod J. Sadalage
• Continuous Delivery by Jez Humble and David Farley