Zero downtime starts with compatibility, not a clever SQL statement
“Zero downtime” should be treated as an engineering objective, not a guarantee that every migration is invisible. A short metadata operation can still wait behind a long transaction, a backfill can saturate storage, and an otherwise safe index build can increase CPU and I/O pressure. The practical goal is to keep the application available, bound the blast radius, and make every phase observable and reversible.
The reliable default is expand–migrate–contract. Instead of changing the schema and application in one release, introduce a compatible shape first, move data and traffic gradually, and remove the old shape only after no supported application version depends on it.
Expand
Add nullable columns, tables, or indexes. Deploy code that understands both old and new shapes.
Migrate
Dual-write where needed, backfill existing rows in bounded batches, and verify parity.
Contract
Switch reads, stop legacy writes, enforce invariants, then remove obsolete objects later.
This sequence matters whenever old and new application instances overlap during a rolling deployment. It also gives queued jobs, mobile clients, read replicas, analytics consumers, and rollback releases time to catch up. The same compatibility discipline helps teams evolving a multi-tenant SaaS architecture or modernizing an existing platform.
Classify each PostgreSQL operation before it reaches production
Migration risk is determined by more than table size. Review the lock mode, whether PostgreSQL scans or rewrites the table, expected WAL volume, replication impact, transaction duration, and compatibility with every live application version. PostgreSQL notes that ALTER TABLE takes an ACCESS EXCLUSIVE lock unless a subcommand documents a weaker one; that lock conflicts with every table-level lock, including ordinary reads. Use the PostgreSQL 18 ALTER TABLE reference and table-lock documentation to review the exact operation.
| Change | Main risk | Production approach |
|---|---|---|
| Add a nullable column | Brief lock acquisition can queue behind older transactions | Use a short transaction and a bounded lock_timeout; retry deliberately |
| Add a column with a default | A volatile default requires existing rows to be updated | Prefer a nullable expansion and backfill when values vary; constant defaults are optimized by current PostgreSQL |
| Create an index | A regular build blocks writes | Use CREATE INDEX CONCURRENTLY when its caveats fit |
| Add a foreign key or check | Immediate verification scans existing rows | Add NOT VALID, then validate separately |
| Change a populated column type | May rewrite the table and break older code | Add a new column, dual-write, backfill, switch reads, and retire the old column |
| Rename or drop a column | Older app instances and consumers fail immediately | Treat it as a later contract release after dependency evidence reaches zero |
Worked example: add an organization key to live invoices
Assume invoices currently reaches its tenant through customer_id. We want a direct organization_id for clearer authorization and faster tenant-scoped queries. The table is receiving reads and writes throughout the change.
Release A: expand the schema and deploy compatible writes
Add the column without a volatile default or immediate NOT NULL check. A timeout makes the migration fail instead of waiting indefinitely behind a conflicting transaction. PostgreSQL documents lock_timeout as a session setting that aborts a statement after a bounded lock wait; set it per migration session rather than globally.
BEGIN;
SET LOCAL lock_timeout = '2s';
SET LOCAL statement_timeout = '30s';
ALTER TABLE public.invoices
ADD COLUMN organization_id bigint;
COMMIT;The exact timeout values are operational choices, not universal defaults. Choose them from your latency budget and rehearse the retry behavior. The current PostgreSQL client timeout documentation also warns that a lock_timeout equal to or greater than statement_timeout is ineffective because the statement timeout fires first.
Next, deploy code that writes customer_id and organization_id in the same transaction. Reads should still work when organization_id is null because old rows have not been migrated. Derive the organization from trusted server-side data, not an unverified request field. If the database uses tenant policies, align the change with the isolation controls in our PostgreSQL row-level security guide.
Backfill in bounded, restartable batches
A single update of every historical row creates one large transaction, holds row locks, generates a burst of WAL, delays vacuum cleanup, and makes recovery expensive. Use a background migration that commits small batches, pauses or reduces concurrency when the database is under pressure, and can resume safely after interruption.
WITH batch AS (
SELECT i.id, c.organization_id
FROM public.invoices AS i
JOIN public.customers AS c ON c.id = i.customer_id
WHERE i.organization_id IS NULL
ORDER BY i.id
LIMIT 1000
FOR UPDATE OF i SKIP LOCKED
)
UPDATE public.invoices AS i
SET organization_id = batch.organization_id
FROM batch
WHERE i.id = batch.id
RETURNING i.id;Run one batch per transaction until it returns no rows. Tune batch size and pause duration from measured query latency, lock waits, WAL generation, replica lag, dead tuples, and connection-pool pressure. SKIP LOCKED is appropriate for this queue-like maintenance loop because another writer can own a row temporarily; it is not a general-purpose way to obtain a consistent user-facing result.
- Make the predicate idempotent: already-migrated rows are ignored.
- Record progress and errors without logging sensitive invoice data.
- Compare source and target ownership before writing; stop on ambiguous relationships.
- Keep new application writes populating both shapes so the backfill does not race a live insert.
- Run
ANALYZEwhen data distribution changes enough to affect query planning.
Backfills share the same database capacity as API traffic. Coordinate worker concurrency with the connection strategy described in our Node.js database connection-pooling guide.
Build indexes and validate constraints as separate phases
Create the supporting index without blocking normal writes
CREATE INDEX CONCURRENTLY invoices_organization_id_idx
ON public.invoices (organization_id);PostgreSQL's concurrent-index documentation explains the trade-off: normal inserts, updates, and deletes can continue, but PostgreSQL performs extra work and waits for relevant transactions. The command cannot run inside a transaction block, only one concurrent index build can run on a table at a time, and failure can leave an invalid index that still adds update overhead. Check validity before considering this phase complete.
SELECT indexrelid::regclass AS index_name,
indisready,
indisvalid
FROM pg_index
WHERE indexrelid = 'public.invoices_organization_id_idx'::regclass;Add the foreign key without scanning under the initial lock
ALTER TABLE public.invoices
ADD CONSTRAINT invoices_organization_fk
FOREIGN KEY (organization_id)
REFERENCES public.organizations (id)
NOT VALID;
ALTER TABLE public.invoices
VALIDATE CONSTRAINT invoices_organization_fk;NOT VALID skips the initial scan of existing rows while still enforcing the foreign key for subsequent inserts and updates. VALIDATE CONSTRAINT later scans the table with a weaker SHARE UPDATE EXCLUSIVE lock. It still consumes resources and can conflict with other maintenance, so schedule and monitor it deliberately.
Prove non-null data before the final metadata change
ALTER TABLE public.invoices
ADD CONSTRAINT invoices_organization_present
CHECK (organization_id IS NOT NULL)
NOT VALID;
ALTER TABLE public.invoices
VALIDATE CONSTRAINT invoices_organization_present;
ALTER TABLE public.invoices
ALTER COLUMN organization_id SET NOT NULL;
ALTER TABLE public.invoices
DROP CONSTRAINT invoices_organization_present;The valid check constraint proves that no null value exists, allowing current PostgreSQL to skip another full-table scan when SET NOT NULL runs. The final metadata operations can still wait for their required locks, so keep the timeout and retry policy in place.
For a new unique rule, build a UNIQUE INDEX CONCURRENTLY first and attach it with ADD CONSTRAINT ... UNIQUE USING INDEX where PostgreSQL's restrictions permit. Do not assume an ORM's generated “add unique” migration uses that online sequence.
Observe the migration as a production workload
Watch the system before, during, and after every phase. Database availability can degrade while every command is technically “online.” At minimum, track application error rate and latency alongside database connections, active and waiting sessions, lock age, transaction age, WAL generation, replica lag, CPU, storage latency, dead tuples, and autovacuum activity.
SELECT a.pid,
a.state,
a.wait_event_type,
a.wait_event,
now() - a.xact_start AS transaction_age,
left(a.query, 120) AS query
FROM pg_stat_activity AS a
WHERE a.datname = current_database()
AND a.pid <> pg_backend_pid()
ORDER BY a.xact_start NULLS LAST;While an index is building, pg_stat_progress_create_index reports its command, phase, processed blocks or tuples, and lock wait progress. Pair that view with pg_stat_activity and pg_locks; progress alone does not show the application impact.
Pause signals
Unexpected lock queues, rising API latency, replica lag, storage pressure, or an error-rate change.
Completion signals
No null targets, constraints valid, index valid, read parity stable, and no legacy writes observed.
Cleanup signals
Every supported app and consumer uses the new shape, and the rollback version no longer needs the old one.
Test mixed versions, partial data, failure, and rollback
A migration that passes on an empty development database has not tested the production problem. Restore a sanitized production-like dataset or generate representative cardinality and distributions. Rehearse the operation with realistic traffic, long transactions, replicas, background jobs, and the same migration runner configuration used in production.
- Run the old application against the expanded schema.
- Run the new application while some rows are not yet backfilled.
- Interrupt and restart the backfill; confirm it is idempotent.
- Force the DDL lock timeout and confirm retry or escalation works.
- Inject invalid source relationships and confirm the migration stops safely.
- Verify constraints, indexes, query plans, RLS policies, audit paths, and downstream exports.
- Roll the application back without rolling the schema back destructively.
Prefer roll-forward database recovery: keep additive schema in place, disable the new read path, fix the application or backfill, and resume. A down migration that drops a populated column is not a safe rollback merely because the tool generated it. Take and test backups, but remember that restoring an entire production database is an incident-recovery action, not a routine schema-release strategy.
Release B and C: switch reads, then contract later
After parity checks pass, deploy reads from organization_id while retaining the legacy path temporarily. Stop old writes only when no older deployment or worker remains. Remove customer_id-derived compatibility code, columns, triggers, or indexes in a later release after telemetry and dependency inventories show no use. For high-risk modernization work, the team providing your cloud and infrastructure engineering should own the database runbook together with application engineers—not receive a one-line migration at deployment time.
PostgreSQL zero-downtime migration checklist
✓ Inventory app versions, jobs, replicas, exports, and external consumers
✓ Record lock, scan, rewrite, WAL, disk, and compatibility risks for each command
✓ Expand with additive schema before any old shape is removed
✓ Bound lock waits and keep DDL transactions short
✓ Deploy compatible or dual writes before backfilling historical rows
✓ Backfill in restartable batches with explicit pause signals
✓ Build production indexes concurrently and verify index validity
✓ Add eligible constraints as NOT VALID and validate separately
✓ Test old and new application versions against partially migrated data
✓ Observe application health, locks, WAL, replicas, storage, and vacuum
✓ Roll back application behavior without destructive schema reversal
✓ Contract only after evidence shows the legacy shape is unused
Modernize the database without gambling on the release
Endurance Softwares helps teams plan and deliver reliable SaaS and custom software modernization across PostgreSQL and Supabase data models, Node.js APIs, application releases, testing, and cloud operations.
