When Good Decisions
Outlive Their Context
Two systems managing the same schema simultaneously
While the CI/CD pipeline was being built, manual database migrations were the norm - someone (usually me) would tunnel into staging and run migration commands by hand. To smooth over rough edges during that period, synchronize: true was enabled in the TypeORM config so schema changes were auto-applied during development. This was a pragmatic, intentional decision for that context.
When the DevOps engineer shipped the automated pipeline - GitHub Actions running npm run migration:run before each deploy - it worked perfectly. But synchronize: true was never turned off. Now two systems were managing the schema: migrations via CI/CD before deployment, and TypeORM synchronization at application startup. Most of the time, synchronize found nothing to change and was silent. Until it wasn't.
- synchronize: true left active after CI/CD migrations went live
- TypeORM's introspection detected a minor index naming discrepancy and tried to reconcile
- Attempted: DROP INDEX IDX_yards_created_by ON yards
- MySQL blocked it: the index supported a foreign key constraint and could not be dropped
- Transaction rollback → application startup failure → deploy down in staging
| 2026-02-09 14:40:58 | query failed |
| DROP INDEX IDX_yards_created_by | FK constraint |
| DROP INDEX IDX_yards_updated_by | FK constraint |
| ROLLBACK | - |
| Application end | crash |
TypeORM introspected a discrepancy it shouldn't have touched
The @ManyToOne decorators on the Yard entity create foreign key constraints. MySQL automatically creates an index to support each FK. TypeORM's synchronization algorithm read the existing indexes, detected a minor discrepancy from how it would have named them, and generated DROP INDEX statements to "fix" it. MySQL correctly refused - you cannot drop an index that an active foreign key constraint depends on. The crash was MySQL protecting data integrity, not a bug.
The root cause was the race: migrations (running in CI/CD before deploy) created the correct schema. Synchronize (running at app startup after deploy) tried to modify it. This is the fundamental incompatibility: synchronize and migrationsRun are two competing schema management strategies and should never both be active in the same environment.
Choose one strategy per environment, make it loud
One PR. Zero data loss. Zero recurrence risk.
The crash happened during application startup - before any user traffic hit the new deploy. Staging data was untouched. The ROLLBACK ensured the partial index operations were cleanly reversed.
A single configuration change - synchronize: process.env.NODE_ENV === 'development' - resolved the immediate crash. The safety check and explicit opt-in mechanism were added in the same PR.
The FATAL startup guard means if anyone accidentally sets TYPEORM_SYNCHRONIZE=true in a staging or production environment variable, the app will refuse to start with an explicit error before touching the database.
All schema changes now flow exclusively through the migration system in staging and production. The CI/CD pipeline runs migrations before deploy; the app runs them again at startup as a safety net - no synchronize racing to "fix" things.
Architectural choices, tradeoffs, and what nearly broke production
Why synchronize at all in development?
No. In local development, synchronize is a feature not a bug. You iterate on entity decorators and want the schema to reflect them instantly. Removing it forces developers to write migrations for trivial schema changes in their local flow, killing iteration speed. The key insight is that synchronize and migrationsRun are incompatible at the same environment tier, not incompatible everywhere.
Why not use TypeORM's synchronize within a safety gate?
Theoretically, but operationally risky. TypeORM's synchronize algorithm is a black box you don't control. Even with pre-flight validation, you're hoping the algorithm produces what you expect. By contrast, migrations are explicit SQL. You review every change. This is why the fix disabled synchronize in staging/production - explicit control beats implicit safety gates.
The real problem wasn't the crash
Because the crash was a symptom. The root cause was a decision made years ago when the codebase was young and manual. At that time, synchronize was the right call. Over time, CI/CD became more sophisticated, migrations became standard practice, and the old decision outlived its context.
If I had only fixed the crash, the same race condition could happen again during the next major refactor or schema change. Instead, I treated this as a decision lifecycle issue: good decisions need expiration dates and regular re-evaluation.
Why fail loudly instead of silently skipping?
Silent failure is how configuration mistakes compound into production disasters. By making it loud, you force engineers to consciously decide: 'I'm activating this feature.' The failure is intentional and visible. This prevents accidental re-enablement after a careless config change.
Cost · Risk · Tradeoffs
Estimated values based on production metrics and monitoring data
Transformative Results: Before vs After
- synchronize: true activeSchema conflict risk
- Two systems managing schemaRace conditions
- No safety guardrailsSilent failures
- Manual DB ops eraInconsistent state
- synchronize: false enforcedNo conflict risk
- CI/CD-only migrationsSingle source of truth
- Safety check addedFail-fast protection
- Automated & reliableConsistent state
What was used
Need someone who thinks about the lifecycle of engineering decisions?
I build systems that are maintainable under change - not just correct at the moment of writing.