Alembic experiments
This commit is contained in:
Jonas Linter
174
MIGRATION_REFACTORING.md
Normal file
174
MIGRATION_REFACTORING.md
Normal file
@@ -0,0 +1,174 @@
|
||||
# Database Migration Refactoring
|
||||
|
||||
## Summary
|
||||
|
||||
This refactoring changes the database handling from manual schema migrations in `migrations.py` to using Alembic for proper database migrations. The key improvements are:
|
||||
|
||||
1. **Alembic Integration**: All schema migrations now use Alembic's migration framework
|
||||
2. **Separation of Concerns**: Migrations (schema changes) are separated from startup tasks (data backfills)
|
||||
3. **Pre-startup Migrations**: Database migrations run BEFORE the application starts, avoiding issues with multiple workers
|
||||
4. **Production Ready**: The Conversions/ConversionRoom tables can be safely recreated (data is recoverable from PMS XML imports)
|
||||
|
||||
## Changes Made
|
||||
|
||||
### 1. Alembic Setup
|
||||
|
||||
- **[alembic.ini](alembic.ini)**: Configuration file for Alembic
|
||||
- **[alembic/env.py](alembic/env.py)**: Async-compatible environment setup that:
|
||||
- Loads database URL from config.yaml or environment variables
|
||||
- Supports PostgreSQL schemas
|
||||
- Uses async SQLAlchemy engine
|
||||
|
||||
### 2. Initial Migrations
|
||||
|
||||
Two migrations were created:
|
||||
|
||||
#### Migration 1: `535b70e85b64_initial_schema.py`
|
||||
Creates all base tables:
|
||||
- `customers`
|
||||
- `hashed_customers`
|
||||
- `reservations`
|
||||
- `acked_requests`
|
||||
- `conversions`
|
||||
- `conversion_rooms`
|
||||
|
||||
This migration is idempotent - it only creates missing tables.
|
||||
|
||||
#### Migration 2: `8edfc81558db_drop_and_recreate_conversions_tables.py`
|
||||
Handles the conversion from old production conversions schema to new normalized schema:
|
||||
- Detects if old conversions tables exist with incompatible schema
|
||||
- Drops them if needed (data can be recreated from PMS XML imports)
|
||||
- Allows the initial schema migration to recreate them with correct structure
|
||||
|
||||
### 3. Refactored Files
|
||||
|
||||
#### [src/alpine_bits_python/db_setup.py](src/alpine_bits_python/db_setup.py)
|
||||
- **Before**: Ran manual migrations AND created tables using Base.metadata.create_all
|
||||
- **After**: Only runs startup tasks (data backfills like customer hashing)
|
||||
- **Note**: Schema migrations now handled by Alembic
|
||||
|
||||
#### [src/alpine_bits_python/run_migrations.py](src/alpine_bits_python/run_migrations.py) (NEW)
|
||||
- Wrapper script to run `alembic upgrade head`
|
||||
- Can be called standalone or from run_api.py
|
||||
- Handles errors gracefully
|
||||
|
||||
#### [src/alpine_bits_python/api.py](src/alpine_bits_python/api.py)
|
||||
- **Removed**: `run_all_migrations()` call from lifespan
|
||||
- **Removed**: `Base.metadata.create_all()` call
|
||||
- **Changed**: Now only calls `run_startup_tasks()` for data backfills
|
||||
- **Note**: Assumes migrations have already been run before app start
|
||||
|
||||
#### [src/alpine_bits_python/run_api.py](src/alpine_bits_python/run_api.py)
|
||||
- **Added**: Calls `run_migrations()` BEFORE starting uvicorn
|
||||
- **Benefit**: Migrations complete before any worker starts
|
||||
- **Benefit**: Works correctly with multiple workers
|
||||
|
||||
### 4. Old Files (Can be removed in future cleanup)
|
||||
|
||||
- **[src/alpine_bits_python/migrations.py](src/alpine_bits_python/migrations.py)**: Old manual migration functions
|
||||
- These can be safely removed once you verify the Alembic setup works
|
||||
- The functionality has been replaced by Alembic migrations
|
||||
|
||||
## Usage
|
||||
|
||||
### Development
|
||||
|
||||
Start the server (migrations run automatically):
|
||||
```bash
|
||||
uv run python -m alpine_bits_python.run_api
|
||||
```
|
||||
|
||||
Or run migrations separately:
|
||||
```bash
|
||||
uv run alembic upgrade head
|
||||
uv run python -m alpine_bits_python.run_api
|
||||
```
|
||||
|
||||
### Production with Multiple Workers
|
||||
|
||||
The migrations automatically run before uvicorn starts, so you can safely use:
|
||||
```bash
|
||||
# Migrations run once, then server starts with multiple workers
|
||||
uv run python -m alpine_bits_python.run_api
|
||||
|
||||
# Or with uvicorn directly (migrations won't run automatically):
|
||||
uv run alembic upgrade head # Run this first
|
||||
uvicorn alpine_bits_python.api:app --workers 4 --host 0.0.0.0 --port 8080
|
||||
```
|
||||
|
||||
### Creating New Migrations
|
||||
|
||||
When you modify the database schema in `db.py`:
|
||||
|
||||
```bash
|
||||
# Generate migration automatically
|
||||
uv run alembic revision --autogenerate -m "description_of_change"
|
||||
|
||||
# Or create empty migration to fill in manually
|
||||
uv run alembic revision -m "description_of_change"
|
||||
|
||||
# Review the generated migration in alembic/versions/
|
||||
# Then apply it
|
||||
uv run alembic upgrade head
|
||||
```
|
||||
|
||||
### Checking Migration Status
|
||||
|
||||
```bash
|
||||
# Show current revision
|
||||
uv run alembic current
|
||||
|
||||
# Show migration history
|
||||
uv run alembic history
|
||||
|
||||
# Show pending migrations
|
||||
uv run alembic heads
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Multiple Worker Safe**: Migrations run once before any worker starts
|
||||
2. **Proper Migration History**: All schema changes are tracked in version control
|
||||
3. **Rollback Support**: Can downgrade to previous schema versions if needed
|
||||
4. **Standard Tool**: Alembic is the industry-standard migration tool for SQLAlchemy
|
||||
5. **Separation of Concerns**:
|
||||
- Schema migrations (Alembic) are separate from startup tasks (db_setup.py)
|
||||
- Migrations are separate from application code
|
||||
|
||||
## Migration from Old System
|
||||
|
||||
If you have an existing database with the old migration system:
|
||||
|
||||
1. The initial migration will detect existing tables and skip creating them
|
||||
2. The conversions table migration will detect old schemas and recreate them
|
||||
3. All data in other tables is preserved
|
||||
4. Conversions data will be lost but can be recreated from PMS XML imports
|
||||
|
||||
## Important Notes
|
||||
|
||||
### Conversions Table Data Loss
|
||||
|
||||
The `conversions` and `conversion_rooms` tables will be dropped and recreated with the new schema. This is intentional because:
|
||||
- The production version has a different schema
|
||||
- The data can be recreated by re-importing PMS XML files
|
||||
- This avoids complex data migration logic
|
||||
|
||||
If you need to preserve this data, modify the migration before running it.
|
||||
|
||||
### Future Migrations
|
||||
|
||||
In the future, when you need to change the database schema:
|
||||
|
||||
1. Modify the model classes in `db.py`
|
||||
2. Generate an Alembic migration: `uv run alembic revision --autogenerate -m "description"`
|
||||
3. Review the generated migration carefully
|
||||
4. Test it on a dev database first
|
||||
5. Apply it to production: `uv run alembic upgrade head`
|
||||
|
||||
## Configuration
|
||||
|
||||
The Alembic setup reads configuration from the same sources as the application:
|
||||
- `config.yaml` (via `annotatedyaml` with `secrets.yaml`)
|
||||
- Environment variables (`DATABASE_URL`, `DATABASE_SCHEMA`)
|
||||
|
||||
No additional configuration needed!
|
||||
Reference in New Issue
Block a user