alpinebits_python/MIGRATION_REFACTORING.md

# Database Migration Refactoring

## Summary

This refactoring changes the database handling from manual schema migrations in `migrations.py` to using Alembic for proper database migrations. The key improvements are:

1. **Alembic Integration**: All schema migrations now use Alembic's migration framework
2. **Separation of Concerns**: Migrations (schema changes) are separated from startup tasks (data backfills)
3. **Pre-startup Migrations**: Database migrations run BEFORE the application starts, avoiding issues with multiple workers
4. **Production Ready**: The Conversions/ConversionRoom tables can be safely recreated (data is recoverable from PMS XML imports)

## Changes Made

### 1. Alembic Setup

- **[alembic.ini](alembic.ini)**: Configuration file for Alembic
- **[alembic/env.py](alembic/env.py)**: Async-compatible environment setup that:
  - Loads database URL from config.yaml or environment variables
  - Supports PostgreSQL schemas
  - Uses async SQLAlchemy engine

### 2. Initial Migrations

Two migrations were created:

#### Migration 1: `535b70e85b64_initial_schema.py`
Creates all base tables:
- `customers`
- `hashed_customers`
- `reservations`
- `acked_requests`
- `conversions`
- `conversion_rooms`

This migration is idempotent - it only creates missing tables.

#### Migration 2: `8edfc81558db_drop_and_recreate_conversions_tables.py`
Handles the conversion from old production conversions schema to new normalized schema:
- Detects if old conversions tables exist with incompatible schema
- Drops them if needed (data can be recreated from PMS XML imports)
- Allows the initial schema migration to recreate them with correct structure

### 3. Refactored Files

#### [src/alpine_bits_python/db_setup.py](src/alpine_bits_python/db_setup.py)
- **Before**: Ran manual migrations AND created tables using Base.metadata.create_all
- **After**: Only runs startup tasks (data backfills like customer hashing)
- **Note**: Schema migrations now handled by Alembic

#### [src/alpine_bits_python/run_migrations.py](src/alpine_bits_python/run_migrations.py) (NEW)
- Wrapper script to run `alembic upgrade head`
- Can be called standalone or from run_api.py
- Handles errors gracefully

#### [src/alpine_bits_python/api.py](src/alpine_bits_python/api.py)
- **Removed**: `run_all_migrations()` call from lifespan
- **Removed**: `Base.metadata.create_all()` call
- **Changed**: Now only calls `run_startup_tasks()` for data backfills
- **Note**: Assumes migrations have already been run before app start

#### [src/alpine_bits_python/run_api.py](src/alpine_bits_python/run_api.py)
- **Added**: Calls `run_migrations()` BEFORE starting uvicorn
- **Benefit**: Migrations complete before any worker starts
- **Benefit**: Works correctly with multiple workers

### 4. Old Files (Can be removed in future cleanup)

- **[src/alpine_bits_python/migrations.py](src/alpine_bits_python/migrations.py)**: Old manual migration functions
  - These can be safely removed once you verify the Alembic setup works
  - The functionality has been replaced by Alembic migrations

## Usage

### Development

Start the server (migrations run automatically):
```bash
uv run python -m alpine_bits_python.run_api
```

Or run migrations separately:
```bash
uv run alembic upgrade head
uv run python -m alpine_bits_python.run_api
```

### Production with Multiple Workers

The migrations automatically run before uvicorn starts, so you can safely use:
```bash
# Migrations run once, then server starts with multiple workers
uv run python -m alpine_bits_python.run_api

# Or with uvicorn directly (migrations won't run automatically):
uv run alembic upgrade head  # Run this first
uvicorn alpine_bits_python.api:app --workers 4 --host 0.0.0.0 --port 8080
```

### Creating New Migrations

When you modify the database schema in `db.py`:

```bash
# Generate migration automatically
uv run alembic revision --autogenerate -m "description_of_change"

# Or create empty migration to fill in manually
uv run alembic revision -m "description_of_change"

# Review the generated migration in alembic/versions/
# Then apply it
uv run alembic upgrade head
```

### Checking Migration Status

```bash
# Show current revision
uv run alembic current

# Show migration history
uv run alembic history

# Show pending migrations
uv run alembic heads
```

## Benefits

1. **Multiple Worker Safe**: Migrations run once before any worker starts
2. **Proper Migration History**: All schema changes are tracked in version control
3. **Rollback Support**: Can downgrade to previous schema versions if needed
4. **Standard Tool**: Alembic is the industry-standard migration tool for SQLAlchemy
5. **Separation of Concerns**:
   - Schema migrations (Alembic) are separate from startup tasks (db_setup.py)
   - Migrations are separate from application code

## Migration from Old System

If you have an existing database with the old migration system:

1. The initial migration will detect existing tables and skip creating them
2. The conversions table migration will detect old schemas and recreate them
3. All data in other tables is preserved
4. Conversions data will be lost but can be recreated from PMS XML imports

## Important Notes

### Conversions Table Data Loss

The `conversions` and `conversion_rooms` tables will be dropped and recreated with the new schema. This is intentional because:
- The production version has a different schema
- The data can be recreated by re-importing PMS XML files
- This avoids complex data migration logic

If you need to preserve this data, modify the migration before running it.

### Future Migrations

In the future, when you need to change the database schema:

1. Modify the model classes in `db.py`
2. Generate an Alembic migration: `uv run alembic revision --autogenerate -m "description"`
3. Review the generated migration carefully
4. Test it on a dev database first
5. Apply it to production: `uv run alembic upgrade head`

## Configuration

The Alembic setup reads configuration from the same sources as the application:
- `config.yaml` (via `annotatedyaml` with `secrets.yaml`)
- Environment variables (`DATABASE_URL`, `DATABASE_SCHEMA`)

No additional configuration needed!