alpinebits_python/MIGRATION_REFACTORING.md

Database Migration Refactoring

Summary

This refactoring changes the database handling from manual schema migrations in migrations.py to using Alembic for proper database migrations. The key improvements are:

1. Alembic Integration: All schema migrations now use Alembic's migration framework
2. Separation of Concerns: Migrations (schema changes) are separated from startup tasks (data backfills)
3. Pre-startup Migrations: Database migrations run BEFORE the application starts, avoiding issues with multiple workers
4. Production Ready: The Conversions/ConversionRoom tables can be safely recreated (data is recoverable from PMS XML imports)

Changes Made

1. Alembic Setup

• alembic.ini: Configuration file for Alembic
• alembic/env.py: Async-compatible environment setup that:
  • Loads database URL from config.yaml or environment variables
  • Supports PostgreSQL schemas
  • Uses async SQLAlchemy engine

2. Initial Migrations

Two migrations were created:

Migration 1: 535b70e85b64_initial_schema.py

Creates all base tables:

• customers
• hashed_customers
• reservations
• acked_requests
• conversions
• conversion_rooms

This migration is idempotent - it only creates missing tables.

Migration 2: 8edfc81558db_drop_and_recreate_conversions_tables.py

Handles the conversion from old production conversions schema to new normalized schema:

• Detects if old conversions tables exist with incompatible schema
• Drops them if needed (data can be recreated from PMS XML imports)
• Allows the initial schema migration to recreate them with correct structure

3. Refactored Files

src/alpine_bits_python/db_setup.py

• Before: Ran manual migrations AND created tables using Base.metadata.create_all
• After: Only runs startup tasks (data backfills like customer hashing)
• Note: Schema migrations now handled by Alembic

src/alpine_bits_python/run_migrations.py (NEW)

• Wrapper script to run alembic upgrade head
• Can be called standalone or from run_api.py
• Handles errors gracefully

src/alpine_bits_python/api.py

• Removed: run_all_migrations() call from lifespan
• Removed: Base.metadata.create_all() call
• Changed: Now only calls run_startup_tasks() for data backfills
• Note: Assumes migrations have already been run before app start

src/alpine_bits_python/run_api.py

• Added: Calls run_migrations() BEFORE starting uvicorn
• Benefit: Migrations complete before any worker starts
• Benefit: Works correctly with multiple workers

4. Old Files (Can be removed in future cleanup)

• src/alpine_bits_python/migrations.py: Old manual migration functions
  • These can be safely removed once you verify the Alembic setup works
  • The functionality has been replaced by Alembic migrations

Usage

Development

Start the server (migrations run automatically):

uv run python -m alpine_bits_python.run_api

Or run migrations separately:

uv run alembic upgrade head
uv run python -m alpine_bits_python.run_api

Production with Multiple Workers

The migrations automatically run before uvicorn starts, so you can safely use:

# Migrations run once, then server starts with multiple workers
uv run python -m alpine_bits_python.run_api

# Or with uvicorn directly (migrations won't run automatically):
uv run alembic upgrade head  # Run this first
uvicorn alpine_bits_python.api:app --workers 4 --host 0.0.0.0 --port 8080

Creating New Migrations

When you modify the database schema in db.py:

# Generate migration automatically
uv run alembic revision --autogenerate -m "description_of_change"

# Or create empty migration to fill in manually
uv run alembic revision -m "description_of_change"

# Review the generated migration in alembic/versions/
# Then apply it
uv run alembic upgrade head

Checking Migration Status

# Show current revision
uv run alembic current

# Show migration history
uv run alembic history

# Show pending migrations
uv run alembic heads

Benefits

1. Multiple Worker Safe: Migrations run once before any worker starts
2. Proper Migration History: All schema changes are tracked in version control
3. Rollback Support: Can downgrade to previous schema versions if needed
4. Standard Tool: Alembic is the industry-standard migration tool for SQLAlchemy
5. Separation of Concerns:
   • Schema migrations (Alembic) are separate from startup tasks (db_setup.py)
   • Migrations are separate from application code

Migration from Old System

If you have an existing database with the old migration system:

1. The initial migration will detect existing tables and skip creating them
2. The conversions table migration will detect old schemas and recreate them
3. All data in other tables is preserved
4. Conversions data will be lost but can be recreated from PMS XML imports

Important Notes

Conversions Table Data Loss

The conversions and conversion_rooms tables will be dropped and recreated with the new schema. This is intentional because:

• The production version has a different schema
• The data can be recreated by re-importing PMS XML files
• This avoids complex data migration logic

If you need to preserve this data, modify the migration before running it.

Future Migrations

In the future, when you need to change the database schema:

1. Modify the model classes in db.py
2. Generate an Alembic migration: uv run alembic revision --autogenerate -m "description"
3. Review the generated migration carefully
4. Test it on a dev database first
5. Apply it to production: uv run alembic upgrade head

Configuration

The Alembic setup reads configuration from the same sources as the application:

• config.yaml (via annotatedyaml with secrets.yaml)
• Environment variables (DATABASE_URL, DATABASE_SCHEMA)

No additional configuration needed!