jonas/meta_api_grabber
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c0f24715f426e6abca225f916ff7e4add6cd2355
meta_api_grabber/DEPLOYMENT.md

10 KiB
Raw Blame History

Deployment Guide - Meta API Grabber

Quick Start (Test Deployment for Tonight)

1. Get a Fresh Access Token

Run the OAuth flow to get a new long-lived token (60 days):

uv run python -m meta_api_grabber.auth

This will:

  • Open browser for OAuth authorization
  • Exchange short-lived token for long-lived token (60 days)
  • Save token to .env and .meta_token.json
  • Token will auto-refresh before expiry ✅

2. Verify Your .env File

Ensure your .env has these variables:

# Meta API Credentials
META_APP_ID=your_app_id
META_APP_SECRET=your_app_secret
META_ACCESS_TOKEN=your_long_lived_token  # From step 1

# Database (docker-compose handles this)
DATABASE_URL=postgresql://meta_user:meta_password@localhost:5555/meta_insights

3. Build and Start Everything

# Build the Docker image and start all services
docker-compose up -d --build

This starts:

  • timescaledb: Database for storing insights
  • meta-grabber: Your data collection service ⭐
  • grafana: Visualization dashboard (optional)

4. Monitor the Logs

# Watch the grabber logs in real-time
docker-compose logs -f meta-grabber

# Expected output:
# ============================================================
# SCHEDULED INSIGHTS GRABBER STARTED
# ============================================================
# ✅ Token valid (X days remaining)
# Loading accessible ad accounts...
# Loaded X ad account(s)
# Collection interval: 2.0 hours
# ============================================================
#
# COLLECTION CYCLE - 2025-10-21T...
# ============================================================
# Processing X ad account(s)
# ...

5. Verify It's Running

# Check container status
docker-compose ps

# Should show:
# NAME                 STATUS              PORTS
# meta_timescaledb    Up (healthy)        0.0.0.0:5555->5432/tcp
# meta_api_grabber    Up
# meta_grafana        Up                  0.0.0.0:3555->3000/tcp

6. Let It Run Overnight

The service will:

  • ✅ Collect "today" data every 2 hours
  • ✅ Detect when a new day starts
  • ✅ Fetch "yesterday" data immediately when new day is detected
  • ✅ Update "yesterday" data every 12 hours
  • ✅ Auto-refresh the access token before it expires
  • ✅ Restart automatically if it crashes (restart: unless-stopped)

Token Auto-Refresh

How It Works

The system uses MetaTokenManager which:

  1. On startup: Checks if token expires within 7 days
  2. If expiring soon: Exchanges current token for a new long-lived token
  3. Saves new token: Updates both .env and .meta_token.json
  4. Every cycle: Re-checks token validity before fetching data

Token Lifecycle

New Token (via OAuth)
    ↓
60 days validity
    ↓
Day 53 (7 days before expiry)
    ↓
Auto-refresh triggered
    ↓
New 60-day token issued
    ↓
Cycle repeats indefinitely ♾️

What If Token Expires?

If the token somehow expires (e.g., manual revocation):

  • Container will error out immediately with clear message
  • Logs will show: ❌ Fatal error - Token validation failed
  • Container stops (won't waste API calls)
  • You'll see it in: docker-compose logs meta-grabber

To fix:

  1. Stop the container: docker-compose stop meta-grabber
  2. Get new token: uv run python -m meta_api_grabber.auth
  3. Restart: docker-compose up -d meta-grabber

Data Collection Schedule

Normal Operation (Same Day)

00:00 - Cycle 1:  Fetch "today" (2025-10-21)
02:00 - Cycle 2:  Fetch "today" (2025-10-21)
04:00 - Cycle 3:  Fetch "today" (2025-10-21)
...
22:00 - Cycle 12: Fetch "today" (2025-10-21)

When New Day Starts

00:00 - Cycle 13:
  - Fetch "today" (2025-10-22) ← New date detected!
  - 📅 New day detected: 2025-10-21 -> 2025-10-22
  - Fetch "yesterday" (2025-10-21) immediately

02:00 - Cycle 14:
  - Fetch "today" (2025-10-22)
  - Skip "yesterday" (< 12h since last fetch)

...

12:00 - Cycle 19:
  - Fetch "today" (2025-10-22)
  - Update "yesterday" (12h passed since last fetch)

Checking Data in Database

Connect to Database

# From host machine
docker exec -it meta_timescaledb psql -U meta_user -d meta_insights

# Or using psql directly
psql -h localhost -p 5555 -U meta_user -d meta_insights
# Password: meta_password

Query Today's Data

SELECT
    time,
    account_id,
    date_preset,
    date_start,
    impressions,
    spend
FROM account_insights
WHERE date_preset = 'today'
ORDER BY time DESC
LIMIT 10;

Query Yesterday's Data

SELECT
    time,
    account_id,
    date_preset,
    date_start,
    impressions,
    spend
FROM account_insights
WHERE date_preset = 'yesterday'
ORDER BY time DESC
LIMIT 10;

Check Last Collection Time

SELECT
    date_preset,
    MAX(fetched_at) as last_fetch,
    COUNT(*) as total_records
FROM account_insights
GROUP BY date_preset;

Stopping and Restarting

Stop Everything

docker-compose down

This stops all containers but preserves data:

  • ✅ Database data (in volume timescale_data)
  • ✅ Token files (mounted from host: .env, .meta_token.json)
  • ✅ Grafana dashboards (in volume grafana_data)

Stop Just the Grabber

docker-compose stop meta-grabber

Restart the Grabber

docker-compose restart meta-grabber

View Logs

# Follow logs in real-time
docker-compose logs -f meta-grabber

# Last 100 lines
docker-compose logs --tail=100 meta-grabber

# All services
docker-compose logs -f

Configuration

Adjusting Collection Interval

Edit scheduled_grabber.py line 522:

await grabber.run_scheduled(
    interval_hours=2.0,  # ← Change this (in hours)
    refresh_metadata_every_n_cycles=12,
)

Then rebuild:

docker-compose up -d --build meta-grabber

Adjusting Number of Accounts

Edit scheduled_grabber.py line 519:

grabber = ScheduledInsightsGrabber(
    max_accounts=3,  # ← Change this (None = all accounts)
)

Adjusting Yesterday Fetch Interval

Currently hardcoded to 12 hours in _should_fetch_yesterday() method at line 175.

To change, edit:

return hours_since_last_fetch >= 12.0  # ← Change to 6.0 for 6 hours, etc.

Troubleshooting

Container Keeps Restarting

# Check logs for error
docker-compose logs meta-grabber

# Common issues:
# 1. Token invalid → Get new token
# 2. Database not ready → Wait for timescaledb health check
# 3. Missing .env file → Create from .env.example

No Data Being Collected

# Check if grabber is running
docker-compose ps

# Check logs for API errors
docker-compose logs meta-grabber | grep "Error"

# Verify token
uv run python -m meta_api_grabber.token_manager

Database Connection Failed

# Check if TimescaleDB is healthy
docker-compose ps timescaledb

# Should show: "Up (healthy)"

# If not healthy, check TimescaleDB logs
docker-compose logs timescaledb

Yesterday Data Not Appearing

Check logs for:

📅 New day detected: YYYY-MM-DD -> YYYY-MM-DD
Fetching yesterday's data (first time)

If you don't see this, the system hasn't detected a new day yet.

To force a test:

  1. Stop grabber: docker-compose stop meta-grabber
  2. Manually insert yesterday data (see manual testing section)
  3. Restart: docker-compose start meta-grabber

Manual Testing (Before Overnight Run)

Test Token Validity

# This will check token and auto-refresh if needed
uv run python -m meta_api_grabber.token_manager

Test Single Collection Cycle

# Run one cycle without Docker
uv run python -c "
import asyncio
from src.meta_api_grabber.scheduled_grabber import ScheduledInsightsGrabber

async def test():
    grabber = ScheduledInsightsGrabber(max_accounts=1)
    await grabber.db.connect()
    await grabber.db.initialize_schema()
    await grabber.load_ad_accounts()
    await grabber.run_collection_cycle()
    await grabber.db.close()

asyncio.run(test())
"

Verify Database Schema

docker exec -it meta_timescaledb psql -U meta_user -d meta_insights -c "\dt"

# Should show:
# account_insights
# campaign_insights
# adset_insights
# ad_accounts
# campaigns
# adsets

Monitoring in Production

Health Checks

The container has a built-in health check:

docker inspect meta_api_grabber | grep -A 5 Health

Resource Usage

# Monitor container resources
docker stats meta_api_grabber

Log Rotation

Logs are automatically rotated (see docker-compose.yml):

  • Max size: 10MB per file
  • Max files: 3
  • Total max: ~30MB of logs

Backup Considerations

What to Backup

  1. Database (most important):

    docker exec meta_timescaledb pg_dump -U meta_user meta_insights > backup.sql

  2. Token files:

    cp .env .env.backup
cp .meta_token.json .meta_token.json.backup

  3. Configuration:

    • .env
    • docker-compose.yml

Restore from Backup

# Restore database
docker exec -i meta_timescaledb psql -U meta_user meta_insights < backup.sql

# Restore token files
cp .env.backup .env
cp .meta_token.json.backup .meta_token.json

# Restart
docker-compose restart meta-grabber

Production Checklist

Before leaving it running overnight:

  • Fresh access token obtained (60 days validity)
  • .env file has all required variables
  • .meta_token.json exists with token metadata
  • docker-compose up -d --build succeeded
  • All containers show "Up" in docker-compose ps
  • Logs show successful data collection
  • Database contains data (SELECT COUNT(*) FROM account_insights)
  • Token auto-refresh is enabled (auto_refresh_token=True)
  • Restart policy is set (restart: unless-stopped)

Summary

To deploy for overnight testing:

# 1. Get token
uv run python -m meta_api_grabber.auth

# 2. Start everything
docker-compose up -d --build

# 3. Verify it's working
docker-compose logs -f meta-grabber

# 4. Let it run!
# Come back tomorrow and check:
docker-compose logs meta-grabber | grep "New day detected"

The system will handle everything automatically:

  • ✅ Data collection every 2 hours
  • ✅ New day detection
  • ✅ Yesterday data collection
  • ✅ Token auto-refresh
  • ✅ Auto-restart on failures

Sleep well! 😴