Ratelimiting and timescaledb changes

README.md

@@ -41,17 +41,24 @@ Edit `.env` and add:
 - **META_AD_ACCOUNT_ID** from Meta Ads Manager (format: `act_1234567890`)
 - **DATABASE_URL** is pre-configured for local Docker setup
 
-### 4. Get Access Token
+### 4. Get Long-Lived Access Token
 
-**Option A: OAuth2 Flow (Recommended)**
+**OAuth2 Flow (Recommended - Gets 60-day token)**
 ```bash
 uv run python src/meta_api_grabber/auth.py
 ```
-Follow the prompts to authorize and save your token.
 
-**Option B: Manual Token**
+This will:
+1. Open OAuth2 authorization in your browser
+2. Exchange the code for a short-lived token
+3. **Automatically exchange for a long-lived token (60 days)**
+4. Save token to `.env`
+5. Save token metadata to `.meta_token.json` (for auto-refresh)
+
+**Manual Token (Not Recommended)**
 - Get a token from [Graph API Explorer](https://developers.facebook.com/tools/explorer/)
 - Add it to `.env` as `META_ACCESS_TOKEN`
+- Note: Manual tokens won't have auto-refresh capability
 
 ### 5. Start Scheduled Collection
 ```bash
@@ -59,6 +66,7 @@ uv run python src/meta_api_grabber/scheduled_grabber.py
 ```
 
 This will:
+- **Automatically refresh tokens** before they expire (checks every cycle)
 - Collect data every 2 hours using the `today` date preset (recommended by Meta)
 - Cache metadata (accounts, campaigns, ad sets) twice daily
 - Store time-series data in TimescaleDB
@@ -87,8 +95,15 @@ uv run python src/meta_api_grabber/insights_grabber.py
 ```bash
 uv run python src/meta_api_grabber/auth.py
 ```
-- Interactive flow to get access token
-- Saves token to `.env` automatically
+- Interactive flow to get long-lived token (60 days)
+- Saves token to `.env` and metadata to `.meta_token.json`
+
+### 4. Check Token Status
+```bash
+uv run python src/meta_api_grabber/token_manager.py
+```
+- Shows token expiry and validity
+- Manually refresh if needed
 
 ## Data Collected
 
@@ -164,12 +179,69 @@ WHERE account_id = 'act_your_id'
 ORDER BY bucket;
 ```
 
-## Rate Limiting
+## Rate Limiting & Backoff
 
-The system is configured to be very conservative:
-- **2 seconds delay** between API requests
-- **Only 1 concurrent request** at a time
+The system implements Meta's best practices for rate limiting:
+
+### Intelligent Rate Limiting
+- **Monitors `x-fb-ads-insights-throttle` header** from every API response
+- Tracks both app-level and account-level usage percentages
+- **Auto-throttles** when usage exceeds 75%
+- **Progressive delays** based on usage (75%: 2x, 85%: 3x, 90%: 5x, 95%: 10x)
+
+### Exponential Backoff
+- **Automatic retries** on rate limit errors (up to 5 attempts)
+- **Exponential backoff**: 2s → 4s → 8s → 16s → 32s
+- Max backoff: 5 minutes
+- Recognizes Meta error codes 17 and 80004
+
+### Conservative Defaults
+- **2 seconds base delay** between API requests
+- **1 concurrent request** at a time
 - **Top 50 campaigns/adsets** per collection
-- **2 hour intervals** between collections
+- **2 hour intervals** between scheduled collections
 
-This ensures you stay well within Meta's API rate limits.
+### Best Practices Applied
+Based on [Meta's official recommendations](https://developers.facebook.com/docs/marketing-api/insights/best-practices/):
+- ✅ Monitor rate limit headers
+- ✅ Pace queries with wait times
+- ✅ Implement backoff when approaching limits
+- ✅ Use date presets (e.g., 'today') instead of custom ranges
+- ✅ Limit query scope and metrics
+
+## Token Management
+
+### Automatic Token Refresh
+
+The system automatically manages token lifecycle:
+
+**Token Types:**
+- **Short-lived tokens**: Valid for 1-2 hours (obtained from OAuth)
+- **Long-lived tokens**: Valid for 60 days (automatically exchanged)
+
+**Auto-Refresh Logic:**
+1. OAuth flow automatically exchanges for 60-day token
+2. Token metadata saved to `.meta_token.json` (includes expiry)
+3. Scheduled grabber checks token before each cycle
+4. Auto-refreshes when < 7 days remaining
+5. New token saved and API reinitialized seamlessly
+
+**Files Created:**
+- `.env` - Contains `META_ACCESS_TOKEN` (updated on refresh)
+- `.meta_token.json` - Token metadata (expiry, issued_at, etc.)
+- Both files are gitignored for security
+
+**Manual Token Operations:**
+
+Check token status:
+```bash
+uv run python src/meta_api_grabber/token_manager.py
+```
+
+Re-authenticate (if token expires):
+```bash
+uv run python src/meta_api_grabber/auth.py
+```
+
+**Long-Running Collection:**
+The scheduled grabber runs indefinitely without manual intervention. Token refresh happens automatically every ~53 days (7 days before the 60-day expiry).