meta_api_grabber/RATE_LIMITER_ENHANCEMENTS.md

# Meta API Rate Limiter Enhancements

## Summary

Enhanced the rate limiter in [rate_limiter.py](src/meta_api_grabber/rate_limiter.py) to monitor **all** Meta API rate limit headers as documented in the [official Meta documentation](https://developers.facebook.com/docs/graph-api/overview/rate-limiting).

## New Headers Monitored

### 1. **X-App-Usage** (Platform Rate Limits)
Tracks application-level rate limits across all users.

**Fields:**
- `call_count`: Percentage of calls made (0-100)
- `total_cputime`: Percentage of CPU time used (0-100)
- `total_time`: Percentage of total time used (0-100)

**Example:**
```json
{
  "call_count": 28,
  "total_time": 25,
  "total_cputime": 25
}
```

### 2. **X-Ad-Account-Usage** (Ad Account Specific)
Tracks rate limits for specific ad accounts.

**Fields:**
- `acc_id_util_pct`: Percentage of ad account usage (0-100)
- `reset_time_duration`: Time in seconds until rate limit resets
- `ads_api_access_tier`: Access tier (e.g., "standard_access", "development_access")

**Example:**
```json
{
  "acc_id_util_pct": 9.67,
  "reset_time_duration": 100,
  "ads_api_access_tier": "standard_access"
}
```

### 3. **X-Business-Use-Case-Usage** (Business Use Case Limits)
Tracks rate limits per business use case (ads_insights, ads_management, etc.).

**Fields:**
- `business_id`: Business object ID
- `type`: Type of BUC (ads_insights, ads_management, custom_audience, etc.)
- `call_count`: Percentage of calls made (0-100)
- `total_cputime`: Percentage of CPU time (0-100)
- `total_time`: Percentage of total time (0-100)
- `estimated_time_to_regain_access`: Time in minutes until access is restored
- `ads_api_access_tier`: Access tier

**Example:**
```json
{
  "66782684": [{
    "type": "ads_management",
    "call_count": 95,
    "total_cputime": 20,
    "total_time": 20,
    "estimated_time_to_regain_access": 0,
    "ads_api_access_tier": "development_access"
  }]
}
```

### 4. **x-fb-ads-insights-throttle** (Legacy)
Original header still supported for backward compatibility.

**Fields:**
- `app_id_util_pct`: App usage percentage
- `acc_id_util_pct`: Account usage percentage

## Key Enhancements

### 1. Intelligent Throttling
The rate limiter now uses `estimated_time_to_regain_access` and `reset_time_duration` to calculate optimal delays:

```python
# If we have estimated_time_to_regain_access from BUC header
if self.estimated_time_to_regain_access > 0:
    delay = self.estimated_time_to_regain_access * 60  # Convert minutes to seconds

# If we have reset_time_duration from Ad Account header
elif self.reset_time_duration > 0:
    delay = self.reset_time_duration * 0.5  # Use fraction as safety margin
```

### 2. Comprehensive Error Code Detection
Expanded error code detection to include all Meta rate limit error codes:

- **4**: App rate limit
- **17**: User rate limit
- **32**: Pages rate limit
- **613**: Custom rate limit
- **80000-80014**: Business Use Case rate limits (Ads Insights, Ads Management, Custom Audience, Instagram, LeadGen, Messenger, Pages, WhatsApp, Catalog)

### 3. Debug Logging
All headers are now logged in DEBUG mode with detailed parsing information:

```python
logger.debug(f"X-App-Usage header: {header_value}")
logger.debug(f"Parsed X-App-Usage: {result}")
```

### 4. Enhanced Statistics
The `get_stats()` and `print_stats()` methods now display comprehensive metrics from all headers:

```
======================================================================
RATE LIMITER STATISTICS
======================================================================
Total Requests:              0
Throttled Requests:          0
Rate Limit Errors:           0

X-App-Usage (Platform Rate Limits):
  Call Count:                95.0%
  Total CPU Time:            90.0%
  Total Time:                88.0%

X-Ad-Account-Usage:
  Account Usage:             97.5%
  Reset Time Duration:       300s
  API Access Tier:           standard_access

X-Business-Use-Case-Usage:
  Type: ads_insights
    Call Count:              98.0%
    Total CPU Time:          95.0%
    Total Time:              92.0%
    Est. Time to Regain:     15 min

Legacy (x-fb-ads-insights-throttle):
  App Usage:                 93.0%
  Account Usage:             96.0%

Max Usage Across All Metrics: 98.0%
Currently Throttled:          True
======================================================================
```

## Usage

### Enable Debug Logging
To see all header parsing in debug mode:

```python
import logging

logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
```

### Access New Metrics
All metrics are available through the `get_stats()` method:

```python
stats = limiter.get_stats()

print(f"App call count: {stats['app_call_count']}%")
print(f"Ad account usage: {stats['ad_account_usage_pct']}%")
print(f"Reset in: {stats['reset_time_duration']}s")
print(f"Regain access in: {stats['estimated_time_to_regain_access']} min")
print(f"API tier: {stats['ads_api_access_tier']}")

# Business use case details
for buc in stats['buc_usage']:
    print(f"BUC {buc['type']}: {buc['call_count']}%")
```

## Testing

Run the test script to see the rate limiter in action:

```bash
uv run python test_rate_limiter.py
```

This will demonstrate:
- Parsing all four header types
- Intelligent throttling based on usage
- Comprehensive statistics display
- Debug logging output

## References

- [Meta Graph API Rate Limiting](https://developers.facebook.com/docs/graph-api/overview/rate-limiting)
- [Meta Marketing API Best Practices](https://developers.facebook.com/docs/marketing-api/insights/best-practices/)