Updated meta rate limiter logs

2025-11-04 11:42:18 +01:00
parent 638b466daf
commit 6127ea4650
10 changed files with 1304 additions and 58 deletions
--- a/RATE_LIMITER_ENHANCEMENTS.md
+++ b/RATE_LIMITER_ENHANCEMENTS.md
@@ -0,0 +1,193 @@
+# Meta API Rate Limiter Enhancements
+
+## Summary
+
+Enhanced the rate limiter in [rate_limiter.py](src/meta_api_grabber/rate_limiter.py) to monitor **all** Meta API rate limit headers as documented in the [official Meta documentation](https://developers.facebook.com/docs/graph-api/overview/rate-limiting).
+
+## New Headers Monitored
+
+### 1. **X-App-Usage** (Platform Rate Limits)
+Tracks application-level rate limits across all users.
+
+**Fields:**
+- `call_count`: Percentage of calls made (0-100)
+- `total_cputime`: Percentage of CPU time used (0-100)
+- `total_time`: Percentage of total time used (0-100)
+
+**Example:**
+```json
+{
+  "call_count": 28,
+  "total_time": 25,
+  "total_cputime": 25
+}
+```
+
+### 2. **X-Ad-Account-Usage** (Ad Account Specific)
+Tracks rate limits for specific ad accounts.
+
+**Fields:**
+- `acc_id_util_pct`: Percentage of ad account usage (0-100)
+- `reset_time_duration`: Time in seconds until rate limit resets
+- `ads_api_access_tier`: Access tier (e.g., "standard_access", "development_access")
+
+**Example:**
+```json
+{
+  "acc_id_util_pct": 9.67,
+  "reset_time_duration": 100,
+  "ads_api_access_tier": "standard_access"
+}
+```
+
+### 3. **X-Business-Use-Case-Usage** (Business Use Case Limits)
+Tracks rate limits per business use case (ads_insights, ads_management, etc.).
+
+**Fields:**
+- `business_id`: Business object ID
+- `type`: Type of BUC (ads_insights, ads_management, custom_audience, etc.)
+- `call_count`: Percentage of calls made (0-100)
+- `total_cputime`: Percentage of CPU time (0-100)
+- `total_time`: Percentage of total time (0-100)
+- `estimated_time_to_regain_access`: Time in minutes until access is restored
+- `ads_api_access_tier`: Access tier
+
+**Example:**
+```json
+{
+  "66782684": [{
+    "type": "ads_management",
+    "call_count": 95,
+    "total_cputime": 20,
+    "total_time": 20,
+    "estimated_time_to_regain_access": 0,
+    "ads_api_access_tier": "development_access"
+  }]
+}
+```
+
+### 4. **x-fb-ads-insights-throttle** (Legacy)
+Original header still supported for backward compatibility.
+
+**Fields:**
+- `app_id_util_pct`: App usage percentage
+- `acc_id_util_pct`: Account usage percentage
+
+## Key Enhancements
+
+### 1. Intelligent Throttling
+The rate limiter now uses `estimated_time_to_regain_access` and `reset_time_duration` to calculate optimal delays:
+
+```python
+# If we have estimated_time_to_regain_access from BUC header
+if self.estimated_time_to_regain_access > 0:
+    delay = self.estimated_time_to_regain_access * 60  # Convert minutes to seconds
+
+# If we have reset_time_duration from Ad Account header
+elif self.reset_time_duration > 0:
+    delay = self.reset_time_duration * 0.5  # Use fraction as safety margin
+```
+
+### 2. Comprehensive Error Code Detection
+Expanded error code detection to include all Meta rate limit error codes:
+
+- **4**: App rate limit
+- **17**: User rate limit
+- **32**: Pages rate limit
+- **613**: Custom rate limit
+- **80000-80014**: Business Use Case rate limits (Ads Insights, Ads Management, Custom Audience, Instagram, LeadGen, Messenger, Pages, WhatsApp, Catalog)
+
+### 3. Debug Logging
+All headers are now logged in DEBUG mode with detailed parsing information:
+
+```python
+logger.debug(f"X-App-Usage header: {header_value}")
+logger.debug(f"Parsed X-App-Usage: {result}")
+```
+
+### 4. Enhanced Statistics
+The `get_stats()` and `print_stats()` methods now display comprehensive metrics from all headers:
+
+```
+======================================================================
+RATE LIMITER STATISTICS
+======================================================================
+Total Requests:              0
+Throttled Requests:          0
+Rate Limit Errors:           0
+
+X-App-Usage (Platform Rate Limits):
+  Call Count:                95.0%
+  Total CPU Time:            90.0%
+  Total Time:                88.0%
+
+X-Ad-Account-Usage:
+  Account Usage:             97.5%
+  Reset Time Duration:       300s
+  API Access Tier:           standard_access
+
+X-Business-Use-Case-Usage:
+  Type: ads_insights
+    Call Count:              98.0%
+    Total CPU Time:          95.0%
+    Total Time:              92.0%
+    Est. Time to Regain:     15 min
+
+Legacy (x-fb-ads-insights-throttle):
+  App Usage:                 93.0%
+  Account Usage:             96.0%
+
+Max Usage Across All Metrics: 98.0%
+Currently Throttled:          True
+======================================================================
+```
+
+## Usage
+
+### Enable Debug Logging
+To see all header parsing in debug mode:
+
+```python
+import logging
+
+logging.basicConfig(
+    level=logging.DEBUG,
+    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+)
+```
+
+### Access New Metrics
+All metrics are available through the `get_stats()` method:
+
+```python
+stats = limiter.get_stats()
+
+print(f"App call count: {stats['app_call_count']}%")
+print(f"Ad account usage: {stats['ad_account_usage_pct']}%")
+print(f"Reset in: {stats['reset_time_duration']}s")
+print(f"Regain access in: {stats['estimated_time_to_regain_access']} min")
+print(f"API tier: {stats['ads_api_access_tier']}")
+
+# Business use case details
+for buc in stats['buc_usage']:
+    print(f"BUC {buc['type']}: {buc['call_count']}%")
+```
+
+## Testing
+
+Run the test script to see the rate limiter in action:
+
+```bash
+uv run python test_rate_limiter.py
+```
+
+This will demonstrate:
+- Parsing all four header types
+- Intelligent throttling based on usage
+- Comprehensive statistics display
+- Debug logging output
+
+## References
+
+- [Meta Graph API Rate Limiting](https://developers.facebook.com/docs/graph-api/overview/rate-limiting)
+- [Meta Marketing API Best Practices](https://developers.facebook.com/docs/marketing-api/insights/best-practices/)