Fix field schema validation test and update database schema
- Fixed field extraction logic in test_field_schema_validation.py to properly parse methods with docstrings - Previous regex was too greedy and matched across multiple method definitions - Now uses proper parenthesis and docstring matching to isolate method bodies - Correctly handles both 'fields = [...]' and 'fields = common_fields + [...]' patterns - Updated db_schema.sql to include missing columns: - campaign_insights: added frequency, cpp, cost_per_action_type columns - adset_insights: added account_currency column - campaign_insights_by_country: added frequency, cpp, cost_per_action_type columns - All field schema validation tests now pass - Test dynamically extracts fields from scheduled_grabber.py source code - Compares against actual database schema from db_schema.sql - Properly filters metadata-only fields (campaign_id, campaign_name, etc.) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -19,20 +19,44 @@ Run specific test file:
|
||||
uv run pytest tests/test_field_schema_validation.py -v
|
||||
```
|
||||
|
||||
Run with coverage:
|
||||
Run with verbose output:
|
||||
```bash
|
||||
uv run pytest --cov=meta_api_grabber
|
||||
uv run pytest tests/test_field_schema_validation.py -v -s
|
||||
```
|
||||
|
||||
## Test Files
|
||||
|
||||
### `test_field_schema_validation.py`
|
||||
Validates that all fields requested by the grab_* methods in `scheduled_grabber.py` exist in the database schema. This ensures:
|
||||
- Field compatibility between Meta API and database
|
||||
- Early detection of schema mismatches
|
||||
- Consistency across all insight levels (account, campaign, adset, country)
|
||||
### `test_field_schema_validation.py` (Integration Test)
|
||||
|
||||
**Why this test is important:** When new fields are added to the Meta API field lists, this test quickly alerts you if the corresponding database columns need to be added.
|
||||
This is a critical integration test that validates all fields requested by the grab_* methods in `scheduled_grabber.py` exist in the actual database schema.
|
||||
|
||||
**What it does:**
|
||||
1. Parses `db_schema.sql` to extract actual table columns
|
||||
2. Checks fields requested by each grab method:
|
||||
- `grab_account_insights()` → `account_insights` table
|
||||
- `grab_campaign_insights()` → `campaign_insights` table
|
||||
- `grab_adset_insights()` → `adset_insights` table
|
||||
- `grab_campaign_insights_by_country()` → `campaign_insights_by_country` table
|
||||
3. Verifies all requested fields exist in the corresponding database table
|
||||
|
||||
**Why this test is important:** When new fields are added to the Meta API field lists, this test quickly alerts you if the corresponding database columns need to be added. Since fields are only added (never removed), the test helps catch schema mismatches early.
|
||||
|
||||
**Test methods:**
|
||||
- `test_account_insights_fields()` - Validates account-level insight fields
|
||||
- `test_campaign_insights_fields()` - Validates campaign-level insight fields
|
||||
- `test_adset_insights_fields()` - Validates ad set-level insight fields
|
||||
- `test_campaign_insights_by_country_fields()` - Validates country breakdown fields
|
||||
- `test_all_tables_exist()` - Ensures all required insight tables exist
|
||||
- `test_schema_documentation()` - Prints out the parsed schema for reference
|
||||
|
||||
**Output example:**
|
||||
```
|
||||
Table: account_insights
|
||||
Columns (17): account_id, actions, clicks, cost_per_action_type, cpc, cpm, cpp, ctr, ...
|
||||
|
||||
Table: campaign_insights
|
||||
Columns (15): account_id, actions, campaign_id, clicks, cpc, cpm, ctr, ...
|
||||
```
|
||||
|
||||
## Writing Tests
|
||||
|
||||
@@ -56,3 +80,36 @@ Run everything except integration tests:
|
||||
```bash
|
||||
uv run pytest -m "not integration"
|
||||
```
|
||||
|
||||
## Schema Validation Workflow
|
||||
|
||||
When you add new fields to a grab method:
|
||||
|
||||
1. **Add fields to `scheduled_grabber.py`:**
|
||||
```python
|
||||
fields = [
|
||||
...
|
||||
AdsInsights.Field.new_field, # New field added
|
||||
]
|
||||
```
|
||||
|
||||
2. **Run tests to see what's missing:**
|
||||
```bash
|
||||
uv run pytest tests/test_field_schema_validation.py -v -s
|
||||
```
|
||||
|
||||
3. **Test output will show:**
|
||||
```
|
||||
adset_insights table missing columns: {'new_field'}
|
||||
Available: [account_id, actions, adset_id, ...]
|
||||
```
|
||||
|
||||
4. **Update `db_schema.sql` with the new column:**
|
||||
```sql
|
||||
ALTER TABLE adset_insights ADD COLUMN IF NOT EXISTS new_field TYPE;
|
||||
```
|
||||
|
||||
5. **Run tests again to verify:**
|
||||
```bash
|
||||
uv run pytest tests/test_field_schema_validation.py -v
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user