meta_api_grabber/TIMESTAMP_LOGIC.md

Timestamp Logic for Meta Insights Data

Overview

The system now uses intelligent timestamp assignment based on the date_preset and account timezone to ensure accurate day-by-day plotting while handling Meta's timezone-based data reporting.

Key Concepts

Meta's Timezone Behavior

Meta API reports data based on the ad account's timezone:

• "today" = today in the account's timezone
• "yesterday" = yesterday in the account's timezone
• An account in America/Los_Angeles (PST/PDT) will have different "today" dates than an account in Europe/London (GMT/BST)

The Timestamp Challenge

When storing time-series data, we need timestamps that:

1. Reflect the actual date of the data (not when we fetched it)
2. Account for the ad account's timezone
3. Allow for accurate day-by-day plotting
4. Use current time for "today" (live, constantly updating data)
5. Use historical timestamps for past data (fixed point in time)

Implementation

The _compute_timestamp() Method

Located in scheduled_grabber.py, this method computes the appropriate timestamp for each data point:

def _compute_timestamp(
    self,
    date_preset: str,
    date_start_str: Optional[str],
    account_timezone: str
) -> datetime:
    """
    Compute the appropriate timestamp for storing insights data.

    For 'today': Use current time (data is live, constantly updating)
    For historical presets: Use noon of that date in the account's timezone,
                           then convert to UTC for storage
    """

Logic Flow

For "today" Data:

date_preset = "today"
    ↓
Use datetime.now(timezone.utc)
    ↓
Store with current timestamp
    ↓
Multiple fetches during the day overwrite each other
(database ON CONFLICT updates existing records)

Why: Today's data changes throughout the day. Using the current time ensures we can see when data was last updated.

For Historical Data (e.g., "yesterday"):

date_preset = "yesterday"
date_start = "2025-10-20"
account_timezone = "America/Los_Angeles"
    ↓
Create datetime: 2025-10-20 12:00:00 in PST
    ↓
Convert to UTC: 2025-10-20 19:00:00 UTC (PST is UTC-7 in summer)
    ↓
Store with this timestamp
    ↓
Data point will plot on the correct day

Why: Historical data is fixed. Using noon in the account's timezone ensures:

1. The timestamp falls on the correct calendar day
2. Timezone differences don't cause data to appear on wrong days
3. Consistent time (noon) for all historical data points

Timezone Handling

Account timezones are:

1. Cached during metadata collection in the ad_accounts table
2. Retrieved from database using _get_account_timezone()
3. Cached in memory to avoid repeated database queries

Example timezone conversion:

# Account in Los Angeles (PST/PDT = UTC-8/UTC-7)
date_start = "2025-10-20"  # Yesterday in account timezone
account_tz = ZoneInfo("America/Los_Angeles")

# Create datetime at noon LA time
timestamp_local = datetime(2025, 10, 20, 12, 0, 0, tzinfo=account_tz)
# Result: 2025-10-20 12:00:00-07:00 (PDT)

# Convert to UTC for storage
timestamp_utc = timestamp_local.astimezone(timezone.utc)
# Result: 2025-10-20 19:00:00+00:00 (UTC)

Examples

Example 1: Same Account, Multiple Days

Ad Account: act_123 in America/New_York (EST = UTC-5)

Scenario:

• Fetch "yesterday" data on Oct 21, 2025
• date_start from API: "2025-10-20"

Timestamp Calculation:

2025-10-20 12:00:00 EST (noon in NY)
    ↓ convert to UTC
2025-10-20 17:00:00 UTC (stored in database)

Result: Data plots on October 20 regardless of viewer's timezone

Example 2: Different Timezones

Account A: America/Los_Angeles (PDT = UTC-7) Account B: Europe/London (BST = UTC+1)

Both fetch "yesterday" on Oct 21, 2025:

Account	date_start	Local Time	UTC Stored
A (LA)	2025-10-20	12:00 PDT	19:00 UTC
B (London)	2025-10-20	12:00 BST	11:00 UTC

Result: Both plot on October 20, even though stored at different UTC times

Example 3: "Today" Data Updates

Account: Any timezone Fetches: Every 2 hours

Fetch Time (UTC)	date_preset	date_start	Stored Timestamp
08:00 UTC	"today"	2025-10-21	08:00 UTC (current)
10:00 UTC	"today"	2025-10-21	10:00 UTC (current)
12:00 UTC	"today"	2025-10-21	12:00 UTC (current)

Result: Latest data always has the most recent timestamp, showing when it was fetched

Database Schema Implications

Primary Key Constraint

All insights tables use:

PRIMARY KEY (time, account_id)  -- or (time, campaign_id), etc.

With ON CONFLICT DO UPDATE:

INSERT INTO account_insights (time, account_id, ...)
VALUES (...)
ON CONFLICT (time, account_id)
DO UPDATE SET
    impressions = EXCLUDED.impressions,
    spend = EXCLUDED.spend,
    ...

Behavior by Date Preset

"today" data:

• Multiple fetches in same day have different timestamps
• No conflicts (different time values)
• Creates multiple rows, building time-series
• Can see data evolution throughout the day

"yesterday" data:

• All fetches use same timestamp (noon in account TZ)
• Conflicts occur (same time value)
• Updates existing row with fresh data
• Only keeps latest version

Querying Data

Query by Day (Recommended)

-- Get all data for a specific date range
SELECT
    DATE(time AT TIME ZONE 'America/Los_Angeles') as data_date,
    account_id,
    AVG(spend) as avg_spend,
    MAX(impressions) as max_impressions
FROM account_insights
WHERE time >= '2025-10-15' AND time < '2025-10-22'
GROUP BY data_date, account_id
ORDER BY data_date DESC;

Filter by Date Preset

-- Get only historical (yesterday) data
SELECT * FROM account_insights
WHERE date_preset = 'yesterday'
ORDER BY time DESC;

-- Get only live (today) data
SELECT * FROM account_insights
WHERE date_preset = 'today'
ORDER BY time DESC;

Plotting Considerations

When creating day-by-day plots:

Option 1: Use date_start Field

SELECT
    date_start,  -- Already a DATE type
    SUM(spend) as total_spend
FROM account_insights
GROUP BY date_start
ORDER BY date_start;

Option 2: Extract Date from Timestamp

SELECT
    DATE(time) as data_date,  -- Convert timestamp to date
    SUM(spend) as total_spend
FROM account_insights
GROUP BY data_date
ORDER BY data_date;

For "Today" Data (Multiple Points Per Day)

-- Get latest "today" data for each account
SELECT DISTINCT ON (account_id)
    account_id,
    time,
    spend,
    impressions
FROM account_insights
WHERE date_preset = 'today'
ORDER BY account_id, time DESC;

Benefits

1. Accurate Day Assignment: Historical data always plots on correct calendar day
2. Timezone Aware: Respects Meta's timezone-based reporting
3. Live Updates: "Today" data shows progression throughout the day
4. Historical Accuracy: Yesterday data uses consistent timestamp
5. Update Tracking: Can see when "yesterday" data was last refreshed
6. Query Flexibility: Can query by date_start or extract date from time

Troubleshooting

Data Appears on Wrong Day

Symptom: Yesterday's data shows on wrong day in graphs Cause: Timezone not being considered Solution: Already handled! Our _compute_timestamp() uses account timezone

Multiple Entries for Yesterday

Symptom: Multiple rows for same account and yesterday's date Cause: Database conflict resolution not working Check:

• Primary key includes time and account_id
• ON CONFLICT clause exists in insert statements
• Timestamp is actually the same (should be: noon in account TZ)

Timezone Errors

Symptom: ZoneInfo errors or invalid timezone names Cause: Invalid timezone in database or missing timezone data Solution: Code falls back to UTC if timezone is invalid

except Exception as e:
    print(f"Warning: Could not parse timezone '{account_timezone}': {e}")
    return datetime.now(timezone.utc)

Summary

The timestamp logic ensures:

• ✅ "Today" data uses current time (live updates)
• ✅ Historical data uses noon in account's timezone
• ✅ Timezone conversions handled automatically
• ✅ Data plots correctly day-by-day
• ✅ Account timezone cached for performance
• ✅ Fallback handling for missing/invalid timezones

This provides accurate, timezone-aware time-series data ready for visualization!