Debug: Campaign Processing Steps

Field	Value
ID	239
Title	Debeers
Keywords	Debeers
Platforms	twitter youtube instagram news blogs facebook
Status	active
Date From	2026-05-19
Date To	2026-05-19
Client Keywords
Monitor Links
Relevancy Threshold	0.70

Step 2: Collect Data from Platforms

API Endpoint: api/collect_data_step2.php (POST)

Database Tables: twitter_raw, youtube_raw, instagram_raw, news_raw, blogs_raw, facebook_raw (INSERT, SELECT COUNT)

Database Table: background_jobs (SELECT)

ℹ️ What this means: Background jobs track long-running scraping processes. If no jobs are found, it means either:
• Scraping completed and jobs were cleaned up
• Scraping happened directly without creating job records (this is normal)
• No background scraping has been started yet
✅ Check Step 3 (Data Collection Status) to see if data was actually collected - that's what matters!

No background jobs found for this campaign

This is normal! Background jobs are optional tracking records. The important thing is whether data was collected - check Step 3 below.

ℹ️ This collects 100 results from each selected platform. This may take 10-20 minutes.

Data Collection Status by Platform

ℹ️ Limits: Loaded from platform_limits table (managed via Settings > Platform Limit)

Platform	Table Name	Records Count	Limit	Status
Twitter	`twitter_raw`	1	`1000`	✓ Data Collected
Youtube	`youtube_raw`	12	`100`	✓ Data Collected
Instagram	`instagram_raw`	397	`100`	✓ Data Collected
Facebook	`facebook_raw`	17	`100`	✓ Data Collected
News	`news_raw`	0	`1000`	⚠ No Data
Blogs	`blogs_raw`	0	`1000`	⚠ No Data
Total Records	427	-	✓ Data Available	-

Step 3: AI Relevancy Analysis

API Endpoint: api/check_raw_mentions.php (POST)

OpenAI key: loaded from api_keys_db.api_keys (then OPENAI_API_KEY env, then config). Model: gpt-4o-mini.

Database Tables: twitter_raw, youtube_raw, instagram_raw, news_raw, blogs_raw, facebook_raw (SELECT), ai_relevancy_results (INSERT)

ℹ️ Run sends every raw row to the API with force_reprocess (full pass, re-scores by AI). Re-Run does the same. Batches of 50; large campaigns can take 15+ minutes.

AI Relevancy Results

Database Table: ai_jobs (SELECT)

No AI jobs found for this campaign. Click the button above to create AI jobs from raw data.

Breakdown by Platform:

Breakdown by Relevancy:

Breakdown by Sentiment:

Breakdown by Processing State:

Sample Results (Latest 5):

Platform	Results Count
instagram	387
twitter	1
youtube	12

Sentiment	Count
neutral	107
positive	262
negative	31

Processing State	Count
fetched	400

ID	Platform	Platform Post ID	Relevancy Score	Relevancy Label	Sentiment	Processing State	Raw Item ID	Raw Table	Created At
103340	instagram	N/A	0.8000	N/A	positive	fetched	N/A	N/A	2026-05-19 12:43:35
103365	instagram	N/A	0.9000	N/A	positive	fetched	N/A	N/A	2026-05-19 12:43:35
103364	instagram	N/A	0.7000	N/A	neutral	fetched	N/A	N/A	2026-05-19 12:43:35
103363	instagram	N/A	0.8500	N/A	positive	fetched	N/A	N/A	2026-05-19 12:43:35
103362	instagram	N/A	0.9000	N/A	positive	fetched	N/A	N/A	2026-05-19 12:43:35

Step 4: Complete Processing & Save Results

API Endpoint: api/sync_campaign_articles.php (POST)

Database Table: campaign_articles (INSERT/UPDATE, SELECT COUNT, SELECT)

ℹ️ This finalizes all collected data and syncs relevant items to campaign_articles.

Campaign Articles (Final Processed Results)

Total Campaign Articles: 320

ℹ️ This will sync relevant items (relevancy score ≥ 70% - campaign threshold: 0.7) from ai_relevancy_results to campaign_articles. Note: 320 relevant items found, 0 can be synced (others may already exist in campaign_articles).

Breakdown by Platform:

Platform	Articles Count
instagram	316
twitter	1
youtube	3

Sample Articles (Latest 5):

ID	Platform	Platform Post ID	Title/Content	Relevancy Score	Relevancy Label	Sentiment	Sentiment Score	Raw Item ID	Raw Table	Created At
51202	instagram	18304	Our latest #jacketring - truly stunning! Why? Beca...	0.9000	very_high	positive	0.8000	18304	instagram_raw	2026-05-19 12:43:35
51201	instagram	18339	Follow @johnsfinancetips for more Personal Finance...	0.7000	high	negative	0.2000	18339	instagram_raw	2026-05-19 12:43:35
51200	instagram	18275	AND THAT’S WHAT IT LOOKS LIKE WHEN ITS FLAWLESS!!!...	0.8500	high	positive	0.8000	18275	instagram_raw	2026-05-19 12:43:35
51203	instagram	18603	Nerd alert: this post about the DeBeers Cullinan B...	0.8500	high	positive	0.8000	18603	instagram_raw	2026-05-19 12:43:35
51229	instagram	18409	I DON’T IS NOT AN OPTION!!! @debeersofficial in Ba...	0.9000	very_high	positive	0.8000	18409	instagram_raw	2026-05-19 12:43:35

Campaign Articles Breakdown by Relevancy Label:

Relevancy Label	Count
high	195
very_high	125

Additional: Background Jobs Status

Database Table: background_jobs (SELECT)

ℹ️ Background jobs track long-running scraping processes. This is optional tracking.

No background jobs found for this campaign

Additional: AI Processing Jobs Status

Database Table: ai_jobs (SELECT)

No AI jobs found for this campaign

📋 Complete Summary Report

📊 Processing Status Summary

Metric	Value
Campaign ID	239
Campaign Status	active
Relevancy Threshold	0.70
Total Raw Records Collected	427
AI Relevancy Results	400
Campaign Articles (Final)	320
Background Jobs	0
AI Jobs	0

🌐 External APIs Used

API Service	Provider	Usage
`apidojo~tweet-scraper`	Apify	Twitter data collection (Step 3)
`streamers~youtube-scraper`	Apify	YouTube data collection (Step 3)
`apify~instagram-post-scraper`	Apify	Instagram data collection (Step 3)
ScrapingDog API	ScrapingDog	News & Blogs data collection (Step 3)
OpenAI API	OpenAI	AI relevancy analysis (Step 5)
Gemini API	Google	AI relevancy analysis (Step 5, alternative)

📊 Database Tables Used

Step	API Endpoint	Table Name	Operation
Step 1	`api/start_background_scraping.php`	`campaigns_impact` (relevancy_threshold)	SELECT
Step 2	`api/collect_data_step2.php`	`twitter_raw`, `youtube_raw`, `instagram_raw`, `news_raw`, `blogs_raw`, `facebook_raw`	INSERT, SELECT COUNT
Step 3	`api/check_raw_mentions.php`	`twitter_raw`, `youtube_raw`, `instagram_raw`, `news_raw`, `blogs_raw`, `facebook_raw`, `ai_relevancy_results` (platform_post_id, content_hash, processing_state, raw_item_id, raw_table_name)	SELECT, INSERT
Step 4	`api/sync_campaign_articles.php`	`campaign_articles` (platform_post_id, matched_terms, ai_reasoning, relevancy_label, sentiment_score, raw_item_id, raw_table_name)	INSERT/UPDATE, SELECT COUNT, SELECT

🔄 Processing Flow Summary

Step	Description	API Endpoint	Duration
Step 1	Initialize scraping process	`api/start_background_scraping.php`	10-20 seconds
Step 2	Collect data from platforms (100 results per platform)	`api/collect_data_step2.php`	10-20 minutes
Step 3	AI relevancy analysis (50 mentions per batch)	`api/check_raw_mentions.php`	5-7 minutes
Step 4	Complete processing & save results	`api/sync_campaign_articles.php`	1-2 minutes

Debug: Campaign Processing Steps

Step 1: Initialize Scraping Process

Step 2: Collect Data from Platforms

Data Collection Status by Platform

Step 3: AI Relevancy Analysis

AI Relevancy Results

Breakdown by Platform:

Breakdown by Relevancy:

Breakdown by Sentiment:

Breakdown by Processing State:

Sample Results (Latest 5):

Step 4: Complete Processing & Save Results

Campaign Articles (Final Processed Results)

Breakdown by Platform:

Sample Articles (Latest 5):

Campaign Articles Breakdown by Relevancy Label:

Additional: Background Jobs Status

Additional: AI Processing Jobs Status

📋 Complete Summary Report

📊 Processing Status Summary

🌐 External APIs Used

📊 Database Tables Used

🔄 Processing Flow Summary

✓ Complete!