Debug: Campaign Processing Steps

Step 1: Initialize Scraping Process

API Endpoint: api/start_background_scraping.php (POST)

Database Table: campaigns_impact (SELECT)

Field	Value
ID	202
Title	Test Twitter Campaign - 2026-05-01 07:49:27
Keywords	"Bombay Sweet Shop" OR #bombaysweetshop OR @bombaysweetshop OR bombaysweetshop.com
Platforms	["twitter"]
Status	active
Date From	2026-04-24
Date To	2026-05-01
Client Keywords	BSS, Bombay Sweets
Monitor Links	N/A
Relevancy Threshold	0.70

ℹ️ This prepares the campaign for data collection. This step must be completed before Step 2.

Step 2: Collect Data from Platforms

API Endpoint: api/collect_data_step2.php (POST)

Database Tables: twitter_raw, youtube_raw, instagram_raw, news_raw, blogs_raw, facebook_raw (INSERT, SELECT COUNT)

Database Table: background_jobs (SELECT)

ℹ️ What this means: Background jobs track long-running scraping processes. If no jobs are found, it means either:
• Scraping completed and jobs were cleaned up
• Scraping happened directly without creating job records (this is normal)
• No background scraping has been started yet
✅ Check Step 3 (Data Collection Status) to see if data was actually collected - that's what matters!

Job ID	Status	Created At	Started At
53ee47e1-c5af-4299-ab37-780ee428ae87	failed	2026-05-01 07:49:27	2026-05-01 09:57:31

ℹ️ This collects 100 results from each selected platform. This may take 10-20 minutes.

Data Collection Status by Platform

ℹ️ Limits: Loaded from platform_limits table (managed via Settings > Platform Limit)

Platform	Table Name	Records Count	Limit	Status
Twitter	`twitter_raw`	0	`1000`	⚠ No Data
Youtube	`youtube_raw`	0	`100`	⚠ No Data
Instagram	`instagram_raw`	0	`100`	⚠ No Data
Facebook	`facebook_raw`	0	`100`	⚠ No Data
News	`news_raw`	0	`1000`	⚠ No Data
Blogs	`blogs_raw`	0	`1000`	⚠ No Data
Total Records	0	-	⚠ No Data Collected	-

Step 3: AI Relevancy Analysis

API Endpoint: api/check_raw_mentions.php (POST)

OpenAI key: loaded from api_keys_db.api_keys (then OPENAI_API_KEY env, then config). Model: gpt-4o-mini.

Database Tables: twitter_raw, youtube_raw, instagram_raw, news_raw, blogs_raw, facebook_raw (SELECT), ai_relevancy_results (INSERT)

ℹ️ Run sends every raw row to the API with force_reprocess (full pass, re-scores by AI). Re-Run does the same. Batches of 50; large campaigns can take 15+ minutes.

AI Relevancy Results

Database Table: ai_jobs (SELECT)

No AI jobs found for this campaign. Click the button above to create AI jobs from raw data.

Step 4: Complete Processing & Save Results

API Endpoint: api/sync_campaign_articles.php (POST)

Database Table: campaign_articles (INSERT/UPDATE, SELECT COUNT, SELECT)

ℹ️ This finalizes all collected data and syncs relevant items to campaign_articles.

Campaign Articles (Final Processed Results)

Total Campaign Articles: 0

ℹ️ To create campaign articles:
1. Process AI jobs (Step 3) to analyze relevancy
2. Items with relevancy score ≥ 70% (campaign threshold: 0.7) are considered relevant
3. Use the sync button above if relevant items exist in ai_relevancy_results

No campaign articles found. Final processing may not have completed yet.

Additional: Background Jobs Status

Database Table: background_jobs (SELECT)

ℹ️ Background jobs track long-running scraping processes. This is optional tracking.

Job ID	Status	Created At	Started At
53ee47e1-c5af-4299-ab37-780ee428ae87	failed	2026-05-01 07:49:27	2026-05-01 09:57:31

Additional: AI Processing Jobs Status

Database Table: ai_jobs (SELECT)

No AI jobs found for this campaign

📋 Complete Summary Report

📊 Processing Status Summary

Metric	Value
Campaign ID	202
Campaign Status	active
Relevancy Threshold	0.70
Total Raw Records Collected	0
AI Relevancy Results	0
Campaign Articles (Final)	0
Background Jobs	1
AI Jobs	0

🌐 External APIs Used

API Service	Provider	Usage
`apidojo~tweet-scraper`	Apify	Twitter data collection (Step 3)
`streamers~youtube-scraper`	Apify	YouTube data collection (Step 3)
`apify~instagram-post-scraper`	Apify	Instagram data collection (Step 3)
ScrapingDog API	ScrapingDog	News & Blogs data collection (Step 3)
OpenAI API	OpenAI	AI relevancy analysis (Step 5)
Gemini API	Google	AI relevancy analysis (Step 5, alternative)

📊 Database Tables Used

Step	API Endpoint	Table Name	Operation
Step 1	`api/start_background_scraping.php`	`campaigns_impact` (relevancy_threshold)	SELECT
Step 2	`api/collect_data_step2.php`	`twitter_raw`, `youtube_raw`, `instagram_raw`, `news_raw`, `blogs_raw`, `facebook_raw`	INSERT, SELECT COUNT
Step 3	`api/check_raw_mentions.php`	`twitter_raw`, `youtube_raw`, `instagram_raw`, `news_raw`, `blogs_raw`, `facebook_raw`, `ai_relevancy_results` (platform_post_id, content_hash, processing_state, raw_item_id, raw_table_name)	SELECT, INSERT
Step 4	`api/sync_campaign_articles.php`	`campaign_articles` (platform_post_id, matched_terms, ai_reasoning, relevancy_label, sentiment_score, raw_item_id, raw_table_name)	INSERT/UPDATE, SELECT COUNT, SELECT

🔄 Processing Flow Summary

Step	Description	API Endpoint	Duration
Step 1	Initialize scraping process	`api/start_background_scraping.php`	10-20 seconds
Step 2	Collect data from platforms (100 results per platform)	`api/collect_data_step2.php`	10-20 minutes
Step 3	AI relevancy analysis (50 mentions per batch)	`api/check_raw_mentions.php`	5-7 minutes
Step 4	Complete processing & save results	`api/sync_campaign_articles.php`	1-2 minutes