Scraper Workers

Webhook flow

Every scraper worker posts its results back to the platform when a run completes. The webhook endpoint authenticates the call, dispatches by source, and upserts the results into the right database table. The skill that fired the run reads from those tables on its next turn; the workflow continues without polling.

Updated today

Overview

At a glance

Endpoint: POST /api/apify-webhook
Handler: src/app/api/apify-webhook/route.ts
Auth header: x-webhook-secret
Shared secret: APIFY_WEBHOOK_SECRET env
Source param: ?source=gmaps | jobs | linkedin | twitter | email | profile-changes
Upstream: Every scraper worker after a successful run

The webhook endpoint is the integration point between the scraper workers and the rest of the platform. The workers do not write to the application database directly because they live in different runtimes (Apify for actors, Cloudflare for the marketing worker) and the application database is the platform's source of truth. The webhook centralises every write through one auth-gated route.

The webhook endpoint

One route. Authenticate, parse, dispatch.

src/app/api/apify-webhook/route.tsts

1export async function POST(req: Request) {

2 // 1. Auth

3 if (req.headers.get("x-webhook-secret") !== process.env.APIFY_WEBHOOK_SECRET) {

4 return new Response("unauthorized", { status: 401 })

5 }

7 // 2. Parse

8 const url = new URL(req.url)

9 const source = url.searchParams.get("source") ?? ""

10 const payload = await req.json()

12 // 3. Dispatch

13 switch (source) {

14 case "gmaps": return handleGmaps(payload)

15 case "jobs": return handleJobs(payload)

16 case "linkedin":return handleLinkedIn(payload)

17 case "twitter": return handleTwitter(payload)

18 case "email": return handleEmailEnrichment(payload)

19 case "profile-changes": return handleProfileChanges(payload)

20 default: return new Response("unknown source", { status: 400 })

21 }

22}

Authentication

One header. One env var. No exceptions.

Warning

Every worker that posts to the webhook is configured with the same x-webhook-secret value as the platform's APIFY_WEBHOOK_SECRET env. A mismatch returns 401 immediately. Rotate the secret by updating both sides in the same deploy window.

Source dispatch

One handler per scraper.

source	Handler	Target table	Dedup key
gmaps	handleGmaps	leads	(user_id, enrichment_data ->> place_id) OR (name + address)
jobs	handleJobs	job_signals	(user_id, jd_url)
linkedin	handleLinkedIn	leads	(user_id, enrichment_data ->> profile_id)
twitter	handleTwitter	leads	(user_id, enrichment_data ->> twitter_user_id)
email	handleEmailEnrichment	leads (update)	lead_id from input
profile-changes	handleProfileChanges	audit row + reprocess	-

gmaps handler

One row per place. Upsert into leads.

src/app/api/apify-webhook/route.tsts

63async function handleGmaps(payload: { rows: GmapsRow[]; user_id: string; conversation_id: string }) {

64 const upserts = payload.rows.map((r) => ({

65 user_id: payload.user_id,

66 name: r.name,

67 company: r.name,

68 email: null,

69 phone: r.phone,

70 website: r.website,

71 location: r.address,

72 tags: ["gmaps", `gmaps:${r.search_term}`],

73 enrichment_data: { place_id: r.place_id, lat: r.lat, lng: r.lng, categories: r.categories, rating: r.rating },

74 }))

75 await supabaseAdmin.from("leads").upsert(upserts, { onConflict: "user_id,enrichment_data->>place_id" })

76 await logActivity(payload.user_id, payload.conversation_id, `gmaps run ingested ${upserts.length} places`)

77 return new Response(null, { status: 204 })

78}

jobs handler

One row per posting. Upsert into job_signals.

src/app/api/apify-webhook/route.tsts

157async function handleJobs(payload: { rows: JobRow[]; user_id: string; conversation_id: string }) {

158 const upserts = payload.rows.map((r) => ({

159 user_id: payload.user_id,

160 company_name: r.company_name,

161 company_domain: r.company_domain,

162 title: r.title,

163 location: r.location,

164 posted_at: r.posted_at,

165 jd_url: r.jd_url,

166 snippet: r.jd_text_snippet,

167 raw: r,

168 }))

169 await supabaseAdmin.from("job_signals").upsert(upserts, { onConflict: "user_id,jd_url" })

170 return new Response(null, { status: 204 })

171}

email handler

Joins enrichment results back to existing leads by lead_id.

The email handler is the only one that updates rather than inserts. It reads lead_id on every row and patches the corresponding lead with the best email, the verification status, the confidence, and the alternates. The activity log gets a single line per run summarising how many leads moved from no-email to email.

linkedin handler

One row per profile. Upsert into leads with profile_id as the dedup key.

The LinkedIn handler is structurally identical to gmaps but with a different dedup column. Profile id is opaque and stable across LinkedIn rewrites, which makes it a more reliable key than name plus company.

twitter handler

One row per author. Upsert into leads with twitter_user_id as the dedup key.

Twitter authors are upserted with an empty email and a Twitter handle in enrichment_data. Downstream skills can fire the email finder to attach addresses where possible; the leads stay even when no email materialises because Twitter DMs are a viable outreach channel for the angel persona.

Failure modes

What goes wrong on the receive side and how the endpoint responds.

Failure	Endpoint response
Missing or wrong x-webhook-secret	401 unauthorized
Unknown source query param	400 unknown source
Payload missing user_id or conversation_id	400 missing user_id
Upstream worker timed out and resent the same batch	Idempotent upsert; second call writes nothing new
Database transient error	500 with the error string; the worker retries per Apify's webhook retry policy
Activity log write fails after upsert succeeded	Logged but does not fail the webhook response (we keep the data, lose only the log line)

File map

src/app/api/apify-webhook

route.tsauth + dispatch + per-source handlers

src/lib

supabase-admin.tsservice-role client used by the handlers

activity-log.tsshared logger for run-summary rows

apify-actors

gmaps/src/webhook.jsclient side: posts to /api/apify-webhook?source=gmaps

jobs/src/...client side

linkedin/src/main.jsposts to /api/apify-webhook?source=linkedin