Workers

Scraper orchestrator

Coordinates scraper runs end to end. Spawns scraper jobs, tracks their lifecycle, polls for completion, pulls datasets back, routes rows to the right workspace and skill run. Hosts two workflow classes: ScraperOrchestrator (the master) and PollerWorkflow (the per-run watcher).

Updated today

Overview

At a glance

Worker: orchestrator
Workflows: ScraperOrchestrator (master), PollerWorkflow (per-run)
D1: ultron-scrapes
Reads: Scrape platform API to spawn runs and pull datasets
Writes: scrape_runs, scrape_items, scrape_events

Without an orchestrator, every caller would have to handle actor IDs, dataset polling, webhook payloads, and retry policy themselves. The orchestrator centralises that. A skill that needs lead data calls orchestrator with a brief; the orchestrator picks the actors, spawns them, watches them, and pushes rows back when they finish.

Worker shape

wrangler.toml (key bindings)toml

1name = "orchestrator"

2main = "src/index.ts"

4[[workflows]]

5name = "scraper"

6binding = "SCRAPER"

7class_name = "ScraperOrchestrator"

9[[workflows]]

10name = "poller"

11binding = "POLLER"

12class_name = "PollerWorkflow"

14[[d1_databases]]

15binding = "SCRAPES_DB"

16database_name = "ultron-scrapes"

18[[kv_namespaces]]

19binding = "SHARED_CONFIG"

ScraperOrchestrator

The master workflow.

src/scraper.ts (concept)ts

1export class ScraperOrchestrator extends WorkflowEntrypoint<Env, ScraperParams> {

2 async run(event, step) {

3 const { runs } = event.payload // runs[] = { actor, input, options }

5 // 1. Persist run header

6 const headerId = await step.do('init-header',

7 () => insertRunHeader(this.env, event.payload)

8 )

10 // 2. Spawn every actor run in parallel

11 const spawned = await Promise.all(runs.map((r, i) =>

12 step.do(`spawn-${i}`, () => spawnActorRun(this.env, r))

13 ))

15 // 3. For each spawned run, kick off a PollerWorkflow

16 await Promise.all(spawned.map(s =>

17 step.do(`poll-init-${s.actorRunId}`, () =>

18 this.env.POLLER.create({ params: { actorRunId: s.actorRunId, headerId } })

19 )

20 ))

21 }

22}

PollerWorkflow

Per-actor-run watcher with sleep + retry.

src/poller.ts (concept)ts

1export class PollerWorkflow extends WorkflowEntrypoint<Env, PollerParams> {

2 async run(event, step) {

3 const { actorRunId, headerId } = event.payload

4 const MAX_TICKS = 240 // 240 * 60s = 4 hours max

6 for (let tick = 0; tick < MAX_TICKS; tick++) {

7 const status = await step.do(`poll-${tick}`, () =>

8 readActorRunStatus(this.env, actorRunId)

9 )

11 if (status === 'SUCCEEDED') {

12 const datasetId = await step.do(`dataset-${actorRunId}`,

13 () => readDatasetId(this.env, actorRunId)

14 )

15 await step.do(`ingest-${actorRunId}`,

16 () => ingestDataset(this.env, datasetId, headerId)

17 )

18 return { ok: true }

19 }

21 if (status === 'FAILED' || status === 'ABORTED' || status === 'TIMED_OUT') {

22 await step.do(`mark-failed-${actorRunId}`,

23 () => markRunFailed(this.env, actorRunId, status)

24 )

25 return { ok: false, status }

26 }

28 await step.sleep(`sleep-${tick}`, '60 seconds')

29 }

31 await markRunTimedOut(this.env, actorRunId)

32 }

33}

Note

The webhook from the scraper platform also routes through the scraper-webhook worker, which can signal an early completion event into the orchestrator. The poller is the backstop: it ensures the orchestrator advances even when the webhook is lost.

Input shape

POST /runsjson

1{

2 "workspaceId": "ws_<id>",

3 "conversationId": "conv_<id>", // optional, for chat-driven runs

4 "skillId": "skill_<id>", // optional, for skill-driven runs

5 "callbackWebhook": "https://...", // optional

6 "runs": [

7 {

8 "actor": "ultron-leadgen",

9 "input": { "brief": { ... }, "icp": { ... } },

10 "options": { "memoryMb": 1024, "timeoutSec": 1800 }

11 },

12 {

13 "actor": "ultron-similarweb-bulk",

14 "input": { "domains": [...] }

15 }

16 ]

17}

Run lifecycle

State	Meaning
queued	Header inserted, child runs not yet spawned
spawning	Calling the scraper platform to start each run
running	All children spawned, pollers active
partial	At least one child failed; others still running
succeeded	All children succeeded, dataset rows ingested
failed	All children failed
timed_out	Backstop poller cap reached without any final status

File map

cf-workers/orchestrator

wrangler.toml

src

index.ts

scraper.tsScraperOrchestrator class

poller.tsPollerWorkflow class

platform.tsspawnActorRun + readActorRunStatus + readDatasetId

ingest.tsingestDataset into ultron-scrapes

migrations

0001_scrape_runs.sql

0002_scrape_items.sql

0003_scrape_events.sql