Scrapers

Leadgen engine

The composite engine that turns a brief plus an ICP into a vetted lead list. Chains the maps + jobs + research + email-finder + similarweb actors behind a single entry point, deduplicates across sources, and emits one row per qualified lead.

Updated today

Overview

At a glance

Worker: ultron-leadgen
Pattern: Composite. Spawns child runs of other actors, merges results.
Upstream scrapers: maps, jobs (auth + guest), research, email-finder, similarweb
Output: Saved-lead rows ready for CRM ingest
Trigger: Specter's outreach pipeline, manual chat invocation, scheduled task

Where most scrapers are a single-source crawl, the leadgen engine is an orchestrator on top of them. It reads the brief, decides which sources to query, fans out the child runs in parallel, then merges. The merge step is the value-add: deduplication across sources (so the same company landing from both Maps and Wellfound is one lead, not two), entity resolution (same domain, different surface forms of the company name), and intent ranking.

Input schema

actor inputjson

1{

2 "brief": {

3 "industry": ["fintech", "B2B SaaS"],

4 "geo": ["US", "EU"],

5 "sizeBandHc": [11, 200],

6 "intentSignals": ["hiring backend", "raised in last 12mo"],

7 "exclusions": ["acme.com", "competitor.io"]

8 },

9 "icp": {

10 "personaTitles": ["VP Engineering", "Head of Platform"],

11 "techStack": ["postgres", "kubernetes"]

12 },

13 "limits": {

14 "maxCandidates": 500,

15 "maxBudget": 50.00 // USD ceiling for the run

16 },

17 "sources": ["jobs", "research", "similarweb", "wellfound"] // optional override

18}

Pipeline flow

pipelinetxt

1# 1. Brief expansion

2# Parse brief into structured filters (industry, geo, size, intent).

3# Resolve geo strings to LinkedIn geoIds and country codes.

4#

5# 2. Source selection

6# Based on intent signals, pick the right scraper subset.

7# "hiring backend" -> jobs (auth or guest depending on volume)

8# "raised recently" -> research + dealroom (if EU)

9# "high web traffic" -> similarweb + research

10# "category leader" -> g2-reviews or capterra-reviews

11# "local presence" -> maps

12#

13# 3. Parallel fan-out

14# Spawn child actor runs. Each run inherits a slice of the brief.

15#

16# 4. Merge phase

17# Wait for all children to finish (with a hard timeout). Read each

18# child's dataset. Normalise into canonical Lead shape.

19#

20# 5. Dedupe + entity resolution

21# Key on domain when present, fall back to (name, country).

22# Pick the highest-fidelity source row to seed the canonical row.

23#

24# 6. Enrichment

25# For each survivor, fire email-finder for the persona titles.

26# Drop candidates where no email candidate verifies.

27#

28# 7. Intent scoring

29# Score 0 to 100 based on which signals matched in the brief.

30# Sort. Truncate to limits.maxCandidates.

31#

32# 8. Push to dataset

33# One kind="lead" row per qualified candidate.

Output shape

dataset-row.jsonjson

1{

2 "kind": "lead",

3 "source": "leadgen",

4 "leadId": "lg_<hash>",

5 "domain": "linear.app",

6 "companyName": "Linear",

7 "industry": "B2B SaaS",

8 "country": "US",

9 "size": "51-200",

10 "intentSignals": ["hiring backend", "raised series_c"],

11 "intentScore": 87,

12 "personaCandidates": [

13 {

14 "title": "VP Engineering",

15 "name": "Jane Doe",

16 "email": "jane@linear.app",

17 "verified": true,

18 "source": "email-finder"

19 }

20 ],

21 "sources": {

22 "jobs": true,

23 "research": true,

24 "similarweb": { "globalRank": 2843 },

25 "wellfound": { "fundingStage": "Series C" }

26 },

27 "scrapedAt": "2026-05-19T12:00:00Z"

28}

Use cases

The Specter persona's outreach pipeline. When the operator says "find me 50 backend-hiring fintech startups in the EU we can pitch", Specter calls leadgen with a brief and gets back a ranked list of candidates ready to feed into a sequence builder.

Scheduled top-of-funnel. A daily schedule fires leadgen with the workspace's saved brief, then pushes net-new candidates into the CRM with a tag for review.

Ad-hoc research. The marketing swarm uses leadgen to find comparable companies for a positioning audit, treating candidates as competitive set rather than as outreach targets.

Webhook contract

webhookjson

1{

2 "eventTypes": ["ACTOR.RUN.SUCCEEDED", "ACTOR.RUN.FAILED", "ACTOR.RUN.TIMED_OUT", "ACTOR.RUN.ABORTED"],

3 "requestUrl": "https://scraper-webhook.51ultron.com/apify",

4 "payload": { "actorName": "leadgen", "source": "leadgen", "datasetId": "<id>" }

5}

Note

Child runs also webhook independently. The webhook handler recognises the parent run id via a header and skips ingest for children, letting the parent's merge step own the canonical rows.

File map

actors/leadgen

.actor

actor.json

INPUT_SCHEMA.json

src

__main__.pybrief parser, source selector, fan-out, merge

merge.pydedupe + entity resolution

score.pyintent scoring

Dockerfile

requirements.txt

README.md