Workers

Discovery aggregator

Hourly aggregator that pulls signals from many surfaces, filters against the workspace ICP, scores, dedupes, and writes a unified feed for the Discovery Lab pages. Fans out to the scrape worker for HN, news, GitHub trending, SEC, Product Hunt, Polymarket, and Yahoo, plus reads scraper datasets from the social and jobs actors.

Updated today

Overview

At a glance

Worker: discovery
Workflow: DiscoveryAggregatorWorkflow
Trigger: 0 * * * * (top of every hour)
D1 (ops): ultron-operations (writes feed + dedup keys)
D1 (scrapes): ultron-scrapes (reads scraper_items)
KV: SHARED_CONFIG (ICP rules, source toggles)
Calls: scrape.51ultron.com for web-fetched sources

Worker shape

wrangler.toml (key bindings)toml

1name = "discovery"

2main = "src/index.ts"

4[[workflows]]

5name = "discovery-aggregator"

6binding = "DISCOVERY"

7class_name = "DiscoveryAggregatorWorkflow"

9[[d1_databases]]

10binding = "OPS_DB"

11database_name = "ultron-operations"

13[[d1_databases]]

14binding = "SCRAPES_DB"

15database_name = "ultron-scrapes"

17[[kv_namespaces]]

18binding = "SHARED_CONFIG"

20[[analytics_engine_datasets]]

21binding = "EVENTS"

22dataset = "ultron_events"

24[triggers]

25crons = ["0 * * * *"]

Source surfaces

Where each tick reads from.

Source	How it is reached	What it produces
Hacker News	scrape worker (top + best + ask + show)	Stories with score + comment thread digest
TechCrunch / news pool	scrape worker (per-feed)	Articles with summary
GitHub trending	scrape worker (lang + period selectors)	Repo cards with description + stars delta
SEC filings	scrape worker (EDGAR full-text)	Recent filings by company watch-list
Product Hunt	scrape worker (daily + weekly)	Launches
Polymarket	scrape worker (markets endpoint)	Active markets matching workspace keywords
Yahoo Finance	scrape worker (per ticker)	Earnings dates, analyst movement
TikTok / IG / YouTube trends	ultron-scrapes D1 (already populated by actors)	Trend rows from the dataset
LinkedIn jobs	ultron-scrapes D1	Hiring signal rows

Workflow steps

DiscoveryAggregatorWorkflow.run sketchts

1async run(event, step) {

2 const sources = await step.do('load-source-config',

3 () => loadSources(this.env)

4 )

5 const icpRules = await step.do('load-icp',

6 () => loadIcpRules(this.env)

7 )

9 // One step per source. A broken source fails only its step.

10 const raw = await Promise.all(sources.map(src =>

11 step.do(`fetch-${src.id}`, () => fetchSource(this.env, src))

12 ))

14 const filtered = await step.do('icp-filter',

15 () => raw.flat().filter(item => matchesIcp(item, icpRules))

16 )

18 const scored = await step.do('score',

19 () => scoreItems(filtered, icpRules)

20 )

22 const deduped = await step.do('dedup',

23 () => dedupeAgainstFeed(this.env, scored)

24 )

26 await step.do('persist',

27 () => writeFeed(this.env, deduped)

28 )

29}

ICP filter and scoring

Per-workspace rules narrow the firehose.

Each workspace defines an ICP rule set: industries, geos, size bands, intent signal keywords, and exclusions. The filter step drops items that miss every dimension; the score step rewards items that hit multiple dimensions and weights them by source-specific signal strength (e.g. a hiring signal that matches both an industry and a tech-stack tag scores higher than one that matches only the industry).

Dimension	Weight
Industry match	+2
Tech-stack match	+2
Geo match	+1
Size-band match	+1
Intent keyword match (per match)	+2 each, capped at 6
Recency bonus (< 24h old)	+1
Excluded entity match	set score to 0 (drop)

Unified feed

The shape the Discovery Lab UI reads.

ultron-operations.discovery_feedsql

1CREATE TABLE discovery_feed (

2 id TEXT PRIMARY KEY,

3 workspace_id TEXT NOT NULL,

4 source TEXT NOT NULL, -- 'hn' | 'news' | 'gh-trending' | ...

5 source_url TEXT NOT NULL,

6 source_id TEXT NOT NULL, -- per-source stable id, used for dedup

7 title TEXT NOT NULL,

8 summary TEXT,

9 score INTEGER NOT NULL,

10 signals TEXT NOT NULL, -- JSON array of matched signal keys

11 published_at TEXT NOT NULL,

12 ingested_at TEXT NOT NULL DEFAULT (datetime('now')),

13 UNIQUE (workspace_id, source, source_id)

14);

File map

cf-workers/discovery

wrangler.toml

src

index.tsfetch + scheduled handler

workflow.tsDiscoveryAggregatorWorkflow

sources

hn.ts

news.ts

gh-trending.ts

sec.ts

product-hunt.ts

polymarket.ts

yahoo.ts

icp.tsmatchesIcp + scoreItems

dedup.ts

migrations

0001_discovery_feed.sql