Utility workers
The single-purpose, small-surface workers that keep the rest of the platform glued together. Each does one thing - scrape execution, webhook receipt, search, auth, dashboard backend, log ingest, URL signing, model bridging, SMTP verify, fundraising - and gets out of the way.
Overview
- Pattern
- One endpoint, one job, no state of its own (mostly)
- Total count
- 11 workers covered here
- Common header
- Every utility worker is auth-gated against the platform's edge-auth tokens
Utility fleet
| Worker | Endpoint | What it does |
|---|---|---|
| scrape | POST /scrape { url, options? } | Generic scrape executor. Browser-rendered or fetch-only. |
| scraper-webhook | POST /apify | Receives terminal-status webhooks from the scraper platform, routes datasets. |
| search | POST /search { q, scopes? } | Federated search across web + brain + workspace artefacts. |
| edge-auth | POST /token, /verify | Issues and verifies short-lived tokens for the extension and the widget. |
| dashboard | GET /api/* (read-only) | Read-heavy dashboard endpoints. Caches aggressively. |
| observability-ingest | POST /events | Receives logs and metrics from surfaces that cannot write to analytics datasets directly. |
| tiktok-sign | GET /sign?url=... | Stateless X-Bogus signer wrapper for the TikTok scrapers. |
| kimi-anthropic-shim | POST /v1/messages | Translates between two different inference API request shapes. |
| kimi-bridge | POST /v1/messages | Successor to the shim. New routing rules and headers. |
| ultron-smtp-verify | POST /verify { email } | SMTP-handshake verification with conn caching. Used by email-finder + leadgen. |
| ultron-fundraising | POST /research { domain } | Fundraising-targeted research pipeline. Composite over research + pitchbook + dealroom. |
scrape
Generic scrape executor.
Browser-rendered or fetch-only at the caller's request. Used by the discovery aggregator and the competitor watcher so neither of them has to maintain its own browser pool. Output is the raw HTML plus metadata; the caller does the parsing.
scraper-webhook
Webhook receiver from the scraper platform.
Every actor in the scraper fleet POSTs to the same endpoint (scraper-webhook.51ultron.com/apify) on terminal status. The worker validates the auth header, reads the dataset id, pulls rows into the ultron-scrapes D1, and routes the result to the correct workspace and skill run via metadata stored at run-spawn time.
search
Federated search.
Single entry point that fans out to a web search provider, the brain (semantic + lexical), and workspace-scoped artefacts (notes, files, canvases). Results are merged with per-scope weights so a single query surfaces the right blend.
edge-auth
Edge token mint and verify.
Issues short-lived bearer tokens scoped to a (user, workspace, scopes) tuple. The extension and the widget both use these tokens; the chat-v2 route also accepts them so a terminal can hold a session without holding the full Supabase JWT.
dashboard
Read-heavy dashboard backend.
Caches aggressively. Most dashboard panels read aggregate data that does not change minute by minute; this worker serves them from a cache fed by the analytics dataset roll-up. Cuts the app's read pressure significantly for the dashboard surface.
observability-ingest
Log + metric ingest.
Some surfaces - the static widget, the extension on a slow network - cannot write directly to the analytics dataset. They POST events to this worker, which validates and forwards. Single concern: the public ingest endpoint that cannot live on the analytics dataset itself.
tiktok-sign
Stateless X-Bogus signer.
A managed Chrome instance with the TikTok webmssdk shim loaded. Other workers that need an X-Bogus header (the tiktok-trends actor when running in fetch-only mode) call this endpoint with the request URL; it returns the signed URL.
Kimi bridges
Two workers, one purpose.
| Worker | Status | Notes |
|---|---|---|
| kimi-anthropic-shim | Legacy. Still serving traffic. | Translates between two different inference API request shapes. Used by the shell as its default backend. |
| kimi-bridge | Successor. | New routing rules, header normalisation, support for an alternate-backend override. New deployments use this. |
ultron-smtp-verify
SMTP handshake email verification.
Opens a TCP socket to the recipient mailbox provider's SMTP, performs RCPT TO without sending an actual message, and reports whether the address would accept mail. Used by the email-finder and leadgen scrapers to drop candidates whose addresses bounce.
ultron-fundraising
Fundraising-targeted research pipeline.
Composite over the research actor, the Pitchbook scraper, and the Dealroom scraper, with prompts tuned for fundraising context. Given a company domain, returns stage, last round details, lead investors, board, recent press, and an outreach-readiness signal.