Try for free
Scrapers

Similarweb traffic

Bulk traffic and rank lookups for a domain list. Hits Similarweb's free per-domain data endpoint, rotates residential IP per request to defeat per-IP rate limiting, returns rank plus traffic plus engagement metrics per domain.

Updated today

Overview

At a glance
Worker
ultron-similarweb-bulk
Runtime
Python + httpx, no browser
Endpoint
data.similarweb.com/api/v1/data
IP rotation
Per request (default on)
Proxy
Residential US
Default delay
500 ms, +50% jitter
Safety
Aborts after 10 consecutive failures (configurable)

Used as a free enrichment surface for company research and competitor sets. Single-domain throughput is unimpressive, but with IP rotation the bulk endpoint clears thousands of domains per run with no auth and no fingerprinting concerns.

Input schema

.actor/INPUT_SCHEMA.jsonjson
1{
2 "domains": ["netflix.com", "github.com", "linear.app"],
3 "rotateIpPerRequest": true,
4 "proxyConfiguration": {
5 "useApifyProxy": true,
6 "apifyProxyGroups": ["RESIDENTIAL"],
7 "apifyProxyCountry": "US"
8 },
9 "requestDelayMs": 500,
10 "stopOnConsecutiveFailures": 10
11}

IP rotation

The single biggest performance lever.

Tip
Similarweb caches lookups per-IP and starts rate-limiting after 10 to 30 requests from the same IP. With per-request IP rotation the actor effectively has no rate ceiling at the IP layer; the limiter becomes the delay floor plus the proxy pool's IP diversity.
rotateIpPerRequestEffective ceiling
true (default)Hundreds of req/min sustained. One TLS handshake per lookup is the cost.
falseAbout 10 to 30 successful lookups, then 429s until the IP cools down.

Crawl flow

src/__main__.pypy
1# 1. Read inputs. Compute jitter floor = max(requestDelayMs, 0).
2# 2. For each domain:
3# a. Build proxy session with fresh IP if rotateIpPerRequest = true.
4# b. GET data.similarweb.com/api/v1/data?domain=<d>.
5# c. Parse JSON: globalRank, countryRank, countryCode, category,
6# totalVisits, avgVisitDuration, pagesPerVisit, bounceRate,
7# trafficSources, topCountryShares, topReferrers.
8# d. Push row.
9# 3. On failure (network, 429, 5xx): increment consecutiveFailures.
10# Reset on next success. Abort if >= stopOnConsecutiveFailures.
11# 4. Sleep requestDelayMs + 50% jitter between lookups.

Output shape

dataset-row.jsonjson
1{
2 "domain": "linear.app",
3 "globalRank": 2843,
4 "countryRank": 312,
5 "countryCode": "US",
6 "category": "Computers Electronics and Technology",
7 "totalVisits": 1284322,
8 "avgVisitDuration": 342, // seconds
9 "pagesPerVisit": 4.3,
10 "bounceRate": 0.38,
11 "trafficSources": {
12 "directVisitsShare": 0.42,
13 "searchVisitsShare": 0.31,
14 "socialVisitsShare": 0.05,
15 "referralsShare": 0.18,
16 "mailShare": 0.04
17 },
18 "topCountryShares": [
19 { "country": "US", "share": 0.42 },
20 { "country": "GB", "share": 0.09 }
21 ],
22 "snapshotDate": "2026-05-19",
23 "scrapedAt": "2026-05-19T12:00:00Z"
24}

Webhook contract

actor.jsonjson
1{
2 "eventTypes": ["ACTOR.RUN.SUCCEEDED", "ACTOR.RUN.FAILED", "ACTOR.RUN.TIMED_OUT", "ACTOR.RUN.ABORTED"],
3 "requestUrl": "https://scraper-webhook.51ultron.com/apify",
4 "payload": { "actorName": "similarweb-bulk", "source": "similarweb", "datasetId": "<id>" }
5}

File map

actors/similarweb-bulk
.actor
actor.json
INPUT_SCHEMA.json
src
__main__.py
Dockerfile
requirements.txt
README.md