How to automate website screenshots at scale
Architecture for capturing thousands of URLs reliably.
Taking one screenshot is trivial. Taking 100,000 screenshots reliably is an engineering problem. Whether you're building a website directory, a monitoring service, or a portfolio showcase, here's how to architect screenshot automation that works at scale.
Use cases that need scale
Directory sites (Product Hunt, Dribbble, SaaS marketplaces) need fresh thumbnails for every listed site. Monitoring services capture production pages on a schedule to detect regressions. Reporting tools generate visual snapshots for dashboards. SEO tools show clients what their pages look like in search results. Content pipelines auto-generate social cards and blog thumbnails.
Architecture: queue + workers + storage
The fundamental pattern is: a queue of URLs to capture, a pool of workers that process the queue, and object storage for the results.
// Simplified architecture:
//
// [URL Queue (Redis/SQS)]
// ↓
// [Worker Pool (N processes)]
// ↓ ↓ ↓
// [Browser 1] [Browser 2] [Browser 3]
// ↓ ↓ ↓
// [S3/R2 Storage] ← images saved here
// ↓
// [CDN] ← images served from hereThe queue
Redis with BullMQ is the most common choice for Node.js. SQS if you're on AWS. The queue provides: retry logic for failed captures, rate limiting (don't overwhelm target sites), priority (some captures are more urgent), and deduplication (don't capture the same URL twice in 10 minutes).
Workers
Each worker runs a headless browser and processes URLs from the queue. The key decisions: how many workers, how many browser instances per worker, and how to handle failures.
A single Chromium instance uses 200–500MB of RAM. On a 16GB server, you can comfortably run 8–10 browsers concurrently (leaving room for the OS and queue). Each browser can reuse its process across multiple captures — launch once, create new pages for each URL.
Storage
Screenshots are large (500KB–2MB for PNG) and you're generating thousands. Use object storage: S3, Cloudflare R2, or Google Cloud Storage. Serve via CDN for fast delivery. Key by URL hash + timestamp for versioning.
Self-hosted vs API
Self-hosted means managing browser infrastructure, handling crashes, security sandboxing, and keeping Chromium updated. An API handles all of this.
Cost comparison at different scales:
| Volume/month | Self-hosted (est.) | nightglass API |
|---|---|---|
| 1,000 | $20–50 (server time) | $5 |
| 10,000 | $50–100 | $50 |
| 100,000 | $200–400 | $500 |
| 1,000,000 | $500–1,500 | $5,000 |
Below 10,000/month, the API is cheaper when you factor in engineering time. Above 100,000, self-hosting starts winning on raw cost — but you're paying in operational complexity instead.
Handling failures
At scale, failures are guaranteed. Pages time out, DNS fails, servers return 500s, Chromium crashes. Build in: exponential retry (wait 1s, 2s, 4s between retries), dead letter queue (after 3 retries, move to a "failed" queue for human review), health checks (restart workers that stop responding), and resource limits (kill any browser that uses more than 1GB or runs for more than 60 seconds).
Rate limiting yourself
Don't hammer target sites with concurrent requests. Implement per-domain rate limiting — max 2 concurrent requests to the same domain. This prevents getting blocked and is just polite. Your queue should be domain-aware.
The simple path: If you're under 50,000 screenshots/month, use an API. The engineering cost of building and maintaining screenshot infrastructure exceeds the API cost by a wide margin. nightglass handles the browsers, scaling, and reliability — you just send URLs and get images back.
If building a self-hosted service, start with the Node.js screenshot microservice guide. Two common use cases for scale: visual change monitoring and visual regression testing. Note: serverless can't run headless browsers — you need dedicated infra or an API at scale.