Case study
A website analyzer that takes one public URL and returns scores and findings across SEO, accessibility, security, performance, content, domain and contact signals. This is how it fetches pages safely, and the trade-offs behind it.
Why
A first review of a web page usually means juggling several tools: one for SEO tags, another for security headers, a WHOIS lookup, a link checker. Each gives a verdict in its own format, often without showing what it actually saw.
SiteScopia does it in one pass from one URL. Findings carry the evidence they were based on and a suggested fix, so the report can be checked rather than taken on trust.
Constraints
Only public addresses
A URL that resolves to a private, loopback or internal address is refused before any request is made.
Fetch once, parse once
The page is downloaded one time and parsed into a single structure every check reads from.
One broken check cannot sink the report
Each check runs in isolation, and a failure becomes a note in the report instead of an error.
Never score a bot wall
If the site answers with a challenge page, the result is marked blocked rather than graded.
Architecture
In the request
Job202 and a token
PollEvery 1.5 sIn the background job
Into the report
The request only validates and queues. Everything slow happens in the job the browser polls.
Submit, then poll
The browser posts a URL and gets back 202 with a job ID and a random token. It polls every 1.5 seconds until the job is done, failed or blocked. Reading a job needs its token, compared in constant time, and there is no endpoint that lists other people's jobs.
Check the address before every hop
Only http and https are allowed, internal hostnames are refused, and the name is resolved so any non-public address is rejected. Redirects are followed by hand, up to five, and each new host is checked again. The body is streamed with a 5 MB cap and a 15 second timeout.
Get past bot walls, or say so
A detector recognises challenge pages from Cloudflare, DataDome, Akamai, Imperva and others. The fetcher retries once with a Chrome TLS fingerprint through a different proxy, and if the wall is still there the job is marked blocked with no score.
Parse once, enrich in parallel
The HTML is parsed a single time. Then the domain lookup over RDAP and a check of up to 25 links, eight at a time within a 12 second budget, run side by side with asyncio.
Run the checks, then score
Every check is a plain function registered in a list for its category. Each one runs inside its own try/except. A check scores on its worst finding, weighted 3 for critical, 1 for normal and 0.4 for minor, and categories roll up into a 0 to 100 score.
Trade-offs
Why
Some sites take seconds to answer, and link checks add more. Replying 202 straight away means a slow target never holds the HTTP request open, and the UI can show progress.
The cost
Jobs live in memory in one process, so a restart loses them and a second instance could not see them. Delivery is polling, not push.
Why
Adding a check is one function and one list entry. Checks do no network work, so they are fast and easy to reason about, and the public list of checks is generated from the same registry.
The cost
Checks for different categories take different inputs, and the broken-links check runs outside the registry, so it is not in the published count.
Why
A bug in one check, or an odd page that trips it up, should cost one line of the report, not the whole analysis.
The cost
The isolation covers the checks only. An error while parsing, extracting contacts or scoring still fails the job.
Why
It is simple to explain: a critical check counts seven and a half times as much as a minor one, and the category score is the share of credit earned.
The cost
A check that finds nothing counts as a pass, and the overall score is a plain average, so a three-check category weighs as much as an eight-check one.
Why
Response time, size, compression, caching headers and script counts come straight from the response. No browser is needed, so an analysis is fast and cheap, and every result traces back to what was received.
The cost
Layout shift, rendered contrast and anything built by JavaScript are out of reach, and response time is measured from the server, so the network path is included.
Why
Many real sites sit behind bot protection. Rotating proxies, burning a proxy for a host after a failure, and one retry with a browser TLS fingerprint let far more of them be analyzed.
The cost
It adds real complexity and a paid proxy provider, and timings and headers can reflect the proxy path rather than a visitor's.
Security
Honest limits
Close the remaining SSRF gaps
Connect to the address that was checked instead of resolving again, and re-validate redirects in the link checker and the TLS fallback.
Move jobs out of memory
A store like Redis and a worker would let jobs survive restarts and let more than one instance serve results.
Add tests and CI
The checks are pure functions, which makes them easy to test. A suite in CI would guard the scoring as checks are added.
I am happy to go deeper on any of these trade-offs.
Next case study
JobHarvester