Case study
A job board for Pakistan's tech market. Scheduled crawlers read 38 company career pages, push what they find through an authenticated API, and a React app makes it searchable. This is how the pipeline works, and the trade-offs behind it.
Why
Tech jobs in Lahore are spread across dozens of company career pages, each with its own layout, and many of them never reach the big job boards. Finding them means checking site after site by hand.
JobHarvester checks them on a schedule instead and puts every listing in one searchable feed, aimed at developers, interns and fresh graduates.
Constraints
Crawlers never touch the database
They submit jobs through the API with a key, so validation and storage rules live in one place.
One broken site cannot stop the run
Each crawler runs on its own, and a crash is recorded while the others carry on.
Running twice is harmless
Jobs are matched on their source URL, so a repeat run updates listings instead of copying them.
No crawler server to keep alive
The crawl is a scheduled GitHub Actions job that starts, runs and exits.
Architecture
Every six hours on GitHub Actions
ScheduleCron, every 6 h
Fetchrequests, Selenium, JSONInto the API
UpsertMatched on source URL
PostgreSQLSSL requiredOut to people
FrontendDebounced, shareable URLsCrawling runs on its own schedule, so an ingest that takes minutes never slows a page load.
Run every crawler, one at a time
A cron job starts every six hours. The runner imports each of the 38 crawler modules, marks the company as seen, and calls its crawl function inside a try/except, so a crash is logged as one failed row and the loop moves on.
Fetch each site the cheapest way that works
Nine companies publish jobs through a hiring platform's JSON API, such as Greenhouse or Workable, so those are read directly. Five pages only render with JavaScript and need headless Chrome through Selenium. The rest are plain HTML fetched with requests, with a 30 second timeout and up to four retries.
Submit through the API
Each job is posted to an ingest endpoint with a secret key in a header. The API rejects a wrong key with 401, validates the payload with a serializer, and throttles ingest to 60 requests a minute.
Upsert on the source URL
The company and then the job are looked up by their source URL. A new URL is created; an existing one is compared field by field and either updated or counted as a duplicate. In the logged runs, about 125,000 submissions were duplicates against roughly 1,760 creates and updates, which is exactly what repeated crawls should produce.
Serve and search
Django REST Framework serves paginated lists of jobs and companies with search. The React app debounces typing and keeps the view, search and page in the URL, so any result can be shared as a link.
Trade-offs
Why
The API owns validation and the upsert rules, the crawlers never hold database credentials, and the ingest endpoint can be throttled like any other.
The cost
It costs one HTTP request per job. When the API moved hosts, its address was hard-coded in the crawlers, and about 26,700 submissions failed until it was updated.
Why
Every career site is different. Keeping each in its own module means a site's quirks stay in one file, and a site that changes breaks only its own crawler.
The cost
There is no shared base class, so headers and helpers are repeated across modules, and a change to what crawlers return needs 38 edits.
Why
Headless Chrome is slow and heavy, so it is used for the five sites that need it. JSON APIs are used where companies have them, because they are far less brittle than scraped HTML.
The cost
The Selenium crawlers start a fresh browser for every page, so they are the slowest part of each run.
Why
It is stable, it comes with every listing, and it makes repeated runs idempotent without any extra bookkeeping.
The cost
The job table has no unique index on that column yet, so two simultaneous submissions of the same job could both insert it.
Why
Each run stamps when a company's page was last crawled, and the app shows it, so people can see how current a listing is.
The cost
Individual jobs are never marked expired, so the total count grows over time. The site shows about 1,400 listings collected, while a recent run found 366 open ones.
Why
Production turns on HTTPS redirects, HSTS, SSL to the database and a JSON-only API, and the app refuses to start without an ingest key.
The cost
Throttle counters live in each process's memory, so they are not shared if the API ever runs on more than one instance.
Security
Honest limits
Expire stale listings
Mark a job inactive when a run no longer sees it, and filter on that, so the listing count reflects what is actually open.
Batch ingest and a unique index
Send each crawler’s jobs in one request, read the API address from configuration, retry failures, and add a unique index on the source URL.
A shared crawler contract with tests
A small base class and tests for what every crawler returns would catch drift between the runner and the 38 modules.
I am happy to go deeper on any of these trade-offs.
Next case study
BOMWatcher