Case study
An AI bill of materials for any GitHub repo, generated on that repo's own Actions runners, so the source code never leaves GitHub. This is how it works, and the trade-offs behind it.
Why
AI features are landing in codebases faster than anyone is tracking them. Most teams cannot quickly answer simple questions: which models do we call, from which providers, in which repos, and under what licenses?
At Broadstone I built scanning and inventory as stages every build had to pass, so problems showed up before release instead of after it. BOMWatcher takes the same idea, a scan as a pipeline stage rather than an afterthought, and turns it into something anyone can connect a GitHub account to.
Constraints
Code never leaves GitHub
BOMWatcher never clones a repo. It only ever receives the finished report.
Nothing runs without consent
The scan is added through a pull request the owner reviews. Nothing happens until it is merged.
Read-only in the repo
The workflow runs with contents: read and does not keep checkout credentials.
An open format
Results are CycloneDX 1.6, so any tool that reads it can use them.
Architecture
Once per repo
InstallGitHub AppEvery push to the default branch, on GitHub
RunnerCheckout, read-only
DetectorAI models, SDKsBack on BOMWatcher
IngestDownload, validate
StoreKeyed by run IDThe only thing that crosses from the customer’s repo to BOMWatcher is the report itself.
Connect and pick repos
The user installs the BOMWatcher GitHub App and chooses which repos it can see. The API authenticates as the app with a short-lived JWT, exchanges it for an installation token, and caches that token per installation so a busy dashboard does not mint a new one on every call.
Open a pull request, not a commit
For each repo the API creates a branch, writes a roughly twenty-line workflow file to it, and opens a PR that explains what the workflow does. If the workflow already exists it skips the PR, and if a previous branch is left over it resets it rather than failing.
Scan on the repo’s own runner
Once merged, every push to the default branch runs a composite Action. Syft inventories dependencies into CycloneDX, then a Python script walks the source for AI model names, provider SDK imports and Hugging Face loaders, merges both into one BOM and uploads it as a build artifact.
Hear about it through a signed webhook
When the run finishes, GitHub sends a workflow_run event. The API checks the HMAC-SHA256 signature in constant time, caps the body at 5 MB, records the scan and replies 202 immediately.
Ingest after replying
A background task downloads the artifact with a fresh installation token, enforces a 20 MB limit on the listed size, the zip and the file inside it, checks that the document really is CycloneDX, and stores it against the scan. Failures are recorded on the scan, not swallowed.
Trade-offs
Why
It is the only way to promise the code never leaves GitHub: nothing is cloned or stored on my side. It also means scanning capacity grows with the number of users at no cost to me.
The cost
Scans spend the customer’s Actions minutes, take as long as a CI job, and I cannot see anything beyond the artifact the run produces.
Why
The owner reads exactly what will run before anything runs, and stopping is as simple as deleting one file. That consent step is what makes a stranger comfortable installing the app.
The cost
There is an extra step before the first result, and some people never merge. The API has to track PR state and handle repos where the workflow already exists.
Why
If the Action posted results to my API, every repo would need a BOMWatcher secret to authenticate with. Instead the Action only uploads an artifact, GitHub tells me the run finished, and I fetch the artifact with a token GitHub issues to the app. No secret lives in user repos.
The cost
It adds a round trip, depends on webhook delivery, and relies on the artifact still existing. Artifacts are kept for 30 days.
Why
GitHub expects a webhook response within a few seconds. Verifying the signature and answering 202 before downloading anything means a slow artifact never makes GitHub mark the delivery as failed.
The cost
The ingest runs as a FastAPI background task in the same process. If the process restarts mid-ingest, that scan is left marked as running.
Why
GitHub sends several workflow_run events per run and can redeliver them. With the run ID as the primary key, every event for a run lands on the same row, so handling them is idempotent.
The cost
Re-running a workflow keeps the same run ID, so the newest attempt overwrites the previous one rather than adding a second scan.
Why
Regular expressions for model names, SDK imports and Hugging Face loaders run anywhere Python 3 does, need no dependencies, and catch the common case: a model name written in the code.
The cost
Names built at runtime or read from environment variables are missed, and each new model family needs a pattern.
Security
Quality
A pytest suite drives the whole lifecycle against a fake GitHub client: connect, enable a repo, merge the PR, receive workflow_run, ingest the BOM. Around it are tests for the trial limit, repos that already have the workflow, a PR that fails to open, an invalid BOM, one user trying to read another user’s repos, unsigned webhooks, and uninstall cleanup.
Honest limits
Move ingest onto a real queue
Celery with RabbitMQ or Redis would survive restarts and retry failed downloads, instead of relying on in-process background tasks.
Share state across instances
The rate limiter and token cache live in process memory. That is fine on one instance, but a second one needs them in Redis.
Validate the full schema
Ingest checks the CycloneDX envelope. Validating against the full 1.6 schema would catch malformed components earlier.
I am happy to go deeper on any of these trade-offs.
Next case study
SiteScopia