CASE STUDY — AI SECURITY AUDIT ENGINE
← Back to portfolio
← Ankit Jangir

AI Security Audit Engine

internal codename: Kotwal — the medieval Indian city watch

One command audits the code, pen-tests the running deployment, and writes the fix.

Solo build Python · 19K lines 4 specialist agents AI triage layer Runs in CI

Outcome

264 repositories swept for a supply-chain advisory, in one afternoon

When a compromised release of a widely-used npm package was disclosed, every engineering lead had the same urgent question: are we exposed, anywhere?

Answering by hand means opening each repository, locating the lockfile, resolving the transitive tree, and checking the pinned version. Across a few hundred repositories that is days of work — and the kind where attention fails before the list does.

The engine swept 264 repositories, parsed lockfiles rather than manifests so transitive pins were caught and not just direct dependencies, and returned the exposed set the same afternoon.

Why lockfiles, not manifests

A manifest states the version range you asked for. The lockfile states what actually got installed. In a supply-chain event the difference is the whole answer — a compromised version usually arrives as a transitive pin nobody declared.

Why it exists

Security tooling is priced for enterprises and built for specialists

Commercial application-security platforms start around $50–100 per developer per month and climb into six figures a year. That maths works for a bank. It does not work for a team of eight shipping a product for NGOs.

Open-source alternatives exist — Semgrep, ZAP, Trivy, Bandit, Nuclei — but each covers a fragment. A small team has to stitch six tools together, write the glue, schedule it, de-duplicate findings, and turn the output into something a client's auditor will accept.

I was doing that stitching by hand, repeatedly, across every platform I was responsible for.

One command should audit the code, probe what's running, and hand back the fix.

Decisions & trade-offs

Scanners find. The model judges.

The obvious way to build this in 2026 is to hand the repository to a language model and ask what's wrong. I deliberately didn't.

Detection stays rule-based and deterministic. The same commit produces the same findings every time — the property an auditor actually needs. A pattern either matches a credential or it doesn't; that result is reproducible, explainable, and defensible in review.

The model is confined to the three jobs where judgement genuinely beats rules:

  1. 1Triage. Rule-based scanners are noisy by design — better to over-report than miss. The model reads each finding in its surrounding code and dismisses false positives, cutting most of the noise.
  2. 2Remediation. It writes the actual fix in the context of the file being fixed — not a generic "use parameterised queries" link.
  3. 3Summary. It drafts the executive summary a non-engineer will read, from findings a scanner produced.
The trade-off this buys

A security tool whose findings change between runs is worse than no tool — you cannot track closure, and you cannot show an assessor that a finding was fixed. Keeping detection deterministic preserves that audit trail, while still using the model where non-determinism costs nothing.

Other choices worth naming. OSV.dev over a commercial vulnerability feed, because it is free, API-first, and covers the ecosystems we actually ship. Playwright over raw HTTP probes for the dynamic scanners, because modern apps render client-side and a plain request sees an empty shell. OpenSSF Scorecard as the scoring model rather than my own invention, so the number means something to someone who has never seen this tool.

Architecture

One orchestrator, four specialists

Each agent owns a different attack surface. The orchestrator sequences them and merges findings into a single scored report.

$ audit --repo my-org/my-app --url https://app.example.com
Code Auditor
Static · repo

16 secret patterns, 5 PII classes, entropy detection, dependency CVEs via OSV.dev, 22 SAST patterns, workflow drift and branch policy.

Pen-Tester
Dynamic · live

10 scanners against the running deployment: SQLi, XSS, CSRF, IDOR, clickjacking, open redirect, traversal, session and upload flaws.

Runtime Monitor
Posture · production

Uptime, TLS certificate expiry, and the security headers assessors ask about — HSTS, CSP, X-Frame-Options.

Policy Inspector
Compliance · org

Branch protection, secret scanning, Dependabot, CODEOWNERS, disclosure policy. Scored against OpenSSF Scorecard's 13 checks.

Ships: branded DOCX report Top-10 email digest Auto-PR into the target repo OpenSSF Scorecard 0–100 Scheduled runs in GitHub Actions

What I'd do differently

My own viability gate failed it

Before extending this past internal use I wrote the strategy work a product needs: competitive pricing analysis, market sizing, and an explicit set of go/no-go gates set before the assessment rather than rationalised after it.

One gate failed. The tool was still carrying internal branding, hardcoded organisation defaults, and error messages that leaked internal infrastructure names. Ship it in that state and the first external user finds someone else's company inside their security report — which destroys trust on day one, in the one product category where trust is the entire value proposition.

That is why it hasn't shipped externally. The honest finding was more useful than the launch.

What I would change structurally: branding and tenancy should have been configuration from the first commit, not a retrofit. Building it as an internal tool first made every default a hidden assumption, and hidden assumptions are exactly what a codebase audit has to go hunting for later.

Stack

Built with

Python 3.11+TyperPydantichttpx PlaywrightBanditSemgrepOSV.dev python-docxGitHub REST APIGitHub ActionsAWS SES OpenSSF ScorecardClaude