Regex Finds Candidates. AI Decides What's Noise.
We ran Gitleaks, TruffleHog, GitGuardian, and Puaro on four open-source repos. Puaro's AI suppressed 334 candidate findings and left 33 real alerts for humans.
Regex Finds Candidates. AI Decides What's Noise.
Most secret scanners are good at one job: spotting strings that look like credentials. The harder job is deciding which of those strings are noise—sample keys in docs, test fixtures, placeholders—so your team is not the filter.
We ran that question on four open-source repositories with Gitleaks, TruffleHog, GitGuardian, and Puaro, then measured how many candidates Puaro's proprietary AI marked as false positives before anyone opened the queue.
Key takeaways
- Across four repos, Puaro AI suppressed 334 candidate findings and left 33 real alerts for users.
- On bitcoin, the AI suppressed all 253 candidates. User-facing alerts: 0. Other tools still returned raw hits (Gitleaks 25, TruffleHog 13, GitGuardian 12).
- On python-docs-samples, TruffleHog returned 98 raw findings; Puaro showed 5 real alerts after suppressing 46 candidates.
- Pattern engines find candidates. Someone still has to decide what is noise. That someone should not be your on-call engineer.
- Puaro's AI FP labels are Puaro's own classifier—not a claim that every other tool's raw hit is a false positive.
The problem is triage, not detection
When Gitleaks fires on a line, it is doing what regex engines do: matching a pattern. TruffleHog and GitGuardian also return findings you then triage. None of that is useless. Detection without classification just shifts the cost onto humans.
Alert fatigue is the practical failure mode. If the queue is mostly noise, people stop trusting the tool. Real leaks hide in the same list.
Puaro still uses detectors to find candidates. The difference in this run is what happens next: a proprietary AI pass that marks false positives and keeps them out of the active alert list (puaro_real_alerts vs puaro_ai_suppressed_fp in our comparison export).
How we measured
We reused the security-scanner-comparison toolkit:
- Scan each repo with TruffleHog, Gitleaks, and GitGuardian (Docker runners).
- Export Puaro findings from the product (including the AI false-positive flag).
- Cluster matches by file path + overlapping line range (default ±2 lines).
- For Puaro, split totals into real alerts (what a user sees) and AI-suppressed FPs.
Repos in this run:
| Repository | Why it matters | |---|---| | bitcoin | Mature C++ codebase—classic noise risk for pattern engines | | python-docs-samples | Docs and samples—known FP trap | | WrongSecrets | Intentionally planted secrets (OWASP) | | leaky-repo | Small leak-oriented test repo |
This is a focused four-repo slice, not a claim about every repository on GitHub. For an earlier, broader volume comparison, see our April 2026 benchmark and the workflow comparison of Puaro vs Gitleaks vs TruffleHog.
Headline results
| Repository | TruffleHog | Gitleaks | GitGuardian | Puaro real alerts | Puaro AI-suppressed FP |
|---|---|---|---|---|---|
| bitcoin | 13 | 25 | 12 | 0 | 253 |
| python-docs-samples | 98 | 27 | 8 | 5 | 46 |
| wrongsecrets | 21 | 46 | 27 | 2 | 31 |
| leaky-repo | 7 | 19 | 25 | 26 | 4 |
| TOTAL (Puaro) | raw tool totals vary by detector set | 33 | 334 |
On this run, 91% of Puaro's candidates (334 of 367) were suppressed as false positives by the AI layer. Users were left with 33 alerts.
Other tools' columns are raw finding counts. We are not labeling those rows as "all FPs." The point is structural: they return a list; Puaro returns a list after AI FP recognition.
Deep dive: bitcoin
Bitcoin Core is a large, mature codebase. It is a bad place for a naive pattern matcher to look smart.
| Tool | What you get | |---|---| | Gitleaks | 25 raw findings | | TruffleHog | 13 | | GitGuardian | 12 | | Puaro (user-facing) | 0 | | Puaro AI-suppressed | 253 |
Puaro's detectors still found 253 candidates. The AI classified every one as a false positive, so the inventory queue stayed empty for this repo. That is the product story we care about: detection plus context, not detection alone.
Read the numbers carefully. A lower raw count from another tool does not mean it "won" on bitcoin. It means it fired fewer patterns—or different ones—and left classification to whoever opens the report.
Deep dive: python-docs-samples
Documentation and sample code are where secret scanners earn a bad reputation. Example keys, tutorial snippets, and demo configs look like the real thing to a regex.
TruffleHog returned 98 findings on this repo. Gitleaks returned 27. GitGuardian returned 8. Puaro showed 5 real alerts after the AI suppressed 46 candidates.
If your pipeline blocks merges on every pattern match in a docs folder, you either tune endlessly or people turn the scanner off. Context-aware FP recognition is meant to avoid that choice.
WrongSecrets and leaky-repo in brief
WrongSecrets is built to hide intentional secrets. On that repo, Gitleaks returned 46 findings, GitGuardian 27, TruffleHog 21. Puaro showed 2 real alerts after suppressing 31 candidates.
leaky-repo flipped the shape: Puaro left 26 real alerts and suppressed only 4. That matters for the story—AI FP recognition is not a blunt "suppress everything" switch. When the content looks like real leaks, more of the queue stays visible.
Taken together with bitcoin and python-docs-samples, the run shows both sides: heavy suppression where noise dominates, and retention where findings look actionable.
What Puaro's AI changes for triage
The practical difference is who does the first pass:
- Find candidates with detectors (patterns, known formats).
- Classify noise with proprietary AI that reads surrounding code and usage context.
- Show real alerts with severity and reasoning in the dashboard / PR workflow.
Gitleaks remains a strong regex-first tool—fast, local, deterministic. TruffleHog adds verification depth for many credential types. GitGuardian is a commercial platform; we do not treat it as a pure regex engine. We name all three because they were in the run; we do not pretend they are the same product.
What they share for this comparison: the human still receives a finding list that needs triage. Puaro's AI FP layer is the step that turns "looks like a secret" into "show this to a human" or "suppress as noise."
If you already run Gitleaks in CI, that is still a reasonable first gate. The gap this benchmark highlights is what happens after the gate: does your platform reduce the pile, or do you?
Limits and caveats
- AI FP labels are Puaro's.
puaro_ai_suppressed_fpis not an independent lab's ground-truth label for every cluster. - Four repos. Useful for illustrating FP behavior; not a full market share study.
- Detector sets differ. Tools do not search for identical secret types, so raw totals are not a pure apples-to-apples recall contest.
- WrongSecrets / leaky-repo include intentional or test secrets. Low user-facing counts there can reflect classification and product filters, not "missed everything."
- We do not claim a fixed false-positive rate for every customer repository.
If you want reproducibility, the comparison CSVs live under comparison_results/2026-08-12_15-55-47/ in the security-scanner-comparison toolkit.
Try it on your own pull requests
Secret scanning only helps if people trust the queue. Pattern engines find candidates. Puaro's proprietary AI decides what is noise so developers are not the first filter.
Start free on Puaro (no card required within Free plan limits), or read the product overview.