- On precision vs. noise: yeah, this is a core challenge. Quick answer is the scanner tries to be conservative and lean towards not flagging borderline issues. There's a custom guidance field in the config that lets users adjust sensitivity and severity based on domain/preferences.
- CI times: on a medium-sized PR (say 10k lines) in a fairly large codebase (say a few hundred K lines), it will generally run in 5-15 minutes, and run in parallel with other CI actions. In our case, we have other actions that already take this long, so it doesn't increase total CI time at all.
- Vulnerability types: the post goes into this a bit, but I would look at scanning and red teaming as working together for defense in depth. RAG and tool misuse vulnerabilities are definitely things the scanner can catch. Red teaming is better for vulnerabilities that might not be visible in the code and/or require complex setup state or back and forth to successfully attack.
Comments
Thanks for the comment.
- On precision vs. noise: yeah, this is a core challenge. Quick answer is the scanner tries to be conservative and lean towards not flagging borderline issues. There's a custom guidance field in the config that lets users adjust sensitivity and severity based on domain/preferences.
- CI times: on a medium-sized PR (say 10k lines) in a fairly large codebase (say a few hundred K lines), it will generally run in 5-15 minutes, and run in parallel with other CI actions. In our case, we have other actions that already take this long, so it doesn't increase total CI time at all.
- Vulnerability types: the post goes into this a bit, but I would look at scanning and red teaming as working together for defense in depth. RAG and tool misuse vulnerabilities are definitely things the scanner can catch. Red teaming is better for vulnerabilities that might not be visible in the code and/or require complex setup state or back and forth to successfully attack.