A finding needs something engineers can verify
AWS published guidance October 7 on configuring an AI vulnerability-triage harness through a persistent steering file. The authors describe checks for actual files, functions, dataflow and call paths before accepting a finding. Their approach also considers deployed controls and exploitation signals rather than relying on the model’s own confidence label.
The test is narrower than the ambition
AWS reports testing the configuration against a deliberately vulnerable application containing ten known issues. With steering, the model identified nine and documented its reasoning about mitigations; the omitted case was a credential intentionally placed in test code and never called. This is a bounded demonstration, not evidence that the configuration will eliminate false findings across arbitrary production applications. The published scoring weights are described as a starting point requiring calibration.
Treat configuration as a reviewed security artifact
Our assessment for software agencies and e-commerce development teams is that AI triage instructions deserve the same change discipline as other security tooling. Assign an owner, preserve the version used for an assessment and keep a small evaluation set with known outcomes. When instructions change, compare the resulting decisions rather than judging the writing style of the reports. Include examples that should be rejected as well as examples that should be escalated. Otherwise a configuration can appear more productive simply because it opens more tickets, while making the engineering queue harder to trust.
Give developers a reproducible handoff
A useful report should tell an engineer what code was examined, which deployment assumptions mattered and what evidence supports the proposed priority. Keep unresolved questions explicit. If a finding depends on a control being active, the reviewer needs a way to verify that control in the relevant environment. Do not let a generated risk score authorize production changes by itself. The accountable security owner should decide how the report enters the existing remediation process and what evidence closes the ticket. For an MSP or development partner, that produces a deliverable the customer can audit and repeat, instead of an impressive narrative with no reliable connection to the running system.
Sources & further reading
Our practical suggestions are editorial guidance. Verify applicability against current source and vendor instructions.
