Use AI safely
Review generated code for security
Check the new behavior from an attacker’s perspective before treating working AI-generated code as safe to ship.
8 minute lesson
A generated feature can work perfectly for the intended user and still be unsafe. Security review asks what happens when input, identity, or sequence differs from the happy path.
Suppose a generated route does this:
app.get('/api/invoices/:id', requireLogin, async (req, res) => {
const invoice = await db.invoice.findById(req.params.id)
res.json(invoice)
})
The route checks authentication: the caller is signed in. It does not check authorization: whether that caller may read this invoice.
A valid customer can change the URL to another invoice ID. The feature works, but the trust boundary is broken.
Start with a small threat model
Ask:
- What input can an attacker control?
- What data or action does this code protect?
- Which identity performs the operation?
- Where must the server enforce the boundary?
- What happens for missing, oversized, malformed, or cross-account input?
The invoice query should be scoped by both invoice ID and the current account. The test should sign in as one customer and prove that another customer’s invoice is not returned.
Inspect convenience changes
Generated code often solves friction by weakening a boundary. Look carefully for:
- permissive CORS or wildcard permissions
- disabled certificate or signature checks
- SQL or shell commands built from strings
- secrets added to source or browser code
- verbose errors that expose internal data
- destructive operations without confirmation
- new dependencies added for a tiny task
Also treat content read by agents as untrusted. A document or web page may contain prompt injection that tries to misuse connected tools. Normal authorization and validation must still apply after the model selects an action.
Verify the abuse case
A security suggestion is not a fix until you exercise the dangerous path. Add the cross-account test, malformed input test, or injection test and confirm it fails safely.
A second reviewer or fresh model session can help find omissions, but it does not replace the test or the trust-boundary analysis.
Try this
Review the invoice route above. Describe one abuse request, the server-side boundary that should stop it, and the negative test that proves the fix.
Lesson completed