OpenSafe — OSS Agent Evaluation Sandbox

Challenge

Once an agent gets shell, HTTP, or file-write tools, prompt injection and runaway recursion can produce real side effects, and root-causing them from logs is painful. Existing frameworks hard-code tool permissions, so every policy tweak requires a redeploy.

Solution

Each LangGraph node transition is gated against an OPA/Rego policy declaring allowed tools, arguments, callers, and cost ceilings. Execution itself runs inside gVisor with egress allow-lists enforced at Envoy. Prompts, responses, and tool_calls are emitted as structured OpenTelemetry traces using the gen_ai semantic conventions.

Results

• Blocked 198 of 200 adversarial prompts at the policy layer in the internal test suite
• Permission changes now ship via Rego only — code-bearing deploys dropped from 4/month to 0
• gVisor isolation defeated all 23 escape patterns we threw at it (host never reached)
• p95 tool-call latency surfaced at 320 ms via OTel traces