Kagami — Multi-provider LLM Gateway

Challenge

When apps call OpenAI, Anthropic, Bedrock, and local vLLM directly, API-key management, cost rollups, PII-leak risk, and rate limits scatter across teams — making audit and governance painful.

Solution

Envoy ext_authz aggregates traffic; the Go control plane evaluates per-tenant rules and budget caps; the Rust data plane redacts PII inline on streaming responses. Presidio gained Japanese recognizers (names, addresses, My Number), and weighted routing balances per-token cost against latency.

Results

• Consolidated API-key management surface from 12 apps to 1
• 96.5% PII-detection rate on test documents (internal evaluation)
• Estimated -22% monthly token spend via cost-aware routing (internal simulation)
• Every request stamped with trace_id and audit log, wired into SIEM