Skip to main content

Governance & Observability Overview

Status

Owner: TBD · Status: draft · Canonical source: flezi.app.frontend.main/src/pages/GovernanceOverview.tsx

Who is this for?​

Organization Administrators (ORG_ADMIN), System Administrators (SYSTEM_ADMIN), and AI Builders (AI_BUILDER).

What can you do?​

  • Gain holistic visibility into organizational LLM expenditures.
  • Monitor response accuracy, latency, and agent task completion rates.
  • Deep-dive into individual runs via distributed tracing.
  • Organize automated benchmark evaluation runs and manage golden test datasets.

Governance Modules​

  1. Overview: Summary cards of total spend, token volume, and model call metrics.
  2. Tracing: Visual span trees breaking down agent execution down to millisecond accuracy.
  3. Usage & Costs: Granular expense allocation across workspaces, users, and models.
  4. Quality Dashboard: User satisfaction sentiment and automated accuracy benchmarks.
  5. Evaluations: Benchmark test suites with LLM-as-a-judge scorers.