AgenticLens
{ width="420" }
Open-source observability, evaluation, and operational intelligence for production AI systems.
AgenticLens helps teams profile LLM applications, capture structured agent traces, calculate cost, diagnose memory and retry overhead, and compare candidate configurations against reviewed baselines.
What It Measures
| Dimension | What AgenticLens captures |
|---|---|
| Step token use | Prompt tokens, completion tokens, total tokens by workflow step |
| Cost | Provider pricing, dollar-per-run, monthly projections |
| Latency | Step-level runtime and tokens per second |
| Workflow shape | Planner, retriever, tool, memory, and final-response steps |
| Waste patterns | Repeated prompts, excessive chunks, duplicate tool calls, long history |
| Quality risk | Confidence and risk notes for optimization recommendations |
| Resilience | Fault-injection outcomes through the chaos_events schema extension |
| Tracing | Hierarchical runs and spans with status and error evidence |
| Comparison | Success, tokens, latency, cost, variability, and regression deltas |
| Diagnosis | Failure taxonomy, faulty step and agent, incident timeline, baseline anomalies |
| Evaluation quality | Deterministic vs model-based scores, judge cost, evaluator disagreement |
| Regression suites | Cases from CSV/JSON, traces, and feedback; PII masking, duplicates, splits, approved immutable versions |
Token Optimization Focus
AgenticLens reports token savings at the step where the waste occurs:
| Area | Optimization signal |
|---|---|
| Prompting | Repeated prompt prefixes that should be cached or deduplicated |
| RAG | Excessive top-k retrieval and low-utility retrieved chunks |
| Memory | Conversation history that should be summarized or truncated |
| Tools | Duplicate tool calls that should be cached |
| Multi-agent handoffs | Oversized context passed between agents |
| Workflow | Estimated reducible tokens, cost per run, and monthly savings |
Multi-agent workflows can attach agent_name, agent_role, and handoff metadata
to each step. AgenticLens then reports token usage by agent and flags oversized
handoffs that should be summarized before passing context to the next agent.
Why It Matters
Production agent systems fail in ways that ordinary request logs rarely explain. Token cost can drift across memory, retrieval, planning, and tool use. Reliability can degrade silently when an upstream tool fails. AgenticLens keeps these signals local, inspectable, and exportable so teams can compare workflows across versions.
Documentation
- Workflow schema specification
- AIOS validation and OTel export
- Export formats
- RAG chunk utility
- Trace and comparison
- Evaluation and release gates
- Test-suite and dataset management
- Diagnosis and failure attribution
- Experiments and statistical comparison
- Multi-agent reference workflows
- Product roadmap
- Research roadmap
Quickstart
from agenticlens import profile, step
with profile("Customer Support"):
with step("Planner", type="planner") as s:
response = planner_llm.invoke(prompt)
s.record(response)
uv run agenticlens profile examples/recommendations_demo.py --save workflow.json
uv run agenticlens analyze workflow.json
Current Status
AgenticLens is early-stage open-source software. Workflow profiling, live and offline pricing, optimization recommendations, structured tracing, deterministic memory and retry findings, repeated-run comparison, evaluation suites with versioned evaluators and evidence audits, release gates with webhook alerts, judge calibration, multi-variant experiments with pass@k and significance tests, failure diagnosis with faulty-step attribution and incident timelines, trace-aware evaluators, judge cost and evaluator-disagreement reporting, AIOS draft validation/conformance with rule-cited summaries, OTLP trace export and ingestion, HTML dashboards, CLI commands, and export formats are implemented. The trace, comparison, evaluation, and experiment APIs remain experimental. Framework adapters remain roadmap work.