TracePilot
Evidence-grounded incident investigation for distributed systems
The problem this had to solve.
Turn fragmented incident telemetry into reviewable root-cause candidates without allowing a diagnostic model or public demo to become an unrestricted remediation control plane.
TracePilot brings service topology and operational telemetry into one incident context, correlating Prometheus metrics, Loki logs, Jaeger traces, and change evidence.
The platform ranks plausible root causes, retrieves runbooks, cites supporting evidence, and tracks model evaluation and human feedback. Reproducible Kubernetes fault experiments and controlled sandbox recovery keep investigation behavior testable and reviewable.
Constraints
- Telemetry collection uses fixed, bounded, read-only queries and persists immutable checksummed bundles before analysis.
- The learned models are trained on generated laboratory incidents, not real production incidents.
- Experiments and the single recovery action run only through separately trusted local workers with closed, typed interfaces.
- The hosted profile is a deployment template; it has not been deployed to a cloud provider or measured under production load.
How the system is put together.
Service graph
Distributed-system topology and alert context
sourceIncident
Unifies the active investigation scope
processEvidence correlation
Logs, metrics, traces, topology, and changes
processRoot-cause ranking
Orders candidate causes against collected evidence
decisionEvidence-cited diagnosis
Returns auditable conclusions and runbook context
outputWhat was chosen, and what it cost.
Replay immutable evidence
- Choice
- Store checksum-addressed telemetry bundles in MinIO and perform RCA against replayed artifacts rather than querying live systems during analysis.
- Rationale
- An investigation can be reproduced and its evidence remains tied to the incident window that produced it.
- Cost
- Collection failures and missing signals stay visible; the system does not silently backfill them from a later live state.
Gate learned models against a deterministic baseline
- Choice
- Retain the root-cause ranker as an unpromoted candidate when its held-out result underperforms the graph baseline.
- Rationale
- A passing absolute score is insufficient when a simpler deterministic method performs better.
Close the action surface
- Choice
- Expose seven read-only investigations and one typed, human-approved local recovery action instead of arbitrary shell, Kubernetes, or provider commands.
- Rationale
- Diagnosis and recovery remain separate reviewable capabilities with a bounded blast radius.
- Cost
- The safety boundary intentionally limits automation and is not a general remediation framework.
How it was checked.
- The repository records passing lint, strict type checking, backend/frontend tests, production builds, deterministic OpenAPI export, and eight Compose-backed integration tests.
- A generated 180-incident laboratory dataset evaluates the ranker at Top-1 0.854 and the deterministic baseline at Top-1 1.0; the ranker correctly remains ineligible for promotion.
- The production profile validation, deterministic security scan, PostgreSQL backup/restore test, and local Kind readiness checks are documented as passing.
- The bounded localhost liveness smoke test completed 40/40 requests with p50 52.628 ms and p95 97.919 ms; it is not a capacity benchmark.
What was measured.
Each figure states the conditions it was measured under. None of them describe production traffic.
5
evidence channels
Metrics, logs, traces, topology, and change evidence are correlated.
Auditable
root-cause ranking
Diagnoses retain evidence citations for review.
Where this stops being true.
- No real incident corpus, telemetry shift study, privacy review, fairness assessment, threshold tuning, or production SLO evidence exists.
- The checked-in models cover five synthetic fault classes and must not be used for operational decision-making.
- The hosted template still requires provider-managed secrets, TLS, ingress controls, durable object storage, and digest-specific vulnerability scanning.
- The Kubernetes and Chaos Mesh environment is a local Kind laboratory, not a production deployment target.
What it is built with.
Runtime & interface
- Python
- FastAPI
- Next.js
Data & evaluation
- PostgreSQL
- pgvector
- Redis
- MLflow
Infrastructure & providers
- Kubernetes
- Prometheus
- Loki
- Jaeger
- OpenTelemetry
- MinIO