Skip to main content

Monitoring

This guide covers monitoring and observability for your Actyze deployment.

Overview​

Actyze follows an emit-only observability model: every service emits structured JSON logs to stdout and exposes Prometheus metrics, and you bring your own backend (Prometheus, Grafana, Datadog, Splunk, Grafana Loki — anything that scrapes a /metrics endpoint or ingests stdout). No observability backend is bundled with Actyze, and none is required to run it.

A shared observability library instruments every service the same way, so logging, metrics, and health checks are consistent across:

  • Nexus — FastAPI orchestration service
  • Schema Service — FAISS-powered schema recommendations
  • Prediction Workers — XGBoost / LightGBM / AutoGluon
  • Frontend — React app served by nginx
  • PostgreSQL and Trino — monitored via their own native tooling (see below)

Monitoring your deployment helps you track system health, debug issues quickly, optimize resource usage, and plan for scaling.

Scope

Today's instrumentation is operational / SRE-facing — health probes, metrics, and logs for whoever runs the deployment. End-user-facing observability (per-query execution timelines, query history with timings surfaced in the UI, cache/freshness indicators) is on the roadmap and not yet shipped.

Health Checks​

Every Actyze service exposes Kubernetes-compatible health endpoints:

EndpointPurpose
/healthzLiveness — the process is up
/readyzReadiness — dependencies reachable; gates traffic
/healthDetailed aggregated status (Nexus rolls up Schema Service, LLM, Trino, and cache)
/metricsPrometheus metrics
# Nexus
curl http://<nexus-service>:8000/healthz
curl http://<nexus-service>:8000/readyz
curl -s http://<nexus-service>:8000/health | jq '.status, .details.services[].name'

# Schema Service
curl http://<schema-service>:8001/health

The aggregated /health returns each dependency's status, e.g.:

{
  "status": "healthy",
  "service": "nexus",
  "details": {
    "services": [
      { "name": "schema-service", "healthy": true },
      { "name": "llm_service",   "healthy": true },
      { "name": "trino-service", "healthy": true },
      { "name": "cache-service", "healthy": true }
    ]
  }
}

Metrics​

Each service exposes Prometheus-format metrics at GET /metrics:

  • HTTP — request rate, latency histograms, in-flight requests
  • Queries — NL-query and SQL-execution counters, execution duration, result rows
  • Predictions — pipeline run counts and durations (XGBoost / LightGBM / AutoGluon)
  • Process / runtime — CPU, memory, GC, file descriptors
curl -s http://<nexus-service>:8000/metrics | head -30

Scraping with Prometheus​

For Kubernetes deployments using the Prometheus Operator, add a ServiceMonitor (or PodMonitor) per service targeting its HTTP port and the /metrics path:

# Install kube-prometheus-stack (Prometheus + Grafana + Alertmanager)
helm install prometheus-operator prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

See shared/observability/docs/KUBERNETES.md in the main repo for ready-to-use ServiceMonitor and log-collection examples.

Application Logs​

All Actyze services write structured JSON logs to stdout with propagated context variables (request_id, user_id, query_id, session_id) so requests can be correlated across services.

# Nexus (FastAPI orchestration)
kubectl logs -f deployment/dashboard-nexus -n actyze | jq '.'

# Schema Service
kubectl logs -f deployment/dashboard-schema-service -n actyze | jq 'select(.event)'

# Frontend (nginx access/error logs)
kubectl logs -f deployment/dashboard-frontend -n actyze

Collect and ship these with your cluster log agent (Fluent Bit, Vector, etc.) to Loki, ELK, Datadog, or your aggregator of choice.

Kubernetes Resource Monitoring​

# Pod status and restarts
kubectl get pods -n actyze

# Resource usage
kubectl top pods -n actyze
kubectl top nodes

Database & Query Engine Monitoring​

PostgreSQL​

Check active connections:

kubectl exec -it <postgres-pod> -n actyze -- \
  psql -U postgres -c "SELECT * FROM pg_stat_activity;"

Monitor database size:

kubectl exec -it <postgres-pod> -n actyze -- \
  psql -U postgres -c "SELECT pg_size_pretty(pg_database_size('dashboard'));"

For Prometheus-based monitoring, run postgres_exporter as a sidecar or standalone deployment.

Trino​

Trino ships its own Web UI showing per-query execution stages, rows processed, wall time, and memory:

kubectl port-forward svc/dashboard-trino -n actyze 8081:8080

Then open http://localhost:8081. Trino also exposes JMX metrics that can be scraped into Prometheus.

Alerts & Notifications​

Once metrics are flowing into Prometheus, alert on what matters:

  1. Resource usage — CPU/memory thresholds, disk space, pod restart counts
  2. Application — API error rates, query failure rates, request latency (p95/p99)
  3. Database — connection-pool exhaustion, slow queries, replication lag

Key Performance Metrics​

  • Request latency — API response times (p50/p95/p99)
  • Throughput — requests per second
  • Error rate — failed-request percentage
  • Query performance — SQL execution duration
  • Resource usage — CPU, memory, disk I/O

Troubleshooting​

High memory usage​

kubectl top pods -n actyze --sort-by=memory

Frequent pod restarts​

kubectl get pods -n actyze
kubectl logs <pod-name> -n actyze --previous

A service reports unhealthy​

Query the aggregated health endpoint to see which dependency is failing:

curl -s http://<nexus-service>:8000/health | jq '.details.services'

Best Practices​

  1. Keep structured (JSON) logging on — easier parsing and request correlation via context IDs
  2. Set resource limits — define CPU/memory limits in Helm values to prevent exhaustion
  3. Scrape /metrics into Prometheus — and build dashboards/alerts in Grafana
  4. Configure log retention — use a log aggregator for long-term storage