Laminar is an open-source, self-hosted LLMOps observability platform providing OpenTelemetry tracing, evaluations, and loop detection for AI agents.
Laminar was founded by infrastructure engineers from Palantir, Amazon, and Bloomberg during Y Combinator's S24 batch to address the debugging challenges of autonomous AI agents. The founders built the engine in Rust around OpenTelemetry standards to keep high-throughput trace ingestion efficient on private hardware. That origin yields a self-hosted platform that tracks non-deterministic agent executions, tool calls, and cost metrics without exporting internal prompts to third-party clouds.
LangChain centers its commercial telemetry around LangSmith, a proprietary SaaS platform that records execution steps, manages evaluation datasets, and meters billing by trace volume. For teams deploying agents across diverse frameworks—or making direct API calls—relying on LangSmith creates vendor lock-in and exports prompts outside internal infrastructure. Laminar replaces that hosted dependency with an open-source, self-hosted telemetry engine built on OpenTelemetry standards. Its SDK instruments multiple runtimes, capturing execution trees from LangChain, Vercel AI SDK, Browser Use, Stagehand, and direct model calls through one interface.
The operational divide centers on framework neutrality. While LangSmith favors native LangChain components, Laminar treats every agent architecture equally through standardized trace spans, ClickHouse storage, and Model Context Protocol endpoints. Engineering squads can inspect reasoning graphs, configure Slack alerts for runaway loops, and run CI evaluations without paying SaaS seat fees or adopting LangChain's wrappers. However, Laminar covers only observability and evaluations; teams still use LangChain or alternative libraries to structure prompts and dispatch tools.
Laminar vs LangChain — which should I choose? Choose LangChain's hosted tooling if your stack relies exclusively on LangChain and you want a managed SaaS dashboard in LangSmith; choose Laminar if you need a framework-agnostic, self-hosted observability platform that captures OpenTelemetry traces on your own servers.
Can I self-host Laminar in production? Yes. Teams can deploy Laminar into Kubernetes clusters using official Helm charts or Docker Compose, retaining full control over data storage and private network boundaries.
Is Laminar free for commercial use? Yes. The core software is open source and available for commercial deployment without licensing costs, though teams provide their own server infrastructure and LLM API keys for automated evaluations.
Laminar is a practical match for engineering teams running multi-step AI agents that require deep OpenTelemetry inspection and private data residency, though organizations wanting zero infrastructure maintenance may prefer fully hosted vendor offerings. Active project maintenance through late 2026 and an Apache-2.0 license ensure long-term stability and legal flexibility for teams embedding self-hosted agent observability directly into their production clusters.
We started this exact setup from scratch in a clean Linux machine on Nov 3, 2026 and checked it through Laminar's own API: every check passed. Recipes are re-checked regularly and removed if they stop working.
mkdir -p laminar && cd laminar
curl -fsSL https://open-source.store/tools/laminar/compose.yml -o compose.ymlcat > .env <<EOF
POSTGRES_USER=postgres
POSTGRES_PASSWORD=$(openssl rand -hex 12)
POSTGRES_DB=postgres
CLICKHOUSE_USER=ch_user
CLICKHOUSE_PASSWORD=$(openssl rand -hex 12)
AEAD_SECRET_KEY=$(openssl rand -hex 32)
NEXTAUTH_SECRET=$(openssl rand -hex 16)
SHARED_SECRET_TOKEN=$(openssl rand -hex 16)
NEXTAUTH_URL=http://localhost:5667
NEXT_PUBLIC_URL=http://localhost:5667
EOFPOSTGRES_USER — Username for PostgreSQL database.POSTGRES_PASSWORD — Password for PostgreSQL database. Generated above; keep it secret.POSTGRES_DB — Database name for PostgreSQL.CLICKHOUSE_USER — Default user for ClickHouse analytics store.CLICKHOUSE_PASSWORD — Password for ClickHouse user. Generated above; keep it secret.AEAD_SECRET_KEY — 32-byte (64 hex characters) AEAD encryption key for sensitive credentials. Generated above; keep it secret.NEXTAUTH_SECRET — Secret key used to encrypt and sign authentication sessions and tokens. Generated above; keep it secret.SHARED_SECRET_TOKEN — Shared secret token between frontend and backend services. Generated above; keep it secret.NEXTAUTH_URL — Canonical URL where the frontend web UI is reached. Set to your public domain in production.NEXT_PUBLIC_URL — Public URL for links and redirects. Set to your public domain in production.docker compose up -dservices:
postgres:
image: postgres:16-alpine
environment:
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: ${POSTGRES_DB}
volumes:
- postgres-data:/var/lib/postgresql/data
restart: always
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 3s
timeout: 5s
retries: 10
clickhouse:
image: clickhouse/clickhouse-server:26.5
environment:
CLICKHOUSE_USER: ${CLICKHOUSE_USER}
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
volumes:
- clickhouse-data:/var/lib/clickhouse/
- clickhouse-logs:/var/log/clickhouse-server/
configs:
- source: clickhouse_profiles_config
target: /etc/clickhouse-server/users.d/lmnr.xml
- source: clickhouse_server_config
target: /etc/clickhouse-server/config.d/lmnr.xml
ulimits:
nofile:
soft: 262144
hard: 262144
restart: always
healthcheck:
test: ["CMD-SHELL", "wget --no-verbose --tries=1 -O - http://127.0.0.1:8123/ping || exit 1"]
interval: 5s
timeout: 5s
retries: 10
quickwit:
image: quickwit/quickwit:v0.9.1
environment:
QW_DATA_DIR: /quickwit/qwdata
command: ["run"]
volumes:
- quickwit-data:/quickwit/qwdata
restart: always
healthcheck:
test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/127.0.0.1/7280 && printf 'GET /health/livez HTTP/1.0\\r\\n\\r\\n' >&3 && grep -q '200 OK' <&3"]
interval: 5s
timeout: 5s
retries: 10
frontend:
image: ghcr.io/lmnr-ai/frontend:v0.3.3
ports:
- "127.0.0.1:5667:5667"
depends_on:
postgres:
condition: service_healthy
clickhouse:
condition: service_healthy
quickwit:
condition: service_healthy
app-server:
condition: service_healthy
environment:
DATABASE_URL: postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}
PORT: 5667
BACKEND_URL: http://app-server:8000
BACKEND_RT_URL: http://app-server:8002
SHARED_SECRET_TOKEN: ${SHARED_SECRET_TOKEN}
NEXTAUTH_URL: ${NEXTAUTH_URL}
NEXTAUTH_SECRET: ${NEXTAUTH_SECRET}
NEXT_PUBLIC_URL: ${NEXT_PUBLIC_URL}
ENVIRONMENT: LITE
LAMINAR_TELEMETRY_DISABLED: "true"
CLICKHOUSE_URL: http://clickhouse:8123
CLICKHOUSE_USER: ${CLICKHOUSE_USER}
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
CLICKHOUSE_RO_USER: ${CLICKHOUSE_USER}
CLICKHOUSE_RO_PASSWORD: ${CLICKHOUSE_PASSWORD}
AEAD_SECRET_KEY: ${AEAD_SECRET_KEY}
QUICKWIT_SEARCH_URL: http://quickwit:7280
restart: always
healthcheck:
test: ["CMD-SHELL", "wget -q -O - http://127.0.0.1:5667/sign-in | grep -q 'Laminar' || exit 1"]
interval: 5s
timeout: 5s
retries: 20
start_period: 20s
app-server:
image: ghcr.io/lmnr-ai/app-server:v0.3.3
ports:
- "127.0.0.1:8000:8000"
- "127.0.0.1:8001:8001"
depends_on:
postgres:
condition: service_healthy
clickhouse:
condition: service_healthy
quickwit:
condition: service_healthy
environment:
PORT: 8000
GRPC_PORT: 8001
DATABASE_URL: postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}
SHARED_SECRET_TOKEN: ${SHARED_SECRET_TOKEN}
CLICKHOUSE_URL: http://clickhouse:8123
CLICKHOUSE_USER: ${CLICKHOUSE_USER}
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
CLICKHOUSE_RO_USER: ${CLICKHOUSE_USER}
CLICKHOUSE_RO_PASSWORD: ${CLICKHOUSE_PASSWORD}
ENVIRONMENT: LITE
AEAD_SECRET_KEY: ${AEAD_SECRET_KEY}
QUICKWIT_SEARCH_URL: http://quickwit:7280
QUICKWIT_INGEST_URL: http://quickwit:7281
QUICKWIT_SPANS_INDEX_ID: spans_v2
restart: always
healthcheck:
test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/127.0.0.1/8000 && printf 'GET /health HTTP/1.0\\r\\n\\r\\n' >&3 && grep -q '200 OK' <&3"]
interval: 5s
timeout: 5s
retries: 10
configs:
clickhouse_profiles_config:
content: |
<?xml version="1.0"?>
<clickhouse>
<profiles>
<default>
<date_time_input_format replace="replace">best_effort</date_time_input_format>
</default>
</profiles>
</clickhouse>
clickhouse_server_config:
content: |
<?xml version="1.0"?>
<clickhouse>
<trace_log remove="1"/>
<text_log remove="1"/>
<part_log remove="1"/>
<metric_log remove="1"/>
<asynchronous_metric_log remove="1"/>
<background_schedule_pool_log remove="1"/>
<query_metric_log remove="1"/>
<query_thread_log remove="1"/>
<query_views_log remove="1"/>
<session_log remove="1"/>
<crash_log remove="1"/>
<opentelemetry_span_log remove="1"/>
<zookeeper_log remove="1"/>
<blob_storage_log remove="1"/>
<processors_profile_log remove="1"/>
<asynchronous_insert_log remove="1"/>
<query_log>
<ttl>event_date + INTERVAL 3 DAY DELETE</ttl>
</query_log>
<error_log>
<ttl>event_date + INTERVAL 3 DAY DELETE</ttl>
</error_log>
</clickhouse>
volumes:
postgres-data:
clickhouse-data:
clickhouse-logs:
quickwit-data:
Lint, codegen, breaking-change checks and dependency management in one Git-native CLI. Open source, no registry lock-in.
Show that Laminar is listed as an open-source alternative. Paste into your README:
Ubuntu 24.04 · Apple container, Docker 29.1.3, Compose 2.40.3+ds1-0ubuntu1~24.04.1. Images pulled in 187 s: ghcr.io/lmnr-ai/app-server:v0.3.3, postgres:16-alpine, clickhouse/clickhouse-server:26.5, quickwit/quickwit:v0.9.1, ghcr.io/lmnr-ai/frontend:v0.3.3.
Based on the upstream setup. Official Docker Compose configuration (lite mode). Pinned images to v0.3.3, postgres to 16-alpine, clickhouse to 26.5, and quickwit to v0.9.1; embedded ClickHouse configuration via Compose configs; ports 5667, 8000, and 8001 are bound to 127.0.0.1 because passwordless local-email login is enabled without an OAuth provider and port 8000 exposes unauthenticated internal routes; ENVIRONMENT=LITE uses in-memory queues/cache instead of RabbitMQ/Valkey from docker-compose-full.yml; parameterized secrets and database passwords via .env.
postgres-1 | 2026-11-03T07:48:00.011012376Z 2026-11-03 07:48:00.010 UTC [40] LOG: background worker "logical replication launcher" (PID 46) exited with exit code 1
postgres-1 | 2026-11-03T07:48:00.011059584Z 2026-11-03 07:48:00.011 UTC [41] LOG: shutting down
postgres-1 | 2026-11-03T07:48:00.012873584Z 2026-11-03 07:48:00.012 UTC [41] LOG: checkpoint starting: shutdown immediate
postgres-1 | 2026-11-03T07:48:00.016367501Z 2026-11-03 07:48:00.016 UTC [41] LOG: checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.006 s; sync files=2, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=0 kB; lsn=0/14F7160, redo lsn=0/14F7160
postgres-1 | 2026-11-03T07:48:00.017742084Z 2026-11-03 07:48:00.017 UTC [40] LOG: database system is shut down
postgres-1 | 2026-11-03T07:48:00.107598668Z done
postgres-1 | 2026-11-03T07:48:00.107635960Z server stopped
postgres-1 | 2026-11-03T07:48:00.108125168Z
postgres-1 | 2026-11-03T07:48:00.108128168Z PostgreSQL init process complete; ready for start up.
postgres-1 | 2026-11-03T07:48:00.108128960Z
postgres-1 | 2026-11-03T07:48:00.117691210Z 2026-11-03 07:48:00.117 UTC [1] LOG: starting PostgreSQL 16.15 on aarch64-unknown-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit
postgres-1 | 2026-11-03T07:48:00.117704918Z 2026-11-03 07:48:00.117 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres-1 | 2026-11-03T07:48:00.117706210Z 2026-11-03 07:48:00.117 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres-1 | 2026-11-03T07:48:00.118972293Z 2026-11-03 07:48:00.118 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres-1 | 2026-11-03T07:48:00.120424710Z 2026-11-03 07:48:00.120 UTC [54] LOG: database system was shut down at 2026-11-03 07:48:00 UTC
postgres-1 | 2026-11-03T07:48:00.122791210Z 2026-11-03 07:48:00.122 UTC [1] LOG: database system is ready to accept connections
app-server-1 | 2026-11-03T07:48:05.167005837Z 2026-11-03T07:48:05.166996Z INFO app_server::batch_worker::worker: Worker d9aedc43-8487-4ed7-8adb-7ec0ab12010b (Spans) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.167023045Z 2026-11-03T07:48:05.167016Z INFO app_server::batch_worker::worker: Worker c85413e8-30fc-4f0b-94f9-2c0d9fb0d9fc (Spans) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.167031587Z 2026-11-03T07:48:05.167027Z INFO app_server::batch_worker::worker: Worker 64fd5cb1-d971-49fe-bba1-0f6124b52526 (Spans) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.167041629Z 2026-11-03T07:48:05.167037Z INFO app_server::worker: Worker 55177fd4-6333-47f0-883d-9c29ffdceac3 (SpansIndexer) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.167059754Z 2026-11-03T07:48:05.167048Z INFO app_server::worker: Worker 1474d538-12f2-46c0-8a80-e2b1adb319da (SpansIndexer) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.167075295Z 2026-11-03T07:48:05.167067Z INFO app_server::worker: Worker 18759d80-1b38-4453-8936-c627674c2087 (SpansIndexer) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.167098795Z 2026-11-03T07:48:05.167091Z INFO app_server::batch_worker::worker: Worker 77b656a8-e3d7-48a5-9e45-e7cc223a7747 (Spans) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.167157379Z 2026-11-03T07:48:05.166803Z WARN app_server::auth::cli_user: NEXT_INTERNAL_URL/NEXT_PUBLIC_URL unset; CLI-auth JWKS defaults to https://laminar.sh
app-server-1 | 2026-11-03T07:48:05.173130420Z 2026-11-03T07:48:05.172992Z INFO app_server::worker: Queue 'notifications' prefetch_count=128 (override via NOTIFICATIONS_PREFETCH_COUNT)
app-server-1 | 2026-11-03T07:48:05.173165462Z 2026-11-03T07:48:05.173018Z INFO app_server::worker: Spawning worker fc74172f-1606-4a98-beae-64b15d0a85d9 (Notifications) instance 0
app-server-1 | 2026-11-03T07:48:05.176821837Z 2026-11-03T07:48:05.173101Z INFO app_server::worker: Worker fc74172f-1606-4a98-beae-64b15d0a85d9 (Notifications) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.176852004Z 2026-11-03T07:48:05.173158Z INFO app_server::worker: Spawning worker e8c69f5e-705f-4f7c-9c81-b8766ae33141 (Notifications) instance 1
app-server-1 | 2026-11-03T07:48:05.176856129Z 2026-11-03T07:48:05.173172Z INFO app_server::worker: Worker e8c69f5e-705f-4f7c-9c81-b8766ae33141 (Notifications) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.177158170Z 2026-11-03T07:48:05.177107Z INFO app_server: Spinning up full HTTP server
app-server-1 | 2026-11-03T07:48:05.177369004Z 2026-11-03T07:48:05.177349Z INFO actix_server::builder: starting 2 workers
app-server-1 | 2026-11-03T07:48:05.177398670Z 2026-11-03T07:48:05.177389Z INFO actix_server::server: Tokio runtime found; starting in existing Tokio runtime
app-server-1 | 2026-11-03T07:48:05.177407337Z 2026-11-03T07:48:05.177402Z INFO actix_server::server: starting service: "actix-web-service-0.0.0.0:8000", workers: 2, listening on: 0.0.0.0:8000
app-server-1 | 2026-11-03T07:48:05.193138254Z 2026-11-03T07:48:05.193056Z INFO app_server::worker: Queue 'notification_deliveries' prefetch_count=128 (override via NOTIFICATION_DELIVERIES_PREFETCH_COUNT)
app-server-1 | 2026-11-03T07:48:05.193157504Z 2026-11-03T07:48:05.193075Z INFO app_server::worker: Spawning worker f044d1b0-2716-4637-9983-dfc16b4d7070 (NotificationDeliveries) instance 0
app-server-1 | 2026-11-03T07:48:05.193171045Z 2026-11-03T07:48:05.193119Z INFO app_server::worker: Worker f044d1b0-2716-4637-9983-dfc16b4d7070 (NotificationDeliveries) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193172212Z 2026-11-03T07:48:05.193129Z INFO app_server::worker: Spawning worker 0e1ec949-831a-4cea-81dc-568e5b2dd812 (NotificationDeliveries) instance 1
app-server-1 | 2026-11-03T07:48:05.193180129Z 2026-11-03T07:48:05.193136Z INFO app_server::worker: Worker 0e1ec949-831a-4cea-81dc-568e5b2dd812 (NotificationDeliveries) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193181295Z 2026-11-03T07:48:05.193139Z INFO app_server::worker: Queue 'input_extraction_queue' prefetch_count=128 (override via INPUT_EXTRACTION_QUEUE_PREFETCH_COUNT)
app-server-1 | 2026-11-03T07:48:05.193182045Z 2026-11-03T07:48:05.193141Z INFO app_server::worker: Spawning worker 9dcdc103-725a-480b-896c-2b1e8d868a8e (InputExtraction) instance 0
app-server-1 | 2026-11-03T07:48:05.193182712Z 2026-11-03T07:48:05.193151Z INFO app_server::worker: Worker 9dcdc103-725a-480b-896c-2b1e8d868a8e (InputExtraction) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193183379Z 2026-11-03T07:48:05.193156Z INFO app_server::worker: Spawning worker b8ad57f7-f873-4433-aba6-8d5071981325 (InputExtraction) instance 1
app-server-1 | 2026-11-03T07:48:05.193184004Z 2026-11-03T07:48:05.193162Z INFO app_server::worker: Worker b8ad57f7-f873-4433-aba6-8d5071981325 (InputExtraction) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193184670Z 2026-11-03T07:48:05.193164Z INFO app_server::worker: Queue 'user_task_regex_queue' prefetch_count=128 (override via USER_TASK_REGEX_QUEUE_PREFETCH_COUNT)
app-server-1 | 2026-11-03T07:48:05.193190379Z 2026-11-03T07:48:05.193165Z INFO app_server::worker: Spawning worker 1c8200eb-7b7b-402d-b111-3966cb7d98c7 (UserTaskRegex) instance 0
app-server-1 | 2026-11-03T07:48:05.193198545Z 2026-11-03T07:48:05.193170Z INFO app_server::worker: Worker 1c8200eb-7b7b-402d-b111-3966cb7d98c7 (UserTaskRegex) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193199504Z 2026-11-03T07:48:05.193174Z INFO app_server::worker: Spawning worker d9e6b2c2-9c07-431c-9eb8-1c88b8b9b457 (UserTaskRegex) instance 1
app-server-1 | 2026-11-03T07:48:05.193200170Z 2026-11-03T07:48:05.193180Z INFO app_server::worker: Worker d9e6b2c2-9c07-431c-9eb8-1c88b8b9b457 (UserTaskRegex) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193200837Z 2026-11-03T07:48:05.193183Z INFO app_server::worker: Queue 'logs_queue' prefetch_count=128 (override via LOGS_QUEUE_PREFETCH_COUNT)
app-server-1 | 2026-11-03T07:48:05.193201504Z 2026-11-03T07:48:05.193184Z INFO app_server::worker: Spawning worker 5933f700-94fe-46b0-82eb-dedbf52e90df (Logs) instance 0
app-server-1 | 2026-11-03T07:48:05.193202129Z 2026-11-03T07:48:05.193189Z INFO app_server::worker: Worker 5933f700-94fe-46b0-82eb-dedbf52e90df (Logs) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193203837Z 2026-11-03T07:48:05.193191Z INFO app_server::worker: Spawning worker 625b9e03-a0eb-4b73-91cb-a57e3cf21088 (Logs) instance 1
app-server-1 | 2026-11-03T07:48:05.193211712Z 2026-11-03T07:48:05.193195Z INFO app_server::worker: Worker 625b9e03-a0eb-4b73-91cb-a57e3cf21088 (Logs) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193212962Z 2026-11-03T07:48:05.193197Z INFO app_server::worker: Spawning worker 1784d140-e521-456e-80f5-f19a1928b70f (Logs) instance 2
app-server-1 | 2026-11-03T07:48:05.193213587Z 2026-11-03T07:48:05.193200Z INFO app_server::worker: Worker 1784d140-e521-456e-80f5-f19a1928b70f (Logs) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193214254Z 2026-11-03T07:48:05.193202Z INFO app_server::worker: Spawning worker fc9b76ef-65e2-4190-96d5-836a2270c7fc (Logs) instance 3
app-server-1 | 2026-11-03T07:48:05.193221420Z 2026-11-03T07:48:05.193207Z INFO app_server::worker: Worker fc9b76ef-65e2-4190-96d5-836a2270c7fc (Logs) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193222587Z 2026-11-03T07:48:05.193209Z INFO app_server::worker: Queue 'static_prompt_queue' prefetch_count=128 (override via STATIC_PROMPT_QUEUE_PREFETCH_COUNT)
app-server-1 | 2026-11-03T07:48:05.193223295Z 2026-11-03T07:48:05.193210Z INFO app_server::worker: Spawning worker f7f894e0-6cb1-4537-ab14-aa32df857436 (StaticPrompt) instance 0
app-server-1 | 2026-11-03T07:48:05.193228879Z 2026-11-03T07:48:05.193218Z INFO app_server::worker: Worker f7f894e0-6cb1-4537-ab14-aa32df857436 (StaticPrompt) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193229754Z 2026-11-03T07:48:05.193221Z INFO app_server::worker: Spawning worker c3866638-4cb5-4516-b218-5a927390631b (StaticPrompt) instance 1
app-server-1 | 2026-11-03T07:48:05.193233295Z 2026-11-03T07:48:05.193226Z INFO app_server::worker: Worker c3866638-4cb5-4516-b218-5a927390631b (StaticPrompt) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193234087Z 2026-11-03T07:48:05.193228Z INFO app_server::worker: Queue 'sp_versioning_queue' prefetch_count=128 (override via SP_VERSIONING_QUEUE_PREFETCH_COUNT)
app-server-1 | 2026-11-03T07:48:05.193234795Z 2026-11-03T07:48:05.193229Z INFO app_server::worker: Spawning worker bbced2c6-8ba0-4637-8c27-04019f86a972 (SpVersioning) instance 0
app-server-1 | 2026-11-03T07:48:05.193238087Z 2026-11-03T07:48:05.193233Z INFO app_server::worker: Worker bbced2c6-8ba0-4637-8c27-04019f86a972 (SpVersioning) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193238837Z 2026-11-03T07:48:05.193235Z INFO app_server::worker: Spawning worker 12f3474d-58c1-44d2-8452-02ea87da157a (SpVersioning) instance 1
app-server-1 | 2026-11-03T07:48:05.193243004Z 2026-11-03T07:48:05.193239Z INFO app_server::worker: Worker 12f3474d-58c1-44d2-8452-02ea87da157a (SpVersioning) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193246504Z 2026-11-03T07:48:05.193241Z INFO app_server::worker: Queue 'user_template_versioning_queue' prefetch_count=128 (override via USER_TEMPLATE_VERSIONING_QUEUE_PREFETCH_COUNT)
app-server-1 | 2026-11-03T07:48:05.193247420Z 2026-11-03T07:48:05.193242Z INFO app_server::worker: Spawning worker 2203c8b2-75cc-4e9c-b4b1-a14cee2ccfdf (UserTemplateVersioning) instance 0
app-server-1 | 2026-11-03T07:48:05.193251337Z 2026-11-03T07:48:05.193246Z INFO app_server::worker: Worker 2203c8b2-75cc-4e9c-b4b1-a14cee2ccfdf (UserTemplateVersioning) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193252129Z 2026-11-03T07:48:05.193248Z INFO app_server::worker: Spawning worker 197ef589-39da-4cdd-8ca1-17a5b6483d80 (UserTemplateVersioning) instance 1
app-server-1 | 2026-11-03T07:48:05.193255462Z 2026-11-03T07:48:05.193251Z INFO app_server::worker: Worker 197ef589-39da-4cdd-8ca1-17a5b6483d80 (UserTemplateVersioning) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193262462Z 2026-11-03T07:48:05.193253Z INFO app_server::worker: Queue 'sp_regex_extraction_queue' prefetch_count=128 (override via SP_REGEX_EXTRACTION_QUEUE_PREFETCH_COUNT)
app-server-1 | 2026-11-03T07:48:05.193263379Z 2026-11-03T07:48:05.193254Z INFO app_server::worker: Spawning worker 4a711f7a-dcb6-47bd-b710-639aa58de80d (SpRegexExtraction) instance 0
app-server-1 | 2026-11-03T07:48:05.193268295Z 2026-11-03T07:48:05.193259Z INFO app_server::worker: Worker 4a711f7a-dcb6-47bd-b710-639aa58de80d (SpRegexExtraction) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193269212Z 2026-11-03T07:48:05.193261Z INFO app_server::worker: Spawning worker ae437b3e-efa3-411d-90a7-2ee5f4bf75e6 (SpRegexExtraction) instance 1
app-server-1 | 2026-11-03T07:48:05.193277212Z 2026-11-03T07:48:05.193265Z INFO app_server::worker: Worker ae437b3e-efa3-411d-90a7-2ee5f4bf75e6 (SpRegexExtraction) connected and ready to process messages
app-server-1 | 2026-11-03T07:48:05.193353962Z 2026-11-03T07:48:05.193318Z INFO actix_server::builder: starting 2 workers
app-server-1 | 2026-11-03T07:48:05.193358212Z 2026-11-03T07:48:05.193328Z INFO actix_server::server: Tokio runtime found; starting in existing Tokio runtime
app-server-1 | 2026-11-03T07:48:05.193359462Z 2026-11-03T07:48:05.193330Z INFO actix_server::server: starting service: "actix-web-service-0.0.0.0:8002", workers: 2, listening on: 0.0.0.0:8002
frontend-1 | 2026-11-03T07:48:11.443949048Z line: '12298',
frontend-1 | 2026-11-03T07:48:11.443949590Z routine: 'ATExecDropConstraint'
frontend-1 | 2026-11-03T07:48:11.443950090Z }
frontend-1 | 2026-11-03T07:48:11.465114465Z ✓ Postgres migrations applied successfully
frontend-1 | 2026-11-03T07:48:11.465826340Z Skipped provisioning the ClickHouse Postgres role: CLICKHOUSE_POSTGRES_USER and CLICKHOUSE_POSTGRES_PASSWORD are not set (see README, SQL access to Postgres entities)
frontend-1 | 2026-11-03T07:48:11.467855673Z ✓ Postgres data initialized successfully
frontend-1 | 2026-11-03T07:48:11.480919715Z Fetching model costs...
frontend-1 | 2026-11-03T07:48:22.003069678Z Continuing without model costs data...
frontend-1 | 2026-11-03T07:48:22.003137262Z Applying ClickHouse schema. This may take a while...
frontend-1 | 2026-11-03T07:48:22.003154137Z Failed to initialize model costs: [TypeError: fetch failed] {
frontend-1 | 2026-11-03T07:48:22.003156845Z [cause]: Error [ConnectTimeoutError]: Connect Timeout Error (attempted addresses: 185.199.111.133:443, timeout: 10000ms)
frontend-1 | 2026-11-03T07:48:22.003158470Z at ignore-listed frames {
frontend-1 | 2026-11-03T07:48:22.003159762Z code: 'UND_ERR_CONNECT_TIMEOUT'
frontend-1 | 2026-11-03T07:48:22.003178845Z }
frontend-1 | 2026-11-03T07:48:22.003180595Z }
frontend-1 | 2026-11-03T07:48:22.017929845Z Turning off the ClickHouse `pg` database: CLICKHOUSE_POSTGRES_USER and CLICKHOUSE_POSTGRES_PASSWORD are not set (see README, SQL access to Postgres entities)
frontend-1 | 2026-11-03T07:48:28.964836000Z clickhouse-migrations : The migration(s) 1_squashed.sql, 2_labels-to-tags.sql, 3_eval-scores-span-tags.sql, 4_views.sql, 5_trace-summaries.sql, 6_trace-summary-views.sql, 7_versioned-datasets.sql, 8_eval_to_dataset_link.sql, 9_success_status.sql, 10_trace_browser_session.sql, 11_tags_re_index.sql, 12_events_source.sql, 13_traces_replacing.sql, 14_signal_runs.sql, 15_span_names.sql, 16_logs.sql, 17_signal_runs_fix.sql, 18_remove_span_ngram_idx.sql, 19_denorm_evals.sql, 20_span_events_column.sql, 21_signal_events_update.sql, 22_events_to_clusters.sql, 23_new_evals_datapoints_columns.sql, 24_evals_rmt_final.sql, 25_clusters.sql, 26_root_span_input_output.sql, 27_notification_logs.sql, 28_indexed_clusters.sql, 29_events_to_clusters_rmt.sql, 30_hotfix_clusters_view.sql, 31_signal_runs_mode.sql, 32_evaluation_datapoints_order_key.sql, 33_spans_new_order_with_projection.sql, 34_hide_l0_clusters.sql, 35_signal_messages_ttl.sql, 36_trace_tags.sql, 37_signal_runs_steps_processed.sql, 38_notifications.sql, 39_signal_events_severity.sql, 40_all_clusters_events_views.sql, 41_evaluation_datapoints_view_top_span_id.sql, 42_labeling_queue_items.sql, 43_llm_messages.sql, 44_spans_input_new_message_indices.sql, 45_drop_llm_messages_dict.sql, 46_deduped_content_project_scope.sql, 47_clusters_v0_drop_centroid.sql, 48_clusters_centroid_dim_768.sql, 49_trace_cache_reasoning_tokens.sql, 50_drop_events_tags.sql, 51_signal_token_columns.sql, 52_clusters_centroid_at_naming.sql, 53_traces_v0_start_time_params.sql, 54_traces_agg.sql, 55_evaluation_datapoints_v0_traces_agg.sql, 56_system_prompt_versions.sql, 57_signal_event_summaries.sql, 58_drop_unused_tables.sql, 59_signal_runs_stages.sql, 60_clusters_size_at_naming.sql, 61_spans_token_details.sql, 62_signal_version.sql, 63_evaluation_datapoints_v0_pushdown.sql, 64_unique_content.sql, 65_trace_signal_clusters.sql, 66_pii_dual_storage.sql, 67_user_template_versions.sql, 68_clusters_parent_id_index.sql, 69_signal_run_credit_applied.sql, 70_spans_original_size.sql, 71_clusters_replacing_deleted.sql, 72_trace_agent_versions.sql, 73_pg_entity_views.sql, 74_tool_definition_versions.sql, 75_signal_events_named_cluster_details.sql, 76_agent_prompts.sql, 77_drop_tool_definition_versions.sql, 78_pg_dashboard_views.sql, 79_trace_release.sql, 80_task_id.sql was successfully applied!
frontend-1 | 2026-11-03T07:48:29.011571166Z ✓ ClickHouse schema applied successfully
frontend-1 | 2026-11-03T07:48:29.011631250Z CLICKHOUSE_RO_USER equals CLICKHOUSE_USER; skipping source-privilege revoke. Use a dedicated read-only ClickHouse user in production.
frontend-1 | 2026-11-03T07:48:29.104238041Z 2026-11-03T07:48:29.103Z WARN [Better Auth]: [better-auth] Warning: your BETTER_AUTH_SECRET appears low-entropy. Use a randomly generated secret for production.
frontend-1 | 2026-11-03T07:48:29.105058875Z [traces_agg_backfill_lock] REDIS_URL is not set, so there is no cross-replica lock. If you run MORE THAN ONE frontend replica, set REDIS_URL or run a single replica for the first boot — concurrent replicas would duplicate this backfill's writes.
frontend-1 | 2026-11-03T07:48:29.113762250Z No projects need default signals, skipping seeding
frontend-1 | 2026-11-03T07:48:29.118237333Z [traces-agg-backfill] traces_replacing is empty; nothing to migrate
frontend-1 | 2026-11-03T07:48:29.118420541Z [signal_clusters_backfill_lock] REDIS_URL is not set, so there is no cross-replica lock. If you run MORE THAN ONE frontend replica, set REDIS_URL or run a single replica for the first boot — concurrent replicas would duplicate this backfill's writes.
frontend-1 | 2026-11-03T07:48:29.125170541Z Found 3 Quickwit index configuration(s)
frontend-1 | 2026-11-03T07:48:29.139375541Z Found 2 existing Quickwit index(es)
frontend-1 | 2026-11-03T07:48:29.139980708Z [signal_clusters_rebuild_lock] REDIS_URL is not set, so there is no cross-replica lock. If you run MORE THAN ONE frontend replica, set REDIS_URL or run a single replica for the first boot — concurrent replicas would duplicate this backfill's writes.
frontend-1 | 2026-11-03T07:48:29.156075250Z Creating index "events"...
frontend-1 | 2026-11-03T07:48:29.167294291Z ✓ Index "events" created successfully
frontend-1 | 2026-11-03T07:48:29.169260791Z Creating index "signal_events"...
frontend-1 | 2026-11-03T07:48:29.181608875Z [signal-clusters-rebuild] copied 0 clusters into signal_event_clusters_v2
frontend-1 | 2026-11-03T07:48:29.188117041Z [signal-clusters-rebuild] delta: re-copied 0 changed clusters, tombstoned 0 deleted ones
frontend-1 | 2026-11-03T07:48:29.192464208Z [signal-clusters-rebuild] swapped; restart the app-server to start clustering on the new table
frontend-1 | 2026-11-03T07:48:29.196449000Z [signal-clusters-rebuild] delta2: re-copied 0 clusters written during the swap; the old table is now old_unpartitioned_signal_event_clusters
frontend-1 | 2026-11-03T07:48:31.173172709Z ✓ Index "signal_events" created successfully
frontend-1 | 2026-11-03T07:48:31.178344667Z Creating index "spans_v2"...
frontend-1 | 2026-11-03T07:48:33.170761793Z ✓ Index "spans_v2" created successfully
frontend-1 | 2026-11-03T07:48:33.171061002Z ✓ Quickwit indexes initialized successfully
frontend-1 | 2026-11-03T07:48:33.321607127Z 2026-11-03T07:48:33.321Z WARN [Better Auth]: [better-auth] Warning: your BETTER_AUTH_SECRET appears low-entropy. Use a randomly generated secret for production.
frontend-1 | 2026-11-03T07:48:33.479947419Z 2026-11-03T07:48:33.479Z WARN [Better Auth]: [better-auth] Warning: your BETTER_AUTH_SECRET appears low-entropy. Use a randomly generated secret for production.