Gigapipe

Gigapipe routes logs, metrics, traces, and continuous profiles into ClickHouse through a Go binary that implements the native ingestion and query protocols of Loki, Prometheus, Tempo, and Pyroscope. Originally developed under the project name qryn, the platform transpiles LogQL, PromQL, and TraceQL queries into vectorized ClickHouse SQL in real time. This architecture allows engineering teams to preserve existing Grafana dashboards while storing all telemetry in a shared analytical database.

Why Gigapipe instead of Datadog

Datadog charges per host, per metric series, and per gigabyte of logs, forcing teams to drop debug logs or sample traces to control spend. Gigapipe eliminates volume-based billing by directing all four telemetry signals into a self-hosted ClickHouse cluster. ClickHouse executes queries using columnar tables and SIMD vectorization, maintaining fast response times across billions of records. Columnar compression routinely reduces log storage five- to ten-fold, enabling long retention periods that would be cost-prohibitive on managed SaaS.

The trade-off is operational ownership. Datadog delivers a managed SaaS suite with automated anomaly detection, synthetic browser checks, and turnkey alerting across cloud providers. Gigapipe provides the storage and query translation engine, leaving dashboards to Grafana and alerts to Alertmanager. Teams migrating from Datadog can redirect Datadog agents or OpenTelemetry collectors to Gigapipe without re-instrumenting code, exchanging SaaS subscription costs for ClickHouse cluster maintenance.

What it does well

• Drop-in Grafana support: Emulates native endpoints for Prometheus, Loki, Tempo, and Pyroscope, connecting to standard Grafana datasources without custom plugins.
• Multi-protocol ingestion: Ingests OpenTelemetry (OTLP), Prometheus remote write, Loki push, Zipkin, InfluxDB, and Datadog agent formats without intermediate proxies.
• Vectorized query engine: Translates LogQL, PromQL, and TraceQL into ClickHouse SQL, executing aggregations across disk partitions with high throughput.
• Continuous profiling: Ingests Pyroscope profiles and serves Graphviz DOT graphs with configurable node limits for rapid call-stack bottleneck analysis.
• Trace-derived metrics: Computes latency distributions and error rates dynamically from span data via TraceQL metrics endpoints without duplicate counters.