Swirl Search

Sid Probstein built Swirl Search after serving as CTO of Attivio and VP of Technology at FAST Search & Transfer, drawing on decades in enterprise discovery. Frustrated by the operational overhead and security risks of replicating corporate records into central indexes, he designed an engine that queries data where it lives. Today, Swirl coordinates live queries across external APIs to return cited answers without duplicating documents into separate storage.

Why Swirl Search instead of ElasticSearch

Deploying ElasticSearch for workplace discovery demands constant ETL maintenance. Platform teams must build extractors, configure message queues, and write synchronization jobs to copy documentation from tools like GitHub, Jira, and Confluence into central indices. Mirroring permissions is equally fragile, as stale access control lists risk exposing restricted files. Swirl Search replaces that pipeline with zero-copy federation. Instead of syncing data, developers configure SearchProviders that dispatch queries directly to native APIs using the user's active credentials, maintaining source-level security without duplicated permissions tables.

The tradeoff centers on latency and architectural purpose. ElasticSearch indexes documents for millisecond retrieval across billions of records, making it indispensable for primary application backends and log analysis. Swirl Search cannot serve as a primary transactional index; its response times depend on the slowest external API, typically taking several seconds. Compared to commercial platforms like Coveo or AWS Kendra, Swirl Search gives DevOps teams an open alternative that eliminates ingestion pipelines and recurring per-seat fees, provided operations can accept federated response times.

What it does well

• Zero-copy federated queries execute against source APIs in parallel, removing the need for ETL extractors or vector database synchronization.
• Authentication pass-through forwards user credentials to upstream services like Microsoft 365 and GitHub, enforcing native permissions without maintaining sync tables.
• Extensible Python processors let developers subclass SearchProvider, QueryProcessor, and ResultProcessor to integrate internal endpoints and custom transforms.
• Multi-source result ranking blends heterogeneous JSON payloads into a unified list using spaCy embeddings and NLTK linguistic processing.