quickwit-1 | 2026-11-02T03:29:05.577120888Z 2026-11-02T03:29:05.577Z INFO quickwit_config::node_config: ingestion shard throughput limit: 5.2 MB
quickwit-1 | 2026-11-02T03:29:05.577563471Z 2026-11-02T03:29:05.577Z WARN quickwit_config::node_config::serialize: cluster ID is not set, falling back to default value `quickwit-default-cluster`
quickwit-1 | 2026-11-02T03:29:05.577566346Z 2026-11-02T03:29:05.577Z WARN quickwit_config::node_config::serialize: peer seeds are empty
quickwit-1 | 2026-11-02T03:29:05.577569305Z 2026-11-02T03:29:05.577Z INFO quickwit_cli: loaded node config config_uri=file:///quickwit/config/quickwit.yaml config=NodeConfig { cluster_id: "quickwit-default-cluster", node_id: NodeId("4bd715b19413"), availability_zone: None, enabled_services: {Metastore, Indexer, Janitor, Searcher, ControlPlane}, gossip_listen_addr: 0.0.0.0:7280, grpc_listen_addr: 0.0.0.0:7281, gossip_advertise_addr: 172.18.0.2:7280, grpc_advertise_addr: 172.18.0.2:7281, gossip_interval: 1s, peer_seeds: [], data_dir_path: "/quickwit/qwdata", metastore_uri: Uri { uri: "file:///quickwit/qwdata/indexes#polling_interval=30s" }, metastore_read_replica_uri: None, default_index_root_uri: Uri { uri: "file:///quickwit/qwdata/indexes" }, rest_config: RestConfig { listen_addr: 0.0.0.0:7280, cors_allow_origins: [], extra_headers: {}, tls_config: None, max_connection_age: None, max_connection_age_grace: None }, health_config: None, grpc_config: GrpcConfig { max_message_size: 20.0 MiB (20971520 bytes), tls_config: None, keep_alive: None, max_connection_age: None, max_connection_age_grace: None }, storage_configs: StorageConfigs([]), metastore_configs: MetastoreConfigs([]), indexer_config: IndexerConfig { split_store_max_num_bytes: 100.0 GiB (107374182400 bytes), split_store_max_num_splits: 1000, max_concurrent_split_uploads: 12, max_merge_write_throughput: None, merge_concurrency: 1, enable_otlp_endpoint: true, enable_cooperative_indexing: false, cpu_capacity: CpuCapacity(2000), parquet_merge_use_streaming_engine: false }, searcher_config: SearcherConfig { aggregation_memory_limit: 476.8 MiB (500000000 bytes), aggregation_bucket_limit: 65000, fast_field_cache: CacheConfig { capacity: Some(953.7 MiB (1000000000 bytes)), policy: None, virtual_caches: [] }, split_footer_cache: CacheConfig { capacity: Some(476.8 MiB (500000000 bytes)), policy: None, virtual_caches: [] }, partial_request_cache: CacheConfig { capacity: Some(61.0 MiB (64000000 bytes)), policy: None, virtual_caches: [] }, predicate_cache: CacheConfig { capacity: Some(244.1 MiB (256000000 bytes)), policy: None, virtual_caches: [] }, max_num_concurrent_split_searches: 100, max_splits_per_search: None, _max_num_concurrent_split_streams: None, split_cache: None, request_timeout_secs: 30, leaf_request_timeout_secs: 30, storage_timeout_policy: None, use_metastore_read_replica: false, warmup_memory_budget: 93.1 GiB (100000000000 bytes), warmup_single_split_initial_allocation: 286.1 MiB (300000000 bytes), lambda: None }, ingest_api_config: IngestApiConfig { max_queue_memory_usage: 2.0 GiB (2147483648 bytes), max_queue_disk_usage: 4.0 GiB (4294967296 bytes), replication_factor: 1, content_length_limit: 10.0 MiB (10485760 bytes), shard_throughput_limit: 5.0 MiB (5242880 bytes), shard_burst_limit: 50.0 MiB (52428800 bytes), shard_scale_up_factor: 1.5, grpc_compression_algorithm: None }, jaeger_config: JaegerConfig { enable_endpoint: true, lookback_period_hours: 72, max_trace_duration_secs: 3600, max_fetch_spans: 10000 }, compactor_config: CompactorConfig { max_concurrent_merge_executions: 1, pipeline_slots_per_merge_execution: 2, max_concurrent_split_uploads: 12, max_merge_write_throughput: None }, enable_standalone_compactors: false }
quickwit-1 | 2026-11-02T03:29:05.577580930Z 2026-11-02T03:29:05.577Z INFO quickwit_common::env: using environment variable `QW_DISABLE_TOKIO_LIFO_SLOT` default value default_value=true
quickwit-1 | 2026-11-02T03:29:05.577848096Z 2026-11-02T03:29:05.577Z INFO quickwit_cluster::cluster: joining cluster cluster_id=quickwit-default-cluster node_id=4bd715b19413 generation_id=1793590145577794430 enabled_services={Metastore, Indexer, Janitor, Searcher, ControlPlane} gossip_listen_addr=0.0.0.0:7280 gossip_advertise_addr=172.18.0.2:7280 grpc_advertise_addr=172.18.0.2:7281 peer_seed_addrs=
quickwit-1 | 2026-11-02T03:29:05.578012221Z 2026-11-02T03:29:05.577Z INFO quickwit_serve::metastore: starting local metastore service metastore_kind="primary"
quickwit-1 | 2026-11-02T03:29:05.578409013Z 2026-11-02T03:29:05.578Z INFO quickwit_cluster::change: node `4bd715b19413` has joined the cluster node_id=4bd715b19413 generation_id=1793590145577794430
quickwit-1 | 2026-11-02T03:29:05.580041721Z 2026-11-02T03:29:05.580Z INFO quickwit_control_plane::control_plane: starting control plane
quickwit-1 | 2026-11-02T03:29:05.580995513Z 2026-11-02T03:29:05.580Z INFO load_from_metastore: quickwit_common::env: using environment variable `QW_ENABLE_INGEST_V2` default value default_value=true
quickwit-1 | 2026-11-02T03:29:05.581000221Z 2026-11-02T03:29:05.580Z INFO load_from_metastore: quickwit_control_plane::model: adding ingest v2 source to 0 indexes
quickwit-1 | 2026-11-02T03:29:05.581001513Z 2026-11-02T03:29:05.580Z INFO load_from_metastore: quickwit_control_plane::model: added ingest v2 source to 0 indexes in 0ms
quickwit-1 | 2026-11-02T03:29:05.581002513Z 2026-11-02T03:29:05.580Z INFO load_from_metastore: quickwit_control_plane::model: enabling 0 ingest v2 sources
quickwit-1 | 2026-11-02T03:29:05.581003513Z 2026-11-02T03:29:05.580Z INFO load_from_metastore: quickwit_control_plane::model: enabled 0 ingest v2 sources in 0ms
quickwit-1 | 2026-11-02T03:29:05.581006471Z 2026-11-02T03:29:05.580Z INFO load_from_metastore: quickwit_control_plane::model: synced control plane model with metastore in 0ms (0 indexes, 0 sources, 0 shards)
quickwit-1 | 2026-11-02T03:29:05.581054346Z 2026-11-02T03:29:05.581Z INFO quickwit_common::env: using environment variable `QW_DISABLE_INGEST_V1` default value default_value=false
quickwit-1 | 2026-11-02T03:29:05.581057305Z 2026-11-02T03:29:05.581Z WARN quickwit_control_plane::indexing_scheduler: no ready indexer available, falling back to retiring indexers for shard draining
quickwit-1 | 2026-11-02T03:29:05.581058430Z 2026-11-02T03:29:05.581Z INFO quickwit_serve: control plane is ready
quickwit-1 | 2026-11-02T03:29:05.581513596Z 2026-11-02T03:29:05.581Z INFO quickwit_ingest::ingest_api_service: Ingest API partition id ingest_partition_id=ingest_partition_01M6D4VCHDY232WG26CWEC71YD
quickwit-1 | 2026-11-02T03:29:05.582005680Z 2026-11-02T03:29:05.581Z INFO quickwit_indexing: starting indexer service
quickwit-1 | 2026-11-02T03:29:05.582415513Z 2026-11-02T03:29:05.582Z INFO quickwit_ingest::ingest_v2::state: opening WAL located at `/quickwit/qwdata/wal`
quickwit-1 | 2026-11-02T03:29:05.583197805Z 2026-11-02T03:29:05.582Z INFO quickwit_ingest::ingest_v2::ingester: subscribing ingester to shard positions updates
quickwit-1 | 2026-11-02T03:29:05.583201680Z 2026-11-02T03:29:05.582Z INFO quickwit_ingest::ingest_v2::state: opened WAL successfully in 0ms
quickwit-1 | 2026-11-02T03:29:05.583202930Z 2026-11-02T03:29:05.582Z INFO ingester.reset_shards: quickwit_ingest::ingest_v2::ingester: resetting shards
quickwit-1 | 2026-11-02T03:29:05.583204180Z 2026-11-02T03:29:05.582Z INFO quickwit_indexing::models::shard_positions: initialized shard positions in 0ms (0 keys)
quickwit-1 | 2026-11-02T03:29:05.583205430Z 2026-11-02T03:29:05.582Z INFO quickwit_control_plane::ingest::ingest_controller: received advise reset shards request from `4bd715b19413`
quickwit-1 | 2026-11-02T03:29:05.583206638Z 2026-11-02T03:29:05.582Z INFO ingester.reset_shards: quickwit_ingest::ingest_v2::state: resetting shards
quickwit-1 | 2026-11-02T03:29:05.583208055Z 2026-11-02T03:29:05.582Z INFO ingester.reset_shards: quickwit_ingest::ingest_v2::ingester: deleted 0 and truncated 0 shard(s) in 0ms
quickwit-1 | 2026-11-02T03:29:05.583211846Z 2026-11-02T03:29:05.582Z INFO quickwit_config::index_config::serialize: index config does not specify `index_uri`, falling back to default value index_id=otel-logs-v0_9 index_uri=file:///quickwit/qwdata/indexes/otel-logs-v0_9
quickwit-1 | 2026-11-02T03:29:05.583213638Z 2026-11-02T03:29:05.583Z INFO quickwit_config::index_config::serialize: index config does not specify `index_uri`, falling back to default value index_id=otel-traces-v0_9 index_uri=file:///quickwit/qwdata/indexes/otel-traces-v0_9
quickwit-1 | 2026-11-02T03:29:05.587084346Z 2026-11-02T03:29:05.587Z INFO quickwit_control_plane::control_plane: created index index_uid=otel-logs-v0_9:01M6D4VCHFM5DNRP4N92E9R7RK
quickwit-1 | 2026-11-02T03:29:07.583480639Z 2026-11-02T03:29:07.582Z WARN quickwit_control_plane::indexing_scheduler: no ready indexer available, falling back to retiring indexers for shard draining
quickwit-1 | 2026-11-02T03:29:07.583573556Z 2026-11-02T03:29:07.582Z WARN quickwit_control_plane::indexing_scheduler: no indexing capacity available, cannot schedule an indexing plan
quickwit-1 | 2026-11-02T03:29:07.595089889Z 2026-11-02T03:29:07.594Z INFO quickwit_control_plane::control_plane: created index index_uid=otel-traces-v0_9:01M6D4VEFZKYF8DMV9CSR1RVGX
quickwit-1 | 2026-11-02T03:29:09.585244875Z 2026-11-02T03:29:09.585Z WARN quickwit_control_plane::indexing_scheduler: no ready indexer available, falling back to retiring indexers for shard draining
quickwit-1 | 2026-11-02T03:29:09.585280625Z 2026-11-02T03:29:09.585Z WARN quickwit_control_plane::indexing_scheduler: no indexing capacity available, cannot schedule an indexing plan
quickwit-1 | 2026-11-02T03:29:09.586524250Z 2026-11-02T03:29:09.585Z INFO quickwit_serve: configured search metastore client metastore_kind="primary"
quickwit-1 | 2026-11-02T03:29:09.586530209Z 2026-11-02T03:29:09.585Z INFO quickwit_common::env: using environment variable `QW_DISABLE_DELETE_TASK_SERVICE` default value default_value=false
quickwit-1 | 2026-11-02T03:29:09.586532000Z 2026-11-02T03:29:09.585Z INFO quickwit_janitor: starting janitor service
quickwit-1 | 2026-11-02T03:29:09.588642209Z 2026-11-02T03:29:09.586Z INFO quickwit_janitor::actors::retention_policy_executor: loaded 2 indexes from the metastore
quickwit-1 | 2026-11-02T03:29:09.588647417Z 2026-11-02T03:29:09.587Z INFO quickwit_serve: configured metastore readiness dependency metastore_kind="primary"
quickwit-1 | 2026-11-02T03:29:09.588652917Z 2026-11-02T03:29:09.588Z INFO quickwit_serve::grpc: starting gRPC server listening on 0.0.0.0:7281 enabled_grpc_services={"control-plane", "developer", "health", "indexing", "ingest-api", "ingest-router", "ingester", "jaeger", "jaeger-v2", "metastore", "otlp-logs", "otlp-traces", "reflection", "search"} grpc_listen_addr=0.0.0.0:7281
quickwit-1 | 2026-11-02T03:29:09.588667000Z 2026-11-02T03:29:09.588Z INFO quickwit_serve: gRPC server is ready
quickwit-1 | 2026-11-02T03:29:09.588667875Z 2026-11-02T03:29:09.588Z INFO quickwit_common::env: using environment variable `QW_ENABLE_CORS_DEBUG` default value default_value=false
quickwit-1 | 2026-11-02T03:29:09.588668750Z 2026-11-02T03:29:09.588Z INFO quickwit_serve::rest: starting REST server listening on 0.0.0.0:7280 listen_addr=0.0.0.0:7280
quickwit-1 | 2026-11-02T03:29:09.588669667Z 2026-11-02T03:29:09.588Z INFO quickwit_serve: REST server is ready
quickwit-1 | 2026-11-02T03:29:09.588670334Z 2026-11-02T03:29:09.588Z INFO quickwit_janitor::actors::garbage_collector: loaded 2 indexes from the metastore
quickwit-1 | 2026-11-02T03:29:09.588671084Z 2026-11-02T03:29:09.588Z INFO quickwit_common::env: using environment variable `QW_SPLIT_DELETION_GRACE_PERIOD_SECS` default value default_value=1920
quickwit-1 | 2026-11-02T03:29:09.588671959Z 2026-11-02T03:29:09.588Z INFO quickwit_janitor::actors::delete_task_pipeline: spawning delete tasks pipeline index_uid=otel-logs-v0_9:01M6D4VCHFM5DNRP4N92E9R7RK root_dir=/quickwit/qwdata/delete_task_service
quickwit-1 | 2026-11-02T03:29:09.588798000Z 2026-11-02T03:29:09.588Z INFO quickwit_janitor::actors::delete_task_pipeline: spawning delete tasks pipeline index_uid=otel-traces-v0_9:01M6D4VEFZKYF8DMV9CSR1RVGX root_dir=/quickwit/qwdata/delete_task_service
quickwit-1 | 2026-11-02T03:29:10.580197501Z 2026-11-02T03:29:10.579Z INFO quickwit_cluster::change: node `4bd715b19413` has transitioned to ready state node_id=4bd715b19413 generation_id=1793590145577794430
quickwit-1 | 2026-11-02T03:29:10.580228001Z 2026-11-02T03:29:10.580Z INFO quickwit_serve: adding node `4bd715b19413` to searcher pool node_id=4bd715b19413 generation_id=1793590145577794430
quickwit-1 | 2026-11-02T03:29:10.580233126Z 2026-11-02T03:29:10.580Z INFO quickwit_serve: adding node `4bd715b19413` with ingester status `ready` to indexer pool node_id=4bd715b19413 generation_id=1793590145577794430
quickwit-1 | 2026-11-02T03:29:10.580249043Z 2026-11-02T03:29:10.580Z INFO quickwit_serve: adding node `4bd715b19413` with ingester status `ready` to ingester pool node_id=4bd715b19413 generation_id=1793590145577794430
quickwit-1 | 2026-11-02T03:29:10.584146043Z 2026-11-02T03:29:10.583Z INFO quickwit_indexing::actors::indexing_service: fetched 0 splits candidates for merge
quickwit-1 | 2026-11-02T03:29:10.584157543Z 2026-11-02T03:29:10.583Z INFO spawn_merge_pipeline{index_uid=otel-logs-v0_9:01M6D4VCHFM5DNRP4N92E9R7RK generation=0}: quickwit_indexing::actors::merge_pipeline: spawning merge pipeline index_uid=otel-logs-v0_9:01M6D4VCHFM5DNRP4N92E9R7RK source_id=_ingest-api-source root_dir=/quickwit/qwdata/indexing/otel-logs-v0_9%01M6D4VCHFM5DNRP4N92E9R7RK%_ingest-api-source%01M6D4VHDPS4RE0SRPQYS7EC3D%Rtw8vw merge_policy=StableLogMergePolicy { config: StableLogMergePolicyConfig { min_level_num_docs: 100000, merge_factor: 10, max_merge_factor: 12, maturation_period: 172800s }, split_num_docs_target: 10000000 }
quickwit-1 | 2026-11-02T03:29:10.584162709Z 2026-11-02T03:29:10.583Z INFO spawn_pipeline{index=otel-logs-v0_9 gen=0}: quickwit_indexing::actors::indexing_pipeline: spawning indexing pipeline index_id="otel-logs-v0_9" source_id="_ingest-api-source" pipeline_uid=01M6D4VHDPS4RE0SRPQYS7EC3D root_dir=/quickwit/qwdata/indexing/otel-logs-v0_9%01M6D4VCHFM5DNRP4N92E9R7RK%_ingest-api-source%01M6D4VHDPS4RE0SRPQYS7EC3D%Rtw8vw
quickwit-1 | 2026-11-02T03:29:10.584166043Z 2026-11-02T03:29:10.583Z INFO spawn_pipeline{index=otel-logs-v0_9 gen=0}: quickwit_common::env: using environment variable `QW_DISABLE_PER_INDEX_METRICS` default value default_value=false
quickwit-1 | 2026-11-02T03:29:10.584195126Z 2026-11-02T03:29:10.583Z INFO spawn_merge_pipeline{index_uid=otel-traces-v0_9:01M6D4VEFZKYF8DMV9CSR1RVGX generation=0}: quickwit_indexing::actors::merge_pipeline: spawning merge pipeline index_uid=otel-traces-v0_9:01M6D4VEFZKYF8DMV9CSR1RVGX source_id=_ingest-api-source root_dir=/quickwit/qwdata/indexing/otel-traces-v0_9%01M6D4VEFZKYF8DMV9CSR1RVGX%_ingest-api-source%01M6D4VHDP3YJDNF1ZNTTJ04SM%cL4gpL merge_policy=StableLogMergePolicy { config: StableLogMergePolicyConfig { min_level_num_docs: 100000, merge_factor: 10, max_merge_factor: 12, maturation_period: 172800s }, split_num_docs_target: 10000000 }
quickwit-1 | 2026-11-02T03:29:10.584203626Z 2026-11-02T03:29:10.583Z INFO spawn_pipeline{index=otel-traces-v0_9 gen=0}: quickwit_indexing::actors::indexing_pipeline: spawning indexing pipeline index_id="otel-traces-v0_9" source_id="_ingest-api-source" pipeline_uid=01M6D4VHDP3YJDNF1ZNTTJ04SM root_dir=/quickwit/qwdata/indexing/otel-traces-v0_9%01M6D4VEFZKYF8DMV9CSR1RVGX%_ingest-api-source%01M6D4VHDP3YJDNF1ZNTTJ04SM%cL4gpL
quickwit-1 | 2026-11-02T03:29:10.585639251Z 2026-11-02T03:29:10.585Z INFO spawn_pipeline{index=otel-logs-v0_9 gen=0}: quickwit_indexing::source::ingest_api_source: created queue successfully index_id="otel-logs-v0_9" partition_id=ingest_partition_01M6D4VCHDY232WG26CWEC71YD
quickwit-1 | 2026-11-02T03:29:10.586516418Z 2026-11-02T03:29:10.586Z INFO spawn_pipeline{index=otel-traces-v0_9 gen=0}: quickwit_indexing::source::ingest_api_source: created queue successfully index_id="otel-traces-v0_9" partition_id=ingest_partition_01M6D4VCHDY232WG26CWEC71YD
quickwit-1 | 2026-11-02T03:29:11.713711710Z 2026-11-02T03:29:11.713Z INFO http_request{otel.kind="Server" http.request.method=POST url.path=/api/v1/indexes}: quickwit_config::index_config::serialize: index config does not specify `index_uri`, falling back to default value index_id=books index_uri=file:///quickwit/qwdata/indexes/books
quickwit-1 | 2026-11-02T03:29:11.713756835Z 2026-11-02T03:29:11.713Z INFO http_request{otel.kind="Server" http.request.method=POST url.path=/api/v1/indexes}: quickwit_serve::index_api::index_resource: create-index index_id=books overwrite=false
quickwit-1 | 2026-11-02T03:29:11.727882918Z 2026-11-02T03:29:11.727Z INFO quickwit_control_plane::control_plane: created index index_uid=books:01M6D4VJH1WNADXD95ZT1TMD1C
quickwit-1 | 2026-11-02T03:29:11.728297751Z 2026-11-02T03:29:11.728Z INFO quickwit_indexing::actors::indexing_service: fetched 0 splits candidates for merge
quickwit-1 | 2026-11-02T03:29:11.728900710Z 2026-11-02T03:29:11.728Z INFO spawn_merge_pipeline{index_uid=books:01M6D4VJH1WNADXD95ZT1TMD1C generation=0}: quickwit_indexing::actors::merge_pipeline: spawning merge pipeline index_uid=books:01M6D4VJH1WNADXD95ZT1TMD1C source_id=_ingest-api-source root_dir=/quickwit/qwdata/indexing/books%01M6D4VJH1WNADXD95ZT1TMD1C%_ingest-api-source%01M6D4VJHFM1P697R3TJHVZC02%ohT6Hs merge_policy=StableLogMergePolicy { config: StableLogMergePolicyConfig { min_level_num_docs: 100000, merge_factor: 10, max_merge_factor: 12, maturation_period: 172800s }, split_num_docs_target: 10000000 }
quickwit-1 | 2026-11-02T03:29:11.728945501Z 2026-11-02T03:29:11.728Z INFO spawn_pipeline{index=books gen=0}: quickwit_indexing::actors::indexing_pipeline: spawning indexing pipeline index_id="books" source_id="_ingest-api-source" pipeline_uid=01M6D4VJHFM1P697R3TJHVZC02 root_dir=/quickwit/qwdata/indexing/books%01M6D4VJH1WNADXD95ZT1TMD1C%_ingest-api-source%01M6D4VJHFM1P697R3TJHVZC02%ohT6Hs
quickwit-1 | 2026-11-02T03:29:11.729718126Z 2026-11-02T03:29:11.729Z INFO http_request{otel.kind="Server" http.request.method=GET url.path=/api/v1/indexes/books}: quickwit_serve::index_api::index_resource: get-index-metadata index_id=books
quickwit-1 | 2026-11-02T03:29:11.730761335Z 2026-11-02T03:29:11.730Z INFO ingest.ingester.init_shards: quickwit_ingest::ingest_v2::ingester: init primary shard index_uid=books:01M6D4VJH1WNADXD95ZT1TMD1C source_id="_ingest-source" shard_id=01M6D4VJHJ0W1C5WYQCE9PE98Q
quickwit-1 | 2026-11-02T03:29:11.730772876Z 2026-11-02T03:29:11.730Z INFO ingest.ingester.init_shards: quickwit_ingest::ingest_v2::doc_mapper: inserted doc mapper `01M6D4VJH17695QAZWCX2Y42FK` into cache`
quickwit-1 | 2026-11-02T03:29:11.731867960Z 2026-11-02T03:29:11.731Z INFO spawn_pipeline{index=books gen=0}: quickwit_indexing::source::ingest_api_source: created queue successfully index_id="books" partition_id=ingest_partition_01M6D4VCHDY232WG26CWEC71YD
quickwit-1 | 2026-11-02T03:29:11.733270251Z 2026-11-02T03:29:11.733Z INFO metastore.open_shards:metastore.open_shards:metastore.file_backed.open_shards{num_subrequests=1}: quickwit_metastore::metastore::file_backed::file_backed_index::shards: opened shard index_uid=books:01M6D4VJH1WNADXD95ZT1TMD1C source_id=_ingest-source shard_id=01M6D4VJHJ0W1C5WYQCE9PE98Q leader_id=4bd715b19413 follower_id=None
quickwit-1 | 2026-11-02T03:29:11.739520126Z 2026-11-02T03:29:11.739Z INFO http_request{otel.kind="Server" http.request.method=POST url.path=/api/v1/books/ingest}:ingest.router.ingest{commit_type=3}:ingest.ingester.persist{leader_id=4bd715b19413 commit_type=3}: quickwit_common::env: using environment variable `QW_DISABLE_DOCUMENT_VALIDATION` default value default_value=false
quickwit-1 | 2026-11-02T03:29:13.728988961Z 2026-11-02T03:29:13.728Z INFO quickwit_control_plane::indexing_scheduler: QW_ENABLE_VARIABLE_SHARD_LOAD not set, defaulting to false
quickwit-1 | 2026-11-02T03:29:13.729013002Z 2026-11-02T03:29:13.728Z INFO quickwit_common::env: using environment variable `QW_DEFAULT_LOAD_PER_SHARD` default value default_value=1000
quickwit-1 | 2026-11-02T03:29:13.729954919Z 2026-11-02T03:29:13.729Z INFO quickwit_indexing::actors::indexing_service: fetched 0 splits candidates for merge
quickwit-1 | 2026-11-02T03:29:13.729969127Z 2026-11-02T03:29:13.729Z INFO spawn_merge_pipeline{index_uid=books:01M6D4VJH1WNADXD95ZT1TMD1C generation=0}: quickwit_indexing::actors::merge_pipeline: spawning merge pipeline index_uid=books:01M6D4VJH1WNADXD95ZT1TMD1C source_id=_ingest-source root_dir=/quickwit/qwdata/indexing/books%01M6D4VJH1WNADXD95ZT1TMD1C%_ingest-source%01M6D4VMG0DDGSD0RMH44Z3AAD%XcISPO merge_policy=StableLogMergePolicy { config: StableLogMergePolicyConfig { min_level_num_docs: 100000, merge_factor: 10, max_merge_factor: 12, maturation_period: 172800s }, split_num_docs_target: 10000000 }
quickwit-1 | 2026-11-02T03:29:13.729976794Z 2026-11-02T03:29:13.729Z INFO spawn_pipeline{index=books gen=0}: quickwit_indexing::actors::indexing_pipeline: spawning indexing pipeline index_id="books" source_id="_ingest-source" pipeline_uid=01M6D4VMG0DDGSD0RMH44Z3AAD root_dir=/quickwit/qwdata/indexing/books%01M6D4VJH1WNADXD95ZT1TMD1C%_ingest-source%01M6D4VMG0DDGSD0RMH44Z3AAD%XcISPO
quickwit-1 | 2026-11-02T03:29:13.730171002Z 2026-11-02T03:29:13.730Z INFO quickwit_indexing::actors::indexing_pipeline: assigning shards to indexing pipeline shard_ids={ShardId("01M6D4VJHJ0W1C5WYQCE9PE98Q")}
quickwit-1 | 2026-11-02T03:29:14.731976211Z 2026-11-02T03:29:14.731Z INFO quickwit_indexing::source::ingest: resetting indexing pipeline index_uid=books:01M6D4VJH1WNADXD95ZT1TMD1C pipeline_uid=01M6D4VMG0DDGSD0RMH44Z3AAD
quickwit-1 | 2026-11-02T03:29:14.732017628Z 2026-11-02T03:29:14.731Z INFO quickwit_indexing::source::ingest: adding shards assignment added_shards=[ShardId("01M6D4VJHJ0W1C5WYQCE9PE98Q")]
quickwit-1 | 2026-11-02T03:29:14.735729670Z 2026-11-02T03:29:14.735Z INFO ingest.ingester.open_fetch_stream: quickwit_common::env: using environment variable `QW_INGEST_BATCH_NUM_BYTES` default value default_value=1048576
quickwit-1 | 2026-11-02T03:29:15.580807378Z 2026-11-02T03:29:15.580Z INFO quickwit_cluster::grpc_gossip: no peer nodes to pull the cluster state from
quickwit-1 | 2026-11-02T03:29:15.739348628Z 2026-11-02T03:29:15.738Z INFO quickwit_common::env: using environment variable `QW_RANDOM_SPLIT_PREFIX` default value default_value=false
quickwit-1 | 2026-11-02T03:29:15.769455962Z 2026-11-02T03:29:15.769Z INFO index-doc-batches{index_id=books source_id=_ingest-source pipeline_uid=01M6D4VMG0DDGSD0RMH44Z3AAD workbench_id=01M6D4VPESEAJ2Y5YQCKMZ92DF}:publisher{split_update=SplitsUpdate { index_id: "books", new_splits: "01M6D4VPETR4JJS2NDSD8C4KF0", checkpoint_delta: Some(_ingest-source:∆(01M6D4VJHJ0W1C5WYQCE9PE98Q:(..00000000000000000001])) }}: quickwit_indexing::actors::log_publisher_impl: publish-new-splits num_splits=1 num_docs=1 split_size_bytes=6012
quickwit-1 | 2026-11-02T03:29:16.741938421Z 2026-11-02T03:29:16.740Z INFO ingest.ingester.truncate_shards:ingester.truncate_shard: quickwit_ingest::ingest_v2::state: truncated shard `books:01M6D4VJH1WNADXD95ZT1TMD1C/_ingest-source/01M6D4VJHJ0W1C5WYQCE9PE98Q` at 00000000000000000001 initiated via `indexer RPC`
quickwit-1 | 2026-11-02T03:29:16.746560671Z 2026-11-02T03:29:16.746Z INFO http_request{otel.kind="Server" http.request.method=GET url.path=/api/v1/books/search}: quickwit_serve::search_api::rest_handler: search request=SearchRequestQueryString { query: "title:Hitchhiker", aggs: None, search_fields: None, snippet_fields: None, start_timestamp: None, end_timestamp: None, max_hits: 20, start_offset: 0, format: PrettyJson, sort_by: SortBy { sort_fields: [] }, count_all: CountAll, allow_failed_splits: false }
quickwit-1 | 2026-11-02T03:29:16.749244546Z 2026-11-02T03:29:16.749Z INFO http_request{otel.kind="Server" http.request.method=GET url.path=/api/v1/books/search}:root_search: quickwit_search::root: root_search query_ast="{\"type\":\"full_text\",\"field\":\"title\",\"text\":\"Hitchhiker\",\"params\":{\"mode\":{\"type\":\"phrase_fallback_to_intersection\"}},\"lenient\":false}" agg="" start_ts=0..0 count_required="COUNT_ALL" num_docs=1 num_splits=1
quickwit-1 | 2026-11-02T03:29:16.749342212Z 2026-11-02T03:29:16.749Z INFO http_request{otel.kind="Server" http.request.method=GET url.path=/api/v1/books/search}:root_search: quickwit_common::env: using environment variable `QW_DISABLE_LOAD_ESTIMATION` default value default_value=false
quickwit-1 | 2026-11-02T03:29:16.753254837Z 2026-11-02T03:29:16.753Z INFO http_request{otel.kind="Server" http.request.method=DELETE url.path=/api/v1/indexes/books}: quickwit_serve::index_api::index_resource: delete-index index_id=books dry_run=false
quickwit-1 | 2026-11-02T03:29:16.756632712Z 2026-11-02T03:29:16.756Z INFO http_request{otel.kind="Server" http.request.method=DELETE url.path=/api/v1/indexes/books}:metastore.delete_splits:metastore.control_plane.delete_splits{index_uid=books:01M6D4VJH1WNADXD95ZT1TMD1C}:metastore.delete_splits:metastore.delete_splits:metastore.file_backed.delete_splits{index_uid=books:01M6D4VJH1WNADXD95ZT1TMD1C}: quickwit_metastore::metastore::file_backed::file_backed_index: deleted 0 splits from index index_id=books
quickwit-1 | 2026-11-02T03:29:16.762316254Z 2026-11-02T03:29:16.762Z INFO quickwit_control_plane::ingest::ingest_controller: sync_with_ingester ingester=4bd715b19413
quickwit-1 | 2026-11-02T03:29:16.762331129Z 2026-11-02T03:29:16.762Z INFO quickwit_control_plane::ingest::ingest_controller: retain shards ingester ingester_id=4bd715b19413
quickwit-1 | 2026-11-02T03:29:16.762342629Z 2026-11-02T03:29:16.762Z INFO quickwit_control_plane::control_plane: deleted index index_uid=books:01M6D4VJH1WNADXD95ZT1TMD1C
quickwit-1 | 2026-11-02T03:29:16.762651962Z 2026-11-02T03:29:16.762Z INFO ingest.ingester.retain_shards: quickwit_ingest::ingest_v2::ingester: removing queues queues={"books:01M6D4VJH1WNADXD95ZT1TMD1C/_ingest-source/01M6D4VJHJ0W1C5WYQCE9PE98Q"}
quickwit-1 | 2026-11-02T03:29:16.765718754Z 2026-11-02T03:29:16.765Z INFO ingest.ingester.retain_shards:ingester.delete_shard: quickwit_ingest::ingest_v2::state: deleted shard `books:01M6D4VJH1WNADXD95ZT1TMD1C/_ingest-source/01M6D4VJHJ0W1C5WYQCE9PE98Q` initiated via `control-plane-retain-shards-rpc`
quickwit-1 | 2026-11-02T03:29:16.765730421Z 2026-11-02T03:29:16.765Z INFO ingest.ingester.retain_shards:ingester.delete_shard: quickwit_ingest::ingest_v2::state: evicted doc mapper `01M6D4VJH17695QAZWCX2Y42FK` from cache`
quickwit-1 | 2026-11-02T03:29:16.765732921Z 2026-11-02T03:29:16.765Z WARN quickwit_ingest::ingest_v2::fetch: fetch stream ended before reaching end of shard client_id=indexer/4bd715b19413/books:01M6D4VJH1WNADXD95ZT1TMD1C/_ingest-source/01M6D4VMG0DDGSD0RMH44Z3AAD index_uid=books:01M6D4VJH1WNADXD95ZT1TMD1C source_id=_ingest-source shard_id=01M6D4VJHJ0W1C5WYQCE9PE98Q
quickwit-1 | 2026-11-02T03:29:16.765735629Z 2026-11-02T03:29:16.765Z ERROR quickwit_ingest::ingest_v2::fetch: failed to fetch records from ingester `4bd715b19413`: closing fetch stream client_id=indexer/4bd715b19413/books:01M6D4VJH1WNADXD95ZT1TMD1C/_ingest-source/01M6D4VMG0DDGSD0RMH44Z3AAD index_uid=books:01M6D4VJH1WNADXD95ZT1TMD1C source_id=_ingest-source shard_id=01M6D4VJHJ0W1C5WYQCE9PE98Q error=internal error: fetch stream ended before reaching end of shard
quickwit-1 | 2026-11-02T03:29:17.739329463Z 2026-11-02T03:29:17.738Z INFO quickwit_indexing::actors::indexing_service: shutdown indexing pipelines pipeline_uids=[Pipeline(01M6D4VJHFM1P697R3TJHVZC02), Pipeline(01M6D4VMG0DDGSD0RMH44Z3AAD)]
quickwit-1 | 2026-11-02T03:29:17.739414171Z 2026-11-02T03:29:17.738Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="IndexingPipeline-divine-Whgk" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739420879Z 2026-11-02T03:29:17.738Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="IndexUploader-dark-k4DN" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739457088Z 2026-11-02T03:29:17.738Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="quickwit_indexing::actors::doc_processor::DocProcessor-nameless-sk1v" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739463171Z 2026-11-02T03:29:17.738Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="quickwit_indexing::actors::sequencer::Sequencer<quickwit_indexing::actors::publisher::Publisher>-winter-C1HI" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739466921Z 2026-11-02T03:29:17.738Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="Publisher-shy-RnE6" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739469838Z 2026-11-02T03:29:17.739Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="IndexingPipeline-silent-nees" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739472754Z 2026-11-02T03:29:17.739Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="quickwit_indexing::actors::doc_processor::DocProcessor-aged-KoBc" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739476088Z 2026-11-02T03:29:17.739Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="IndexUploader-patient-SqKI" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739480129Z 2026-11-02T03:29:17.739Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="quickwit_indexing::actors::sequencer::Sequencer<quickwit_indexing::actors::publisher::Publisher>-crimson-BWl9" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739483671Z 2026-11-02T03:29:17.739Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="Indexer-autumn-iH2N" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739486463Z 2026-11-02T03:29:17.739Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="Publisher-lively-tFmd" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739489296Z 2026-11-02T03:29:17.739Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="Indexer-rough-10ej" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739496504Z 2026-11-02T03:29:17.739Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="quickwit_indexing::actors::index_serializer::IndexSerializer-divine-VAjU" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739499838Z 2026-11-02T03:29:17.739Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="quickwit_indexing::actors::index_serializer::IndexSerializer-polished-7GAl" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739502963Z 2026-11-02T03:29:17.739Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="Packager-old-4JB3" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.739505796Z 2026-11-02T03:29:17.739Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="Packager-spring-TCgX" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.741068463Z 2026-11-02T03:29:17.740Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="SourceActor-lingering-wHzN" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.744011546Z 2026-11-02T03:29:17.743Z INFO quickwit_actors::spawn_builder: actor-exit actor_id="SourceActor-fragrant-dQJB" phase=running exit_status=Killed
quickwit-1 | 2026-11-02T03:29:17.746150588Z 2026-11-02T03:29:17.746Z INFO quickwit_indexing::actors::indexing_service: deleted queue successfully index_id=books partition_id="ingest_partition_01M6D4VCHDY232WG26CWEC71YD"
