Quickwit 0.7: Elasticsearch API compatibility and 30% performance gains

Today, we are proud to announce the release of Quickwit 0.7, which boasts performance improvements, extended Elasticsearch API compatibility, and better integration with key observability tools such as Grafana and Jaeger.

Over the past few months, we have been delighted to witness the users of the nightly version of Quickwit deploy clusters with hundreds of nodes ingesting hundreds of terabytes of data daily, experience massive performance gains, and enjoy additional cost savings. We are now very excited to put it in the hands of all of our users.

This gives us great confidence in our ability to deliver a robust OSS alternative to Elasticsearch, Datadog, or Splunk in the coming years.

First, let’s dive into the performance enhancements and main features of Quickwit 0.7 before looking at our roadmap for 2024.

Pushing Quickwit's performance forward​

We made huge strides forward in indexing and search performance by reducing the number of allocations during indexing, tweaking some internal data structures, and implementing various query optimization techniques. In the next weeks, we will publish a handful of detailed blog posts about how we achieved those gains, but for now, let us take a shortcut and share with you some figures reported by our users.

The most compelling report comes from a company ingesting hundreds of terabytes of logs daily and migrating from Elasticsearch to Quickwit, dividing their compute costs by 5x and storage costs by 2x while increasing retention from 3 to 30 days.

In this screenshot shared by an engineer, a Quickwit cluster consisting of 14 pods running on AWS Graviton instances, with each pod limited to 3500 milliCPU and 4GB of RAM, is indexing logs at 540MB/s. That’s approximately 46 terabytes per day!

But there's more to consider than just cost reductions. Quickwit also delivers increased durability, accuracy, and elasticity for this user:

• Index data is now stored on Amazon S3 as opposed to EBS volumes, which is more durable (replication occurs across two availability zones instead of one), less complex operationally (no volumes to manage), and more flexible (storage classes, cross-region replication, …)

• Quickwit ensures end-to-end exactly-once semantics using its native Kafka support. Concretely, Quickwit keeps track of Kafka positions internally and transactionally updates them when a new piece of index is published.

• Stateless indexer nodes make it easy to quickly adjust cluster capacity, whether for handling traffic spikes or backfilling an index. In the screenshot below, 30 pods are temporarily added to a cluster to consume a backlog of 1 billion events in 20 minutes.

New Elasticsearch-compatible API endpoints​

In 0.7, we have expanded the coverage of our Elasticsearch-compatible API and added support for the following endpoints:

• scroll and search after APIs to allow for deep pagination
• multi-index search
• field capabilities API, which is useful for listing the field names and types of schemaless indexes

With these new additions, several of our users now seamlessly interact with Quickwit exclusively through the Elasticsearch API using Elasticsearch or OpenSearch clients. They did not have to migrate their applications when transitioning to Quickwit!

Here is a summary of our current Elasticsearch API coverage:

• _bulk API
• _search API with scroll and search_after APIs and support for multi-target
• _msearch API
• Query DSL (query_string, bool, range, match, …)
• Bucket aggregations (range, terms, histogram, date_histogram)
• Metric aggregations (avg, count, min, max, sum, stats, percentiles)
• _field_caps API

You can access the exhaustive list of Elasticsearch-compatible endpoints supported by Quickwit in our documentation. If you identify a missing endpoint or parameter for migrating your application, please let us know: we will be happy to extend our coverage further.

More ways to ingest OpenTelemetry events​

Since 0.6, Quickwit can ingest OTel data via OTLP over gRPC. In this new release, we added support for ingestion via OTLP over HTTP or Kafka in either Protobuf or JSON formats. You can also send your OpenTelemetry events to the other types of distributed queues currently supported: Amazon Kinesis, Apache Pulsar, or Google Cloud PubSub. We are also planning on adding a NATS source in the near future.

Grafana and Jaeger UX improvements​

We worked on improving the integration of Quickwit with Grafana and Jaeger, two ubiquitous tools in the observability space.

First, we have just released a new version of our Grafana plugin, version 0.3, which brings the following main features:

• Trace search and trace view
• Trace analytics dashboards
• Correlation between logs and traces
• Context view in Explore.

Grafana trace views

Explore context view

The updated data source is available on the Grafana plugins registry. You can follow its roadmap on the dedicated GitHub repository. We would love to hear your feedback.

Finally, we also improved the UX with Jaeger and Grafana by adding the capability of querying one or several custom indexes to power those UIs. This is particularly useful in multi-tenant environments or when migrating from one index to another.

Miscellaneous features and improvements​

Quickwit 0.7 contains some features, improvements, and bug fixes we could not discuss in this blog post. Please refer to our CHANGELOG to review the exhaustive list of changes shipped in this release.

2024 Roadmap​

Our 2024 roadmap is ambitious, and we have already made tremendous progress in some areas. One of our biggest milestones for this year will be the release of Quickwit 1.0 planned for Q3. It will guarantee the stability of our index format and APIs for the coming years. Additionally, we intend to deliver the following functionalities:

• Quickwit on AWS Lambda (Q1 - January)
• Native distributed ingest (Q1 - February)
• Index config and schema update (Q2)
• Pipe-based query language (Q3)
• Metrics support (Q4)

We will also attend and speak at multiple events this year, starting with the London Cloud Native meetup in January, FOSDEM in February, and KubeCon Europe in March. We hope to meet you there!