Skip to main content

Quickwit 0.6: Elasticsearch compatibility, Grafana support, and more...

Quickwit 0.6

This is with pride and delight that we are today releasing Quickwit 0.6!

We started Quickwit with the objective of offering a more cost-efficient solution to search and analyze large amounts of logs. This is not a new problem really, is it? The most established solution today is the very mature and versatile Elasticsearch.

As you may expect, many of our users come from the Elasticsearch ecosystem. Naturally, they expect Quickwit to offer the set of features they used and loved... Too often, Quickwit lacked a few of these features to be a valid replacement for their Elasticsearch cluster. Quickwit 0.6 is precisely bridging this gap!

Let's unpack Quickwit 0.6 main new features together!

Elasticsearch-compatible search API

A customer asked us if we could offer an Elasticsearch-compatible API. The idea was to facilitate their migration. Of course, Quickwit will most likely never cover the gigantic Elasticsearch API exhaustively, but in this first iteration, you will find that we already cover most of the popular endpoints, query DSL and aggregations.

You can check at the exhaustive list of endpoints in Quickwit's Reference documentation. If you spot a feature that is missing and that you need, please let us know.

Grafana support

So far, Quickwit has only offered a minimalist UI to explore your logs. Starting wth Quickwit 0.6, you will be able to visualize your logs in Grafana.

The Grafana Explore view will let you explore/filter massive amounts of logs using the power of Quickwit.

Explore View

Quickwit 0.6 - Grafana Explorer view

Now, it is also possible to build dashboards and alerts using Quickwit and Grafana.

Dashboard View

Quickwit 0.6 - Grafana Dashboard view

The data source is not published on the Grafana plugins registry yet, but if you are ready to install it manually and give it a try, it is available on GitHub. We would love to hear your feedback.

If you prefer to wait for it to be published, stay tuned! We will announce its officially in a few weeks.

New schemaless columnar format

In order to support operations such as aggregations or sorting by a field, search engines usually come with a columnar storage. In Quickwit, declaring that a field should also be stored in a columar format is done by setting it as fast: true in the index configuration. The previous approach had several limitations:

  • All fast fields had to be listed one by one in the schema. In other words, while Quickwit 0.5 made it possible to index documents in a schemaless fashion, "dynamic" fields could not be used in aggregations.
  • Fast fields had to have exactly one value per document. Multivalued fields, or optional fields, were forbidden.

In Quickwit 0.6, we introduced a new columnar format. We will describe its specificity in a different blog post, but functionally, this new columnar format removes all of the previous limitations listed above.

In addition:

  • it has no limitations on the number of fields supported
  • it is meant to be consumed straight from object storage.

As a result, the following minimal configuration is a solid solution for most of your use cases.

version: 0.6
index_id: my-dynamic-index
- name: ts
type: datetime
indexed: false
store: true
fast: true
timestamp_field: ts
# by default this is dynamic mode.
# Everything is indexed, stored and fast.

More queries and aggregations

Quickwit 0.6 includes the new following query types and aggregations:

  • date histogram aggregation
  • percentiles aggregation
  • prefix phrase query
  • range query

Better object storages' support

Our users operate Quickwit with object storages from many vendors: Amazon S3, Google Cloud Storage, Azure, Alibaba, MinIO, Garage, Scaleway... These vendors usually offer an S3-compatible API. In practice, however, the API is usually not entirely supported. In Quickwit 0.6, we improved compatibility and performance for some of these vendors and added a section in the configuration to fine-tune Quickwit's interaction with object storage.

In particular,

  • it is now possible to use virtual-hosted style or to force the use of path-style: the latter not being supported by Alibaba's Cloud Object Storage,
  • it is now possible to disable the use of the multi-object delete API, which is not supported by Google cloud storage.

Many other features and performance improvements

Quickwit 0.6 includes many other features, performance improvements, and bugfixes. You can have a look at the CHANGELOG here.