Quickwit 0.6: Elasticsearch compatibility, Grafana support, and more...

This is with pride and delight that we are today releasing Quickwit 0.6!

We started Quickwit with the objective of offering a more cost-efficient solution to search and analyze large amounts of logs. This is not a new problem really, is it? The most established solution today is the very mature and versatile Elasticsearch.

As you may expect, many of our users come from the Elasticsearch ecosystem. Naturally, they expect Quickwit to offer the set of features they used and loved... Too often, Quickwit lacked a few of these features to be a valid replacement for their Elasticsearch cluster. Quickwit 0.6 is precisely bridging this gap!

Let's unpack Quickwit 0.6 main new features together!

Elasticsearch-compatible search API​

A customer asked us if we could offer an Elasticsearch-compatible API. The idea was to facilitate their migration. Of course, Quickwit will most likely never cover the gigantic Elasticsearch API exhaustively, but in this first iteration, you will find that we already cover most of the popular endpoints, query DSL and aggregations.

You can check the exhaustive list of endpoints in Quickwit's Reference documentation. If you spot a feature that is missing and that you need, please let us know.

Grafana support​

So far, Quickwit has only offered a minimalist UI to explore your logs. Starting with Quickwit 0.6, you will be able to visualize your logs in Grafana.

The Grafana Explore view will let you explore/filter massive amounts of logs using the power of Quickwit.

Explore View

Now, it is also possible to build dashboards and alerts using Quickwit and Grafana.

Dashboard View

The data source is not published on the Grafana plugins registry yet, but if you are ready to install it manually and give it a try, it is available on GitHub. We would love to hear your feedback.

If you prefer to wait for it to be published, stay tuned! We will announce it officially in a few weeks.

New schemaless columnar format​

In order to support operations such as aggregations or sorting by a field, search engines usually come with a columnar storage. In Quickwit, declaring that a field should also be stored in a columnar format is done by setting it as fast: true in the index configuration. The previous approach had several limitations:

• All fast fields had to be listed one by one in the schema. In other words, while Quickwit 0.5 made it possible to index documents in a schemaless fashion, "dynamic" fields could not be used in aggregations.
• Fast fields had to have exactly one value per document. Multivalued fields, or optional fields, were forbidden.

In Quickwit 0.6, we introduced a new columnar format. We will describe its specificity in a different blog post, but functionally, this new columnar format removes all of the previous limitations listed above.

In addition:

• it has no limitations on the number of fields supported
• it is meant to be consumed straight from object storage.

As a result, the following minimal configuration is a solid solution for most of your use cases.

version: 0.6
index_id: my-dynamic-index
doc_mapping:
    field_mappings:
        - name: ts
          type: datetime
          indexed: false
          store: true
          fast: true
    timestamp_field: ts
    # by default this is dynamic mode.
    # Everything is indexed, stored and fast.

More queries and aggregations​

Quickwit 0.6 includes the new following query types and aggregations:

• date histogram aggregation
• percentiles aggregation
• prefix phrase query
• range query

Better object storages' support​

Our users operate Quickwit with object storages from many vendors: Amazon S3, Google Cloud Storage, Azure, Alibaba, MinIO, Garage, Scaleway... These vendors usually offer an S3-compatible API. In practice, however, the API is usually not entirely supported. In Quickwit 0.6, we improved compatibility and performance for some of these vendors and added a section in the configuration to fine-tune Quickwit's interaction with object storage.

In particular,

• it is now possible to use virtual-hosted style or to force the use of path-style: the latter not being supported by Alibaba's Cloud Object Storage,
• it is now possible to disable the use of the multi-object delete API, which is not supported by Google cloud storage.

Many other features and performance improvements​

Quickwit 0.6 includes many other features, performance improvements, and bug fixes. You can have a look at the CHANGELOG here.