Skip to main content
Version: main branch

Storage URI

In Quickwit, Storage URIs refer to different kinds of storage.

Generally speaking, you can use a storage URI or a regular file path wherever you would have expected a file path.

For instance:

  • when configuring the index storage. (Passed as the index_uri in the index command line.)
  • when configuring a file-backed metastore. (metastore_uri in the QuickwitConfig).
  • when passing a config file in the command line. (you can store your quickwit.yaml on Amazon S3 if you want)

Right now, only two types of storage are supported.

Local file system​

One can refer to the file system storage by using a file path directly, or a URI with the file:// protocol. Relative file paths are allowed and are resolved relatively to the current working directory (CWD). ~ can be used as a shortcut to refer to the current user directory.

The following are valid local file system URIs

- /var/quickwit
- file:///var/quickwit
- /home/quickwit/data
- ~/data
- ./quickwit
caution

When using the file:// protocol, a third / is necessary to express an absolute path.

For instance, the following URI file://home/quickwit/ is interpreted as ./home/quickwit

Amazon S3​

It is also possible to refer to Amazon S3 using a S3 URI. S3 URIs must have to follow the following format:

s3://<bucket name>/<key>

For instance

s3://quickwit-prod/quickwit-indexes

The credentials, as well as the region or the custom endpoint, have to be configured separately, using the methods described below.

S3 credentials​

Quickwit will detect the S3 credentials using the first successful method in this list (order matters)

Region​

The region or custom endpoint will be detected using the first successful method in this list (order matters)

S3-compatible Object Storage​

Quickwit can target other S3-compatible storage. This is done by setting an endpoint url in the QW_S3_ENDPOINT environment variable.

Depending on the object storage, you will also need to set the region and force path-style access (Things like self-hosted MinIO instances, which are not behind a domain name.)

Example:

export QW_S3_ENDPOINT=http://localhost:9000/
export QW_S3_FORCE_PATH_STYLE_ACCESS=true

Google Cloud Storage​

Example for Google Cloud Storage:

export QW_S3_ENDPOINT=https://storage.googleapis.com

See our Google Cloud Storage Setup Guide for the detailed steps to configure Quickwit with Google Cloud Storage.

Scaleway object storage​

Example:

export QW_S3_ENDPOINT=https://s3.{your-region}.scw.cloud
export AWS_REGION={your-region}

See our Scaleway Setup Guide for the detailed steps to configure Quickwit with Scaleway object storage.

Garage​

Garage is an Open-Source lightweight and efficient object storage.

To use it with Quickwit, you will need to setup the region, as mentioned in Garage documentation, it's often set just to garage.

Example for a local garage server:

export QW_S3_ENDPOINT=http://127.0.0.1:3900
export QW_S3_FORCE_PATH_STYLE_ACCESS=true
export AWS_REGION=garage

MinIO, Ceph and more!​

We support other S3-compatible storages and are welcoming PRs to enrich the documentation with new storage backends.

Azure blob storage​

Quickwit supports Azure URIs formatted as azure://{storage-account}/{container}/{prefix} where:

  • storage-account is your Azure storage account name.
  • container is the container name (or bucket in S3 parlance).
  • prefix is optional and can be any prefix.

See our Azure Setup Guide for the detailed steps to configure Quickwit with Azure.