Skip to main content
Version: v0.3.1

Storage URI

In Quickwit, Storage URIs refer to different kinds of storage.

Generally speaking, you can use a storage URI or a regular file path wherever you would have expected a file path.

For instance

  • when configuring the index storage. (Passed as the index_uri in the index command line.)
  • when configuring a file-backed metastore. (metastore_uri in the QuickwitConfig).
  • when passing a config file in the command line. (you can store your quickwit.yaml on Amazon S3 if you want)

Right now, only two types of storage are supported.

Local file system

One can refer to the file system storage by using a file path directly, or a URI with the file:// protocol. Relative file paths are allowed and are resolved relatively to the current working directory (CWD). ~ can be used as a shortcut to refer to the current user directory.

The following are valid local file system URIs

- /var/quickwit
- file:///var/quickwit
- /home/quickwit/data
- ~/data
- ./quickwit
caution

When using the file:// protocol, a third / is necessary to express an absolute path.

For instance, the following URI file://home/quickwit/ is interpreted as ./home/quickwit

Amazon S3

It is also possible to refer to Amazon S3 using a S3 URI. S3 URIs must have to follow the following format:

s3://<bucket name>/<key>

For instance

s3://quickwit-prod/quickwit-indexes

The credentials, as well as the region or the custom endpoint, have to be configured separately, using the methods described below.

S3 credentials

Quickwit will detect the S3 credentials using the first successful method in this list (order matters)

Region

The region or custom endpoint will be detected using the first successful method in this list (order matters)

S3-compatible Objective storage like Minio

Quickwit can target other S3-compatible storage like MinIO. This is done by setting an endpoint url in the QW_S3_ENDPOINT environment variable.

In this case, the region will be ignored.

For instance:

export QW_S3_ENDPOINT=http://localhost:9000/