Skip to main content
Version: 0.6.5

Storage configuration

Supported Storage Providers

Quickwit currently supports three types of storage providers:

  • Amazon S3 and S3-compatible (Garage, MinIO, ...)
  • Azure Blob Storage
  • Local file storage*

Storage URIs

Storage URIs refer to different storage providers identified by a URI "protocol" or "scheme". Quickwit supports the following storage URI protocols:

  • s3:// for Amazon S3 and S3-compatible
  • azure:// for Azure Blob Storage
  • file:// for local file systems

In general, you can use a storage URI or a file path anywhere you would intuitively expect a file path. For instance:

  • when setting the index_uri of an index to specify the storage provider and location; _ when setting the metastore_uri in a node config to set up a file-backed metastore;
  • when passing a file path as a command line argument.

Local file storage URIs

Quickwit interprets regular file paths as local file system URIs. Relative file paths are allowed and are resolved relatively to the current working directory (CWD). ~ can be used as a shortcut to refer to the user’s home directory. The following are valid local file system URIs:

- /var/quickwit
- file:///var/quickwit
- /home/quickwit/data
- ~/data
- ./quickwit
caution

When using the file:// protocol, a third / is necessary to express an absolute path. For instance, the following URI file://home/quickwit/ is interpreted as ./home/quickwit

Storage configuration

This section contains one configuration subsection per storage provider. If a storage configuration parameter is not explicitly set, Quickwit relies on the default values provided by the storage provider SDKs (Azure SDK for Rust, AWS SDK for Rust).

S3 storage configuration

PropertyDescriptionDefault value
flavorThe optional storage flavor to use. Available flavors are digital_ocean, garage, gcs, and minio.
access_key_idThe AWS access key ID.
secret_access_keyThe AWS secret access key.
regionThe AWS region to send requests to.us-east-1 (SDK default)
endpointCustom endpoint for use with S3-compatible providers.SDK default
force_path_style_accessDisables virtual-hosted–style requests. Required by some S3-compatible providers (Ceph, MinIO).false
disable_multi_object_deleteDisables Multi-Object Delete requests. Required by some S3-compatible providers (GCS).false
disable_multipart_uploadDisables multipart upload of objects. Required by some S3-compatible providers (GCS).false
danger

Hardcoding credentials into configuration files is not secure and strongly discouraged. Prefer the alternative authentication methods that your storage backend may provide.

Environment variables

Env variableDescription
QW_S3_ENDPOINTCustom S3 endpoint.
QW_S3_MAX_CONCURRENCYLimit the number of concurent requests to S3

Storage flavors

Storage flavors ensure that Quickwit works correctly with storage providers that deviate from the S3 API by automatically configuring the appropriate settings. The available flavors are:

  • digital_ocean
  • garage
  • gcs
  • minio

Digital Ocean

The Digital Ocean flavor (digital_ocean) forces path-style access and turns off multi-object delete requests.

Garage flavor

The Garage flavor (garage) overrides the region parameter to garage and forces path-style access.

Google Cloud Storage

The Google Cloud Storage flavor (gcs) turns off multi-object delete requests and multipart uploads.

MinIO flavor

The MinIO flavor (minio) forces path-style access.

Example of a storage configuration for Google Cloud Storage in YAML format:

storage:
s3:
flavor: gcs
region: us-east1
endpoint: https://storage.googleapis.com

Azure storage configuration

PropertyDescriptionDefault value
accountThe Azure storage account name.
access_keyThe Azure storage account access key.

Environment variables

Env variableDescription
QW_AZURE_STORAGE_ACCOUNTAzure Blob Storage account name.
QW_AZURE_STORAGE_ACCESS_KEYAzure Blob Storage account access key.

Example of a storage configuration for Azure in YAML format:

storage:
azure:
account: your-azure-account-name
access_key: your-azure-access-key

Storage configuration examples for various object storage providers

Garage

Garage is an open-source distributed object storage service tailored for self-hosting.

storage:
s3:
flavor: garage
endpoint: http://127.0.0.1:3900

MinIO

MinIO is a high-performance object.

storage:
s3:
flavor: minio
endpoint: http://127.0.0.1:9000