Storage configuration
Supported Storage Providers
Quickwit currently supports three types of storage providers:
- Amazon S3 and S3-compatible (Garage, MinIO, ...)
- Azure Blob Storage
- Local file storage*
Storage URIs
Storage URIs refer to different storage providers identified by a URI "protocol" or "scheme". Quickwit supports the following storage URI protocols:
s3://
for Amazon S3 and S3-compatibleazure://
for Azure Blob Storagefile://
for local file systems
In general, you can use a storage URI or a file path anywhere you would intuitively expect a file path. For instance:
- when setting the
index_uri
of an index to specify the storage provider and location; _ when setting themetastore_uri
in a node config to set up a file-backed metastore; - when passing a file path as a command line argument.
Local file storage URIs
Quickwit interprets regular file paths as local file system URIs. Relative file paths are allowed and are resolved relatively to the current working directory (CWD). ~
can be used as a shortcut to refer to the user’s home directory. The following are valid local file system URIs:
- /var/quickwit
- file:///var/quickwit
- /home/quickwit/data
- ~/data
- ./quickwit
When using the file://
protocol, a third /
is necessary to express an absolute path. For instance, the following URI file://home/quickwit/
is interpreted as ./home/quickwit
Storage configuration
This section contains one configuration subsection per storage provider. If a storage configuration parameter is not explicitly set, Quickwit relies on the default values provided by the storage provider SDKs (Azure SDK for Rust, AWS SDK for Rust).
S3 storage configuration
Property | Description | Default value |
---|---|---|
flavor | The optional storage flavor to use. Available flavors are digital_ocean , garage , gcs , and minio . | |
access_key_id | The AWS access key ID. | |
secret_access_key | The AWS secret access key. | |
region | The AWS region to send requests to. | us-east-1 (SDK default) |
endpoint | Custom endpoint for use with S3-compatible providers. | SDK default |
force_path_style_access | Disables virtual-hosted–style requests. Required by some S3-compatible providers (Ceph, MinIO). | false |
disable_multi_object_delete | Disables Multi-Object Delete requests. Required by some S3-compatible providers (GCS). | false |
disable_multipart_upload | Disables multipart upload of objects. Required by some S3-compatible providers (GCS). | false |
Hardcoding credentials into configuration files is not secure and strongly discouraged. Prefer the alternative authentication methods that your storage backend may provide.
Environment variables
Env variable | Description |
---|---|
QW_S3_ENDPOINT | Custom S3 endpoint. |
QW_S3_MAX_CONCURRENCY | Limit the number of concurent requests to S3 |
Storage flavors
Storage flavors ensure that Quickwit works correctly with storage providers that deviate from the S3 API by automatically configuring the appropriate settings. The available flavors are:
digital_ocean
garage
gcs
minio
Digital Ocean
The Digital Ocean flavor (digital_ocean
) forces path-style access and turns off multi-object delete requests.
Garage flavor
The Garage flavor (garage
) overrides the region
parameter to garage
and forces path-style access.
Google Cloud Storage
The Google Cloud Storage flavor (gcs
) turns off multi-object delete requests and multipart uploads.
MinIO flavor
The MinIO flavor (minio
) forces path-style access.
Example of a storage configuration for Google Cloud Storage in YAML format:
storage:
s3:
flavor: gcs
region: us-east1
endpoint: https://storage.googleapis.com
Azure storage configuration
Property | Description | Default value |
---|---|---|
account | The Azure storage account name. | |
access_key | The Azure storage account access key. |
Environment variables
Env variable | Description |
---|---|
QW_AZURE_STORAGE_ACCOUNT | Azure Blob Storage account name. |
QW_AZURE_STORAGE_ACCESS_KEY | Azure Blob Storage account access key. |
Example of a storage configuration for Azure in YAML format:
storage:
azure:
account: your-azure-account-name
access_key: your-azure-access-key
Storage configuration examples for various object storage providers
Garage
Garage is an open-source distributed object storage service tailored for self-hosting.
storage:
s3:
flavor: garage
endpoint: http://127.0.0.1:3900
MinIO
MinIO is a high-performance object.
storage:
s3:
flavor: minio
endpoint: http://127.0.0.1:9000