Setting up a Quickwit cluster on AWS requires the configuration of three elements:
- AWS credentials
- AWS region
- Network configuration
When starting a node, Quickwit attempts to find AWS credentials using the credential provider chain implemented by rusoto_core::ChainProvider and looks for credentials in this order:
Credential profiles file, typically located at
~/.aws/credentialsor otherwise specified by the
AWS_PROFILEenvironment variables if set and not empty.
Amazon ECS container credentials, loaded from the Amazon ECS container if the environment variable
Instance profile credentials, used on Amazon EC2 instances, and delivered through the Amazon EC2 metadata service.
An error is returned if no credentials are found in the chain.
Quickwit attempts to find an AWS region in multiple locations and with the following order of precedence:
Environment variables (
Config file, typically located at
~/.aws/configor otherwise specified by the
AWS_CONFIG_FILEenvironment variable if set and not empty.
Amazon EC2 instance metadata service determining the region of the currently running Amazon EC2 instance.
AWS credentials or region resolution may take a few seconds, especially if the Amazon EC2 instance metadata service is slow or unavailable.
Required authorized actions:
You can run the following commands to verify that AWS credentials, region, and IAM permissions are property configured for Amazon S3:
aws s3 ls $MY_BUCKET
echo "Hello, World!" | aws s3 cp - $MY_BUCKET/hello
aws s3 ls $MY_BUCKET/hello
aws s3 cp $MY_BUCKET/hello -
aws s3 rm $MY_BUCKET/hello
You can run the following commands to verify that AWS credentials, region, and IAM permissions are property configured for Amazon Kinesis:
MY_STREAM=<my stream name>
# List the shards in the stream and select the first one.
aws kinesis list-shards --stream-name $MY_STREAM \
| jq -r .Shards.ShardId
# Get a shard iterator for the selected shard.
aws kinesis get-shard-iterator --stream-name $MY_STREAM \
--shard-id $SHARD_ID \
--shard-iterator-type TRIM_HORIZON \
| jq -r .ShardIterator
# Fetch some records from the shard and display the first one.
aws kinesis get-records --shard-iterator $SHARD_ITERATOR | jq -r .Records
In order to communicate with each other, nodes must reside in security groups that allow inbound and outbound traffic on one UDP port and two TCP ports. Please, refer to the ports configuration page for more details.
If you set the wrong credentials, you will see this error message with
Unauthorized in your terminal:
Command failed: Another error occurred. `Metastore error`. Cause: `StorageError(kind=Unauthorized, source=Failed to fetch object: s3://quickwit-dev/my-hdfs/metastore.json)`
If you put the wrong region, you will see this one:
Command failed: Another error occurred. `Metastore error`. Cause: `StorageError(kind=InternalError, source=Failed to fetch object: s3://your-bucket/your-index/metastore.json)`.