Skip to main content
Version: 0.5.0

Send logs from Vector

Vector is an amazing piece of software (in Rust obviously) and brings a new fresh wind in the observability space, it is well-known for collecting logs from every part of your infrastructure, transforming and aggregating them, and finally forwarding them to a sink.

In this guide, we will show you how to connect it to Quickwit.

Start Quickwit server

# Create Quickwit data dir.
mkdir qwdata
./quickwit run

Create an index for logs

Let's embrace the OpenTelemetry standard and create an index compatible with its logs data model.

# Index config file for receiving logs in OpenTelemetry format.
# Link:

version: 0.5

index_id: vector-otel-logs

- name: timestamp
type: datetime
- unix_timestamp
output_format: unix_timestamp_secs
fast: true
- name: severity
type: text
tokenizer: raw
fast: true
- name: body
type: text
tokenizer: default
record: position
- name: attributes
type: json
- name: resource
type: json
timestamp_field: timestamp

default_search_fields: [severity, body]

commit_timeout_secs: 10

First download the YAML file:

curl -o vector-otel-logs.yaml

And then create the index with cURL or the CLI:

curl -XPOST http://localhost:7280/api/v1/indexes -H "content-type: application/yaml" --data-binary @vector-otel-logs.yaml

Setup Vector

Our sink here will be Quickwit ingest API To keep it simple in this tutorial, we will use a log source called demo_logs that generates logs in a given format. Let's choose the common syslog format (Vector does not generate logs in the OpenTelemetry format directly!) and use the transform feature to map the syslog format into the OpenTelemetry format.

type = "demo_logs"
format = "syslog"
count = 100000
interval = 0.001

inputs = [ "generate_syslog"]
type = "remap"
source = '''
structured = parse_syslog!(.message)
.timestamp, err = to_unix_timestamp(structured.timestamp, unit: "milliseconds")
.body = .message
.resource.source_type = .source_type = structured.hostname = structured.appname
.attributes.syslog.procid = structured.procid
.attributes.syslog.facility = structured.facility
.attributes.syslog.version = structured.version
.severity = if includes(["emerg", "err", "crit", "alert"], structured.severity) {
} else if structured.severity == "warning" {
} else if structured.severity == "debug" {
} else if includes(["info", "notice"], structured.severity) {
} else {
.name = structured.msgid

# useful to see the logs in the terminal
#inputs = ["remap_syslog"]
#type = "console"
#encoding.codec = "json"

type = "http"
method = "post"
inputs = ["remap_syslog"]
encoding.codec = "json"
framing.method = "newline_delimited"
uri = "http://host.docker.internal:7280/api/v1/vector-otel-logs/ingest"

Now let's start Vector to start send logs to Quickwit.

docker run -v $(pwd)/vector.toml:/etc/vector/vector.toml:ro -p 8383:8383 --add-host=host.docker.internal:host-gateway timberio/vector:0.25.0-distroless-libc

Search logs

Quickwit is now ingesting logs coming from Vector and you can search them either with curl or by using the UI:

  • curl -XGET\?query\=severity:ERROR
  • Open your browser at and play with it!

Compute aggregation on severity

For aggregations, we can't use yet Quickwit UI but we can use cURL.

Let's craft a nice aggregation query to count how many INFO, DEBUG, WARN, and ERROR per minute (all datetime are stored in microseconds thus the interval of 60_000_000 microseconds) we have:

"query": "*",
"max_hits": 0,
"aggs": {
"count_per_minute": {
"histogram": {
"field": "timestamp",
"interval": 60000000
"aggs": {
"severity_count": {
"terms": {
"field": "severity"
curl -XPOST -H "Content-Type: application/json" --data @aggregation-query.json

Further improvements

Coming soon: deploy Vector + Quickwit on your infrastructure, use Grafana to query Quickwit, and more!