Skip to main content
Version: 0.7.1

Send logs from Vector

Vector is an amazing piece of software (in Rust obviously) and brings a new fresh wind in the observability space, it is well-known for collecting logs from every part of your infrastructure, transforming and aggregating them, and finally forwarding them to a sink.

In this guide, we will show you how to connect it to Quickwit.

Start Quickwit server​

# Create Quickwit data dir.
mkdir qwdata
./quickwit run

Taking advantage of Quickwit's native support for logs​

Let's embrace the OpenTelemetry standard and take advantage of Quickwit features. With the native support for OpenTelemetry standards, Quickwit already comes with an index called otel-logs_v0_6 that is compatible with the OpenTelemetry logs data model. This means we can start pushing log data without any prior usual index setup.

Here is the OpenTelemetry index configuration for reference.

otel-logs_v0_6
version: 0.7

index_id: otel-logs-v0_6

doc_mapping:
  mode: strict
  field_mappings:
    - name: timestamp_nanos
      type: datetime
      input_formats: [unix_timestamp]
      output_format: unix_timestamp_nanos
      indexed: false
      fast: true
      fast_precision: milliseconds
    - name: observed_timestamp_nanos
      type: datetime
      input_formats: [unix_timestamp]
      output_format: unix_timestamp_nanos
    - name: service_name
      type: text
      tokenizer: raw
    - name: severity_text
      type: text
      tokenizer: raw
      fast: true
    - name: severity_number
      type: u64
      fast: true
    - name: body
      type: json
    - name: attributes
      type: json
      tokenizer: raw
      fast: true
    - name: dropped_attributes_count
      type: u64
      indexed: false
    - name: trace_id
      type: bytes
    - name: span_id
      type: bytes
    - name: trace_flags
      type: u64
      indexed: false
    - name: resource_attributes
      type: json
      tokenizer: raw
      fast: true
    - name: resource_dropped_attributes_count
      type: u64
      indexed: false
    - name: scope_name
      type: text
      indexed: false
    - name: scope_version
      type: text
      indexed: false
    - name: scope_attributes
      type: json
      indexed: false
    - name: scope_dropped_attributes_count
      type: u64
      indexed: false

  timestamp_field: timestamp_nanos

indexing_settings:
  commit_timeout_secs: 10

search_settings:
  default_search_fields: [body.message]

Setup Vector​

Our sink here will be Quickwit ingest API http://127.0.0.1:7280/api/v1/otel-logs-v0_7/ingest. To keep it simple in this tutorial, we will use a log source called demo_logs that generates logs in a given format. Let's choose the common syslog format (Vector does not generate logs in the OpenTelemetry format directly!) and use the transform feature to map the syslog format into the OpenTelemetry format.

vector.toml
[sources.generate_syslog]
type = "demo_logs"
format = "syslog"
count = 100000
interval = 0.001

[transforms.remap_syslog]
inputs = [ "generate_syslog"]
type = "remap"
source = '''
  structured = parse_syslog!(.message)
  .timestamp_nanos, err = to_unix_timestamp(structured.timestamp, unit: "nanoseconds")
  .body = structured
  .service_name = structured.appname
  .resource_attributes.source_type = .source_type
  .resource_attributes.host.hostname = structured.hostname
  .resource_attributes.service.name = structured.appname
  .attributes.syslog.procid = structured.procid
  .attributes.syslog.facility = structured.facility
  .attributes.syslog.version = structured.version
  .severity_text = if includes(["emerg", "err", "crit", "alert"], structured.severity) {
    "ERROR"
  } else if structured.severity == "warning" {
    "WARN"
  } else if structured.severity == "debug" {
    "DEBUG"
  } else if includes(["info", "notice"], structured.severity) {
    "INFO"
  } else {
   structured.severity
  }
  .scope_name = structured.msgid
  del(.message)
  del(.timestamp)
  del(.source_type)
'''

# useful to see the logs in the terminal
# [sinks.emit_syslog]
# inputs = ["remap_syslog"]
# type = "console"
# encoding.codec = "json"

[sinks.quickwit_logs]
type = "http"
method = "post"
inputs = ["remap_syslog"]
encoding.codec = "json"
framing.method = "newline_delimited"
uri = "http://127.0.0.1:7280/api/v1/otel-logs-v0_7/ingest"

Download the above Vector config file.

curl -o vector.toml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/vector-otel-logs/vector.toml

Now let's start Vector so that we can start sending logs to Quickwit.

docker run -v $(pwd)/vector.toml:/etc/vector/vector.toml:ro -p 8383:8383 --net=host timberio/vector:0.25.0-distroless-libc

Search logs​

Quickwit is now ingesting logs coming from Vector and you can search them either with curl or by using the UI:

  • curl -XGET http://127.0.0.1:7280/api/v1/otel-logs-v0_7/search?query=severity_text:ERROR
  • Open your browser at http://127.0.0.1:7280/ui/search?query=severity_text:ERROR&index_id=otel-logs-v0_7&max_hits=10 and play with it!

Compute aggregation on severity_text​

For aggregations, we can't use yet Quickwit UI but we can use cURL.

Let's craft a nice aggregation query to count how many INFO, DEBUG, WARN, and ERROR per minute (all datetime are stored in microseconds thus the interval of 60_000_000 microseconds) we have:

aggregation-query.json
{
  "query": "*",
  "max_hits": 0,
  "aggs": {
    "count_per_minute": {
      "histogram": {
          "field": "timestamp_nanos",
          "interval": 60000000
      },
      "aggs": {
        "severity_text_count": {
          "terms": {
            "field": "severity_text"
          }
        }
      }
    }
  }
}
curl -XPOST -H "Content-Type: application/json" http://127.0.0.1:7280/api/v1/otel-logs-v0_7/search --data @aggregation-query.json

Going further​

Now you can also deploy Grafana and connect to Quickwit as data source for query, dashboard, alerts and more!