Send logs from OTEL Collector

If you already have your own OpenTelemetry Collector and want to export your logs to Quickwit, you need a new OLTP gRPC exporter in your config.yaml:

macOS/Windows

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  otlp/quickwit:
    endpoint: host.docker.internal:7281
    tls:
      insecure: true  
    # By default, logs are sent to the otel-logs-v0_7.
    # You can customize the index ID By setting this header.
    # headers:
    #   qw-otel-logs-index: otel-logs-v0_7
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/quickwit]

Linux

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  otlp/quickwit:
    endpoint: 127.0.0.1:7281
    tls:
      insecure: true
    # By default, logs are sent to the otel-logs-v0_7.
    # You can customize the index ID By setting this header.
    # headers:
    #   qw-otel-logs-index: otel-logs-v0_7

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/quickwit]

Test your OTEL configuration

1. Install and start a Quickwit server:

./quickwit run

2. Start a collector with the previous config:

macOS/Windows

docker run -v ${PWD}/otel-collector-config.yaml:/etc/otelcol/config.yaml -p 4317:4317 -p 4318:4318 -p 7281:7281 otel/opentelemetry-collector

Linux

docker run -v ${PWD}/otel-collector-config.yaml:/etc/otelcol/config.yaml --network=host -p 4317:4317 -p 4318:4318 -p 7281:7281 otel/opentelemetry-collector

3. Send a log to your collector with cURL:

curl -XPOST "http://localhost:4318/v1/logs" -H "Content-Type: application/json" \
--data-binary @- << EOF
{
 "resource_logs": [
   {
     "resource": {
       "attributes": [
         {
           "key": "service.name",
           "value": {
             "stringValue": "test-with-curl"
           }
         }
       ]
     },
     "scope_logs": [
       {
         "scope": {
           "name": "manual-test"
         },
         "log_records": [
           {
             "time_unix_nano": "1678974011000000000",
             "observed_time_unix_nano": "1678974011000000000",
             "name": "test",
             "severity_text": "INFO"
           }
         ]
       }
     ]
   }
 ]
}
EOF

You should see a log on the Quickwit server similar to the following:

2023-03-16T13:44:09.369Z  INFO quickwit_indexing::actors::indexer: new-split split_id="01GVNAKT5TQW0T2QGA245XCMTJ" partition_id=6444214793425557444

This means that Quickwit has received the log and created a new split. Wait for the split to be published before searching for logs.