Aggregations API
An aggregation summarizes your data as statistics on buckets or metrics.
Aggregations can provide answer to questions like:
- What is the average price of all sold articles?
- How many errors with status code 500 do we have per day?
- What is the average listing price of cars grouped by color?
There are two categories: Metrics and Buckets.
Prerequisite
To be able to use aggregations on a field, the field needs to have a fast field index created. A fast field index is a columnar storage, where documents values are extracted and stored to.
Example to create a fast field on text for term aggregations.
name: category
type: text
tokenizer: raw
record: basic
fast: true
See the index configuration page for more details and examples.
Format
The aggregation request and result de/serialize into elasticsearch compatible JSON. If not documented otherwise you should be able to drop in your elasticsearch aggregation queries.
In some examples below is not the full request shown, but only the payload for aggregations.
Example
Request
{
"query": "*",
"max_hits": 0,
"aggs": {
"sites_and_aqi": {
"terms": {
"field": "County",
"size": 2,
"order": { "average_aqi": "asc" }
},
"aggs": {
"average_aqi": {
"avg": { "field": "AQI" }
}
}
}
}
}
Response
...
"aggs": {
"sites_and_aqi": {
"buckets": [
{
"average_aqi": {
"value": 32.62267569707098
},
"doc_count": 56845,
"key": "臺東縣"
},
{
"average_aqi": {
"value": 35.97893635571055
},
"doc_count": 28675,
"key": "花蓮縣"
}
],
"sum_other_doc_count": 1872055
}
}
Supported Aggregations
Bucket Aggregations
BucketAggregations create buckets of documents. Each bucket is associated with a rule which determines whether or not a document falls into it. In other words, the buckets effectively define document sets. Buckets are not necessarily disjunct, therefore a document can fall into multiple buckets. In addition to the buckets themselves, the bucket aggregations also compute and return the number of documents for each bucket. Bucket aggregations, as opposed to metric aggregations, can hold sub-aggregations. These sub-aggregations will be aggregated for the buckets created by their “parent” bucket aggregation. There are different bucket aggregators, each with a different “bucketing” strategy. Some define a single bucket, some define a fixed number of multiple buckets, and others dynamically create the buckets during the aggregation process.
Example request, histogram with stats in each bucket:
Aggregating on datetime fields
Fields of type datetime are handled the same way as any numeric field. However, all values in the requests such as intervals, offsets, bounds, and range boundaries need to be expressed in microseconds.
Histogram with one bucket per day on a datetime field. interval needs to be provided in microseconds.
In the following example, we grouped documents per day (1 day = 86400000000 microseconds).
The returned format is currently fixed at Rfc3339.
Request
{
"query": "*",
"max_hits": 0,
"aggs": {
"datetime_histogram":{
"histogram":{
"field": "datetime",
"interval": 86400000000
}
}
}
}
Response
{
...
"aggregations": {
"datetime_histogram": {
"buckets": [
{
"doc_count": 1,
"key": 1546300800000000.0,
"key_as_string": "2019-01-01T00:00:00Z"
},
{
"doc_count": 2,
"key": 1546560000000000.0,
"key_as_string": "2019-01-04T00:00:00Z"
}
]
}
}
}
Histogram
Histogram is a bucket aggregation, where buckets are created dynamically for the given interval. Each document value is rounded down to its bucket.
E.g. if we have a price 18 and an interval of 5, the document will fall into the bucket with the key 15. The formula used for this is: ((val - offset) / interval).floor() * interval + offset.
Returned Buckets
By default buckets are returned between the min and max value of the documents, including empty buckets. Setting min_doc_count to != 0 will filter empty buckets.
The value range of the buckets can bet extended via extended_bounds or limit the range via hard_bounds.
Example
{
"query": "*",
"max_hits": 0,
"aggs": {
"prices": {
"histogram": {
"field": "price",
"interval": 10
}
}
}
}
Parameters
field
The field to aggregate on.
Currently this aggregation only works on single value fast fields of type u64, f64, i64, and datetime.
keyed
Change response format from an array to a hashmap, key in the bucket will be the key in the hashmap.
interval
The interval to chunk your data range. Each bucket spans a value range of [0..interval). Must be larger than 0.
offset
Intervals implicitly defines an absolute grid of buckets [interval * k, interval * (k + 1)).
Offset makes it possible to shift this grid into [offset + interval * k, offset + interval (k + 1)). Offset has to be in the range [0, interval).
As an example, if there are two documents with value 8 and 12 and interval 10.0, they would fall into the buckets with the key 0 and 10. With offset 5 and interval 10, they would both fall into the bucket with they key 5 and the range [5..15)
{
"query": "*",
"max_hits": 0,
"aggs": {
"prices": {
"histogram": {
"field": "price",
"interval": 10,
"offset": 2.5
}
}
}
}
min_doc_count
The minimum number of documents in a bucket to be returned. Defaults to 0.
hard_bounds
Limits the data range to [min, max] closed interval.
This can be used to filter values if they are not in the data range.
hard_bounds only limits the buckets, to force a range set both extended_bounds and hard_bounds to the same range.
{
"query": "*",
"max_hits": 0,
"aggs": {
"prices": {
"histogram": {
"field": "price",
"interval": 10,
"hard_bounds": {
"min": 0,
"max": 100
}
}
}
}
}
extended_bounds
Can be set to extend your bounds. The range of the buckets is by default defined by the data range of the values of the documents. As the name suggests, this can only be used to extend the value range. If the bounds for min or max are not extending the range, the value has no effect on the returned buckets.
Cannot be set in conjunction with min_doc_count > 0, since the empty buckets from extended bounds would not be returned.
{
"query": "*",
"max_hits": 0,
"aggs": {
"prices": {
"histogram": {
"field": "price",
"interval": 10,
"extended_bounds": {
"min": 0,
"max": 100
}
}
}
}
}
Range
Provide user-defined buckets to aggregate on. Two special buckets will automatically be created to cover the whole range of values. The provided buckets have to be continuous. During the aggregation, the values extracted from the fast_field field will be checked against each bucket range. Note that this aggregation includes the from value and excludes the to value for each range.
{
"query": "*",
"max_hits": 0,
"aggs": {
"my_ranges": {
"field": "score",
"ranges": [
{ "to": 3.0 },
{ "from": 3.0, "to": 7.0 },
{ "from": 7.0, "to": 20.0 },
{ "from": 20.0 }
]
}
}
}
Limitations/Compatibility
Overlapping ranges are not yet supported.
Parameters
keyed
Change response format from an array to a hashmap, the serialized range will be the key in the hashmap.
field
The field to aggregate on.
Currently this aggregation only works on single value fast fields of type u64, f64, i64, and datetime.
ranges
The list of buckets, with from and to values.
The from value is inclusive in the range.
The to value is not inclusive in the range.
The first bucket can omit the from value, and the last bucket the to value.
Note that this aggregation includes the from value and excludes the to value for each range. Extra buckets will be created until the first to, and last from, if necessary.
Terms
Creates a bucket for every unique term and counts the number of occurrences.
Note that doc_count in the response buckets equals term count here.
If the text is untokenized and single value, that means one term per document and therefore it is in fact doc count.
Request
{
"query": "*",
"max_hits": 0,
"aggs": {
"genres": {
"terms": { "field": "genre" }
}
}
}
Response
...
"aggs": {
"genres": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{ "key": "drumnbass", "doc_count": 6 },
{ "key": "raggae", "doc_count": 4 },
{ "key": "jazz", "doc_count": 2 }
]
}
}
Document count error
In Quickwit, we have one segment per split. Therefore the results returned from a split, is equivalent to results returned from a segment.
To improve performance, results from one split are cut off at split_size.
When combining results of multiple splits, terms that
don't make it in the top n of a result from a split increase the theoretical upper bound error by lowest
term-count.
Even with a larger split_size value, doc_count values for a terms aggregation may be
approximate. As a result, any sub-aggregations on the terms aggregation may also be approximate.
sum_other_doc_count is the number of documents that didn’t make it into the the top size
terms. If this is greater than 0, you can be sure that the terms agg had to throw away some
buckets, either because they didn’t fit into size on the root node or they didn’t fit into
split_size on the leaf node.
Per bucket document count error
If you set the show_term_doc_count_error parameter to true, the terms aggregation will include
doc_count_error_upper_bound, which is an upper bound to the error on the doc_count returned by
each split. It’s the sum of the size of the largest bucket on each split that didn’t fit
into split_size.
Parameters
field
The field to aggregate on.
Currently this aggregation only works on fast text fields.
size
By default, the top 10 terms with the most documents are returned. Larger values for size are more expensive.
split_size
The get more accurate results, we fetch more than size from each segment/split. Increasing this value is will increase the accuracy, but also the CPU/memory usage.
Defaults to size * 1.5 + 10.
show_term_doc_count_error
If you set the show_term_doc_count_error parameter to true, the terms aggregation will include doc_count_error_upper_bound, which is an upper bound to the error on the doc_count returned by each split. It’s the sum of the size of the largest bucket on each split that didn’t fit into split_size.
Defaults to true when ordering by count desc.
min_doc_count
Filter all terms that are lower than min_doc_count. Defaults to 1.
Expensive : When set to 0, this will return all terms in the field.
order
Set the order. String is here a target, which is either “_count”, “_key”, or the name of a metric sub_aggregation. Single value metrics like average can be addressed by its name. Multi value metrics like stats are required to address their field by name e.g. “stats.avg”.
Order alphabetically
{
"query": "*",
"max_hits": 0,
"aggs": {
"genres": {
"terms": {
"field": "genre",
"order": { "_key": "asc" }
}
}
}
}
Order by sub_aggregation
{
"query": "*",
"max_hits": 0,
"aggs": {
"articles_by_price": {
"terms": {
"field": "article_name",
"order": { "average_price": "asc" }
},
"aggs": {
"average_price": {
"avg": { "field": "price" }
}
}
}
}
}
Metric Aggregations
The aggregations in this family compute metrics based on values extracted from the documents that are being aggregated. Values are extracted from the fast field of the document. Some aggregations output a single numeric metric (e.g. Average) and are called single-value numeric metrics aggregation, others generate multiple metrics (e.g. Stats) and are called multi-value numeric metrics aggregation.
In contrast to bucket aggregations, metrics don't allow sub-aggregations, since there is no document set to aggregate on.
Average
A single-value metric aggregation that computes the average of numeric values that are extracted from the aggregated documents.
Supported field types are u64, f64, i64, and datetime.
Request
{
"query": "*",
"max_hits": 0,
"aggs": {
"average_price": {
"avg": { "field": "price" }
}
}
}
Response
{
"num_hits": 9582098,
"hits": [],
"elapsed_time_micros": 101942,
"errors": [],
"aggs": {
"average_price": {
"value": 133.7
}
}
}
Count
A single-value metric aggregation that counts the number of values that are extracted from the aggregated documents.
Supported field types are u64, f64, i64, and datetime.
Request
{
"query": "*",
"max_hits": 0,
"aggs": {
"price_count": {
"value_count": { "field": "price" }
}
}
}
Response
{
"num_hits": 9582098,
"hits": [],
"elapsed_time_micros": 102956,
"errors": [],
"aggs": {
"price_count": {
"value": 9582098
}
}
}
Max
A single-value metric aggregation that computes the maximum of numeric values that are that are extracted from the aggregated documents.
Supported field types are u64, f64, i64, and datetime.
Request
{
"query": "*",
"max_hits": 0,
"aggs": {
"max_price": {
"max": { "field": "price" }
}
}
}
Response
{
"num_hits": 9582098,
"hits": [],
"elapsed_time_micros": 101543,
"errors": [],
"aggs": {
"max_price": {
"value": 1353.23
}
}
}
Min
A single-value metric aggregation that computes the minimum of numeric values that are that are extracted from the aggregated documents.
Supported field types are u64, f64, i64, and datetime.
Request
{
"query": "*",
"max_hits": 0,
"aggs": {
"min_price": {
"min": { "field": "price" }
}
}
}
Response
{
"num_hits": 9582098,
"hits": [],
"elapsed_time_micros": 102342,
"errors": [],
"aggs": {
"min_price": {
"value": 0.01
}
}
}
Stats
A multi-value metric aggregation that computes stats (average, count, min, max, standard deviation, and sum) of numeric values that are extracted from the aggregated documents.
Supported field types are u64, f64, i64, and datetime.
Request
{
"query": "*",
"max_hits": 0,
"aggs": {
"timestamp_stats": {
"stats": { "field": "timestamp" }
}
}
}
Response
{
"num_hits": 10000783,
"hits": [],
"elapsed_time_micros": 65297,
"errors": [],
"aggs": {
"timestamp_stats": {
"avg": 1462320207.9803998,
"count": 10000783,
"max": 1475669670.0,
"min": 1440670432.0,
"standard_deviation": 11867304.28681695,
"sum": 1.4624347076526848e16
}
}
}
Sum
A single-value metric aggregation that that sums up numeric values that are that are extracted from the aggregated documents.
Supported field types are u64, f64, i64, and datetime.
Request
{
"query": "*",
"max_hits": 0,
"aggs": {
"total_price": {
"sum": { "field": "price" }
}
}
}
Response
{
"num_hits": 9582098,
"hits": [],
"elapsed_time_micros": 101142,
"errors": [],
"aggs": {
"total_price": {
"value": 12966782476.54
}
}
}