How Binance built a 100PB log service with Quickwit
Binance Scales its Log Service to 100PB and Saves Millions with Quickwit
Quickwit Blog
How Binance built a 100PB log service with Quickwit
Binance Scales its Log Service to 100PB and Saves Millions with Quickwit
François Massot
- release
- engineering
Introducing Quickwit Glasskube package
You might have heard about an exciting new Kubernetes package manager called Glasskube, as they made the front page of HN last week. If not, you should take a look at what they're doing!
- François Massot
Idriss Neumann
- kubernetes
- helm
Introducing Quickwit Glasskube package
You might have heard about an exciting new Kubernetes package manager called Glasskube, as they made the front page of HN last week. If not, you should take a look at what they're doing!
When allocating unused memory boosts performance by 2x
Term aggregations are one of the key aggregations in our search engine quickwit.
Benchmarking Quickwit vs. Loki
In 2019, Grafana launched Loki, a new log aggregation system, to tackle the challenges commonly faced by teams operating and scaling Elasticsearch:
Tantivy 0.22
Exciting news from the search engine world — Tantivy 0.22 has just been released!
A math puzzle and a better algorithm for top-k
A math brain teaser
Grafana Plugin 0.4: QoL features ahead!
v0.4 has editor completions, improved log context, custom log messages, adhoc filters and many other improvements!
Quickwit 0.8: Indexing and Search at Petabyte Scale
Quickwit 0.8 brings indexing and search at an unprecedented scale, indexing at 1PB/day and searching through 50 PB is a reality.
Building a log search service for under $7/month
In this blog post, we’ll show you how to build a simple log search service using AWS CDK and Quickwit Lambdas on a budget.
Packaging Quickwit for AWS Lambda
In this blog post, we explain why and how Quickwit can run on AWS Lambda efficiently.
Search performance on AWS Lambda
This technical blog post is part of a series about Quickwit's serverless performance on AWS S3 and Lambda.
Scaling search to 0 with AWS Lambda
In this blog post, we introduce you to the new Serverless deployment mode for Quickwit based on AWS Lambda.
Grafana Plugin 0.3: Towards Application Performance Monitoring
Grafana Plugin 0.3 brings Application Performance Monitoring, Trace Views, correlation between your datasources, and more!
Quickwit 0.7: Elasticsearch API compatibility and 30% performance gains
Quickwit 0.7 brings Elasticsearch API compatibility, increased performance, Grafana and Jaeger UX improvements, and many other features.
S3 Express speculations
Amazon just announced at re:Invent 2023 a new S3 tier, S3 Express, and we had the opportunity to beta test it!
Quickwit 101 - Architecture of a distributed search engine on object storage
In this blog post, we dive into Quickwit architecture and its key components. This blog post pairs best with Quickwit's benchmark on a 23TB dataset.
Tantivy 0.21
Oh my, what's that on the horizon? It's a new tantivy release!
Scaling search to terabytes on a budget
In this blog post, we indexed 23 TB of GitHub events and evaluated search performance and costs.
A compressed indexable bitset
In this blog post, we discuss the implementation of an indexable bitset in tantivy.
Tantivy 0.20: Schemaless
People rejoice! A new tantivy release. And it's packed with new features:
Observe your Rust application with Quickwit, Jaeger and Grafana
You may have already come across Lucas Palmieri's blog post Are we observable yet? An introduction to Rust telemetry.
Quickwit 0.6: Elasticsearch compatibility, Grafana support, and more...
This is with pride and delight that we are today releasing Quickwit 0.6!
Whichlang: A fast language detection library for Rust
While the current trend is about smart but slow large language models (LLM), we hope you will find it refreshing to read a blog post about a tiny, stupid but fast model. Less is more, isn’t it?
Efficient indexing with Quickwit Rust actor framework
At Quickwit, we are building the most cost-efficient search engine for logs and traces. Such an engine typically ingests massive amounts of data while serving a comparatively low number of search queries. Under this workload, most of your CPU is spent on indexing, making our indexing pipeline a crucial component.
Quickwit is now an official Jaeger storage backend!
Quickwit, our open-source search engine designed for logs and tracing, has recently been listed as an officially supported Jaeger backend! This is fantastic news for developers who are looking for an efficient, reliable, and cost-effective way to trace and monitor their distributed systems. In this blog post, we will walk you through the steps to trace a Node.js web application with Quickwit and Jaeger.
The Organic Software Manifesto
Reader of the future, this blog post was our 2023 April Fool. The blog post was 85% generated by ChatGPT, and the illustration is the fruit of Stable diffusion.
Quickwit 0.5: Distributed tracing with Open Telemetry and Jaeger, VRL, Pulsar support, and more...
Today, we are thrilled to announce the release of Quickwit 0.5.0!
Tantivy 0.19 is out
Tantivy 0.19 is an exciting release packed with new features! In this blog post, we will explore the most important updates and how they open up new possibilities for using Tantivy.
Quickwit 0.4: Distributed indexing with Kafka, K8s-native, and more!
Today, we are proud to announce the release of Quickwit 0.4. More than 600 pull requests have been merged into the project since our last release six months ago. Notably, some significant contributions have come from the community. This version includes the following new features:
Filtering a Vector with SIMD Instructions (AVX-2 and AVX-512)
This blog post will show how one can use SIMD instructions to optimize the code and highlight differences between AVX2 and AVX512.
Quickwit 0.3: schemaless indexing, aggregations, and more!
Five months have passed since our last release, and our team has been working hard to deliver the next version of Quickwit. Today, we are proud to announce the release of Quickwit 0.3. This new feature-rich version introduces:
Decentralized cluster membership in Rust
Even after 10 years of programming, I still have a relentless curiosity about new software algorithms, reading papers and blog posts, and learning from other engineers. The best part, however, really comes when you have the opportunity to implement one algorithm and even customize it for your specific use case. In this post, I will walk you through my journey from understanding cluster membership fundamentals to the implementation of Chitchat, our Rust implementation of the Scuttlebutt algorithm with a phi accrual failure detector.
How to investigate memory usage of your rust program
At Quickwit we have a mindset to create memory efficient and fast software, something that resonates with me personally. I love to create fast software and optimize it, since
Tantivy 0.17 is out
Tantivy 0.17 is a great update that packs a lot of exciting features. We will describe the main ones in this blog post and open up the possibilities of using Tantivy:
Quickwit raises a $2.6M seed round to build the leading cloud-native search engine for large datasets
Today, we’re thrilled to announce our $2.6M seed funding co-led by FirstMark and firstminute with the participation of Eliot Horowitz (previously, Co-founder and CTO of MongoDB; currently CEO of Viam), Florian Douetteau, and Clément Stenac (CEO and CTO of Dataiku), and Isaac Saldana (previously, Founder and CEO of SendGrid and currently CEO of Memo).
Quickwit 0.2 brings full-text search to ClickHouse and Kafka!
Quickwit wishes you a happy new year 2022!
A cool Rust optimization story
I will be your rabbit guide
Quickwit: A highly cost-efficient search engine in Rust
Meet Quickwit 0.1! A super cost-efficient search engine is born!
Searching the web for < $1000 / month
This blog post pairs best with our common-crawl demo and a glass of vin de Loire.
+Hello +World
Hi there,