Quickwit joins Datadog

The Unexpected Journey of Building A Multi-Petabyte Scale Search Engine​

We are thrilled to announce that Quickwit is joining Datadog! We will be heads down building a new product with Datadog, so to ensure our open-source community can continue on, we will soon release a new version of Quickwit under the Apache License 2.0. Stay tuned!

But first, I want to share our story—a journey that spans four years, three continents, and countless plates of gyoza.

From Friendship to Foundation​

It all began with a decade-long friendship between three engineers—Paul, Adrien, and me—who first met in Paris in 2010. Back then, we'd sit in our favorite gyoza restaurant—the gyoza bar, it still exists. We would typically go there after attending an event called Start in Paris, where rookie founders came to pitch their startup ideas. Think Dragon’s Den with a mustache. There, we would dream about building something revolutionary together. But dreams remained dreams until 2020 when Paul called us to talk about his pet project, tantivy. The pet had sharp teeth - it was already a popular alternative to Lucene.

We knew we had to build something together within minutes in that call. The problem emerged naturally: Elasticsearch wasn't scaling effectively and had become too costly and complex to manage. Our mission crystallized: creating a search engine that would be at least 10x more cost-efficient, scale to multiple petabytes while letting operators sleep peacefully at night, and be significantly easier to manage—the dream of thousands of engineers operating Elasticsearch clusters.

Building Across Borders​

With Adrien in San Francisco, Paul in Tokyo, and me in Paris, we created a "follow-the-sun" development cycle. Our choice of Rust proved transformative—our first demo on the Common Crawl dataset hit the front page of HN with virtually no bugs (except for that one pesky Python error: "'NoneType' object has no attribute"). We launched our first version on July 13, 2021. Choosing open source wasn't just strategic—it was natural for us as engineers. Our front page feature on HN brought our first interesting conversations, but we quickly realized our little experiment needed significant work. Each chat with potential users revealed new missing features we needed to implement. We were struggling to find the shortest path to market, making countless decisions along the way. Even in hindsight, it's hard to tell which choices were good or bad—each day brought its mix of successes and failures, definitely more of the latter than the former.

Breaking Through​

Everything changed when we met the engineering teams at Mezmo and Binance. Thanks to Open-Source. They understood our vision early and trusted us completely. Our partnerships transformed Quickwit from an ambitious idea into a battle-tested product operating at a mind-boggling scale:

• Binance built a 100PB log service with Quickwit, indexing at 1.6PB/day.
• Mezmo recently put in production Quickwit to serve thousands of customers and petabytes of logs, drastically reducing infrastructure cost and complexity while delivering the same user experience.

These partnerships helped us build one of the most cost-efficient, multi-petabyte-scale search engines from scratch. Along the way, we also contributed to several libraries in the Rust ecosystem:

• tantivy: The foundation of Quickwit, a full-text search engine library.
• chitchat: Cluster membership protocol with failure detection inspired by Cassandra and DynamoDB.
• Bitpacking: SIMD algorithms for integer compression via bitpacking.
• Whichlang: A blazingly fast and lightweight language detection library for Rust.
• Mrecordlog: An efficient write-ahead-log (WAL) designed for multitenancy
• And more!

The Path to Datadog​

This summer, the wind started to turn. We witnessed stronger open-source traction, our revenue increased dramatically, and VCs became more insistent. It was time for us to open a new chapter for the company and raise a series A round. However, building across Tokyo, Paris, and New York (Yes, Adrien moved from SF to NYC) stretched us thin, and scaling up would only intensify our challenges. We eventually decided to look for a new home for Quickwit. As we began evaluating acquisition opportunities, Datadog emerged as the clear standout, their proven ability to deliver exceptional user experiences, combined with Quickwit’s petabyte-scale search engine, presented a strong opportunity to create powerful solutions deployed in customer environments.

Initially skeptical about joining a large corporation, our reservations dissolved through interactions with the Datadog team. As Yanbing Li, Datadog's new CPO, candidly observed, "When you start discussing with those guys, you are on a slippery slope". Conversations with Olivier, Alexis, Michael, David, Yanbing, Laurent, Jeromy and others unveiled a refreshingly authentic culture—one characterized by intelligence, humility, and genuine integrity.

The Journey Continues​

Today, we are excited to join Datadog and build something extraordinary together. We will be focused on building a new product with Datadog, and to ensure our open-source community can continue, we will soon release a major update of both Quickwit with a relicense to Apache License 2.0 and tantivy. This new Quickwit version will include several notable features our community has been asking for: distributed ingest, cardinality aggregations, performance and memory improvements, and more. Stay tuned for our upcoming releases!

We are extraordinarily thankful to everyone who invested their time and energy in this journey: employees, investors, contributors, friends, and allies. Special thanks to Pascal, Trinity, Harrison, Loic, Idriss, Shikhar, Vladimir, jY, Raphael, Borat, Evance, Michael, Eugene, Roch, Philippe, Michael, Matt, Sam, and many others who helped shape the project and spread the word.

To our amazing customers—Mezmo, Formal, Radiant Security, MatterLabs, Fly.io, and others—thank you for your trust and support.

To our community, thank you for being part of this adventure.

And if you are ever in Paris, let us know! We'd love to share some gyoza—conveniently close to the Datadog office.

With gratitude and excitement for our next chapter,

François