When the open source streaming service, Apache Kafka, first emerged in 2011 at LinkedIn, the world looked very different. Most companies were still operating on-premises, and the concept of cloud computing was just starting to take shape. WarpStream, a startup in its early stages, recognized the potential of streaming in a cloud-native context. The team set out to create a new solution, built from the ground up using the Apache Kafka protocol, but designed specifically for the modern cloud era.
Today, WarpStream announced a significant milestone with a $20 million investment.
In an interview with TechCrunch, CEO Richard Artoul and co-founder CTO Ryan Worl shared that in their previous roles, they discovered that moving data into Apache Kafka was complex and costly. They were determined to change that. “If we were to build something like Apache Kafka today, with the benefit of hindsight, we would do things differently from when it was first designed in 2011. That is why we believe it is the perfect time for us to create something new that can significantly reduce costs and operational burden for our users,” explained Artoul.
The key to achieving this is by leveraging today’s cloud environment and separating compute from storage, utilizing object storage services such as Amazon S3. This approach allows WarpStream to eliminate inter-zone networking costs, which can often make up 80% or more of the total expenses for large-scale Kafka workloads.
As Artoul stated, “When you interact and store data in cloud object storage, you can avoid these networking fees that often arise when large data systems are moved into the cloud. Furthermore, many of the complex challenges related to data durability and replication that Kafka had to solve on its own are now handled by the object storage layer itself. This makes our system much more cost-effective and easier to manage.”
Artoul and Worl had previously worked together at Datadog, where they collaborated on the development of a storage system called Husky. Interestingly, Datadog now uses Husky to search through application logs for their clients. As Artoul revealed, “Datadog was also a major user of Kafka. With our experience building a storage system on top of object storage, we envisioned that streaming systems should work in a similar way. And so, we left Datadog last year to focus on creating WarpStream.”
WarpStream offers two options for their users: a self-managed “bring your own cloud” installation, or a fully managed serverless alternative. The self-managed option is now available, and the company has even included a pricing calculator on their website to help customers estimate the cost of running WarpStream.
The founders have brought on several individuals who helped build Husky to work on their new system, and currently have nine employees. They are also actively seeking to expand their team and aim to double their headcount by the end of the year.
The $20 million investment was led by Greylock and Amplify Partners, with additional contributions from prominent figures in the data industry.
