Affordable and Cloud-Based Data Streaming: The WarpStream Solution
When the open source streaming service, Apache Kafka, was created in 2011 at LinkedIn, it was a different world.
The way they do that is by taking advantage of today’s cloud environment to separate compute from storage using an object storage service like Amazon S3.
“When you interact and store data in cloud object storage you get to sidestep all these networking fees that plague these big data systems when they get lifted, shifted into the cloud,” he said.
“Based on our experience building the kind of storage system on top of object storage we had built at DataDog, we felt like streaming systems should work the same way.
The founders brought some of the folks who helped build Husky to build the new system, and today they have 9 employees.
