With WarpStream, Confluent Got a New Type of Kafka Platform
https://lnkd.in/gfTxqiYr
The Mountain View California-based company Confluent offers the Kafka streaming software on an enterprise platform. It also offers a version for the cloud, Confluent Cloud. And now it has a third way to deliver Kafka, thanks to the recently acquired WarpStream earlier last month.
Considering the buzz at the company’s annual Current user conference, held in Austin, this buy was anything but the usual routine acquisition of a potential competitor.
WarpStream developed its own real-time data streaming, “Kafka-compatible” distribution, based loosely on — Buzzword Alert! — the Bring-Your-Own-Cloud (BYOC) model, though the WarpStream’s founders insist its not standard BYOC.
Confluent, as always, is bullish on streaming data, and at this year’s conference, they reinforced the idea that most IT systems will eventually move from a batch-processing mindset to that of analyzing and processing stream data. The company’s latest data streaming report shows that 91% of IT leaders are planning on moving their organizations more towards data streaming.
WarpStream fits a particular niche; it is perfect for logging, observability, and feeding data lakes, all cases where a little bit of latency is an acceptable tradeoff for large cloud cost savings, Confluent execs noted.
“Our goal is to make data streaming the central nervous system of every company, and to do that we need to make it something that is a great fit for a vast array of use cases and companies,” noted Jay Kreps, CEO and co-founder of Confluent, in a company statement of the acquisition.
WarpStream Ahead
WarpStream’s approach provides a set of containers to the customers with a fully scalable Kafka deployment, along with supporting tools.
It is managed under what is called a “shared responsibility” model of cloud computing, where both the customer and vendor share in the responsibility of maintaining the application.
Unlike other BYOC approaches, however, it is up to the customer, not the service provider, to scale to the needed level of service. But that can be done automatically.
In a press conference at the conference, WarpStream co-creator Ryan Worl said he did not feel comfortable with a business model where the vendor could scale up its operations on the customer’s behalf. So that part is left to the customer.
“The thing we realized is that we could avoid a lot of the problems with the traditional BYOC model by re-architecting the underlying system in a different way,” explained WarpStream confounder Richard Artoul during the Current keynote.
In this setup, the control plane and data plane are split, with WarpStream managing the control plane from its own cloud account, which manage the transactions, consumer groups, metadata and consensus.
The customer manages the data plane, which is a set of containerized agents, each basically the equivalent of a stateless Kafka broker, handling...