Messaging brokers like Apache Kafka have been known to be called “dumb brokers” because all they do is move streaming data. If the Kafka brokers are dumb, this in turn forces smarter clients. For example Kafka clients need to manage offsets, buffers, compression, ordering, durability, and much more. Oftentimes the smart client requirement is enough to switch to an alternative solution to one that unfortunately may not serve the use case needs.
A follow up to my previous post on HTAP databases. In summary, HTAP databases support both OLTP and OLAP workloads. It’s an all-in-one solution with no ETL. CEO of SingleStore (a fully managed HTAP database), Adam Prout states as a response to my previous post:
I don’t believe there is a place for “Realtime OLAP” long term. I think it will be absorbed by HTAP or OLAP
This is actually an interesting statement that deserves its own blog post. Technically, realtime OLAP is just OLAP. It’s the high QPS (queries per second) and integration to streaming platforms that allow the “realtime” label. But I would like to apply this idea of absorbing technologies in relation to “smarter” data stores, specifically streaming platforms like Kafka.
Smarter Trend
When technology absorbs other technologies, you could say that the previous technology is getting “smarter.” The all-in-one solution actually follows a growing trend of making smarter data stores. Data stores can include any service that persists data - OLAP, OLTP, file systems, and others including specifically message brokers.
Keep reading with a 7-day free trial
Subscribe to SUP! Hubert’s Substack to keep reading this post and get 7 days of free access to the full post archives.
 | A guest post by
|