VAST Data Adds Blocks to Unified Storage Platform
insideBIGDATA
FEBRUARY 19, 2025
VAST also added the VAST Event Broker, an Apache Kafka-compatible event streaming service for real-time data ingestion and […]
Apache Kafka: A Metaphorical Introduction to Event Streaming for Data Scientists and Data Engineers
Analytics Vidhya
NOVEMBER 2, 2020
Overview Learn about viewing data as streams of immutable events in contrast to mutable containers Understand how Apache Kafka captures real-time data through event. The post Apache Kafka: A Metaphorical Introduction to Event Streaming for Data Scientists and Data Engineers appeared first on Analytics Vidhya.
Apache Kafka 
291
291 Join 17,000+
professionals
Sign Up for our Newsletter
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Handling Streaming Data with Apache Kafka – A First Look
Analytics Vidhya
JUNE 21, 2022
The post Handling Streaming Data with Apache Kafka – A First Look appeared first on Analytics Vidhya. Streaming Data is generated continuously, by multiple data sources say, sensors, server logs, stock prices, etc. These records are usually small and in the order […].
Apache Kafka 379
379 
Introduction to Apache Kafka: Fundamentals and Working
Analytics Vidhya
DECEMBER 30, 2022
All these sites use some event streaming tool to monitor user activities. […]. The post Introduction to Apache Kafka: Fundamentals and Working appeared first on Analytics Vidhya.
Apache Kafka 315
315 
Supernovas, Black Holes and Streaming Data
databricks
AUGUST 12, 2024
The blog explores data streams from NASA satellites using Apache Kafka and Databricks. It demonstrates ingestion and transformation with Delta Live Tables in SQL and AI/BI-powered analysis of supernova events.
Apache Kafka 290
290 
Complex Event Processing (CEP)
Dataconomy
MARCH 11, 2025
Complex Event Processing (CEP) is at the forefront of modern analytics, enabling organizations to extract valuable insights from vast streams of real-time data. As industries evolve, the ability to process and respond to events in the moment becomes mission-critical. What is Complex Event Processing (CEP)?
Apache Kafka 91
91 
Maximizing your event-driven architecture investments: Unleashing the power of Apache Kafka with IBM Event Automation
IBM Journey to AI blog
FEBRUARY 12, 2024
Recognizing the need to harness real-time data, businesses are increasingly turning to event-driven architecture (EDA) as a strategic approach to stay ahead of the curve. At the forefront of this event-driven revolution is Apache Kafka, the widely recognized and dominant open-source technology for event streaming.
Apache Kafka 100
100 
Apache Kafka and Apache Flink: An open-source match made in heaven
IBM Journey to AI blog
NOVEMBER 3, 2023
Apache Kafka and Apache Flink working together Anyone who is familiar with the stream processing ecosystem is familiar with Apache Kafka: the de-facto enterprise standard for open-source event streaming. With Apache Kafka, you get a raw stream of events from everything that is happening within your business.
Apache Kafka 79
79 
What If We Could Rebuild Kafka from Scratch?
Hacker News
APRIL 24, 2025
The last few days I spent some time digging into the recently announced KIP-1150 ("Diskless Kafka"), as well AutoMQs Kafka fork, tightly integrating Apache Kafka and object storage, such as S3. Separating storage and compute and object store support would be table stakes, but what else should be there?
Apache Kafka 119
119 
Estimating Pi with Kafka streams
Hacker News
MAY 8, 2024
Recently I wanted to learn a bit about Apache Kafka. It is often used as a way to do event sourcing (or similar message-driven architectures). An “add-on” to.
Apache Kafka 113
113 Apache Kafka 113
113 
Accelerate your speed of business with IBM Event Automation
IBM Journey to AI blog
MAY 9, 2023
There are thousands of business events flowing through every part of your organization every day. These events can provide a wealth of information about what’s actually happening across your business at any moment in time. But operationalizing business events is challenging.
Apache Kafka 110
110 
Apache Kafka use cases: Driving innovation across diverse industries
IBM Journey to AI blog
SEPTEMBER 4, 2024
Apache Kafka is an open-source , distributed streaming platform that allows developers to build real-time, event-driven applications. With Apache Kafka, developers can build applications that continuously use streaming data records and deliver real-time experiences to users. How does Apache Kafka work?
Apache Kafka 58
58 
Show HN: Turn Kafka into an MQTT Broker for IoT with Zilla
Hacker News
DECEMBER 18, 2023
multi-protocol, event-native proxy. Securely interface web apps, IoT clients, and microservices to Apache Kafka® via declaratively defined, stateless APIs. A multi-protocol, event-native proxy. Securely interface web apps, IoT clients, and microservices to Apache Kafka® via declaratively defined, stateless APIs.
Apache Kafka 106
106 Apache Kafka 106
106 
Hybrid Vs. Multi-Cloud: 5 Key Comparisons in Kafka Architectures
Smart Data Collective
AUGUST 17, 2022
You can safely use an Apache Kafka cluster for seamless data movement from the on-premise hardware solution to the data lake using various cloud services like Amazon’s S3 and others. 5 Key Comparisons in Different Apache Kafka Architectures. 5 Key Comparisons in Different Apache Kafka Architectures.
Apache Kafka 130
130 
Streaming Machine Learning Without a Data Lake
ODSC - Open Data Science
MAY 31, 2023
Be sure to check out his talk, “ Apache Kafka for Real-Time Machine Learning Without a Data Lake ,” there! The combination of data streaming and machine learning (ML) enables you to build one scalable, reliable, but also simple infrastructure for all machine learning tasks using the Apache Kafka ecosystem.
Data Lakes 131
131 
The winning combination for real-time insights: Messaging and event-driven architecture
IBM Journey to AI blog
APRIL 2, 2024
A messaging queue technology is essential for businesses to stay afloat, but building out event-driven architecture fueled by messaging might just be your x-factor. However, IBM MQ and Apache Kafka can sometimes be viewed as competitors, taking each other on in terms of speed, availability, cost and skills.
Apache Kafka 75
75 
Real-time artificial intelligence and event processing
IBM Journey to AI blog
NOVEMBER 29, 2023
By leveraging AI for real-time event processing, businesses can connect the dots between disparate events to detect and respond to new trends, threats and opportunities. AI and event processing: a two-way street An event-driven architecture is essential for accelerating the speed of business.
Event-driven architecture (EDA) enables a business to become more aware of everything that’s happening, as it’s happening
IBM Journey to AI blog
JANUARY 8, 2024
In modern enterprises, where operations leave a massive digital footprint, business events allow companies to become more adaptable and able to recognize and respond to opportunities or threats as they occur. Teams want more visibility and access to events so they can reuse and innovate on the work of others.
EDA 92
92 
Data sips and bites: An evening of data insights
Dataconomy
JULY 29, 2024
Hosted at one of Mindspace’s coworking locations, the event was a convergence of insightful talks and professional networking. Mindspace , a global coworking and flexible office provider with over 45 locations worldwide, including 13 in Germany, offered a conducive environment for this knowledge-sharing event.
Apache Kafka 91
91 
Real-Time Sentiment Analysis with Kafka and PySpark
Towards AI
FEBRUARY 29, 2024
Within this article, we will explore the significance of these pipelines and utilise robust tools such as Apache Kafka and Spark to manage vast streams of data efficiently. Apache Kafka Apache Kafka is a distributed event streaming platform used for building real-time data pipelines and streaming applications.
Apache Kafka 142
142 
Level up your Kafka applications with schemas
IBM Journey to AI blog
NOVEMBER 21, 2023
Apache Kafka is a well-known open-source event store and stream processing platform and has grown to become the de facto standard for data streaming. Apache Kafka transfers data without validating the information in the messages. Event Streams on IBM Cloud provides a Schema Registry as part of its Enterprise plan.
Apache Kafka 116
116 
Event-driven architecture, Kafka and CDPs: Joining internal infrastructure with your tech stack
Twilio Segment
SEPTEMBER 7, 2021
Event streaming platforms such as Apache Kafka are gaining in importance across all industries. In this article we'll discuss the benefits Apache Kafka implementations can gain from pairing it with a CDP.
Apache Kafka 40
40 Apache Kafka 40
40 Why your event-driven architecture needs advanced event governance
IBM Journey to AI blog
AUGUST 22, 2024
Event-driven architecture (EDA) has become more crucial for organizations that want to strengthen their competitive advantage through real-time data processing and responsiveness. In recognizing the benefits of event-driven architectures, many companies have turned to Apache Kafka for their event streaming needs.
EDA 40
40 
Big Data – Lambda or Kappa Architecture?
Data Science Blog
JUNE 27, 2023
In this representation, there is a separate store for events within the speed layer and another store for data loaded during batch processing. It is important to note that in the Lambda architecture, the serving layer can be omitted, allowing batch processing and event streaming to remain separate entities.
Big Data 130
130 
Real-time fraud detection using AWS serverless and machine learning services
AWS Machine Learning Blog
MARCH 10, 2023
We show how you can apply this approach to various data streaming and event-driven architectures, depending on the desired outcome and actions to take to prevent fraud (such as alert the user about the fraud or flag the transaction for additional review). The user can then choose to take action to prevent further abuse.
Machine Learning 143
143 
Apache Flink for all: Making Flink consumable across all areas of your business
IBM Journey to AI blog
AUGUST 29, 2024
Event-driven businesses across all industries thrive on real-time data, enabling companies to act on events as they happen rather than after the fact. This is where Apache Flink shines, offering a powerful solution to harness the full potential of an event-driven business model through efficient computing and processing capabilities.
Apache Kafka 45
45 
Getting started with Kafka client metrics
IBM Journey to AI blog
MARCH 14, 2024
Apache Kafka stands as a widely recognized open source event store and stream processing platform. One key advantage of opting for managed Kafka services is the delegation of responsibility for broker and operational metrics, allowing users to focus solely on metrics specific to applications. Take the next step.
Apache Kafka 106
106 
Building a Pizza Delivery Service with a Real-Time Analytics Stack
ODSC - Open Data Science
JUNE 1, 2023
To understand what it means, we should start by thinking of the world in terms of events, where an event is a thing that happens. And we are going to take those events, become aware of them, and understand them. Stores events in a durable manner so that downstream components can process them.
Analytics 98
98 
IBM continues to support OpenSource AsyncAPI in breaking the boundaries of event driven architectures
IBM Journey to AI blog
JULY 12, 2024
IBM® Event Automation’s event endpoint management capability makes it easy to describe and document your Kafka topics (event sources) according to the open source AsyncAPI Specification. Why is this important? Our AsycnAPI applicability is broadening in our implementation.
Apache Kafka 40
40 
Big data engineering simplified: Exploring roles of distributed systems
Data Science Dojo
JULY 24, 2023
Unlike traditional batch processing, where data is processed in fixed intervals, stream processing enables organizations to gain insights and respond to events as they happen in real-time.
Big Data 195
195 
Streaming Data Pipelines: What Are They and How to Build One
Precisely
DECEMBER 28, 2023
Streaming data pipelines, by extension, offer an architecture capable of handling large volumes of data, accommodating millions of events in near real time. One very popular platform is Apache Kafka , a powerful open-source tool used by thousands of companies. How do streaming data pipelines work?
Five scalability pitfalls to avoid with your Kafka application
IBM Journey to AI blog
NOVEMBER 9, 2023
Apache Kafka is a high-performance, highly scalable event streaming platform. To unlock Kafka’s full potential, you need to carefully consider the design of your application. It’s all too easy to write Kafka applications that perform poorly or eventually hit a scalability brick wall.
Apache Kafka 78
78 
How to Unlock Real-Time Analytics with Snowflake?
phData
MAY 3, 2024
How Snowflake Helps Achieve Real-Time Analytics Snowflake is the ideal platform to achieve real-time analytics for several reasons, but two of the biggest are its ability to manage concurrency due to the multi-cluster architecture of Snowflake and its robust connections to 3rd party tools like Kafka. Looking for additional help?
Apache Kafka 52
52 
Bundesliga Match Facts Shot Speed – Who fires the hardest shots in the Bundesliga?
AWS Machine Learning Blog
NOVEMBER 3, 2023
This process comprises two key components: event data and optical tracking data. Event data collection entails gathering the fundamental building blocks of the game. For the precision needed in shot speed calculations, we must ensure that the ball’s position aligns precisely with the moment of the event.
AWS 128
128 
11 Open-Source Data Engineering Tools Every Pro Should Use
ODSC - Open Data Science
FEBRUARY 6, 2024
Apache Kafka For data engineers dealing with real-time data, Apache Kafka is a game-changer. Interested in attending an ODSC event? Learn more about our upcoming events here. Each platform offers unique features and benefits, making it vital for data engineers to understand their differences.
Use streaming ingestion with Amazon SageMaker Feature Store and Amazon MSK to make ML-backed decisions in near-real time
AWS Machine Learning Blog
APRIL 19, 2023
Streaming ingestion – An Amazon Kinesis Data Analytics for Apache Flink application backed by Apache Kafka topics in Amazon Managed Streaming for Apache Kafka (MSK) (Amazon MSK) calculates aggregated features from a transaction stream, and an AWS Lambda function updates the online feature store.
ML 98
98 
How Thomson Reuters delivers personalized content subscription plans at scale using Amazon Personalize
AWS Machine Learning Blog
JANUARY 6, 2023
As the users are interacting with TR’s applications, they generate clickstream events, which are published into Amazon Kinesis Data Streams. Then the events are ingested into TR’s centralized streaming platform, which is built on top of Amazon Managed Streaming for Kafka (Amazon MSK).
AWS 98
98 
Anomaly detection in streaming time series data with online learning using Amazon Managed Service for Apache Flink
AWS Machine Learning Blog
SEPTEMBER 11, 2024
It initially sources input time series data from Amazon Managed Streaming for Apache Kafka (Amazon MSK) using this live stream for model training. He is a clean code advocate and a subject-matter expert on server-less and event-driven architecture. You can follow him on linkedin, syedfurqhan Nirmal Kumar is Sr.
AWS 119
119 
Pictures and Highlights from ODSC Europe 2023
ODSC - Open Data Science
JULY 22, 2023
Expo Hall ODSC events are more than just data science training and networking events. Thank you to everyone who attended for making this event possible, and showing once again why we do what we do — connecting the greater data science community together to push the industry forward. What’s next?
Apache Kafka 52
52 
Know Before You Go: Precisely at Confluent’s Current 2023
Precisely
SEPTEMBER 12, 2023
Precisely data integrity solutions fuel your Confluent and Apache Kafka streaming data pipelines with trusted data that has maximum accuracy, consistency, and context and we’re ready to share more with you at the upcoming Current 2023. Let’s cover some additional information to know before attending.
Data Silos 52
52 
How Netflix Applies Big Data Across Business Verticals: Insights and Strategies
Pickl AI
SEPTEMBER 18, 2024
It utilises Amazon Web Services (AWS) as its main data lake, processing over 550 billion events daily—equivalent to approximately 1.3 Data in Motion Technologies like Apache Kafka facilitate real-time processing of events and data, allowing Netflix to respond swiftly to user interactions and operational needs.
Big Data 52
52 
Watch the Top ODSC Europe 2023 Virtual Sessions Here
ODSC - Open Data Science
JULY 14, 2023
The session participants will learn the theory behind compound sparsity, state-of-the-art techniques, and how to apply it in practice using the Neural Magic platform.
Training Models on Streaming Data [Practical Guide]
The MLOps Blog
FEBRUARY 5, 2023
Streaming data is a continuous flow of information and a foundation of event-driven architecture software model” – RedHat Enterprises around the world are becoming dependent on data more than ever. A streaming data pipeline is an enhanced version which is able to handle millions of events in real-time at scale.
Bringing Mainframe Machine and Log Data into Your ITOA Environment
Precisely
MARCH 16, 2023
Tools like Splunk, Elastic, and Apache Kafka play a central role in IT operations analytics (ITOA). Alongside a leading ITOA platform like Splunk, Elastic, or Kafka, Ironstream helps your organization improve security, lower costs, and pass compliance audits with flying colors.
Apache Kafka 52
52 
Let's personalize your content