Dataconomy

MARCH 11, 2025

Apache Flink: A powerful open-source framework for distributed stream processing with an emphasis on event-driven applications. Apache Kafka: Vital for creating real-time data pipelines and streaming applications. IBM InfoSphere Streams: Provides tailored solutions for real-time data analytics and processing.

Apache Kafka 91

Apache Kafka Machine Learning Machine Learning Data Mining 91

Dataconomy

JULY 29, 2024

Talks and insights Mikhail Epikhin: Navigating the processor landscape for Apache Kafka Mikhail Epikhin began the session by sharing his team’s research on optimizing Managed Service for Apache Kafka. He addressed challenges in data replication and offered solutions to optimize these processes.

Apache Kafka 91

Apache Kafka Data Pipeline Data Warehouse ETL 91

Towards AI

FEBRUARY 29, 2024

Real-time data streaming pipelines play a crutial role in achieving this objective. Within this article, we will explore the significance of these pipelines and utilise robust tools such as Apache Kafka and Spark to manage vast streams of data efficiently.

Apache Kafka 142

Apache Kafka SQL Clustering Data Pipeline 142

Data Science Dojo

MARCH 9, 2023

It requires minimal operational maintenance and allows for rapid development, resulting in significant cost savings and reduced development time for data-focused developers and engineers. Handling too many data sources can become overwhelming, especially with complex schemas. Debugging and troubleshooting can also be challenging.

Apache Kafka 214

Apache Kafka Azure Data Science Data Pipeline 214

Pickl AI

JULY 8, 2024

Summary: This blog explains how to build efficient data pipelines, detailing each step from data collection to final delivery. Introduction Data pipelines play a pivotal role in modern data architecture by seamlessly transporting and transforming raw data into valuable insights.

Data Pipeline 52

Data Pipeline Data Quality Database Apache Kafka 52

Women in Big Data

OCTOBER 9, 2024

These procedures are central to effective data management and crucial for deploying machine learning models and making data-driven decisions. The success of any data initiative hinges on the robustness and flexibility of its big data pipeline. What is a Data Pipeline?

Big Data 59

Big Data Big Data Apache Kafka Data Pipeline 59

IBM Journey to AI blog

AUGUST 29, 2024

Flink jobs, designed to process continuous data streams, are key to making this possible. How Apache Flink enhances real-time event-driven businesses Imagine a retail company that can instantly adjust its inventory based on real-time sales data pipelines. But that’s just the beginning.

Apache Kafka 45

Apache Kafka Hadoop ETL Data Pipeline 45

phData

MAY 3, 2024

How Snowflake Helps Achieve Real-Time Analytics Snowflake is the ideal platform to achieve real-time analytics for several reasons, but two of the biggest are its ability to manage concurrency due to the multi-cluster architecture of Snowflake and its robust connections to 3rd party tools like Kafka. Looking for additional help?

Apache Kafka 52

Apache Kafka Analytics Analytics ETL 52

IBM Journey to AI blog

MARCH 14, 2024

Apache Kafka stands as a widely recognized open source event store and stream processing platform. It has evolved into the de facto standard for data streaming, as over 80% of Fortune 500 companies use it. All major cloud providers provide managed data streaming services to meet this growing demand.

Apache Kafka 106

Apache Kafka Data Pipeline 106

ODSC - Open Data Science

FEBRUARY 6, 2024

Spark offers a versatile range of functionalities, from batch processing to stream processing, making it a comprehensive solution for complex data challenges. Apache Kafka For data engineers dealing with real-time data, Apache Kafka is a game-changer.

Data Engineer 52

Data Engineer Data Engineering Data Engineering Data Engineering 52

The MLOps Blog

MARCH 15, 2023

In this post, you will learn about the 10 best data pipeline tools, their pros, cons, and pricing. A typical data pipeline involves the following steps or processes through which the data passes before being consumed by a downstream process, such as an ML model training process.

Data Pipeline 59

Data Pipeline ETL SQL Data Quality 59

Precisely

SEPTEMBER 12, 2023

As a proud member of the Connect with Confluent program , we help organizations going through digital transformation and IT infrastructure modernization break down data silos and power their streaming data pipelines with trusted data.

Data Silos 52

Data Silos Apache Kafka Data Pipeline Data Quality 52

Pickl AI

NOVEMBER 4, 2024

Effective data governance enhances quality and security throughout the data lifecycle. What is Data Engineering? Data Engineering is designing, constructing, and managing systems that enable data collection, storage, and analysis. They are crucial in ensuring data is readily available for analysis and reporting.

Data Engineer 52

Data Engineer Data Engineering Data Engineering Data Engineering 52

Precisely

JULY 21, 2023

Organizations face increasing demands for real-time processing and analysis of large volumes of data. Used by more than 75% of the Fortune 500, Apache Kafka has emerged as a powerful open source data streaming platform to meet these challenges. This is where Confluent steps in.

Apache Kafka 52

Apache Kafka Data Silos Data Pipeline Analytics 52

Pickl AI

DECEMBER 9, 2024

What is Apache Kafka, and Why is it Used? Apache Kafka is a distributed messaging system that handles real-time data streaming for building scalable, fault-tolerant data pipelines. Explain the CAP theorem and its relevance in Big Data systems. Have You Ever Worked With Real-Time Data Streams?

Big Data 52

Big Data Big Data Hadoop Apache Kafka 52

Dataconomy

FEBRUARY 23, 2023

This involves creating data validation rules, monitoring data quality, and implementing processes to correct any errors that are identified. Creating data pipelines and workflows Data engineers create data pipelines and workflows that enable data to be collected, processed, and analyzed efficiently.

Big Data 91

Big Data Big Data Data Engineer Data Engineering 91

Pickl AI

JULY 25, 2024

Data Ingestion Tools To facilitate the process, various tools and technologies are available. These tools can automate data collection, transformation, and loading processes, making it easier for organisations to manage their data pipelines effectively. What are Some Popular Data Ingestion Tools?

Apache Kafka 52

Apache Kafka Data Lakes Data Warehouse Data Quality 52

The MLOps Blog

FEBRUARY 5, 2023

This pipeline facilitates the smooth, automated flow of information, preventing many problems that enterprises face, such as data corruption, conflict, and duplication of data entries. A streaming data pipeline is an enhanced version which is able to handle millions of events in real-time at scale. Happy Learning!

Machine Learning 72

Machine Learning Machine Learning Data Pipeline Apache Kafka 72

Pickl AI

JULY 30, 2024

Flow-Based Programming : NiFi employs a flow-based programming model, allowing users to create complex data flows using simple drag-and-drop operations. This visual representation simplifies the design and management of data pipelines.

ETL 52

ETL Data Lakes Big Data Big Data 52

DagsHub

OCTOBER 23, 2024

With proper unstructured data management, you can write validation checks to detect multiple entries of the same data. Continuous learning: In a properly managed unstructured data pipeline, you can use new entries to train a production ML model, keeping the model up-to-date.

Machine Learning 52

Machine Learning Machine Learning AI AI 52

DagsHub

APRIL 7, 2024

Image generated with Midjourney In today’s fast-paced world of data science, building impactful machine learning models relies on much more than selecting the best algorithm for the job. Data scientists and machine learning engineers need to collaborate to make sure that together with the model, they develop robust data pipelines.

Machine Learning 59

Machine Learning Machine Learning ML ML 59

phData

SEPTEMBER 27, 2024

Technologies like Apache Kafka, often used in modern CDPs, use log-based approaches to stream customer events between systems in real-time. Both persistent staging and data lakes involve storing large amounts of raw data. Give your customer data a scrapbook where it can collect memories in their raw, unaltered form.

Data Modeling 52

Data Modeling Data Models Apache Kafka Data Lakes 52

DagsHub

JANUARY 14, 2025

It would help to improve the process in future by creating a clear audit trail of how duplicate records are identified and handled throughout the data pipeline. For Streaming data , use windowed deduplication techniques to identify duplicates within a specific time frame.

Machine Learning 52

Machine Learning Machine Learning Clustering Algorithm 52

The MLOps Blog

AUGUST 11, 2023

Today different stages exist within ML pipelines built to meet technical, industrial, and business requirements. This section delves into the common stages in most ML pipelines, regardless of industry or business function. 1 Data Ingestion (e.g., Apache Kafka, Amazon Kinesis) 2 Data Preprocessing (e.g.,

ML 52

ML ML Machine Learning Machine Learning 52

Pickl AI

MARCH 13, 2025

Two of the most popular message brokers are RabbitMQ and Apache Kafka. In this blog, we will explore RabbitMQ vs Kafka, their key differences, and when to use each. IoT applications : Collecting and distributing sensor data from connected devices. Thats where message brokers come in.

Apache Kafka 52

Apache Kafka Big Data Big Data Data Pipeline 52