Dataconomy

JULY 29, 2024

Talks and insights Mikhail Epikhin: Navigating the processor landscape for Apache Kafka Mikhail Epikhin began the session by sharing his team’s research on optimizing Managed Service for Apache Kafka. He addressed challenges in data replication and offered solutions to optimize these processes.

Apache Kafka 91

Apache Kafka Data Pipeline Data Warehouse ETL 91

Pickl AI

JULY 8, 2024

Summary: This blog explains how to build efficient data pipelines, detailing each step from data collection to final delivery. Introduction Data pipelines play a pivotal role in modern data architecture by seamlessly transporting and transforming raw data into valuable insights.

Data Pipeline 52

Data Pipeline Data Quality Database Apache Kafka 52

phData

MAY 3, 2024

Leveraging real-time analytics to make informed decisions is the golden standard for virtually every business that collects data. If you have the Snowflake Data Cloud (or are considering migrating to Snowflake ), you’re a blog away from taking a step closer to real-time analytics.

Apache Kafka 52

Apache Kafka Analytics Analytics ETL 52

Women in Big Data

OCTOBER 9, 2024

These procedures are central to effective data management and crucial for deploying machine learning models and making data-driven decisions. The success of any data initiative hinges on the robustness and flexibility of its big data pipeline. What is a Data Pipeline?

Big Data 59

Big Data Big Data Apache Kafka Data Pipeline 59

IBM Journey to AI blog

AUGUST 29, 2024

Flink jobs, designed to process continuous data streams, are key to making this possible. How Apache Flink enhances real-time event-driven businesses Imagine a retail company that can instantly adjust its inventory based on real-time sales data pipelines. But that’s just the beginning.

Apache Kafka 45

Apache Kafka Hadoop ETL Data Pipeline 45

ODSC - Open Data Science

FEBRUARY 6, 2024

Spark offers a versatile range of functionalities, from batch processing to stream processing, making it a comprehensive solution for complex data challenges. Apache Kafka For data engineers dealing with real-time data, Apache Kafka is a game-changer.

Data Engineer 52

Data Engineer Data Engineering Data Engineering Data Engineering 52

Pickl AI

NOVEMBER 4, 2024

Data Engineering is designing, constructing, and managing systems that enable data collection, storage, and analysis. It involves developing data pipelines that efficiently transport data from various sources to storage solutions and analytical tools. ETL is vital for ensuring data quality and integrity.

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

Precisely

SEPTEMBER 12, 2023

As a proud member of the Connect with Confluent program , we help organizations going through digital transformation and IT infrastructure modernization break down data silos and power their streaming data pipelines with trusted data.

Data Silos 52

Data Silos Apache Kafka Data Pipeline Data Quality 52

The MLOps Blog

MARCH 15, 2023

In this post, you will learn about the 10 best data pipeline tools, their pros, cons, and pricing. A typical data pipeline involves the following steps or processes through which the data passes before being consumed by a downstream process, such as an ML model training process.

Data Pipeline 59

Data Pipeline ETL SQL Data Quality 59

Precisely

JULY 21, 2023

Organizations face increasing demands for real-time processing and analysis of large volumes of data. Used by more than 75% of the Fortune 500, Apache Kafka has emerged as a powerful open source data streaming platform to meet these challenges. This is where Confluent steps in.

Apache Kafka 52

Apache Kafka Data Silos Data Pipeline Analytics 52

Pickl AI

DECEMBER 9, 2024

Introduction Big Data continues transforming industries, making it a vital asset in 2025. The global Big Data Analytics market, valued at $307.51 Turning raw data into meaningful insights helps businesses anticipate trends, understand consumer behaviour, and remain competitive in a rapidly changing world.

Big Data 52

Big Data Big Data Hadoop Apache Kafka 52

Pickl AI

JULY 25, 2024

Real-Time Processing Also known as stream processing, this method involves continuously ingesting data as it becomes available. This is essential for applications that demand immediate insights, such as fraud detection or real-time analytics. The Importance of Data Ingestion It plays a crucial role in the data lifecycle.

Apache Kafka 52

Apache Kafka Data Lakes Data Warehouse Data Quality 52

The MLOps Blog

FEBRUARY 5, 2023

This pipeline facilitates the smooth, automated flow of information, preventing many problems that enterprises face, such as data corruption, conflict, and duplication of data entries. A streaming data pipeline is an enhanced version which is able to handle millions of events in real-time at scale.

Machine Learning 72

Machine Learning Machine Learning Data Pipeline Apache Kafka 72

Dataconomy

FEBRUARY 23, 2023

This involves creating data validation rules, monitoring data quality, and implementing processes to correct any errors that are identified. Creating data pipelines and workflows Data engineers create data pipelines and workflows that enable data to be collected, processed, and analyzed efficiently.

Big Data 91

Big Data Big Data Data Engineer Data Engineering 91

Pickl AI

JULY 30, 2024

Flow-Based Programming : NiFi employs a flow-based programming model, allowing users to create complex data flows using simple drag-and-drop operations. This visual representation simplifies the design and management of data pipelines.

ETL 52

ETL Data Lakes Big Data Big Data 52

DagsHub

OCTOBER 23, 2024

With proper unstructured data management, you can write validation checks to detect multiple entries of the same data. Continuous learning: In a properly managed unstructured data pipeline, you can use new entries to train a production ML model, keeping the model up-to-date.

Machine Learning 52

Machine Learning Machine Learning Data Lakes AI 52

phData

SEPTEMBER 27, 2024

Technologies like Apache Kafka, often used in modern CDPs, use log-based approaches to stream customer events between systems in real-time. It enables advanced analytics, makes debugging your marketing automations easier, provides natural audit trails for compliance, and allows for flexible, evolving customer data models.

Data Modeling 52

Data Modeling Data Models Apache Kafka Data Lakes 52

The MLOps Blog

AUGUST 11, 2023

Today different stages exist within ML pipelines built to meet technical, industrial, and business requirements. This section delves into the common stages in most ML pipelines, regardless of industry or business function. 1 Data Ingestion (e.g., Apache Kafka, Amazon Kinesis) 2 Data Preprocessing (e.g.,

ML 52

ML ML Machine Learning Machine Learning 52

Pickl AI

MARCH 19, 2025

Summary: Data engineering tools streamline data collection, storage, and processing. Tools like Python, SQL, Apache Spark, and Snowflake help engineers automate workflows and improve efficiency. Learning these tools is crucial for building scalable data pipelines.

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

Pickl AI

MARCH 13, 2025

Two of the most popular message brokers are RabbitMQ and Apache Kafka. In this blog, we will explore RabbitMQ vs Kafka, their key differences, and when to use each. Kafka excels in real-time data streaming and scalability. RabbitMQ uses a push-based model, while Kafka follows a pull-based model.

Apache Kafka 52

Apache Kafka Big Data Big Data Data Pipeline 52