Data Science Blog

JUNE 27, 2023

In practical implementation, the Kappa architecture is commonly deployed using Apache Kafka or Kafka-based tools. Applications can directly read from and write to Kafka or an alternative message queue tool. It offers the advantage of having a single ETL platform to develop and maintain.

Big Data 130

Big Data Big Data Apache Kafka Database 130

Pickl AI

JULY 30, 2024

ETL (Extract, Transform, Load) Processes Apache NiFi can streamline ETL processes by extracting data from multiple sources, transforming it into the desired format, and loading it into target systems such as data warehouses or databases. Its visual interface allows users to design complex ETL workflows with ease.

ETL 52

ETL Data Lakes Big Data Big Data 52

AWS Machine Learning Blog

OCTOBER 9, 2024

To use this feature, you can write rules or analyzers and then turn on anomaly detection in AWS Glue ETL. To learn more about the beta offering, see Anomaly detection in streaming time series data with online learning using Amazon Managed Service for Apache Flink.

AWS 88

AWS ML ML Data Quality 88

ODSC - Open Data Science

OCTOBER 9, 2024

Efficient Incremental Processing with Apache Iceberg and Netflix Maestro Dimensional Data Modeling in the Modern Era Building Big Data Workflows: NiFi, Hive, Trino, & Zeppelin An Introduction to Data Contracts From Data Mess to Data Mesh — Data Management in the Age of Big Data and Gen AI Introduction to Containers for Data Science / Data Engineering (..)

AI 40

AI AI Apache Kafka Machine Learning 40

Mlearning.ai

MAY 16, 2023

ETL Design Pattern The ETL (Extract, Transform, Load) design pattern is a commonly used pattern in data engineering. ETL Design Pattern Here is an example of how the ETL design pattern can be used in a real-world scenario: A healthcare organization wants to analyze patient data to improve patient outcomes and operational efficiency.

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

Pickl AI

JULY 8, 2024

Tools such as Python’s Pandas library, Apache Spark, or specialised data cleaning software streamline these processes, ensuring data integrity before further transformation. This step often involves: ETL Processes: Extracting, transforming, and loading data into a target system.

Data Pipeline 52

Data Pipeline Data Quality Database Apache Kafka 52

Pickl AI

JULY 24, 2023

Data warehousing and ETL (Extract, Transform, Load) procedures frequently involve batch processing. Utilising data streaming platforms such as Apache Kafka, Apache Flink, or Apache Spark Streaming, data is gathered from many sources and processed in real-time or close to real-time.

Internet of Things 52

Internet of Things Apache Kafka ETL Azure 52

The MLOps Blog

MARCH 15, 2023

Typical examples include: Airbyte Talend Apache Kafka Apache Beam Apache Nifi While getting control over the process is an ideal position an organization wants to be in, the time and effort needed to build such systems are immense and frequently exceeds the license fee of a commercial offering.

Data Pipeline 59

Data Pipeline ETL SQL Data Quality 59

Pickl AI

AUGUST 9, 2024

Data Integration Tools Technologies such as Apache NiFi and Talend help in the seamless integration of data from various sources into a unified system for analysis. Understanding ETL (Extract, Transform, Load) processes is vital for students. Knowledge of RESTful APIs and authentication methods is essential.

Big Data 52

Big Data Big Data Big Data Analytics Big Data Analytics 52

DagsHub

OCTOBER 23, 2024

Apache Kafka Apache Kafka is a distributed event streaming platform for real-time data pipelines and stream processing. is similar to the traditional Extract, Transform, Load (ETL) process. Data Processing Tools These tools are essential for handling large volumes of unstructured data. Unstructured.io

Machine Learning 52

Machine Learning Machine Learning AI AI 52

AWS Machine Learning Blog

JANUARY 6, 2023

TR used AWS Glue DataBrew and AWS Batch jobs to perform the extract, transform, and load (ETL) jobs in the ML pipelines, and SageMaker along with Amazon Personalize to tailor the recommendations. Then the events are ingested into TR’s centralized streaming platform, which is built on top of Amazon Managed Streaming for Kafka (Amazon MSK).

AWS 79

AWS Data Warehouse ML ML 79

phData

SEPTEMBER 27, 2024

Technologies like Apache Kafka, often used in modern CDPs, use log-based approaches to stream customer events between systems in real-time. In traditional ETL (Extract, Transform, Load) processes in CDPs, staging areas were often temporary holding pens for data. But the power of logs doesn’t stop there.

Data Models 52

Data Models Data Modeling Apache Kafka Data Lakes 52

DagsHub

APRIL 7, 2024

Flexibility: Its use cases are wider than just machine learning; for example, we can use it to set up ETL pipelines. Also, while it is not a streaming solution, we can still use it for such a purpose if combined with systems such as Apache Kafka. Miscellaneous Workflows are created as directed acyclic graphs (DAGs).

Machine Learning 59

Machine Learning Machine Learning ML ML 59

Dataconomy

FEBRUARY 23, 2023

This involves working with various tools and technologies, such as ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes, to move data from its source to its destination. By creating efficient data pipelines and workflows, data engineers enable organizations to make data-driven decisions quickly and accurately.

Big Data 91

Big Data Big Data Data Engineer Data Engineering 91