The MLOps Blog

MAY 17, 2023

However, efficient use of ETL pipelines in ML can help make their life much easier. This article explores the importance of ETL pipelines in machine learning, a hands-on example of building ETL pipelines with a popular tool, and suggests the best ways for data engineers to enhance and sustain their pipelines.

ETL 59

ETL Data Pipeline ML ML 59

Alation

MAY 24, 2022

Prime examples of this in the data catalog include: Trust Flags — Allow the data community to endorse, warn, and deprecate data to signal whether data can or can’t be used. Data Profiling — Statistics such as min, max, mean, and null can be applied to certain columns to understand its shape.

Data Quality 98

Data Quality Data Governance ETL Data Observability 98

The MLOps Blog

MARCH 15, 2023

This is a difficult decision at the onset, as the volume of data is a factor of time and keeps varying with time, but an initial estimate can be quickly gauged by analyzing this aspect by running a pilot. Also, the industry best practices suggest performing a quick data profiling to understand the data growth.

Data Pipeline 59

Data Pipeline ETL SQL Data Quality 59