Dataconomy

FEBRUARY 23, 2023

Data engineers play a crucial role in managing and processing big data. They are responsible for designing, building, and maintaining the infrastructure and tools needed to manage and process large volumes of data effectively. What is data engineering?

Big Data 91

Big Data Big Data Data Engineer Data Engineering 91

IBM Journey to AI blog

JANUARY 5, 2023

The right data architecture can help your organization improve data quality because it provides the framework that determines how data is collected, transported, stored, secured, used and shared for business intelligence and data science use cases. Reduce data duplication and fragmentation.

Data Quality 52

Data Quality Data Lakes Data Warehouse Big Data 52

Alation

SEPTEMBER 7, 2021

With Snowflake, data stewards have a choice to leverage Snowflake’s governance policies. First, stewards are dependent on data warehouse admins to provide information and to create and edit enforcement policies in Snowflake. Alation’s deep data profiling helps data scientists and analysts get important data profiling insights.

Data Governance 59

Data Governance Data Scientist Data Quality Data Profiling 59

The MLOps Blog

MAY 17, 2023

This article explores the importance of ETL pipelines in machine learning, a hands-on example of building ETL pipelines with a popular tool, and suggests the best ways for data engineers to enhance and sustain their pipelines. It comprises three main areas: Landing area, Staging area, and Data Warehouse area.

ETL 59

ETL Data Pipeline ML ML 59

Alation

MAY 24, 2022

Prime examples of this in the data catalog include: Trust Flags — Allow the data community to endorse, warn, and deprecate data to signal whether data can or can’t be used. Data Profiling — Statistics such as min, max, mean, and null can be applied to certain columns to understand its shape.

Data Quality 98

Data Quality Data Governance ETL Data Observability 98

DagsHub

AUGUST 23, 2024

A data quality standard might specify that when storing client information, we must always include email addresses and phone numbers as part of the contact details. If any of these is missing, the client data is considered incomplete. Data Profiling Data profiling involves analyzing and summarizing data (e.g.

Data Quality 52

Data Quality Data Governance Machine Learning Machine Learning 52

Alation

JANUARY 13, 2022

Data mesh forgoes technology edicts and instead argues for “decentralized data ownership” and the need to treat “data as a product”. Gartner on Data Fabric. Moreover, data catalogs play a central role in both data fabric and data mesh. Let’s turn our attention now to data mesh.

Data Lakes 59

Data Lakes Data Governance Data Quality Data Warehouse 59