The Data Administration Newsletter

OCTOBER 17, 2023

Where exactly within an organization does the primary responsibility lie for ensuring that a data pipeline project generates data of high quality, and who exactly holds that responsibility? Who is accountable for ensuring that the data is accurate? Is it the data engineers? The data scientists?

Data Pipeline 52

Data Pipeline Data Quality Data Governance Data Analyst 52

Pickl AI

NOVEMBER 4, 2024

Summary: The fundamentals of Data Engineering encompass essential practices like data modelling, warehousing, pipelines, and integration. Understanding these concepts enables professionals to build robust systems that facilitate effective data management and insightful analysis. What is Data Engineering?

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

Dataconomy

FEBRUARY 12, 2024

This shift leverages the capabilities of modern data warehouses, enabling faster data ingestion and reducing the complexities associated with traditional transformation-heavy ETL processes. These platforms provide a unified view of data, enabling businesses to derive insights from diverse datasets efficiently. Image credit ) 5.

ETL 195

ETL Data Governance Machine Learning Machine Learning 195

IBM Journey to AI blog

FEBRUARY 26, 2024

The financial services industry has been in the process of modernizing its data governance for more than a decade. But as we inch closer to global economic downturn, the need for top-notch governance has become increasingly urgent. That’s why data pipeline observability is so important.

Data Pipeline 112

Data Pipeline Data Engineer Data Engineering Data Engineering 112

Pickl AI

JULY 25, 2023

Unfolding the difference between data engineer, data scientist, and data analyst. Data engineers are essential professionals responsible for designing, constructing, and maintaining an organization’s data infrastructure. Read more to know.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

Alation

FEBRUARY 17, 2023

This past week, I had the pleasure of hosting Data Governance for Dummies author Jonathan Reichental for a fireside chat , along with Denise Swanson , Data Governance lead at Alation. Can you have proper data management without establishing a formal data governance program?

Data Governance 52

Data Governance Data Quality Data Analyst Data Pipeline 52

Alation

MAY 31, 2023

The Data Governance & Information Quality Conference (DGIQ) is happening soon — and we’ll be onsite in San Diego from June 5-9. If you’re not familiar with DGIQ, it’s the world’s most comprehensive event dedicated to, you guessed it, data governance and information quality. The best part?

Data Governance 52

Data Governance DataOps Data Pipeline Business Intelligence 52

ODSC - Open Data Science

JANUARY 18, 2024

Data engineering is a rapidly growing field, and there is a high demand for skilled data engineers. If you are a data scientist, you may be wondering if you can transition into data engineering. In this blog post, we will discuss how you can become a data engineer if you are a data scientist.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

Smart Data Collective

OCTOBER 17, 2022

Engineering teams, in particular, can quickly get overwhelmed by the abundance of information pertaining to competition data, new product and service releases, market developments, and industry trends, resulting in information anxiety. Explosive data growth can be too much to handle. Data pipeline maintenance.

Big Data 119

Big Data Big Data Data Engineer Data Engineering 119

Alation

MAY 16, 2023

But with the sheer amount of data continually increasing, how can a business make sense of it? Robust data pipelines. What is a Data Pipeline? A data pipeline is a series of processing steps that move data from its source to its destination. The answer?

Data Pipeline 40

Data Pipeline Data Governance Data Lakes Data Warehouse 40

IBM Journey to AI blog

APRIL 20, 2023

This trust depends on an understanding of the data that inform risk models: where does it come from, where is it being used, and what are the ripple effects of a change? Moreover, banks must stay in compliance with industry regulations like BCBS 239, which focus on improving banks’ risk data aggregation and risk reporting capabilities.

Database 86

Database Data Engineer Data Engineering Data Engineering 86

Pickl AI

OCTOBER 11, 2023

What is Data Observability? It is the practice of monitoring, tracking, and ensuring data quality, reliability, and performance as it moves through an organization’s data pipelines and systems. Data quality tools help maintain high data quality standards. Tools Used in Data Observability?

Data Observability 52

Data Observability Data Quality Data Pipeline Data Governance 52

Alation

SEPTEMBER 22, 2022

This new partnership will unify governed, quality data into a single view, granting all stakeholders total visibility into pipelines and providing them with a superior ability to make data-driven decisions. For people to understand and trust data, they need to see it in context. Data Pipeline Strategy.

Data Pipeline 52

Data Pipeline Data Quality Data Governance Data Engineer 52

Alation

JUNE 21, 2022

The audience grew to include data scientists (who were even more scarce and expensive) and their supporting resources (e.g., After that came data governance , privacy, and compliance staff. Power business users and other non-purely-analytic data citizens came after that. Data engineers want to catalog data pipelines.

DataOps 52

DataOps Data Scientist Data Quality Data Pipeline 52

Alation

DECEMBER 23, 2021

The elf teams used data engineering to improve gift matching and deployed big data to scale the naughty and nice list long ago , before either approach was even considered within our warmer climes. The best data was discovered, experts were identified, and conversations were starting. Make data governance an asset.

Data Governance 59

Data Governance Data Pipeline Tableau Big Data 59

Dataversity

SEPTEMBER 28, 2022

Data engineering is a fascinating and fulfilling career – you are at the helm of every business operation that requires data, and as long as users generate data, businesses will always need data engineers. The journey to becoming a successful data engineer […].

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

ODSC - Open Data Science

FEBRUARY 5, 2025

This May, were heading to Boston for ODSC East 2025, where data scientists, AI engineers, and industry leaders will gather to explore the latest advancements in AI, machine learning, and data engineering. This is your chance to gain insights from some of the brightest minds in the industry.

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

AWS Machine Learning Blog

FEBRUARY 13, 2024

Let’s demystify this using the following personas and a real-world analogy: Data and ML engineers (owners and producers) – They lay the groundwork by feeding data into the feature store Data scientists (consumers) – They extract and utilize this data to craft their models Data engineers serve as architects sketching the initial blueprint.

AWS 131

AWS ML ML Machine Learning 131

Alation

MARCH 22, 2022

Accenture calls it the Intelligent Data Foundation (IDF), and it’s used by dozens of enterprises with very complex data landscapes and analytic requirements. Simply put, IDF standardizes data engineering processes. How the IDF Supports a Smarter Data Pipeline. Take a look at figure 1 below.

DataOps 52

DataOps Data Pipeline Data Engineer Data Engineering 52

The MLOps Blog

JUNE 27, 2023

Alignment to other tools in the organization’s tech stack Consider how well the MLOps tool integrates with your existing tools and workflows, such as data sources, data engineering platforms, code repositories, CI/CD pipelines, monitoring systems, etc. For example, neptune.ai

Machine Learning 134

Machine Learning Machine Learning ML ML 134

Precisely

AUGUST 15, 2024

Key Players in AI Development Enterprises increasingly rely on AI to automate and enhance their data engineering workflows, making data more ready for building, training, and deploying AI applications. Data engineers focus on tasks like cleansing and managing data, ensuring its quality and readiness for AI applications.

AI 64

AI AI Data Quality Data Engineer 64

phData

MAY 28, 2024

With Cortex, business analysts, data engineers, and developers can easily incorporate Predictive and Generative AI into their workflows using simple SQL commands and intuitive interfaces. As an example, check out our blog post on implementing a data engineering pipeline for Sentiment Analysis with dbt.

Machine Learning 52

Machine Learning Machine Learning Data Engineering Data Engineering 52

Alation

APRIL 18, 2023

This, in turn, helps them to build new data pipelines, solutions, and products, or clean up the data that’s there. It bears mentioning data profiling has evolved tremendously. Business data stewards benefit from having a breakdown of the patterns that exist in the data.

Data Profiling 52

Data Profiling Data Quality Data Governance Data Pipeline 52

Pickl AI

OCTOBER 17, 2024

Automation Automation plays a pivotal role in streamlining ETL processes, reducing the need for manual intervention, and ensuring consistent data availability. By automating key tasks, organisations can enhance efficiency and accuracy, ultimately improving the quality of their data pipelines.

ETL 52

ETL Data Warehouse Data Quality Data Governance 52

Alation

DECEMBER 7, 2021

Data teams use Bigeye’s data observability platform to detect data quality issues and ensure reliable data pipelines. If there is an issue with the data or data pipeline, the data team is immediately alerted, enabling them to proactively address the issue.

Data Quality 52

Data Quality Data Pipeline Data Observability Data Profiling 52

IBM Journey to AI blog

AUGUST 12, 2024

Consequently, AIOps is designed to harness data and insight generation capabilities to help organizations manage increasingly complex IT stacks. MLOps involves a series of steps that help ensure the seamless deployability, reproducibility, scalability and observability of ML models.

Big Data 106

Big Data Big Data ML ML 106

DataRobot Blog

OCTOBER 3, 2017

Enterprise data architects, data engineers, and business leaders from around the globe gathered in New York last week for the 3-day Strata Data Conference , which featured new technologies, innovations, and many collaborative ideas. 2) When data becomes information, many (incremental) use cases surface.

Data Lakes 52

Data Lakes Azure Data Pipeline Hadoop 52

phData

OCTOBER 25, 2024

It’s common to have terabytes of data in most data warehouses, data quality monitoring is often challenging and cost-intensive due to dependencies on multiple tools and eventually ignored. This results in poor credibility and data consistency after some time, leading businesses to mistrust the data pipelines and processes.

Data Quality 52

Data Quality Data Pipeline Data Governance Database 52

IBM Journey to AI blog

DECEMBER 16, 2022

And because data assets within the catalog have quality scores and social recommendations, Alex has greater trust and confidence in the data she’s using for her decision-making recommendations. This is especially helpful when handling massive amounts of big data. Protected and compliant data.

Data Quality 52

Data Quality Data Governance Data Scientist Data Wrangling 52

ODSC - Open Data Science

APRIL 26, 2023

Tools such as the mentioned are critical for anyone interested in becoming a machine learning engineer. Data Engineer Data engineers are the authors of the infrastructure that stores, processes, and manages the large volumes of data an organization has.

Data Analyst 52

Data Analyst Machine Learning Machine Learning Power BI 52

DagsHub

AUGUST 23, 2024

We already know that a data quality framework is basically a set of processes for validating, cleaning, transforming, and monitoring data. Data Governance Data governance is the foundation of any data quality framework. It primarily caters to large organizations with complex data environments.

Data Quality 52

Data Quality Data Governance Machine Learning Machine Learning 52

IBM Journey to AI blog

JANUARY 5, 2023

The first generation of data architectures represented by enterprise data warehouse and business intelligence platforms were characterized by thousands of ETL jobs, tables, and reports that only a small group of specialized data engineers understood, resulting in an under-realized positive impact on the business.

Data Quality 52

Data Quality Data Lakes Data Warehouse Big Data 52

phData

AUGUST 2, 2024

Snowflake AI Data Cloud is one of the most powerful platforms, including storage services supporting complex data. Integrating Snowflake with dbt adds another layer of automation and control to the data pipeline. Snowflake stored procedures and dbt Hooks are essential to modern data engineering and analytics workflows.

Data Pipeline 52

Data Pipeline Python Database SQL 52

phData

FEBRUARY 14, 2023

Insurance companies often face challenges with data silos and inconsistencies among their legacy systems. To address these issues, they need a centralized and integrated data platform that serves as a single source of truth, preferably with strong data governance capabilities.

Data Governance 52

Data Governance Data Silos Predictive Analytics Data Scientist 52

phData

NOVEMBER 9, 2023

The main goal of a data mesh structure is to drive: Domain-driven ownership Data as a product Self-service infrastructure Federated governance One of the primary challenges that organizations face is data governance.

Data Warehouse 98

Data Warehouse Data Lakes Clustering Cloud Data 98

phData

MARCH 22, 2023

Top Use Cases of Snowpark With Snowpark, bringing business logic to data in the cloud couldn’t be easier. Transitioning work to Snowpark allows for faster ML deployment, easier scaling, and robust data pipeline development. ML Applications For data scientists, models can be developed in Python with common machine learning tools.

ML 52

ML ML Machine Learning Machine Learning 52

phData

AUGUST 10, 2023

In data vault implementations, critical components encompass the storage layer, ELT technology, integration platforms, data observability tools, Business Intelligence and Analytics tools, Data Governance , and Metadata Management solutions. The most important reason for using DBT in Data Vault 2.0

SQL 52

SQL Data Observability Data Quality Data Pipeline 52