ODSC - Open Data Science

FEBRUARY 19, 2025

The stepsinclude: Extraction : Data is collected from multiple sources (databases, APIs, flatfiles). Transformation : Data is cleaned, formatted, and enriched. Loading : The processed data is stored in data warehouses or datalakes. Data privacy & ethics: AI-driven ETL must adhere to governance frameworks.

ETL 52

ETL AI AI Data Warehouse 52

Pickl AI

JUNE 7, 2024

It is a process for moving and managing data from various sources to a central data warehouse. This process ensures that data is accurate, consistent, and usable for analysis and reporting. Definition and Explanation of the ETL Process ETL is a data integration method that combines data from multiple sources.

ETL 40

ETL Data Quality Data Pipeline Data Warehouse 40

Iguazio

JANUARY 22, 2024

Definitions: Foundation Models, Gen AI, and LLMs Before diving into the practice of productizing LLMs, let’s review the basic definitions of GenAI elements: Foundation Models (FMs) - Large deep learning models that are pre-trained with attention mechanisms on massive datasets. This helps cleanse the data.

Data Pipeline 52

Data Pipeline ML ML Data Warehouse 52

phData

JANUARY 19, 2024

Matillion’s Data Productivity Cloud is a versatile platform designed to increase the productivity of data teams. It provides a unified platform for creating and managing data pipelines that are effective for both coders and non-coders. Additional setup is typically optional.

Data Pipeline 52

Data Pipeline Data Warehouse ETL Azure 52

phData

JANUARY 10, 2024

Imagine you wanted to build a dbt project for your existing source data warehouse in your migration to Snowflake. You could leverage the data source tool to profile your source, apply a template against the generated metadata, and automatically create a dbt project with models for each table!

Data Warehouse 52

Data Warehouse Data Profiling Data Pipeline Database 52

phData

SEPTEMBER 5, 2023

Let’s briefly look at the key components and their roles in this process: Azure Data Factory (ADF) : ADF will serve as our data orchestration and integration platform. It enables us to create, schedule, and monitor the data pipeline, ensuring seamless movement of data between the various sources and destinations.

Azure 52

Azure Analytics Analytics Data Pipeline 52

phData

OCTOBER 25, 2024

It’s common to have terabytes of data in most data warehouses, data quality monitoring is often challenging and cost-intensive due to dependencies on multiple tools and eventually ignored. To assign the DMF to the table, we must first add a data metric schedule to the table CUSTOMERS.

Data Quality 52

Data Quality Data Pipeline Data Governance Database 52

phData

MARCH 1, 2024

This process introduces considerable time and effort into the overall data ingestion workflow, delaying the availability of data to end consumers. Fortunately, the client has opted for Snowflake Data Cloud as their target data warehouse.

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

phData

AUGUST 22, 2024

Consider a data pipeline that detects its own failures, diagnoses the issue, and recommends the fix—all automatically. This is the potential of self-healing pipelines, and this blog explores how to implement them using dbt, Snowflake Cortex , and GitHub Actions. This output is less helpful.

SQL 52

SQL Data Quality Python Data Warehouse 52

phData

MAY 25, 2023

One of the easiest ways for Snowflake to achieve this is to have analytics solutions query their data warehouse in real-time (also known as DirectQuery). These additional tools in the Power Platform open up more possible consumption of Snowflake data than there would be otherwise.

Power BI 84

Power BI Analytics Analytics Azure 84

Alation

APRIL 14, 2022

Data pipeline orchestration. Moving/integrating data in the cloud/data exploration and quality assessment. Similar to a data warehouse schema, this prep tool automates the development of the recipe to match. It’s not a simple definition. So how do you take full advantage of the cloud? Scheduling.

Data Governance 59

Data Governance ML ML Cloud Data 59

Alation

JANUARY 26, 2023

That’s why many organizations invest in technology to improve data processes, such as a machine learning data pipeline. However, data needs to be easily accessible, usable, and secure to be useful — yet the opposite is too often the case. Narrow the scope It’s tempting to mark huge swaths of data as critical.

Data Governance 52

Data Governance Data Engineering Data Engineer Data Engineering 52

Mlearning.ai

JULY 10, 2023

In case of complex data pipelines, a combination of Materialized Views, Stored Procedures, and Scheduled Queries could be a better choice than to solely rely on Scheduled Queries by itself. This allows you to use tools like BigQuery to query the data before it’s migrated to a native BigQuery table.

SQL 52

SQL Database Database Administration Data Lakes 52

phData

SEPTEMBER 27, 2024

All this raw data goes into your persistent stage. Then, if you later refine your definition of what constitutes an “engaged” customer, having the raw data in persistent staging allows for easy reprocessing of historical data with the new logic. Your customer data game will never be the same.

Data Modeling 52

Data Modeling Data Models Apache Kafka Data Lakes 52

The MLOps Blog

MARCH 15, 2023

ETL usually stands for “Extract, Transform and Load,” and it refers to a process in data warehousing. Sourcing the data In our case, the data was provided by our client, which was a product-based organization. The data pipelines can be scheduled as event-driven or be run at specific intervals the users choose.

AWS 52

AWS ETL ML ML 52

Mlearning.ai

MARCH 15, 2023

I have checked the AWS S3 bucket and Snowflake tables for a couple of days and the Data pipeline is working as expected. The scope of this article is quite big, we will exercise the core steps of data science, let's get started… Project Layout Here are the high-level steps for this project.

Python 52

Python AWS Exploratory Data Analysis Machine Learning 52

phData

NOVEMBER 9, 2023

This data mesh strategy combined with the end consumers of your data cloud enables your business to scale effectively, securely, and reliably without sacrificing speed-to-market. What is a Cloud Data Warehouse? For example, most data warehouse workloads peak during certain times, say during business hours.

Data Warehouse 98

Data Warehouse Data Lakes Clustering Cloud Data 98

AWS Machine Learning Blog

SEPTEMBER 18, 2024

It simplifies feature access for model training and inference, significantly reducing the time and complexity involved in managing data pipelines. Additionally, Feast promotes feature reuse, so the time spent on data preparation is reduced greatly. The following figure shows schema definition and model which reference it.

AWS 112

AWS Machine Learning Machine Learning ML 112

phData

JULY 18, 2023

With the birth of cloud data warehouses, data applications, and generative AI , processing large volumes of data faster and cheaper is more approachable and desired than ever. First up, let’s dive into the foundation of every Modern Data Stack, a cloud-based data warehouse.

Data Warehouse 52

Data Warehouse Analytics Analytics SQL 52

Flipboard

MARCH 21, 2025

Traditionally, answering this question would involve multiple data exports, complex extract, transform, and load (ETL) processes, and careful data synchronization across systems. The existing Data Catalog becomes the Default catalog (identified by the AWS account number) and is readily available in SageMaker Lakehouse.

SQL 134

SQL Data Analyst Data Warehouse AWS 134