phData

NOVEMBER 28, 2023

In July 2023, Matillion launched their fully SaaS platform called Data Productivity Cloud, aiming to create a future-ready, everyone-ready, and AI-ready environment that companies can easily adopt and start automating their data pipelines coding, low-coding, or even no-coding at all. Why Does it Matter?

Data Warehouse 52

Data Warehouse Data Pipeline ETL Azure 52

phData

MAY 3, 2024

What is Apache Kafka, and How is it Used in Building Real-time Data Pipelines? It is capable of handling high-volume and high-velocity data. It can deliver a high volume of data with latency as low as two milliseconds. Its use cases range from real-time analytics, fraud detection, messaging, and ETL pipelines.

Apache Kafka 52

Apache Kafka Analytics Analytics ETL 52

phData

JULY 12, 2023

In this blog, we’ll explore how Matillion Jobs can simplify the data transformation process by allowing users to visualize the data flow of a job from start to finish. What is Matillion ETL? Whether you’re new to Matillion or just looking to improve your ETL skills, keep reading to learn more!

SQL 52

SQL ETL Database Data Pipeline 52

phData

APRIL 21, 2023

In this blog, we’ll explore how Matillion Jobs can simplify the data transformation process by allowing users to visualize the data flow of a job from start to finish. With that, let’s dive in What is Matillion ETL? Read Components These are the components that define the source of data that is to be transformed.

SQL 52

SQL ETL Database Data Pipeline 52

AWS Machine Learning Blog

SEPTEMBER 18, 2024

Context In early 2023, Zeta’s machine learning (ML) teams shifted from traditional vertical teams to a more dynamic horizontal structure, introducing the concept of pods comprising diverse skill sets. Though it’s worth mentioning that Airflow isn’t used at runtime as is usual for extract, transform, and load (ETL) tasks.

AWS 118

AWS Machine Learning Machine Learning ML 118

ODSC - Open Data Science

SEPTEMBER 29, 2023

It truly is an all-in-one data lake solution. HPCC Systems and Spark also differ in that they work with distinct parts of the big data pipeline. Spark is more focused on data science, ingestion, and ETL, while HPCC Systems focuses on ETL and data delivery and governance. Save the Date!

Data Lakes 52

Data Lakes Clustering Big Data Big Data 52

The MLOps Blog

JANUARY 23, 2023

Dolt LakeFS Delta Lake Pachyderm Git-like versioning Database tool Data lake Data pipelines Experiment tracking Integration with cloud platforms Integrations with ML tools Examples of data version control tools in ML DVC Data Version Control DVC is a version control system for data and machine learning teams.

ML 52

ML ML Data Lakes Machine Learning 52

IBM Journey to AI blog

JULY 17, 2023

The next generation of Db2 Warehouse SaaS and Netezza SaaS on AWS fully support open formats such as Parquet and Iceberg table format, enabling the seamless combination and sharing of data in watsonx.data without the need for duplication or additional ETL. Savings may vary depending on configurations, workloads and vendor.

AI 74

AI AI Machine Learning Machine Learning 74

The MLOps Blog

JANUARY 26, 2024

Reference table for which technologies to use for your FTI pipelines for each ML system. Related article How to Build ETL Data Pipelines for ML See also MLOps and FTI pipelines testing Once you have built an ML system, you have to operate, maintain, and update it.

Machine Learning 72

Machine Learning Machine Learning ML ML 72

Flipboard

MARCH 21, 2025

Traditionally, answering this question would involve multiple data exports, complex extract, transform, and load (ETL) processes, and careful data synchronization across systems. Users can write data to managed RMS tables using Iceberg APIs, Amazon Redshift, or Zero-ETL ingestion from supported data sources.

SQL 134

SQL Data Analyst Data Warehouse AWS 134

phData

SEPTEMBER 27, 2024

In transitional modeling, we’d add new atoms: Subject: Customer#1234 Predicate: hasEmailAddress Object: "john.new@example.com" Timestamp: 2023-07-24T10:00:00Z The old email address atoms are still there, giving us a complete history of how to contact John. Both persistent staging and data lakes involve storing large amounts of raw data.

Data Modeling 52

Data Modeling Data Models Apache Kafka Data Lakes 52

phData

AUGUST 22, 2024

Consider a data pipeline that detects its own failures, diagnoses the issue, and recommends the fix—all automatically. This is the potential of self-healing pipelines, and this blog explores how to implement them using dbt, Snowflake Cortex , and GitHub Actions. This output is less helpful.

SQL 52

SQL Data Quality Python Data Warehouse 52

phData

JULY 18, 2023

The Ultimate Modern Data Stack Migration Guide phData Marketing July 18, 2023 This guide was co-written by a team of data experts, including Dakota Kelley, Ahmad Aburia, Sam Hall, and Sunny Yan. Imagine a world where all of your data is organized, easily accessible, and routinely leveraged to drive impactful outcomes.

Data Warehouse 52

Data Warehouse Analytics Analytics SQL 52

Precisely

JANUARY 6, 2025

If youre like many modern organizations, you may be managing data across an increasingly complex landscape of on-premises platforms, cloud services, and legacy systems and facing challenges in doing so. According to the 2023 Gartner Cloud End-User Behavior Survey, 81% of respondents use multiple cloud providers.

Data Pipeline 59

Data Pipeline ETL Analytics Analytics 59

AWS Machine Learning Blog

MARCH 27, 2025

model_id = "anthropic.claude-3-5-sonnet-20240620-v1:0" # Load the prompt from a file (showed and explained later in the blog) with open('prompt.txt', 'r') as file: data = file.read() def callBedrock(body): # Format the request payload using the model's native structure. The same ETL workflows were running fine before the upgrade.

AWS 107

AWS ETL ML ML 107