phData

SEPTEMBER 2, 2024

In this blog, we will cover the best practices for developing jobs in Matillion, an ETL/ELT tool built specifically for cloud database platforms. Use of Python Component The Python component, including using Jython to connect to various databases, should not be used for resource-intensive data processing.

ETL 52

ETL Data Warehouse SQL Database 52

phData

AUGUST 9, 2023

With ELT, we first extract data from source systems, then load the raw data directly into the data warehouse before finally applying transformations natively within the data warehouse. This is unlike the more traditional ETL method, where data is transformed before loading into the data warehouse.

ETL 52

ETL Data Warehouse Cloud Data Big Data 52

phData

MARCH 8, 2023

Fivetran is an automated data integration platform that offers a convenient solution for businesses to consolidate and sync data from disparate data sources. With over 160 data connectors available, Fivetran makes it easy to move data out of, into, and across any cloud data platform in the market.

Data Warehouse 96

Data Warehouse Data Engineer Data Engineering Data Engineering 96

phData

APRIL 29, 2024

Understanding Fivetran Fivetran is a popular Software-as-a-Service platform that enables users to automate the movement of data and ETL processes across diverse sources to a target destination. For a longer overview, along with insights and best practices, please feel free to jump back to the previous blog.

SQL 52

SQL Data Warehouse Azure Cloud Data 52

phData

JULY 17, 2023

Over the past few decades, the corporate data landscape has changed significantly. The shift from on-premise databases and spreadsheets to the modern era of cloud data warehouses and AI/ LLMs has transformed what businesses can do with data. This is where Fivetran and the Modern Data Stack come in.

Data Warehouse 52

Data Warehouse Data Pipeline Cloud Data ETL 52

Alation

JANUARY 17, 2023

Data ingestion/integration services. Reverse ETL tools. Data orchestration tools. These tools are used to manage big data, which is defined as data that is too large or complex to be processed by traditional means. How Did the Modern Data Stack Get Started? A Note on the Shift from ETL to ELT.

Data Warehouse 52

Data Warehouse ETL Tableau Cloud Data 52

phData

JUNE 14, 2023

In recent years, data engineering teams working with the Snowflake Data Cloud platform have embraced the continuous integration/continuous delivery (CI/CD) software development process to develop data products and manage ETL/ELT workloads more efficiently.

Data Pipeline 52

Data Pipeline Database SQL Data Engineer 52

phData

FEBRUARY 7, 2024

The Snowflake Data Cloud is a leading cloud data platform that provides various features and services for data storage, processing, and analysis. A new feature that Snowflake offers is called Snowpark, which provides an intuitive library for querying and processing data at scale in Snowflake.

Python 52

Python ML ML SQL 52

phData

JANUARY 4, 2023

With Snowflake, organizations can be data consumers, data providers, or both. Complete SQL Database No need to learn new tools as Snowflake supports the tools millions of business users already know how to use today.

Data Warehouse 52

Data Warehouse SQL Database ETL 52

phData

FEBRUARY 5, 2024

The tool converts the templated configuration into a set of SQL commands that are executed against the target Snowflake environment. Replicate can interact with a wide variety of databases, data warehouses, and data lakes (on-premise or based in the cloud). It is also a helpful tool for learning a new SQL dialect.

SQL 52

SQL Database Data Quality Data Warehouse 52

phData

NOVEMBER 30, 2023

The Snowflake Data Cloud is a leading cloud data platform that provides various features and services for data storage, processing, and analysis. A new feature that Snowflake offers is the ability to create alerts based on data in Snowflake.

SQL 52

SQL Cloud Data Database ETL 52

Mlearning.ai

JULY 10, 2023

In my 7 years of Data Science journey, I’ve been exposed to a number of different databases including but not limited to Oracle Database, MS SQL, MySQL, EDW, and Apache Hadoop. Some of the other ways are creating a table 1) using the command line in Google Cloud console, 2) using the APIs, or 3) from Vertex AI Workbench.

SQL 52

SQL Database Apache Hadoop Data Science 52

phData

SEPTEMBER 4, 2024

ThoughtSpot is a cloud-based AI-powered analytics platform that uses natural language processing (NLP) or natural language query (NLQ) to quickly query results and generate visualizations without the user needing to know any SQL or table relations. Suppose your business requires more robust capabilities across your technology stack.

Analytics 52

Analytics Analytics SQL ETL 52

Alation

APRIL 4, 2023

Profiling delivers a birds-eye view of the statistics of the data, such as minimum, maximum, median, and null values. This empowers users to judge data’s quality and fitness for purpose quickly. This expanded connector to Databricks Unity Catalog does just that, delivering to joint customers a comprehensive view of all cloud data.

DataOps 52

DataOps Data Engineer Data Engineering Data Engineering 52

Alation

OCTOBER 27, 2022

Few actors in the modern data stack have inspired the enthusiasm and fervent support as dbt. This data transformation tool enables data analysts and engineers to transform, test and document data in the cloud data warehouse. But what does this mean from a practitioner perspective?

Data Analyst 52

Data Analyst Data Scientist Analytics Analytics 52

phData

FEBRUARY 14, 2023

This may result in data inconsistency when UPDATE and DELETE operations are performed on the target database. For simple and quick replication to Snowflake, Matillion offers Data Loader, a SaaS tool that migrates data from various data sources. Replication of calculated values is not supported during Change Processing.

Data Warehouse 59

Data Warehouse Azure AWS Database 59

IBM Journey to AI blog

JULY 17, 2023

The next generation of Db2 Warehouse SaaS and Netezza SaaS on AWS fully support open formats such as Parquet and Iceberg table format, enabling the seamless combination and sharing of data in watsonx.data without the need for duplication or additional ETL. Savings may vary depending on configurations, workloads and vendor.

AI 74

AI AI Machine Learning Machine Learning 74

Alation

JANUARY 25, 2022

On the policy front, a feature like Policy Center empowers users to enforce and track policies at scale; this ensures that people use data compliantly, and organizations are prepared for compliance audits. For instance, technical power users can explore the actual data through Compose , the intelligent SQL editor.

DataOps 52

DataOps SQL ML ML 52

Dataversity

APRIL 8, 2021

Business intelligence (BI) tools transform the unprocessed data into meaningful and actionable insight. BI tools analyze the data and convert them […]. Click to learn more about author Piyush Goel. What is a BI tool? Which BI tool is best for your organization?

Business Intelligence 75

Business Intelligence Business Intelligence ETL Data Quality 75

phData

SEPTEMBER 27, 2024

Some modern CDPs are starting to incorporate these concepts, allowing for more flexible and evolving customer data models. It also requires a shift in how we query our customer data. Instead of simple SQL queries, we often need to use more complex temporal query languages or rely on derived views for simpler querying.

Data Models 52

Data Models Data Modeling Apache Kafka Data Lakes 52

phData

OCTOBER 28, 2024

Modern low-code/no-code ETL tools allow data engineers and analysts to build pipelines seamlessly using a drag-and-drop and configure approach with minimal coding. One such option is the availability of Python Components in Matillion ETL, which allows us to run Python code inside the Matillion instance.

Python 52

Python ETL AWS Database 52

Flipboard

DECEMBER 18, 2023

Amazon Redshift powers data-driven decisions for tens of thousands of customers every day with a fully managed, AI-powered cloud data warehouse, delivering the best price-performance for your analytics workloads. Learn more about the AWS zero-ETL future with newly launched AWS databases integrations with Amazon Redshift.

AWS 139

AWS Data Warehouse ETL SQL 139

phData

OCTOBER 17, 2023

The story is all too common – a business user requests some data, the data team creates/prioritizes a ticket, and said ticket is completed after some number of months (or weeks if you’re lucky) – just to have the data be wrong, and the whole process starts again. Those are scary for data teams to change.

ETL 52

ETL Data Pipeline Data Engineer Data Engineering 52

phData

JULY 18, 2023

With the birth of cloud data warehouses, data applications, and generative AI , processing large volumes of data faster and cheaper is more approachable and desired than ever. This typically results in long-running ETL pipelines that cause decisions to be made on stale or old data.

Data Warehouse 52

Data Warehouse Analytics Analytics SQL 52

phData

APRIL 20, 2023

Hashed PKs were introduced as a means of eliminating the bottleneck encountered by most database sequence generators, making this DV pattern ideal for customers prioritizing data loading performance and using data warehouse automation tools. If data is present, Tasks runs SQL to push it to the raw data vault objects.

Data Warehouse 72

Data Warehouse Data Governance Clustering Database 72