phData

AUGUST 2, 2024

Snowflake AI Data Cloud is one of the most powerful platforms, including storage services supporting complex data. Integrating Snowflake with dbt adds another layer of automation and control to the data pipeline. Snowflake stored procedures and dbt Hooks are essential to modern data engineering and analytics workflows.

Data Pipeline 52

Data Pipeline Python Database SQL 52

phData

MARCH 1, 2023

The first one we want to talk about is the Toolkit SQL analyze command. When customers are looking to perform a migration, one of the first things that needs to occur is an assessment of the level of effort to migrate existing data definition language (DDL) and data markup language (DML).

SQL 52

SQL Data Pipeline Data Quality Database 52

AWS Machine Learning Blog

SEPTEMBER 18, 2024

It simplifies feature access for model training and inference, significantly reducing the time and complexity involved in managing data pipelines. Additionally, Feast promotes feature reuse, so the time spent on data preparation is reduced greatly. The following figure shows schema definition and model which reference it.

AWS 120

AWS Machine Learning Machine Learning ML 120

phData

SEPTEMBER 7, 2023

Over the last month, we’ve been heavily focused on adding additional support for SQL translations to our SQL Translations tool. Specifically, we’ve been introducing fixes and features for our Microsoft SQL Server to Snowflake translation. Let’s dive in! Let’s take a look at some of the more interesting translations.

SQL 52

SQL Data Profiling Data Pipeline Database 52

phData

JUNE 26, 2023

While many of our customers leverage our UI for tools like our SQL Translation or Privilege Audit tooling, there are limitations when it comes to using a UI. You wouldn’t want to pay someone (or perform yourself) to manually copy/paste each file into a browser window and copy/paste the translated SQL back.

SQL 52

SQL Data Profiling Data Pipeline Data Governance 52

Tableau

OCTOBER 8, 2021

With its LookML modeling language, Looker provides a unique, modern approach to define governed and reusable data models to build a trusted foundation for analytics. Connecting directly to this semantic layer will help give customers access to critical business data in a safe, governed manner. Direct connection to Google BigQuery.

Tableau 98

Tableau Analytics Analytics Machine Learning 98

phData

AUGUST 22, 2024

Consider a data pipeline that detects its own failures, diagnoses the issue, and recommends the fix—all automatically. This is the potential of self-healing pipelines, and this blog explores how to implement them using dbt, Snowflake Cortex , and GitHub Actions.

SQL 52

SQL Data Quality Python Data Warehouse 52

Pickl AI

AUGUST 1, 2023

The primary goal of Data Engineering is to transform raw data into a structured and usable format that can be easily accessed, analyzed, and interpreted by Data Scientists, analysts, and other stakeholders. Future of Data Engineering The Data Engineering market will expand from $18.2

Data Engineer 52

Data Engineer Data Engineering Data Engineering Data Engineering 52

Pickl AI

JUNE 7, 2024

It is a process for moving and managing data from various sources to a central data warehouse. This process ensures that data is accurate, consistent, and usable for analysis and reporting. Definition and Explanation of the ETL Process ETL is a data integration method that combines data from multiple sources.

ETL 40

ETL Data Quality Data Pipeline Data Warehouse 40

AWS Machine Learning Blog

SEPTEMBER 24, 2024

An optional CloudFormation stack to deploy a data pipeline to enable a conversation analytics dashboard. Choose an option for allowing unredacted logs for the Lambda function in the data pipeline. This allows you to control which IAM principals are allowed to decrypt the data and view it. For testing, choose yes.

AWS 101

AWS AI AI Analytics 101

phData

OCTOBER 25, 2024

It’s common to have terabytes of data in most data warehouses, data quality monitoring is often challenging and cost-intensive due to dependencies on multiple tools and eventually ignored. This results in poor credibility and data consistency after some time, leading businesses to mistrust the data pipelines and processes.

Data Quality 52

Data Quality Data Pipeline Data Governance Database 52

phData

MAY 25, 2023

The June 2021 release of Power BI Desktop introduced Custom SQL queries to Snowflake in DirectQuery mode. While the loss of certain DAX functions is definitely a shortcoming that we hope Microsoft will address in the near future, the impact of these lost DAX functions is not necessarily as big as you would expect.

Power BI 84

Power BI Analytics Analytics Azure 84

Mlearning.ai

JUNE 16, 2023

Generative AI can be used to automate the data modeling process by generating entity-relationship diagrams or other types of data models and assist in UI design process by generating wireframes or high-fidelity mockups. GPT-4 Data Pipelines: Transform JSON to SQL Schema Instantly Blockstream’s public Bitcoin API.

AI 52

AI AI Data Analysis Data Analysis 52

phData

MARCH 1, 2024

Sample CSV files (download files here ) Step 1: Load Sample CSV Files Into the Internal Stage Location Open the SQL worksheet and create a stage if it doesn’t exist. From the homepage: Data > Databases > Select your database/schema and select stages. Go back to the SQL worksheet and verify if the files exist.

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

phData

SEPTEMBER 5, 2023

Let’s briefly look at the key components and their roles in this process: Azure Data Factory (ADF) : ADF will serve as our data orchestration and integration platform. It enables us to create, schedule, and monitor the data pipeline, ensuring seamless movement of data between the various sources and destinations.

Azure 52

Azure Analytics Analytics Data Pipeline 52

Mlearning.ai

JULY 10, 2023

In case of complex data pipelines, a combination of Materialized Views, Stored Procedures, and Scheduled Queries could be a better choice than to solely rely on Scheduled Queries by itself. To create a Scheduled Query, the initial step is to ensure your SQL is accurately entered in the Query Editor.

SQL 52

SQL Database Database Administration Data Lakes 52

DagsHub

APRIL 7, 2024

Image generated with Midjourney In today’s fast-paced world of data science, building impactful machine learning models relies on much more than selecting the best algorithm for the job. Data scientists and machine learning engineers need to collaborate to make sure that together with the model, they develop robust data pipelines.

Machine Learning 59

Machine Learning Machine Learning ML ML 59

DagsHub

OCTOBER 23, 2024

Here’s the structured equivalent of this same data in tabular form: With structured data, you can use query languages like SQL to extract and interpret information. In contrast, such traditional query languages struggle to interpret unstructured data. This text has a lot of information, but it is not structured.

Machine Learning 52

Machine Learning Machine Learning Data Lakes AI 52

Alation

APRIL 14, 2022

Data pipeline orchestration. Support for languages and SQL. Moving/integrating data in the cloud/data exploration and quality assessment. It’s not a simple definition. Migration leaders would be wise to enable all the enhancements a cloud environment offers, including: Special requirements for AI/ML.

Data Governance 59

Data Governance ML ML Cloud Data 59

Alation

JANUARY 17, 2023

A legacy data stack usually refers to the traditional relational database management system (RDBMS), which uses a structured query language (SQL) to store and process data. While an RDBMS can still be used in a modern data stack, it is not as common because it is not as well-suited for managing big data.

Data Warehouse 52

Data Warehouse ETL Tableau Cloud Data 52

Flipboard

MARCH 21, 2025

To establish trust between the data producers and data consumers, SageMaker Catalog also integrates the data quality metrics and data lineage events to track and drive transparency in data pipelines. Create a SageMaker Unified Studio domain and three projects using the SQL analytics project profile.

SQL 140

SQL Data Analyst Data Warehouse AWS 140

phData

JULY 18, 2023

Support for Numerous Data Sources: Fivetran supports over 200 data sources, including popular databases, applications, and cloud platforms like Salesforce, Google Analytics, SQL Server, Snowflake, and many more. Additionally, unsupported data sources can be integrated using Fivetran’s cloud function connectors.

Data Warehouse 52

Data Warehouse Analytics Analytics Cloud Data 52

phData

AUGUST 17, 2023

Data pipeline orchestration tools are designed to automate and manage the execution of data pipelines. These tools help streamline and schedule data movement and processing tasks, ensuring efficient and reliable data flow. What are Orchestration Tools?

Data Engineer 52

Data Engineer Data Engineering Data Engineering Data Engineering 52

phData

SEPTEMBER 27, 2024

Some modern CDPs are starting to incorporate these concepts, allowing for more flexible and evolving customer data models. It also requires a shift in how we query our customer data. Instead of simple SQL queries, we often need to use more complex temporal query languages or rely on derived views for simpler querying.

Data Models 52

Data Models Data Modeling Apache Kafka Data Lakes 52

AWS Machine Learning Blog

OCTOBER 24, 2024

The agent can generate SQL queries using natural language questions using a database schema DDL (data definition language for SQL) and execute them against a database instance for the database tier. Make sure to add a semicolon after the end of the SQL statement generated. Generate UI and backend code with LLMs.

AWS 96

AWS SQL Database AI 96

Dataconomy

MARCH 19, 2025

Data science is an interdisciplinary field that utilizes advanced analytics techniques to extract meaningful insights from vast amounts of data. This helps facilitate data-driven decision-making for businesses, enabling them to operate more efficiently and identify new opportunities.

Data Science 91

Data Science Citizen Data Scientist Data Scientist Machine Learning 91