phData

AUGUST 2, 2024

Snowflake AI Data Cloud is one of the most powerful platforms, including storage services supporting complex data. Integrating Snowflake with dbt adds another layer of automation and control to the data pipeline. Snowflake stored procedures and dbt Hooks are essential to modern data engineering and analytics workflows.

Data Pipeline 52

Data Pipeline Python Database SQL 52

phData

APRIL 29, 2024

Understanding Fivetran Fivetran is a popular Software-as-a-Service platform that enables users to automate the movement of data and ETL processes across diverse sources to a target destination. The phData team achieved a major milestone by successfully setting up a secure end-to-end data pipeline for a substantial healthcare enterprise.

SQL 52

SQL Data Warehouse Azure Cloud Data 52

phData

AUGUST 10, 2023

In data vault implementations, critical components encompass the storage layer, ELT technology, integration platforms, data observability tools, Business Intelligence and Analytics tools, Data Governance , and Metadata Management solutions. The most important reason for using DBT in Data Vault 2.0

SQL 52

SQL Data Observability Data Quality Data Pipeline 52

phData

AUGUST 5, 2024

Who should have access to sensitive data? How can my analysts discover where data is located? All of these questions describe a concept known as data governance. The Snowflake AI Data Cloud has built an entire blanket of features called Horizon, which tackles all of these questions and more.

Data Governance 52

Data Governance Data Quality Data Lakes ML 52

The MLOps Blog

JUNE 27, 2023

It sits between the data lake and cloud object storage, allowing you to version and control changes to data lakes at scale. LakeFS facilitates data reproducibility, collaboration, and data governance within the data lake environment. Flyte Flyte is a platform for orchestrating ML pipelines at scale.

Machine Learning 134

Machine Learning Machine Learning ML ML 134

phData

JUNE 26, 2023

While many of our customers leverage our UI for tools like our SQL Translation or Privilege Audit tooling, there are limitations when it comes to using a UI. You wouldn’t want to pay someone (or perform yourself) to manually copy/paste each file into a browser window and copy/paste the translated SQL back.

SQL 52

SQL Data Profiling Data Pipeline Data Governance 52

phData

DECEMBER 29, 2022

The phData Toolkit continues to have additions made to it as we work with customers to accelerate their migrations , build a data governance practice , and ensure quality data products are built. Some of the major improvements that have been made are within the data profiling and validation components of the Toolkit CLI.

SQL 52

SQL Database Database Administration Data Profiling 52

Tableau

OCTOBER 8, 2021

Connecting directly to this semantic layer will help give customers access to critical business data in a safe, governed manner. This partnership makes data more accessible and trusted. Our continued investments in connectivity with Google technologies help ensure your data is secure, governed, and scalable. .

Tableau 98

Tableau Analytics Analytics Machine Learning 98

IBM Journey to AI blog

AUGUST 4, 2023

Data democratization instead refers to the simplification of all processes related to data, from storage architecture to data management to data security. It also requires an organization-wide data governance approach, from adopting new types of employee training to creating new policies for data storage.

Data Lakes 73

Data Lakes AI AI Data Governance 73

Alation

APRIL 14, 2022

Semantics, context, and how data is tracked and used mean even more as you stretch to reach post-migration goals. This is why, when data moves, it’s imperative for organizations to prioritize data discovery. Data discovery is also critical for data governance , which, when ineffective, can actually hinder organizational growth.

Data Governance 59

Data Governance ML ML Cloud Data 59

phData

OCTOBER 25, 2024

It’s common to have terabytes of data in most data warehouses, data quality monitoring is often challenging and cost-intensive due to dependencies on multiple tools and eventually ignored. This results in poor credibility and data consistency after some time, leading businesses to mistrust the data pipelines and processes.

Data Quality 52

Data Quality Data Pipeline Data Governance Database 52

Pickl AI

OCTOBER 17, 2024

Automation Automation plays a pivotal role in streamlining ETL processes, reducing the need for manual intervention, and ensuring consistent data availability. By automating key tasks, organisations can enhance efficiency and accuracy, ultimately improving the quality of their data pipelines.

ETL 52

ETL Data Warehouse Data Quality Data Governance 52

phData

JUNE 20, 2024

This oftentimes leads to shadow IT processes and duplicated data pipelines. Data is siloed, and there is no singular source of truth but fragmented data spread across the organization. Establishing a data culture changes this paradigm. Promoting Data Literacy Snowflake is an accessible platform.

Data Governance 52

Data Governance Analytics Analytics Power BI 52

Pickl AI

JULY 3, 2023

Here are steps you can follow to pursue a career as a BI Developer: Acquire a solid foundation in data and analytics: Start by building a strong understanding of data concepts, relational databases, SQL (Structured Query Language), and data modeling.

Business Intelligence 52

Business Intelligence Business Intelligence SQL Data Visualization 52

phData

MAY 28, 2024

With Cortex, business analysts, data engineers, and developers can easily incorporate Predictive and Generative AI into their workflows using simple SQL commands and intuitive interfaces. This not only streamlines operations but also ensures consistency and reliability across the entire data processing lifecycle.

Machine Learning 52

Machine Learning Machine Learning Data Engineer Data Engineering 52

IBM Journey to AI blog

JANUARY 5, 2023

What does a modern data architecture do for your business? A modern data architecture like Data Mesh and Data Fabric aims to easily connect new data sources and accelerate development of use case specific data pipelines across on-premises, hybrid and multicloud environments.

Data Quality 52

Data Quality Data Lakes Data Warehouse Big Data 52

DagsHub

AUGUST 23, 2024

We already know that a data quality framework is basically a set of processes for validating, cleaning, transforming, and monitoring data. Data Governance Data governance is the foundation of any data quality framework. It primarily caters to large organizations with complex data environments.

Data Quality 52

Data Quality Data Governance Machine Learning Machine Learning 52

ODSC - Open Data Science

APRIL 26, 2023

Though scripted languages such as R and Python are at the top of the list of required skills for a data analyst, Excel is still one of the most important tools to be used. Because they are the most likely to communicate data insights, they’ll also need to know SQL, and visualization tools such as Power BI and Tableau as well.

Data Analyst 52

Data Analyst Machine Learning Machine Learning Power BI 52

ODSC - Open Data Science

JANUARY 18, 2024

This individual is responsible for building and maintaining the infrastructure that stores and processes data; the kinds of data can be diverse, but most commonly it will be structured and unstructured data. They’ll also work with software engineers to ensure that the data infrastructure is scalable and reliable.

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

Pickl AI

OCTOBER 17, 2024

Databases These are structured data collections managed by Database Management Systems (DBMS). Organisations often extract data from relational databases like MySQL, Oracle, or SQL Server to facilitate analysis. Choosing the right ETL tool enhances data management efficiency and supports organisational growth.

ETL 52

ETL Data Warehouse SQL Data Quality 52

Precisely

AUGUST 15, 2024

An increasing number of GenAI tools use large language models that automate key data engineering, governance, and master data management tasks. These tools can generate automated outputs including SQL and Python code, synthetic datasets, data visualizations, and predictions – significantly streamlining your data pipeline.

AI 64

AI AI Data Quality Data Engineer 64

phData

OCTOBER 12, 2023

When the data or pipeline configuration needs to be changed, tools like Fivetran and dbt reduce the time required to make the change, and increase the confidence your team can have around the change. These allow you to scale your pipelines quickly. Governance doesn’t have to be scary or preventative to your cloud data warehouse.

Data Warehouse 52

Data Warehouse Database Cloud Data Data Pipeline 52

Mlearning.ai

FEBRUARY 16, 2023

Thus, the solution allows for scaling data workloads independently from one another and seamlessly handling data warehousing, data lakes , data sharing, and engineering. Data warehousing is a vital constituent of any business intelligence operation. Simplify and Win Experienced data engineers value simplicity.

Data Warehouse 52

Data Warehouse Business Intelligence Business Intelligence Database 52

phData

AUGUST 4, 2023

The shared-nothing architecture ensures that users don’t have to worry about distributing data across multiple cluster nodes. Snowflake hides user data objects and makes them accessible only through SQL queries through the compute layer. This includes tasks such as data cleansing, enrichment, and aggregation.

Database 52

Database Azure SQL AWS 52

Alation

MAY 24, 2022

In a sea of questionable data, how do you know what to trust? Data quality tells you the answer. It signals what data is trustworthy, reliable, and safe to use. It empowers engineers to oversee data pipelines that deliver trusted data to the wider organization. Read the blog, Alation 2022.2:

Data Quality 52

Data Quality Data Pipeline DataOps Analytics 52

phData

JULY 18, 2023

Support for Numerous Data Sources: Fivetran supports over 200 data sources, including popular databases, applications, and cloud platforms like Salesforce, Google Analytics, SQL Server, Snowflake, and many more. Additionally, unsupported data sources can be integrated using Fivetran’s cloud function connectors.

Data Warehouse 52

Data Warehouse Analytics Analytics Cloud Data 52

DagsHub

APRIL 7, 2024

Image generated with Midjourney In today’s fast-paced world of data science, building impactful machine learning models relies on much more than selecting the best algorithm for the job. Data scientists and machine learning engineers need to collaborate to make sure that together with the model, they develop robust data pipelines.

Machine Learning 59

Machine Learning Machine Learning ML ML 59

Alation

JANUARY 17, 2023

A legacy data stack usually refers to the traditional relational database management system (RDBMS), which uses a structured query language (SQL) to store and process data. While an RDBMS can still be used in a modern data stack, it is not as common because it is not as well-suited for managing big data.

Data Warehouse 52

Data Warehouse ETL Tableau Cloud Data 52

DagsHub

OCTOBER 23, 2024

Here’s the structured equivalent of this same data in tabular form: With structured data, you can use query languages like SQL to extract and interpret information. In contrast, such traditional query languages struggle to interpret unstructured data. It also aids in identifying the source of any data quality issues.

Machine Learning 52

Machine Learning Machine Learning Data Lakes AI 52

Flipboard

MARCH 21, 2025

To establish trust between the data producers and data consumers, SageMaker Catalog also integrates the data quality metrics and data lineage events to track and drive transparency in data pipelines. Data analysts discover the data and subscribe to the data.

SQL 139

SQL Data Analyst Data Warehouse AWS 139

Dataversity

SEPTEMBER 28, 2022

Data engineering is a fascinating and fulfilling career – you are at the helm of every business operation that requires data, and as long as users generate data, businesses will always need data engineers. The journey to becoming a successful data engineer […]. In other words, job security is guaranteed.

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

phData

AUGUST 17, 2023

Data pipeline orchestration tools are designed to automate and manage the execution of data pipelines. These tools help streamline and schedule data movement and processing tasks, ensuring efficient and reliable data flow. What are Orchestration Tools?

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

phData

SEPTEMBER 27, 2024

Some modern CDPs are starting to incorporate these concepts, allowing for more flexible and evolving customer data models. It also requires a shift in how we query our customer data. Instead of simple SQL queries, we often need to use more complex temporal query languages or rely on derived views for simpler querying.

Data Modeling 52

Data Modeling Data Models Apache Kafka Data Lakes 52