Pickl AI

OCTOBER 18, 2023

It’s essential to ensure that data is not missing critical elements. Consistency Data consistency ensures that data is uniform and coherent across different sources or databases. Timeliness Timeliness relates to the relevance of data at a specific point in time. How to Use AI in Quality Assurance?

Data Quality 52

Data Quality ML ML Machine Learning 52

phData

DECEMBER 29, 2022

Traditionally, database administrators (DBAs) would run scripts that were manually generated through each environment to make changes to the database. This includes things like creating and modifying databases, schemas, and permissions. table1, match on the Snowflake database and table (ignoring the schema).

SQL 52

SQL Database Database Administration Data Profiling 52

Pickl AI

JULY 19, 2023

By maintaining clean and reliable data, businesses can avoid costly mistakes, enhance operational efficiency, and gain a competitive edge in their respective industries. Best Data Hygiene Tools & Software Trifacta Wrangler Pros: User-friendly interface with drag-and-drop functionality. Provides real-time data monitoring and alerts.

Data Quality 40

Data Quality Data Profiling Data Governance Data Preparation 40

IBM Journey to AI blog

JANUARY 5, 2023

Efficiently adopt data platforms and new technologies for effective data management. Apply metadata to contextualize existing and new data to make it searchable and discoverable. Perform data profiling (the process of examining, analyzing and creating summaries of datasets).

Data Quality 52

Data Quality Data Lakes Data Warehouse Big Data 52

IBM Journey to AI blog

MARCH 29, 2024

Each schema specifies the types of data the user can query or modify, and the relationships between the types. The resolver provides instructions for turning GraphQL queries, mutations, and subscriptions into data, and retrieves data from databases, cloud services, and other sources.

Data Profiling 94

Data Profiling Database Data Models Data Modeling 94

phData

JANUARY 10, 2024

This tool provides functionality in a number of different ways based on its metadata and profiling capabilities. The tool now runs on 8 threads as opposed to the original single thread! We highly recommend that you use the phData Advisor Tool within your Snowflake environment.

Data Warehouse 52

Data Warehouse Data Profiling Data Pipeline Database 52

Pickl AI

AUGUST 30, 2024

This is particularly important for organisations that have grown through acquisitions and need to unify disparate data systems. Enhance Performance Moving data to more efficient storage solutions can improve performance and reduce costs. This may involve data profiling and cleansing activities to improve data accuracy.

Data Quality 52

Data Quality Data Governance Azure Database 52

Pickl AI

OCTOBER 19, 2023

Implement Data Validation Rules To maintain data integrity, establish strict validation rules. This ensures that the data entered meets predefined criteria. Implementing validation rules helps prevent incorrect or incomplete data from being added to your databases.

Data Quality 52

Data Quality Data Governance Data Warehouse Machine Learning 52

Iguazio

FEBRUARY 17, 2024

Global Financial Data (GDF) An extensive database of current and historical financial data, providing updated information alongside data from hundreds of years ago. The database covers topics like market indicators, exchange rates, commodities, incomes and more. Get the datasets here 4. Get the datasets here 10.

Machine Learning 40

Machine Learning Machine Learning ML ML 40

Alation

MAY 24, 2022

Prime examples of this in the data catalog include: Trust Flags — Allow the data community to endorse, warn, and deprecate data to signal whether data can or can’t be used. Data Profiling — Statistics such as min, max, mean, and null can be applied to certain columns to understand its shape.

Data Quality 98

Data Quality Data Governance ETL Data Observability 98

phData

SEPTEMBER 28, 2023

Dataflows represent a cloud-based technology designed for data preparation and transformation purposes. Dataflows have different connectors to retrieve data, including databases, Excel files, APIs, and other similar sources, along with data manipulations that are performed using Online Power Query Editor.

Power BI 78

Power BI Data Preparation Machine Learning Machine Learning 78

Dataconomy

JULY 28, 2023

It is known for its ability to connect to almost any database and offers features like reusable data flows, automating repetitive work. Trifacta Trifacta is a data profiling and wrangling tool that stands out with its rich features and ease of use.

Power BI 91

Power BI Data Preparation Exploratory Data Analysis Machine Learning 91

phData

SEPTEMBER 7, 2023

This is commonly handled in code that pulls data from databases, but you can also do this within the SQL query itself. However, in the event that you can’t join those tables together, you would need to concatenate the actual SQL results together.

SQL 52

SQL Data Profiling Data Pipeline Database 52

phData

MARCH 31, 2023

For the Data Source Tool, we’ve addressed the following: Fixed an issue where view filters wouldn’t be disabled when using enabled = false. Fixed an issue when filtering tables in a database where only the first table listed would be scanned.

SQL 52

SQL Data Profiling Data Pipeline Database 52

phData

JUNE 26, 2023

Data Source Tool Updates The data source tool has a number of use cases, as it has the ability to profile your data sources and take the resulting JSON to perform whatever action you want to take. Lately, that has been Microsoft SQL Server (MSSQL) and Snowflake.

SQL 52

SQL Data Profiling Data Pipeline Data Governance 52

The MLOps Blog

MAY 17, 2023

Each subsystem is essential, and sequentially, each sub-system feeds into the next until data reaches its destination. ETL data pipeline architecture | Source: Author Data Discovery: Data can be sourced from various types of systems, such as databases, file systems, APIs, or streaming sources.

ETL 59

ETL Data Pipeline ML ML 59

Dataconomy

FEBRUARY 23, 2023

Collecting, storing, and processing large datasets Data engineers are also responsible for collecting, storing, and processing large volumes of data. This involves working with various data storage technologies, such as databases and data warehouses, and ensuring that the data is easily accessible and can be analyzed efficiently.

Big Data 91

Big Data Big Data Data Engineering Data Engineering 91

Alation

JANUARY 13, 2022

It’s in all types of data management systems, from databases to ERP tools, to data integration software. In fact, data intelligence technologies support building a data fabric and realizing a data mesh. Let’s turn our attention now to data mesh. What Is a Data Mesh?

Data Lakes 59

Data Lakes Data Governance Data Quality Data Warehouse 59

The MLOps Blog

MARCH 15, 2023

This is a difficult decision at the onset, as the volume of data is a factor of time and keeps varying with time, but an initial estimate can be quickly gauged by analyzing this aspect by running a pilot. Also, the industry best practices suggest performing a quick data profiling to understand the data growth.

Data Pipeline 59

Data Pipeline ETL SQL Data Quality 59

phData

AUGUST 17, 2023

Automate Data Quality Checks Integrate data quality checks and validations into your data pipelines. Include tasks to ensure data integrity, accuracy, and consistency. Automate data profiling, data cleansing, and validation steps to identify and address quality issues early in the pipeline.

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

Dataconomy

MARCH 26, 2025

Key applications of ETL pipelines ETL pipelines are utilized across various applications, making them invaluable in the world of data management. Their primary uses include: Data migration: Facilitates the transfer of data from legacy systems to modern databases, ensuring accessibility across platforms.

ETL 91

ETL Data Pipeline Business Intelligence Business Intelligence 91