The MLOps Blog

MAY 17, 2023

However, efficient use of ETL pipelines in ML can help make their life much easier. This article explores the importance of ETL pipelines in machine learning, a hands-on example of building ETL pipelines with a popular tool, and suggests the best ways for data engineers to enhance and sustain their pipelines.

ETL 59

ETL Data Pipeline ML ML 59

Towards AI

JANUARY 28, 2024

To start, get to know some key terms from the demo: Snowflake: The centralized source of truth for our initial data Magic ETL: Domo’s tool for combining and preparing data tables ERP: A supplemental data source from Salesforce Geographic: A supplemental data source (i.e., Instagram) used in the demo Why Snowflake?

ETL 111

ETL Python Database Data Preparation 111

Dataversity

APRIL 1, 2021

In the data-driven world we live in today, the field of analytics has become increasingly important to remain competitive in business. In fact, a study by McKinsey Global Institute shows that data-driven organizations are 23 times more likely to outperform competitors in customer acquisition and nine times […].

Analytics 69

Analytics Analytics Data Silos Data Pipeline 69

Smart Data Collective

APRIL 29, 2020

Data management approaches are varied and may be categorised in the following: Cloud data management. The storage and processing of data through a cloud-based system of applications. Master data management. Extraction, Transform, Load (ETL). Data transformation. Reference data management.

Data Warehouse 98

Data Warehouse SQL Azure ETL 98

phData

FEBRUARY 21, 2025

A data warehouse acts as a single source of truth for an organization’s data, providing a unified view of its operations and enabling data-driven decision-making. A data warehouse enables advanced analytics, reporting, and business intelligence. On the other hand, cloud data warehouses can scale seamlessly.

Data Warehouse 52

Data Warehouse Cloud Data ETL Cloud Computing 52

phData

AUGUST 9, 2023

In short, ELT exemplifies the data strategy required in the era of big data, cloud, and agile analytics. With ELT, we first extract data from source systems, then load the raw data directly into the data warehouse before finally applying transformations natively within the data warehouse.

ETL 52

ETL Data Warehouse Cloud Data Big Data 52

phData

MARCH 8, 2023

Fivetran is an automated data integration platform that offers a convenient solution for businesses to consolidate and sync data from disparate data sources. With over 160 data connectors available, Fivetran makes it easy to move data out of, into, and across any cloud data platform in the market.

Data Warehouse 96

Data Warehouse Data Engineering Data Engineering Data Engineering 96

phData

OCTOBER 9, 2023

As organizations embrace the benefits of data vault, it becomes crucial to ensure optimal performance in the underlying data platform. One such platform that has revolutionized cloud data warehousing is the Snowflake Data Cloud. However, joining tables using a hash key can take longer than a sequential key.

ETL 69

ETL Clustering Data Warehouse SQL 69

Dataversity

MARCH 17, 2021

I do not think it is an exaggeration to say data analytics has come into its own over the past decade or so. What started out as an attempt to extract business insights from transactional data in the ’90s and early 2000s has now transformed into an […]. appeared first on DATAVERSITY.

Analytics 52

Analytics Analytics Data Lakes Data Warehouse 52

Data Science Blog

JUNE 27, 2023

Big Data Analytics stands apart from conventional data processing in its fundamental nature. In the realm of Big Data, there are two prominent architectural concepts that perplex companies embarking on the construction or restructuring of their Big Data platform: Lambda architecture or Kappa architecture.

Big Data 130

Big Data Big Data Apache Kafka Database 130

phData

SEPTEMBER 2, 2024

In this blog, we will cover the best practices for developing jobs in Matillion, an ETL/ELT tool built specifically for cloud database platforms. It can connect to multiple data warehouses, including the Snowflake AI Data Cloud , Delta Lake on Databricks, Amazon Redshift, Google BigQuery, and Azure Synapse Analytics.

ETL 52

ETL Data Warehouse SQL Database 52

Precisely

APRIL 3, 2023

The Long Road from Batch to Real-Time Traditional “extract, transform, load” (ETL) systems were built under certain constraints, stemming from the cost of technology and implementation resources, as well as the inherent limits of computational power. Nevertheless, the result was always yesterday’s news.

ETL 52

ETL Analytics Analytics Database 52

Smart Data Collective

OCTOBER 17, 2022

While growing data enables companies to set baselines, benchmarks, and targets to keep moving ahead, it poses a question as to what actually causes it and what it means to your organization’s engineering team efficiency. What’s causing the data explosion? Big data analytics from 2022 show a dramatic surge in information consumption.

Big Data 119

Big Data Big Data Data Engineering Data Engineering 119

Alation

JANUARY 17, 2023

Data ingestion/integration services. Reverse ETL tools. Data orchestration tools. These tools are used to manage big data, which is defined as data that is too large or complex to be processed by traditional means. How Did the Modern Data Stack Get Started? A Note on the Shift from ETL to ELT.

Data Warehouse 52

Data Warehouse ETL Tableau Cloud Data 52

phData

JULY 17, 2023

Over the past few decades, the corporate data landscape has changed significantly. The shift from on-premise databases and spreadsheets to the modern era of cloud data warehouses and AI/ LLMs has transformed what businesses can do with data. This is where Fivetran and the Modern Data Stack come in.

Data Warehouse 52

Data Warehouse Data Pipeline Cloud Data ETL 52

phData

APRIL 29, 2024

In our previous blog, Top 5 Fivetran Connectors for Financial Services , we explored Fivetran’s capabilities that address the data integration needs of the finance industry. Now, let’s cover the healthcare industry, which also has a surging demand for data and analytics, along with the underlying processes to make it happen.

SQL 52

SQL Data Warehouse Azure Cloud Data 52

phData

JUNE 6, 2024

Data gets ingested, centralized, and deployed within your cloud data warehouse. This allows companies to use their pre-existing data tools and prevents the need for costly setups. Companies need to bring in data from a wide variety of sources to get a holistic view of the customer.

Data Warehouse 52

Data Warehouse Cloud Data ETL Data Modeling 52

Data Science Dojo

JULY 3, 2024

Big Data Technologies : Handling and processing large datasets using tools like Hadoop, Spark, and cloud platforms such as AWS and Google Cloud. Data Processing and Analysis : Techniques for data cleaning, manipulation, and analysis using libraries such as Pandas and Numpy in Python.

Data Science 150

Data Science Machine Learning Machine Learning Data Visualization 150

phData

SEPTEMBER 4, 2024

This article was co-written by Lynda Chao & Tess Newkold With the growing interest in AI-powered analytics, ThoughtSpot stands out as a leader among legacy BI solutions known for its self-service search-driven analytics capabilities. Suppose your business requires more robust capabilities across your technology stack.

Analytics 52

Analytics Analytics SQL ETL 52

Precisely

JUNE 2, 2023

As companies strive to leverage AI/ML, location intelligence, and cloud analytics into their portfolio of tools, siloed mainframe data often stands in the way of forward momentum. At the same time, there is a stronger push for real-time analytics and real-time customer access to data.

Data Silos 52

Data Silos ETL Data Quality Data Pipeline 52

IBM Journey to AI blog

JULY 17, 2023

Visual modeling: Delivers easy-to-use workflows for data scientists to build data preparation and predictive machine learning pipelines that include text analytics, visualizations and a variety of modeling methods. ” Vitaly Tsivin, EVP Business Intelligence at AMC Networks.

AI 74

AI AI Machine Learning Machine Learning 74

Alation

APRIL 14, 2023

Last week, the Alation team had the privilege of joining IT professionals, business leaders, and data analysts and scientists for the Modern Data Stack Conference in San Francisco. Patil also highlighted the need for pragmatic, data-driven leadership, saying “Every boardroom needs a Spock.” Cloud costs are growing prohibitive.

Data Pipeline 52

Data Pipeline Data Warehouse Cloud Data ETL 52

Alation

OCTOBER 27, 2022

This data transformation tool enables data analysts and engineers to transform, test and document data in the cloud data warehouse. To answer this question, I sat down with members of the Alation Data & Analytics team, Bindu, Adrian, and Idris. Let’s dive in. What do you do at Alation?

Data Analyst 52

Data Analyst Data Scientist Analytics Analytics 52

phData

FEBRUARY 7, 2024

The Snowflake Data Cloud is a leading cloud data platform that provides various features and services for data storage, processing, and analysis. A new feature that Snowflake offers is called Snowpark, which provides an intuitive library for querying and processing data at scale in Snowflake.

Python 52

Python ML ML SQL 52

phData

FEBRUARY 14, 2023

Fivetran is the answer for anyone looking to focus their efforts on analytics and not pipeline management. This can provide the efficiency needed to spend more time developing analytic solutions instead of moving data. Replication of calculated values is not supported during Change Processing.

Data Warehouse 59

Data Warehouse Azure AWS Database 59

Alation

MARCH 22, 2022

As the latest iteration in this pursuit of high-quality data sharing, DataOps combines a range of disciplines. It synthesizes all we’ve learned about agile, data quality , and ETL/ELT. Simply put, IDF standardizes data engineering processes. IDF works natively on cloud platforms like AWS.

DataOps 52

DataOps Data Pipeline Data Engineering Data Engineering 52

phData

JANUARY 4, 2023

Snowflake works with an entire ecosystem of tools including Extract Transform and Load (ETL), data integration, and analysis tools. Disaster Recovery Snowflake allows for an easy and automatic backup of data and enables faster disaster recovery of critical IT systems. This makes the task exceedingly difficult.

Data Warehouse 52

Data Warehouse SQL Database ETL 52

phData

JULY 10, 2024

It is important in business to be able to manage and analyze data well. Sigma Computing , a cloud-based analytics platform, helps data analysts and business professionals maximize their data with collaborative and scalable analytics. These tools allow users to handle more advanced data tasks and analyses.

SQL 52

SQL Data Warehouse Analytics Analytics 52

phData

NOVEMBER 28, 2023

Matillion is also built for scalability and future data demands, with support for cloud data platforms such as Snowflake Data Cloud , Databricks, Amazon Redshift, Microsoft Azure Synapse, and Google BigQuery, making it future-ready, everyone-ready, and AI-ready. Why Does it Matter? Contact phData today!

Data Warehouse 52

Data Warehouse Data Pipeline ETL Azure 52

Dataversity

SEPTEMBER 6, 2021

The rush to become data-driven is more heated, important, and pronounced than it has ever been. Businesses understand that if they continue to lead by guesswork and gut feeling, they’ll fall behind organizations that have come to recognize and utilize the power and potential of data. Click to learn more about author Mike Potter.

Data Warehouse 98

Data Warehouse Data Lakes ETL Cloud Data 98

Alation

APRIL 4, 2023

Alation delivers extended connectivity for Databricks Unity Catalog , the lakehouse company, and new connectivity for dbt Cloud by dbt Labs , the pioneer in analytics engineering. Now, joint users will get an enhanced view into cloud and data transformations , with valuable context to guide smarter usage.

DataOps 52

DataOps Data Engineering Data Engineering Data Engineer 52

Alation

JANUARY 25, 2022

As a reminder, here’s Gartner’s definition of data fabric: “A design concept that serves as an integrated layer (fabric) of data and connecting processes. At its best, a data catalog should empower data analysts, scientists, and anyone curious about data with tools to explore and understand it. ” 1.

DataOps 52

DataOps SQL ML ML 52

phData

JANUARY 19, 2024

Matillion is also built for scalability and future data demands, with support for cloud data platforms such as Snowflake Data Cloud , Databricks, Amazon Redshift, Microsoft Azure Synapse, and Google BigQuery, making it future-ready, everyone-ready, and AI-ready. Check out the API documentation for our sample.

Data Pipeline 52

Data Pipeline Data Warehouse ETL Azure 52

Dataversity

APRIL 8, 2021

Business intelligence (BI) tools transform the unprocessed data into meaningful and actionable insight. BI tools analyze the data and convert them […]. Click to learn more about author Piyush Goel. What is a BI tool? Which BI tool is best for your organization?

Business Intelligence 75

Business Intelligence Business Intelligence ETL Data Quality 75

phData

FEBRUARY 5, 2024

Replicate can interact with a wide variety of databases, data warehouses, and data lakes (on-premise or based in the cloud). Matllion can replicate data from sources such as APIs, applications, relational databases, files, and NoSQL databases.

SQL 52

SQL Database Data Quality Data Warehouse 52

phData

SEPTEMBER 27, 2024

It enables advanced analytics, makes debugging your marketing automations easier, provides natural audit trails for compliance, and allows for flexible, evolving customer data models. So next time you’re designing your customer data architecture in your CDP, don’t just think about the current state of your customers.

Data Modeling 52

Data Modeling Data Models Apache Kafka Data Lakes 52

Dataversity

MARCH 12, 2024

While we haven’t built technology that enables real-time matter transfer yet, modern science is pursuing concepts like superposition and quantum teleportation to facilitate information transfer across any distance […] The post 10 Advantages of Real-Time Data Streaming in Commerce appeared first on DATAVERSITY.

Data Pipeline 59

Data Pipeline ETL Cloud Data Analytics 59