IBM Journey to AI blog

MAY 20, 2024

Machine learning (ML) has become a critical component of many organizations’ digital transformation strategy. From predicting customer behavior to optimizing business processes, ML algorithms are increasingly being used to make decisions that impact business outcomes.

Machine Learning 110

Machine Learning Machine Learning Data Scientist ML 110

Applied Data Science

AUGUST 2, 2021

Automation Automating data pipelines and models ➡️ 6. The Data Engineer Not everyone working on a data science project is a data scientist. Data engineers are the glue that binds the products of data scientists into a coherent and robust data pipeline.

Data Science 130

Data Science Data Scientist ML ML 130

Heartbeat

NOVEMBER 6, 2023

Image Source —  Pixel Production Inc In the previous article, you were introduced to the intricacies of data pipelines, including the two major types of existing data pipelines. You might be curious how a simple tool like Apache Airflow can be powerful for managing complex data pipelines.

Data Pipeline 52

Data Pipeline Clean Data ETL Python 52

Flipboard

NOVEMBER 24, 2023

Iris was designed to use machine learning (ML) algorithms to predict the next steps in building a data pipeline. Let’s combine these suggestions to improve upon our original prompt: Human: Your job is to act as an expert on ETL pipelines.

Database 158

Database AWS ETL SQL 158

AWS Machine Learning Blog

SEPTEMBER 18, 2024

The ZMP analyzes billions of structured and unstructured data points to predict consumer intent by using sophisticated artificial intelligence (AI) to personalize experiences at scale. Hosted on Amazon ECS with tasks run on Fargate, this platform streamlines the end-to-end ML workflow, from data ingestion to model deployment.

AWS 121

AWS Machine Learning Machine Learning ML 121

The MLOps Blog

SEPTEMBER 7, 2023

This situation is not different in the ML world. Data Scientists and ML Engineers typically write lots and lots of code. Building a mental model for ETL components Learn the art of constructing a mental representation of the components within an ETL process.

Machine Learning 52

Machine Learning Machine Learning ETL ML 52

AWS Machine Learning Blog

FEBRUARY 21, 2025

Previously, he was a Data & Machine Learning Engineer at AWS, where he worked closely with customers to develop enterprise-scale data infrastructure, including data lakes, analytics dashboards, and ETL pipelines. He specializes in designing, building, and optimizing large-scale data solutions.

AWS 125

AWS Natural Language Processing Machine Learning Machine Learning 125

AWS Machine Learning Blog

JUNE 18, 2024

Despite the challenges, Afri-SET, with limited resources, envisions a comprehensive data management solution for stakeholders seeking sensor hosting on their platform, aiming to deliver accurate data from low-cost sensors. With AWS Glue custom connectors, it’s effortless to transfer data between Amazon S3 and other applications.

AWS 132

AWS AI AI Python 132

The MLOps Blog

MARCH 15, 2023

In this post, you will learn about the 10 best data pipeline tools, their pros, cons, and pricing. A typical data pipeline involves the following steps or processes through which the data passes before being consumed by a downstream process, such as an ML model training process.

Data Pipeline 59

Data Pipeline ETL SQL Data Quality 59

The MLOps Blog

DECEMBER 7, 2022

And we at deployr , worked alongside them to find the best possible answers for everyone involved and build their Data and ML Pipelines. Building data and ML pipelines: from the ground to the cloud It was the beginning of 2022, and things were looking bright after the lockdown’s end.

ML 52

ML ML AWS ETL 52

phData

FEBRUARY 25, 2025

However, one consistent challenge customers face is efficiently integrating and moving data between on-premises systems, cloud environments, and other data sources. Datavolo is more than just an ETL toolit provides functionality for Reverse ETL as well, enabling organizations to push data from Snowflake into other systems.

Data Pipeline 52

Data Pipeline ETL Data Engineering Data Engineering 52

The MLOps Blog

JANUARY 23, 2023

Dolt LakeFS Delta Lake Pachyderm Git-like versioning Database tool Data lake Data pipelines Experiment tracking Integration with cloud platforms Integrations with ML tools Examples of data version control tools in ML DVC Data Version Control DVC is a version control system for data and machine learning teams.

ML 52

ML ML Data Lakes Machine Learning 52

The MLOps Blog

JANUARY 26, 2024

Luckily, we have tried and trusted tools and architectural patterns that provide a blueprint for reliable ML systems. In this article, I’ll introduce you to a unified architecture for ML systems built around the idea of FTI pipelines and a feature store as the central component. But what is an ML pipeline?

Machine Learning 72

Machine Learning Machine Learning ML ML 72

The MLOps Blog

MARCH 15, 2023

This includes the tools and techniques we used to streamline the ML model development and deployment processes, as well as the measures taken to monitor and maintain models in a production environment. Costs: Oftentimes, cost is the most important aspect of any ML model deployment. This includes data quality, privacy, and compliance.

AWS 52

AWS ETL ML ML 52

DagsHub

APRIL 7, 2024

Data scientists and machine learning engineers need to collaborate to make sure that together with the model, they develop robust data pipelines. These pipelines cover the entire lifecycle of an ML project, from data ingestion and preprocessing, to model training, evaluation, and deployment.

Machine Learning 59

Machine Learning Machine Learning ML ML 59

phData

OCTOBER 17, 2023

The story is all too common – a business user requests some data, the data team creates/prioritizes a ticket, and said ticket is completed after some number of months (or weeks if you’re lucky) – just to have the data be wrong, and the whole process starts again. Those are scary for data teams to change.

ETL 52

ETL Data Pipeline Data Engineering Data Engineering 52

Precisely

JUNE 2, 2023

As companies strive to leverage AI/ML, location intelligence, and cloud analytics into their portfolio of tools, siloed mainframe data often stands in the way of forward momentum. The right data integration technology can vastly simplify things. Streaming data pipelines help to make data available and accessible in real time.

Data Silos 52

Data Silos ETL Data Quality Data Pipeline 52

phData

FEBRUARY 7, 2024

On the client side, Snowpark consists of libraries, including the DataFrame API and native Snowpark machine learning (ML) APIs for model development (public preview) and deployment (private preview). Machine Learning Training machine learning (ML) models can sometimes be resource-intensive.

Python 52

Python ML ML SQL 52

DagsHub

OCTOBER 23, 2024

Managing unstructured data is essential for the success of machine learning (ML) projects. Without structure, data is difficult to analyze and extracting meaningful insights and patterns is challenging. This article will discuss managing unstructured data for AI and ML projects. What is Unstructured Data?

Machine Learning 52

Machine Learning Machine Learning Data Lakes AI 52

phData

APRIL 18, 2023

Data movements lead to high costs of ETL and rising data management TCO. The inability to access and onboard new datasets prolong the data pipeline’s creation and time to market. Data co-location enables teams to access, join, query, and analyze internal and external vendor data with minimal to no ETL.

Data Silos 52

Data Silos ETL Clustering Analytics 52

IBM Journey to AI blog

AUGUST 4, 2023

Data mesh Another approach to data democratization uses a data mesh , a decentralized architecture that organizes data by a specific business domain. It uses knowledge graphs, semantics and AI/ML technology to discover patterns in various types of metadata.

Data Lakes 73

Data Lakes AI AI Data Governance 73

IBM Journey to AI blog

JULY 17, 2023

is our enterprise-ready next-generation studio for AI builders, bringing together traditional machine learning (ML) and new generative AI capabilities powered by foundation models. Watsonx.data allows customers to augment data warehouses such as Db2 Warehouse and Netezza and optimize workloads for performance and cost. IBM watsonx.ai

AI 74

AI AI Machine Learning Machine Learning 74

Alation

APRIL 4, 2023

However, the race to the cloud has also created challenges for data users everywhere, including: Cloud migration is expensive, migrating sensitive data is risky, and navigating between on-prem sources is often confusing for users. To build effective data pipelines, they need context (or metadata) on every source.

DataOps 52

DataOps Data Engineering Data Engineering Data Engineer 52

phData

JULY 18, 2023

Why Migrate to a Modern Data Stack? Slow Response to New Information: Legacy data systems often lack the computation power necessary to run efficiently and can be cost-inefficient to scale. This typically results in long-running ETL pipelines that cause decisions to be made on stale or old data.

Data Warehouse 52

Data Warehouse Analytics Analytics Cloud Data 52

phData

JUNE 7, 2023

Data scientists use data-driven approaches to enable AI systems to make better predictions, optimize decision-making, and uncover hidden patterns that ultimately drive innovation and improve performance across various domains. This often involves skills in databases, distributed systems, and ETL (Extract, Transform, Load) processes.

AI 59

AI AI Machine Learning Machine Learning 59

Flipboard

NOVEMBER 22, 2024

Organizations run millions of Apache Spark applications each month to prepare, move, and process their data for analytics and machine learning (ML). During development, data engineers often spend hours sifting through log files, analyzing execution plans, and making configuration changes to resolve issues. Choose your job.

AWS 144

AWS AI AI Data Engineering 144

Alation

APRIL 14, 2023

Last week, the Alation team had the privilege of joining IT professionals, business leaders, and data analysts and scientists for the Modern Data Stack Conference in San Francisco. So, how can a data catalog support the critical project of building data pipelines? What did attendees take away from the event?

Data Pipeline 52

Data Pipeline Data Warehouse Cloud Data ETL 52

phData

SEPTEMBER 27, 2024

If the event log is your customer’s diary, think of persistent staging as their scrapbook – a place where raw customer data is collected, organized, and kept for future reference. In traditional ETL (Extract, Transform, Load) processes in CDPs, staging areas were often temporary holding pens for data.

Data Modeling 52

Data Modeling Data Models Apache Kafka Data Lakes 52

AWS Machine Learning Blog

MARCH 27, 2025

In this post, we explore how you can use Amazon Bedrock to generate high-quality categorical ground truth data, which is crucial for training machine learning (ML) models in a cost-sensitive environment. This use case, solvable through ML, can enable support teams to better understand customer needs and optimize response strategies.

AWS 111

AWS ETL ML ML 111

AWS Machine Learning Blog

NOVEMBER 12, 2024

There are various technologies that help operationalize and optimize the process of field trials, including data management and analytics, IoT, remote sensing, robotics, machine learning (ML), and now generative AI. The first step in developing and deploying generative AI use cases is having a well-defined data strategy.

AWS 122

AWS AI AI Data Lakes 122

ODSC - Open Data Science

MARCH 12, 2025

20212024: Interest declined as deep learning and pre-trained models took over, automating many tasks previously handled by classical ML techniques. This shift suggests that while traditional ML is still relevant, its role is now more supportive rather than cutting-edge.

Data Science 52

Data Science Machine Learning Machine Learning Data Engineer 52