IBM Data Science in Practice

MARCH 8, 2023

Tools like Git and Jenkins are not suited for managing data. By capturing metadata, such as transformations, storage configurations, versions, owners, lineage, statistics, data quality, and other relevant attributes of the data, a feature platform can address these issues. This is where a feature platform comes in handy.

Machine Learning 264

Machine Learning Machine Learning ML ML 264

AWS Machine Learning Blog

DECEMBER 4, 2024

It seems straightforward at first for batch data, but the engineering gets even more complicated when you need to go from batch data to incorporating real-time and streaming data sources, and from batch inference to real-time serving. Without the capabilities of Tecton , the architecture might look like the following diagram.

ML 86

ML ML AWS AI 86

ODSC - Open Data Science

DECEMBER 5, 2023

On Wednesday, Peter Norvig, PhD, Engineering Director at Google and Education Fellow at the Stanford Institute for Human-Centered Artificial Intelligence (HAI) spoke about the human side of AI and how we can focus on using AI for the greater good, improving all stakeholders’ lives and the needs of all users.

Data Science 98

Data Science Artificial Intelligence Artificial Intelligence Machine Learning 98

Precisely

AUGUST 29, 2024

Key Takeaways Data quality ensures your data is accurate, complete, reliable, and up to date – powering AI conclusions that reduce costs and increase revenue and compliance. Data observability continuously monitors data pipelines and alerts you to errors and anomalies. What does “quality” data mean, exactly?

Data Governance 52

Data Governance Data Quality Data Observability AI 52

Precisely

SEPTEMBER 18, 2023

Data observability is a key element of data operations (DataOps). It enables a big-picture understanding of the health of your organization’s data through continuous AI/ML-enabled monitoring – detecting anomalies throughout the data pipeline and preventing data downtime.

Data Observability 52

Data Observability Data Quality Data Pipeline DataOps 52

ODSC - Open Data Science

MAY 15, 2023

A high amount of effort is spent organizing data and creating reliable metrics the business can use to make better decisions. This creates a daunting backlog of data quality improvements and, sometimes, a graveyard of unused dashboards that have not been updated in years. Let’s start with an example.

AI 95

AI AI Data Quality Data Pipeline 95

IBM Journey to AI blog

MAY 9, 2024

How can a healthcare provider improve its data governance strategy, especially considering the ripple effect of small changes? Data lineage can help.With data lineage, your team establishes a strong data governance strategy, enabling them to gain full control of your healthcare data pipeline.

Data Governance 85

Data Governance Data Silos Data Quality Predictive Analytics 85

ODSC - Open Data Science

JUNE 19, 2023

There is no doubt that real-time operating systems (RTOS) have an important role in the future of big data collection and processing. How does RTOS help advance big data processing? It is also important to establish data quality standards and strict access controls. It is the dominant OS used in IoT and embedded systems.

Big Data 96

Big Data Big Data Artificial Intelligence Artificial Intelligence 96

IBM Journey to AI blog

JULY 30, 2024

Generative artificial intelligence (gen AI) is transforming the business world by creating new opportunities for innovation, productivity and efficiency. Data Engineer: A data engineer sets the foundation of building any generating AI app by preparing, cleaning and validating data required to train and deploy AI models.

AI 106

AI AI Data Scientist Data Preparation 106

ODSC - Open Data Science

OCTOBER 15, 2023

AI is quickly scaling through dozens of industries as companies, non-profits, and governments are discovering the power of artificial intelligence. This can be helpful for businesses that need to track data from multiple sources, such as sales, marketing, and customer service.

Machine Learning 52

Machine Learning Machine Learning Data Pipeline AI 52

IBM Journey to AI blog

JANUARY 9, 2024

The entire generative AI pipeline hinges on the data pipelines that empower it, making it imperative to take the correct precautions. 4 key components to ensure reliable data ingestion Data quality and governance: Data quality means ensuring the security of data sources, maintaining holistic data and providing clear metadata.

AI 89

AI AI Data Quality Data Pipeline 89

Precisely

AUGUST 15, 2024

Address common challenges in managing SAP master data by using AI tools to automate SAP processes and ensure data quality. Create an AI-driven data and process improvement loop to continuously enhance your business operations. Let’s dive deeper into data readiness next. Data creation and management processes.

AI 64

AI AI Data Quality Data Engineering 64

Precisely

MARCH 4, 2024

What if every decision, recommendation, and prediction made by artificial intelligence (AI) was as reliable as your most trusted team members? Next, you’ll see valuable AI use cases and how data integrity powers success. Technology-driven insights and capabilities depend on trusted data.

AI 72

AI AI Data Quality Artificial Intelligence 72

Alation

SEPTEMBER 28, 2022

Easy-to-experiment data development environment. Automated testing to ensure data quality. There are many inefficiencies that riddle a data pipeline and DataOps aims to deal with that. DataOps makes processes more efficient by automating as much of the data pipeline as possible.

DataOps 52

DataOps Data Pipeline Data Quality Analytics 52

AWS Machine Learning Blog

DECEMBER 15, 2023

In this post, we discuss how to bring data stored in Amazon DocumentDB into SageMaker Canvas and use that data to build ML models for predictive analytics. Without creating and maintaining data pipelines, you will be able to power ML models with your unstructured data stored in Amazon DocumentDB.

Machine Learning 105

Machine Learning Machine Learning AWS ML 105

Pickl AI

DECEMBER 25, 2024

Introduction In the rapidly evolving field of Artificial Intelligence , datasets like the Pile play a pivotal role in training models to understand and generate human-like text. The dataset is openly accessible, making it a go-to resource for researchers and developers in Artificial Intelligence.

Natural Language Processing 52

Natural Language Processing Machine Learning Machine Learning AI 52

AWS Machine Learning Blog

FEBRUARY 21, 2025

Rajesh Nedunuri is a Senior Data Engineer within the Amazon Worldwide Returns and ReCommerce Data Services team. He specializes in designing, building, and optimizing large-scale data solutions.

AWS 109

AWS Natural Language Processing Machine Learning Machine Learning 109

Dataconomy

MAY 16, 2023

With the explosion of big data and advancements in computing power, organizations can now collect, store, and analyze massive amounts of data to gain valuable insights. Machine learning, a subset of artificial intelligence , enables systems to learn and improve from data without being explicitly programmed.

Data Scientist 156

Data Scientist ML ML Machine Learning 156

IBM Journey to AI blog

APRIL 10, 2023

A data fabric is an architectural approach designed to simplify data access to facilitate self-service data consumption at scale. Data fabric can help model, integrate and query data sources, build data pipelines, integrate data in near real-time, and run AI-driven applications.

AI 103

AI AI Data Governance Data Pipeline 103

Pickl AI

JULY 25, 2023

Data engineers are essential professionals responsible for designing, constructing, and maintaining an organization’s data infrastructure. They create data pipelines, ETL processes, and databases to facilitate smooth data flow and storage. Read more to know. Cloud Platforms: AWS, Azure, Google Cloud, etc.

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

ODSC - Open Data Science

SEPTEMBER 27, 2024

We’re thrilled to announce our first group of Keynote Speakers, representing the groundbreaking AI companies shaking up the industry including Anthropic, Voltron Data, NVIDIA, Google DeepMind, and Microsoft. Develop the tools to build your future in AI at ODSC West. Learn more about this lineup here!

Artificial Intelligence 40

Artificial Intelligence Artificial Intelligence AI AI 40

IBM Journey to AI blog

AUGUST 23, 2023

The recent success of artificial intelligence based large language models has pushed the market to think more ambitiously about how AI could transform many enterprise processes. However, consumers and regulators have also become increasingly concerned with the safety of both their data and the AI models themselves.

Data Governance 66

Data Governance AI AI Artificial Intelligence 66

IBM Journey to AI blog

SEPTEMBER 3, 2024

High hopes Back in the spring of 2023—quite a long time in the artificial intelligence (AI) space—Goldman Sachs released a report estimating that the emergence of generative AI could boost global GDP by 7% annually (link resides outside IBM.com), amounting to more than an additional USD 7 trillion each year. .”

AI 123

AI AI Algorithm Artificial Intelligence 123

ODSC - Open Data Science

OCTOBER 18, 2024

Although data scientists rightfully capture the spotlight, future-focused teams also include engineers building data pipelines, visualization experts, and project managers who integrate efforts across groups. Selecting Technologies The technology landscape enables advanced analytics and artificial intelligence to evolve quickly.

Data Science 52

Data Science Data Scientist Analytics Analytics 52

Pickl AI

OCTOBER 17, 2024

Best Practices for ETL Efficiency Maximising efficiency in ETL (Extract, Transform, Load) processes is crucial for organisations seeking to harness the power of data. Implementing best practices can improve performance, reduce costs, and improve data quality. It also makes predictions for the future of ETL processes.

ETL 52

ETL Data Warehouse Data Quality Data Governance 52

IBM Journey to AI blog

APRIL 20, 2023

Read this e-book on building strong governance foundations Why automated data lineage is crucial for success Data lineage , the process of tracking the flow of data over time from origin to destination within a data pipeline, is essential to understand the full lifecycle of data and ensure regulatory compliance.

Database 86

Database Data Engineering Data Engineering Data Engineer 86

ODSC - Open Data Science

JANUARY 30, 2024

Not only does it involve the process of collecting, storing, and processing data so that it can be used for analysis and decision-making, but these professionals are responsible for building and maintaining the infrastructure that makes this possible; and so much more. Think of data engineers as the architects of the data ecosystem.

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

ODSC - Open Data Science

FEBRUARY 24, 2023

Data Quality Now that you’ve learned more about your data and cleaned it up, it’s time to ensure the quality of your data is up to par. With these data exploration tools, you can determine if your data is accurate, consistent, and reliable. You can watch it on demand here.

Exploratory Data Analysis 126

Exploratory Data Analysis Data Visualization Data Analysis Data Analysis 126

Precisely

MAY 13, 2024

The right data integration solution helps you streamline operations, enhance data quality, reduce costs, and make better data-driven decisions. It synthesizes all the metadata around your organization’s data assets and arranges the information into a simple, easy-to-understand format.

Data Governance 52

Data Governance Data Pipeline Cloud Data Data Quality 52

Pickl AI

OCTOBER 10, 2024

Summary: Data transformation tools streamline data processing by automating the conversion of raw data into usable formats. These tools enhance efficiency, improve data quality, and support Advanced Analytics like Machine Learning. The right tool can significantly enhance efficiency, scalability, and data quality.

Data Quality 52

Data Quality AWS Machine Learning Machine Learning 52

ODSC - Open Data Science

JANUARY 2, 2024

Join us in the city of Boston on April 24th for a full day of talks on a wide range of topics, including Data Engineering, Machine Learning, Cloud Data Services, Big Data Services, Data Pipelines and Integration, Monitoring and Management, Data Quality and Governance, and Data Exploration.

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

IBM Journey to AI blog

OCTOBER 12, 2023

Securing AI models and their access to data While AI models need flexibility to access data across a hybrid infrastructure, they also need safeguarding from tampering (unintentional or otherwise) and, especially, protected access to data.

AI 93

AI AI Data Scientist Data Governance 93

Pickl AI

JULY 25, 2024

Summary: Data ingestion is the process of collecting, importing, and processing data from diverse sources into a centralised system for analysis. This crucial step enhances data quality, enables real-time insights, and supports informed decision-making.

Apache Kafka 52

Apache Kafka Data Lakes Data Warehouse Data Quality 52

Precisely

MAY 23, 2024

Artificial intelligence (AI) algorithms are trained to detect anomalies. It synthesizes all the metadata around your organization’s data assets and arranges the information into a simple, easy-to-understand format. Today’s enterprises need real-time or near-real-time performance, depending on the specific application.

Data Governance 64

Data Governance ETL Data Pipeline Azure 64

IBM Journey to AI blog

JANUARY 17, 2023

As part of a data fabric, IBM’s data integration capability creates a roadmap that helps organizations connect data from disparate data sources, build data pipelines, remediate data issues, enrich data quality, and deliver integrated data to multicloud platforms. Start a trial.

Data Governance 58

Data Governance Data Science AI AI 58

ODSC - Open Data Science

FEBRUARY 14, 2023

Reliable Pipelines and High Quality Data without the Toil Kyle Kirwan | Co-Founder | Big Eye Check out this talk to learn about the history of data quality testing and data observability inside Uber, the differences between data observability and other methods like data pipeline tests, how techniques developed there can be applied by data engineers (..)

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

AWS Machine Learning Blog

MARCH 15, 2024

Federation learning to save the day (and save lives) For good artificial intelligence (AI), you need good data. Legacy systems, which are frequently found in the federal domain, pose significant data processing challenges before you can derive any intelligence or merge them with newer datasets.

AWS 117

AWS ML ML Data Silos 117