Snorkel AI

JUNE 22, 2023

Jacomo Corbo is a Partner and Chief Scientist, and Bryan Richardson is an Associate Partner and Senior Data Scientist, for QuantumBlack AI by McKinsey. They presented “Automating Data Quality Remediation With AI” at Snorkel AI’s The Future of Data-Centric AI Summit in 2022. That is still in flux and being worked out.

Data Quality 52

Data Quality AI AI ML 52

Precisely

SEPTEMBER 18, 2023

It enables a big-picture understanding of the health of your organization’s data through continuous AI/ML-enabled monitoring – detecting anomalies throughout the data pipeline and preventing data downtime. Why is data observability so important? Longer-term data trends also require attention.

Data Observability 52

Data Observability Data Quality Data Pipeline DataOps 52

ODSC - Open Data Science

JANUARY 2, 2024

We couldn’t be more excited to announce two events that will be co-located with ODSC East in Boston this April: The Data Engineering Summit and the Ai X Innovation Summit. These two co-located events represent an opportunity to dive even deeper into the topics and trends shaping these disciplines. Learn more about them below.

Data Engineer 52

Data Engineer Data Engineering Data Engineering Data Engineering 52

The MLOps Blog

MARCH 15, 2023

In this post, you will learn about the 10 best data pipeline tools, their pros, cons, and pricing. A typical data pipeline involves the following steps or processes through which the data passes before being consumed by a downstream process, such as an ML model training process.

Data Pipeline 59

Data Pipeline ETL SQL Data Quality 59

ODSC - Open Data Science

JANUARY 30, 2024

Not only does it involve the process of collecting, storing, and processing data so that it can be used for analysis and decision-making, but these professionals are responsible for building and maintaining the infrastructure that makes this possible; and so much more. Think of data engineers as the architects of the data ecosystem.

Data Engineer 52

Data Engineer Data Engineering Data Engineering Data Engineering 52

ODSC - Open Data Science

DECEMBER 5, 2023

You can see our photos from the event here , and be sure to follow our YouTube for virtual highlights from the conference as well. Over in San Francisco, we had a keynote for each day of the event. Other Events Aside from networking events and all of our sessions, we had a few other special events. What’s next?

Data Science 98

Data Science Artificial Intelligence Artificial Intelligence Machine Learning 98

Precisely

SEPTEMBER 12, 2023

As a proud member of the Connect with Confluent program , we help organizations going through digital transformation and IT infrastructure modernization break down data silos and power their streaming data pipelines with trusted data. Let’s cover some additional information to know before attending.

Data Silos 52

Data Silos Apache Kafka Data Pipeline Data Quality 52

Precisely

DECEMBER 12, 2023

Systems and data sources are more interconnected than ever before. A broken data pipeline might bring operational systems to a halt, or it could cause executive dashboards to fail, reporting inaccurate KPIs to top management. The application of this concept to data is relatively new. Complexity leads to risk.

Data Observability 52

Data Observability Data Quality Data Pipeline Machine Learning 52

The MLOps Blog

JUNE 27, 2023

Data quality control: Robust dataset labeling and annotation tools incorporate quality control mechanisms such as inter-annotator agreement analysis, review workflows, and data validation checks to ensure the accuracy and reliability of annotations. Data monitoring tools help monitor the quality of the data.

Machine Learning 134

Machine Learning Machine Learning ML ML 134

ODSC - Open Data Science

OCTOBER 15, 2023

Elementl / Dagster Labs Elementl and Dagster Labs are both companies that provide platforms for building and managing data pipelines. Elementl’s platform is designed for data engineers, while Dagster Labs’ platform is designed for data scientists. Interested in attending an ODSC event?

Machine Learning 52

Machine Learning Machine Learning Data Pipeline AI 52

Precisely

MAY 13, 2024

The right data integration solution helps you streamline operations, enhance data quality, reduce costs, and make better data-driven decisions. It synthesizes all the metadata around your organization’s data assets and arranges the information into a simple, easy-to-understand format.

Data Governance 52

Data Governance Data Pipeline Data Quality Cloud Data 52

AWS Machine Learning Blog

DECEMBER 15, 2023

In this post, we discuss how to bring data stored in Amazon DocumentDB into SageMaker Canvas and use that data to build ML models for predictive analytics. Without creating and maintaining data pipelines, you will be able to power ML models with your unstructured data stored in Amazon DocumentDB.

Machine Learning 114

Machine Learning Machine Learning AWS ML 114

Dataversity

APRIL 30, 2021

The goal of digital transformation remains the same as ever – to become more data-driven. We have learned how to gain a competitive advantage by capturing business events in data. Events are data snap-shots of complex activity sourced from the web, customer systems, ERP transactions, social media, […].

Data Pipeline 98

Data Pipeline Data Warehouse Data Governance Data Quality 98

phData

OCTOBER 17, 2024

The DAGs can then be scheduled to run at specific intervals or triggered when an event occurs. It even offers a user-friendly interface to visualize the pipelines and monitor progress. The Data Source Tool can automate scanning DDL and profiling tables between source and target, comparing them, and then reporting findings.

AI 52

AI AI SQL Data Quality 52

Iguazio

MAY 29, 2024

The 4 Gen AI Architecture Pipelines The four pipelines are: 1. The Data Pipeline The data pipeline is the foundation of any AI system. It's responsible for collecting and ingesting the data from various external sources, processing it and managing the data.

Data Pipeline 104

Data Pipeline AI AI ML 104

Precisely

FEBRUARY 2, 2023

Business managers are faced with plotting the optimal course in the face of these evolving events. Pipelines must have robust data integration capabilities that integrate data from multiple data silos, including the extensive list of applications used throughout the organization, databases and even mainframes.

Data Pipeline 52

Data Pipeline Data Silos Database Data Observability 52

Alation

MAY 31, 2023

If you’re not familiar with DGIQ, it’s the world’s most comprehensive event dedicated to, you guessed it, data governance and information quality. This year’s DGIQ West will host tutorials, workshops, seminars, general conference sessions, and case studies for global data leaders.

Data Governance 52

Data Governance DataOps Data Pipeline Business Intelligence 52

phData

AUGUST 5, 2024

Horizon addresses key aspects of data governance, including: Compliance Security Access Privacy Interoperability Throughout the remainder of this blog, we will dive deeper into each of the above components and take a look at the ways in which Horizon can help. We will begin with compliance.

Data Governance 52

Data Governance Data Quality Data Lakes ML 52

phData

SEPTEMBER 27, 2024

Methods that allow our customer data models to be as dynamic and flexible as the customers they represent. In this guide, we will explore concepts like transitional modeling for customer profiles, the power of event logs for customer behavior, persistent staging for raw customer data, real-time customer data capture, and much more.

Data Modeling 52

Data Modeling Data Models Apache Kafka Data Lakes 52

phData

FEBRUARY 5, 2024

Setting up the Information Architecture Setting up an information architecture during migration to Snowflake poses challenges due to the need to align existing data structures, types, and sources with Snowflake’s multi-cluster, multi-tier architecture. Moving historical data from a legacy system to Snowflake poses several challenges.

SQL 52

SQL Database Data Quality Data Warehouse 52

phData

JANUARY 24, 2024

In addition to the Application type, Fivetran provides connectors for databases, files, events, and functions. Salesforce – The Salesforce (SFDC) connector streamlines customer relationship management (CRM) data integration. Scalability Considerations: Ensure your data integration setup is scalable and ready for future growth.

Data Warehouse 52

Data Warehouse Data Pipeline Data Governance Data Quality 52

ODSC - Open Data Science

SEPTEMBER 29, 2023

A 2019 survey by McKinsey on global data transformation revealed that 30 percent of total time spent by enterprise IT teams was spent on non-value-added tasks related to poor data quality and availability. It truly is an all-in-one data lake solution. Roxie then consolidates that data and presents the results.

Data Lakes 52

Data Lakes Clustering Big Data Big Data 52

ODSC - Open Data Science

JUNE 19, 2023

As the name suggests, real-time operating systems (RTOS) handle real-time applications that undertake data and event processing under a strict deadline. It is also important to establish data quality standards and strict access controls.

Big Data 96

Big Data Big Data Artificial Intelligence Artificial Intelligence 96

DagsHub

OCTOBER 23, 2024

With proper unstructured data management, you can write validation checks to detect multiple entries of the same data. Continuous learning: In a properly managed unstructured data pipeline, you can use new entries to train a production ML model, keeping the model up-to-date.

Machine Learning 52

Machine Learning Machine Learning AI AI 52

Snorkel AI

JUNE 9, 2023

Snorkel AI wrapped the second day of our The Future of Data-Centric AI virtual conference by showcasing how Snorkel’s data-centric platform has enabled customers to succeed, taking a deep look at Snorkel Flow’s capabilities, and announcing two new solutions. You need to find a place to park your data.

AI 52

AI AI Data Scientist Machine Learning 52

Snorkel AI

JUNE 9, 2023

Snorkel AI wrapped the second day of our The Future of Data-Centric AI virtual conference by showcasing how Snorkel’s data-centric platform has enabled customers to succeed, taking a deep look at Snorkel Flow’s capabilities, and announcing two new solutions. You need to find a place to park your data.

AI 52

AI AI Data Scientist Machine Learning 52

IBM Data Science in Practice

MARCH 8, 2023

Tools like Git and Jenkins are not suited for managing data. By capturing metadata, such as transformations, storage configurations, versions, owners, lineage, statistics, data quality, and other relevant attributes of the data, a feature platform can address these issues. This is where a feature platform comes in handy.

Machine Learning 264

Machine Learning Machine Learning ML ML 264

IBM Journey to AI blog

JANUARY 17, 2023

Due to the convergence of events in the data analytics and AI landscape, many organizations are at an inflection point. From there, it can be easily accessed via dashboards by data consumers or those building into a data product.

Data Governance 59

Data Governance Data Science AI AI 59

Iguazio

FEBRUARY 8, 2024

Continuous monitoring of resources, data, and metrics. Data Pipeline - Manages and processes various data sources. ML Pipeline - Focuses on training, validation and deployment. Application Pipeline - Manages requests and data/model validations. Collecting feedback for further tuning.

ML 52

ML ML AI AI 52

The MLOps Blog

MARCH 15, 2023

For small-scale/low-value deployments, there might not be many items to focus on, but as the scale and reach of deployment go up, data governance becomes crucial. This includes data quality, privacy, and compliance. The data pipelines can be scheduled as event-driven or be run at specific intervals the users choose.

AWS 52

AWS ETL ML ML 52

ODSC - Open Data Science

APRIL 26, 2023

Data Engineer Data engineers are the authors of the infrastructure that stores, processes, and manages the large volumes of data an organization has. The main aspect of their profession is the building and maintenance of data pipelines, which allow for data to move between sources. Well then, you’re in luck.

Data Analyst 52

Data Analyst Machine Learning Machine Learning Power BI 52

The MLOps Blog

AUGUST 11, 2023

Internally within Netflix’s engineering team, Meson was built to manage, orchestrate, schedule, and execute workflows within ML/Data pipelines. Meson managed the lifecycle of ML pipelines, providing functionality such as recommendations and content analysis, and leveraged the Single Leader Architecture.

ML 52

ML ML Machine Learning Machine Learning 52

phData

JUNE 7, 2023

You don’t need massive data sets because “data quality scales better than data size.” ” Small models with good data are better than massive models because “in the long run, the best models are the ones which can be iterated upon quickly.”

AI 59

AI AI Natural Language Processing Machine Learning 59

ODSC - Open Data Science

SEPTEMBER 20, 2023

Monday’s sessions will cover a wide range of topics, from Generative AI and LLMs to MLOps and Data Visualization. Finally, get ready for some All Hallows Eve fun with Halloween Data After Dark , featuring a costume contest, candy, and more. There will also be an in-person career expo where you can find your next job in data science!

Data Wrangling 52

Data Wrangling Data Science Machine Learning Machine Learning 52

Alation

APRIL 19, 2022

Access the resources your data applications need — no more, no less. Data Pipeline Automation. Consolidate all data sources to automate pipelines for processing in a single repository. Efficient Data Processing. To use data, you need the ability to collect and correlate it efficiently.

Data Governance 52

Data Governance Cloud Data Database Data Silos 52

The MLOps Blog

MARCH 21, 2023

Olalekan said that most of the random people they talked to initially wanted a platform to handle data quality better, but after the survey, he found out that this was the fifth most crucial need. And when the platform automates the entire process, it’ll likely produce and deploy a bad-quality model.

Machine Learning 69

Machine Learning Machine Learning Data Scientist ML 69