ODSC - Open Data Science

JANUARY 18, 2024

Data engineering is a rapidly growing field, and there is a high demand for skilled data engineers. If you are a data scientist, you may be wondering if you can transition into data engineering. In this blog post, we will discuss how you can become a data engineer if you are a data scientist.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

ODSC - Open Data Science

FEBRUARY 15, 2024

We couldn’t be more excited to announce the first sessions for our second annual Data Engineering Summit , co-located with ODSC East this April. Join us for 2 days of talks and panels from leading experts and data engineering pioneers. Is Gen AI A Data Engineering or Software Engineering Problem?

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

phData

AUGUST 11, 2023

Data engineering in healthcare is taking a giant leap forward with rapid industrial development. However, data collection and analysis have been commonplace in the healthcare sector for ages. Data Engineering in day-to-day hospital administration can help with better decision-making and patient diagnosis/prognosis.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

AWS Machine Learning Blog

MARCH 11, 2024

The new VistaPrint personalized product recommendation system Figure 1 As seen in Figure 1, the steps in how VistaPrint provides personalized product recommendations with their new cloud-native architecture are: Aggregate historical data in a data warehouse. Transform the data to create Amazon Personalize training data.

AWS 113

AWS Machine Learning Machine Learning Data Warehouse 113

Dataconomy

JUNE 1, 2023

By automating the integration of all Fabric workloads into OneLake, Microsoft eliminates the need for developers, analysts, and business users to create their own data silos. This approach not only improves performance by eliminating the need for separate data warehouses but also results in substantial cost savings for customers.

Power BI 194

Power BI Data Lakes Azure Data Silos 194

IBM Journey to AI blog

NOVEMBER 3, 2023

Apache Kafka and Apache Flink working together Anyone who is familiar with the stream processing ecosystem is familiar with Apache Kafka: the de-facto enterprise standard for open-source event streaming. Apache Kafka streams get data to where it needs to go, but these capabilities are not maximized when Apache Kafka is deployed in isolation.

Apache Kafka 79

Apache Kafka Data Warehouse Data Pipeline Big Data 79

ODSC - Open Data Science

MAY 7, 2024

In this episode, James Serra, author of “Deciphering Data Architectures: Choosing Between a Modern Data Warehouse, Data Fabric, Data Lakehouse, and Data Mesh” joins us to discuss his book and dive into the current state and possible future of data architectures. Interested in attending an ODSC event?

Data Warehouse 52

Data Warehouse Data Lakes Data Science Big Data 52

Dataconomy

APRIL 27, 2023

What is an online transaction processing database: Indexed data sets are used for rapid querying in OLTP systems Regular & incremental backups for data safety Frequent backups are necessary to ensure that data is protected in the event of a system failure or other issue.

Database 159

Database Data Scientist Data Mining Data Mining 159

Mlearning.ai

FEBRUARY 16, 2023

The ultimate need for vast storage spaces manifests in data warehouses: specialized systems that aggregate data coming from numerous sources for centralized management and consistency. In this article, you’ll discover what a Snowflake data warehouse is, its pros and cons, and how to employ it efficiently.

Data Warehouse 52

Data Warehouse Business Intelligence Business Intelligence Database 52

IBM Journey to AI blog

OCTOBER 16, 2023

Curated foundation models, such as those created by IBM or Microsoft, help enterprises scale and accelerate the use and impact of the most advanced AI capabilities using trusted data. In addition to natural language, models are trained on various modalities, such as code, time-series, tabular, geospatial and IT events data.

AI 88

AI AI Data Warehouse ML 88

phData

MARCH 11, 2024

For years, marketing teams across industries have turned to implementing traditional Customer Data Platforms (CDPs) as separate systems purpose-built to unlock growth with first-party data. Event Tracking : Capturing behavioral events such as page views, add-to-cart, signup, purchase, subscription, etc.

Data Warehouse 52

Data Warehouse Cloud Data Data Modeling Data Models 52

IBM Journey to AI blog

SEPTEMBER 19, 2023

Data scientists will typically perform data analytics when collecting, cleaning and evaluating data. By analyzing datasets, data scientists can better understand their potential use in an algorithm or machine learning model. Diagnostic analytics: Diagnostic analytics helps pinpoint the reason an event occurred.

Data Science 94

Data Science Analytics Analytics Data Scientist 94

Pickl AI

JULY 25, 2024

Must Read Blogs: Exploring the Power of Data Warehouse Functionality. Data Lakes Vs. Data Warehouse: Its significance and relevance in the data world. Exploring Differences: Database vs Data Warehouse. It is commonly used in data warehouses for business analytics and reporting.

Data Warehouse 52

Data Warehouse Business Intelligence Business Intelligence Database 52

phData

FEBRUARY 14, 2023

Data integration is essentially the Extract and Load portion of the Extract, Load, and Transform (ELT) process. Data ingestion involves connecting your data sources, including databases, flat files, streaming data, etc, to your data warehouse. Snowflake provides native ways for data ingestion.

Data Warehouse 59

Data Warehouse Azure AWS Database 59

ODSC - Open Data Science

APRIL 4, 2024

ODSC Highlights Announcing the Keynote and Featured Speakers for ODSC East 2024 The keynotes and featured speakers for ODSC East 2024 have won numerous awards, authored books and widely cited papers, and shaped the future of data science and AI with their research. Learn more about them here!

Data Visualization 52

Data Visualization Analytics Analytics Big Data Analytics 52

Alation

MARCH 1, 2022

Through Impact Analysis, users can determine if a problem occurred with data upstream, and locate the impacted data downstream. With robust data lineage, data engineers can find and fix issues fast and prevent them from recurring. Similarly, analysts gain a clear view of how data is created.

Data Warehouse 52

Data Warehouse Data Lakes Cloud Data Database 52

phData

SEPTEMBER 27, 2024

Methods that allow our customer data models to be as dynamic and flexible as the customers they represent. In this guide, we will explore concepts like transitional modeling for customer profiles, the power of event logs for customer behavior, persistent staging for raw customer data, real-time customer data capture, and much more.

Data Modeling 52

Data Modeling Data Models Apache Kafka Data Lakes 52

ODSC - Open Data Science

OCTOBER 7, 2024

Kaggle Grandmaster Jiwei Liu and NVIDIA experts will answer all your questions about the upcoming event. HPCC Systems : Built for data engineers, HPCC provides an open-source platform designed for fast and efficient big data processing.

Data Scientist 52

Data Scientist Data Science Machine Learning Machine Learning 52

phData

MARCH 29, 2024

Founded in 2014 by three leading cloud engineers, phData focuses on solving real-world data engineering, operations, and advanced analytics problems with the best cloud platforms and products. Over the years, one of our primary focuses became Snowflake and migrating customers to this leading cloud data platform.

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

phData

JANUARY 24, 2024

Understanding Fivetran Fivetran is a user-friendly, code-free platform enabling customers to easily synchronize their data by automating extraction, transformation, and loading from many sources. Fivetran automates the time-consuming steps of the ELT process so your data engineers can focus on more impactful projects.

Data Warehouse 52

Data Warehouse Data Pipeline Data Governance Cloud Data 52

phData

FEBRUARY 9, 2023

KPIs for predictive maintenance include: Equipment downtime Mean time between failures (MTBF) Mean time to repair (MTTR) All of these KPIs can be measured by tracking the amount of time that equipment is not in use due to maintenance or repair, as well as the frequency and duration of maintenance and repair events.

Analytics 52

Analytics Analytics Power BI Data Warehouse 52

Alation

FEBRUARY 13, 2020

Andreas Kohlmaier, Head of Data Engineering at Munich Re 1. --> Ron Powell, independent analyst and industry expert for the BeyeNETWORK and executive producer of The World Transformed FastForward Series, interviews Andreas Kohlmaier, Head of Data Engineering at Munich Re.

Data Lakes 52

Data Lakes Analytics Analytics Data Engineering 52

Alation

OCTOBER 27, 2022

Few actors in the modern data stack have inspired the enthusiasm and fervent support as dbt. This data transformation tool enables data analysts and engineers to transform, test and document data in the cloud data warehouse. But what does this mean from a practitioner perspective?

Data Analyst 52

Data Analyst Data Scientist Analytics Analytics 52

phData

OCTOBER 25, 2024

Data Quality Monitoring implements quality checks in operational data processes to ensure that the data meets pre-defined standards and business rules. This results in poor credibility and data consistency after some time, leading businesses to mistrust the data pipelines and processes.

Data Quality 52

Data Quality Data Pipeline Data Governance Database 52

Pickl AI

SEPTEMBER 5, 2024

Data Preparation: Cleaning, transforming, and preparing data for analysis and modelling. Collaborating with Teams: Working with data engineers, analysts, and stakeholders to ensure data solutions meet business needs.

Azure 52

Azure Data Scientist Data Science Machine Learning 52

DagsHub

AUGUST 23, 2024

Data Quality Dimensions Data quality dimensions are the criteria that are used to evaluate and measure the quality of data. These include the following: Accuracy indicates how correctly data reflects the real-world entities or events it represents. Datafold is a tool focused on data observability and quality.

Data Quality 52

Data Quality Data Governance Machine Learning Machine Learning 52

The MLOps Blog

MARCH 15, 2023

Our activities mostly revolved around: 1 Identifying data sources 2 Collecting & Integrating data 3 Developing Analytical/ML models 4 Integrating the above into a cloud environment 5 Leveraging the cloud to automate the above processes 6 Making the deployment robust & scalable Who was involved in the project?

AWS 52

AWS ETL ML ML 52

phData

OCTOBER 28, 2024

Modern low-code/no-code ETL tools allow data engineers and analysts to build pipelines seamlessly using a drag-and-drop and configure approach with minimal coding. Matillion ETL for Snowflake is an ELT/ETL tool that allows for the ingestion, transformation, and building of analytics for data in the Snowflake AI Data Cloud.

Python 52

Python ETL AWS Database 52

AWS Machine Learning Blog

APRIL 3, 2025

Lambda enables serverless, event-driven data processing tasks, allowing for real-time transformations and calculations as data arrives. Step Functions complements this by orchestrating complex workflows, coordinating multiple Lambda functions, and managing error handling for sophisticated data processing pipelines.

AWS 72

AWS Data Governance Data Silos SQL 72

Pickl AI

MARCH 19, 2025

Summary: Data engineering tools streamline data collection, storage, and processing. Tools like Python, SQL, Apache Spark, and Snowflake help engineers automate workflows and improve efficiency. Learning these tools is crucial for building scalable data pipelines. Thats where data engineering tools come in!

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

AWS Machine Learning Blog

NOVEMBER 1, 2024

At the same time, global health awareness and investments in clinical research have increased as a result of motivations by major events like the COVID-19 pandemic. Instead, a core component of decentralized clinical trials is a secure, scalable data infrastructure with strong data analytics capabilities.

AWS 108

AWS Data Lakes Machine Learning Machine Learning 108