ODSC - Open Data Science

SEPTEMBER 27, 2023

In this article, we will delve into the concept of data lakes, explore their differences from data warehouses and relational databases, and discuss the significance of data version control in the context of large-scale data management. This is particularly advantageous when dealing with exponentially growing data volumes.

Data Lakes 98

Data Lakes Data Warehouse Database Big Data 98

Pickl AI

JULY 25, 2023

They create data pipelines, ETL processes, and databases to facilitate smooth data flow and storage. With expertise in programming languages like Python , Java , SQL, and knowledge of big data technologies like Hadoop and Spark, data engineers optimize pipelines for data scientists and analysts to access valuable insights efficiently.

Data Engineer 52

Data Engineer Data Engineering Data Engineering Data Engineering 52

Mlearning.ai

MAY 16, 2023

This article discusses five commonly used architectural design patterns in data engineering and their use cases. ETL Design Pattern The ETL (Extract, Transform, Load) design pattern is a commonly used pattern in data engineering. Finally, the transformed data is loaded into the target system.

Data Engineer 52

Data Engineer Data Engineering Data Engineering Data Engineering 52

Pickl AI

NOVEMBER 4, 2024

This article explores the key fundamentals of Data Engineering, highlighting its significance and providing a roadmap for professionals seeking to excel in this vital field. Key components of data warehousing include: ETL Processes: ETL stands for Extract, Transform, Load. ETL is vital for ensuring data quality and integrity.

Data Engineer 52

Data Engineer Data Engineering Data Engineering Data Engineering 52

DataSeries

AUGUST 15, 2024

In this article, let’s understand an explanation of how to enhance problem-solving skills as a data engineer. Hadoop, Spark). Hadoop, Spark). Understanding these fundamentals is essential for effective problem-solving in data engineering.

Data Engineer 40

Data Engineer Data Engineering Data Engineering Data Engineering 40

Pickl AI

OCTOBER 10, 2024

This article will explore popular data transformation tools, highlighting their key features and how they can enhance data processing in various applications. It integrates well with cloud services, databases, and big data platforms like Hadoop, making it suitable for various data environments. What is Data Transformation?

Data Quality 52

Data Quality AWS Machine Learning Machine Learning 52

Mlearning.ai

JULY 10, 2023

In my 7 years of Data Science journey, I’ve been exposed to a number of different databases including but not limited to Oracle Database, MS SQL, MySQL, EDW, and Apache Hadoop. Now let’s get into the main topic of the article. For the part 1 of this article, I wanted to cover Tables, Views, Stored Procedures, and Materialized Views.

SQL 52

SQL Database Apache Hadoop Data Science 52

IBM Journey to AI blog

JANUARY 18, 2023

This article endeavors to alleviate those confusions. While traditional data warehouses made use of an Extract-Transform-Load (ETL) process to ingest data, data lakes instead rely on an Extract-Load-Transform (ELT) process. This adds an additional ETL step, making the data even more stale. The concepts will be explained.

Data Lakes 52

Data Lakes Data Warehouse Azure Apache Hadoop 52

Dataconomy

FEBRUARY 23, 2023

This involves working with various tools and technologies, such as ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes, to move data from its source to its destination. If you want to learn more about data engineers, check out article called: “ Data is the new gold and the industry demands goldsmiths.”

Big Data 91

Big Data Big Data Data Engineer Data Engineering 91

DagsHub

NOVEMBER 11, 2024

In this article, we’ll explore how AI can transform unstructured data into actionable intelligence, empowering you to make informed decisions, enhance customer experiences, and stay ahead of the competition. These capture the semantic relationships between words, facilitating tasks like classification and clustering within ETL pipelines.

AI 52

AI AI Data Lakes Database 52

DagsHub

OCTOBER 23, 2024

This article will discuss managing unstructured data for AI and ML projects. Popular data lake solutions include Amazon S3 , Azure Data Lake , and Hadoop. Apache Hadoop Apache Hadoop is an open-source framework that supports the distributed processing of large datasets across clusters of computers. Unstructured.io

Machine Learning 52

Machine Learning Machine Learning Data Lakes AI 52

The MLOps Blog

JANUARY 23, 2023

In this article, we will discuss the importance of data versioning control in machine learning and explore various methods and tools for implementing it with different types of data sources. With lakeFS it is possible to test ETLs on top of production data, in isolation, without copying anything.

ML 52

ML ML Data Lakes Machine Learning 52

The MLOps Blog

MAY 31, 2023

In this article, I will share my learnings of how successful ML platforms work in an eCommerce and what are the best practices a Team needs to follow during the course of building it. Final thoughts This article covered the major components of an ML platform and how to build them for an eCommerce business. But how to build it?

ML 59

ML ML Algorithm Machine Learning 59

Alation

JANUARY 25, 2022

On the process side, DataOps is essentially an agile and unified approach to building data movements and transformation pipelines (think streaming and modern ETL). These approaches extend the continuum of enterprise data warehouses, federated data marts, big data (Hadoop), and virtualization on top of distributed cloud file storage.

DataOps 52

DataOps SQL ML ML 52