Analytics Vidhya

MAY 27, 2022

Every time you put on a dog filter, watch cat videos or order food from your favourite restaurant, you generate data. Imagine how much data millions of other people are doing the […]. The post An Introduction to Hadoop Ecosystem for Big Data appeared first on Analytics Vidhya.

Hadoop 376

Hadoop Big Data Big Data Data Science 376

Analytics Vidhya

NOVEMBER 12, 2020

The post Data Engineering for Beginners – Partitioning vs Bucketing in Apache Hive appeared first on Analytics Vidhya. Overview Understand the meaning of partitioning and bucketing in the Hive in detail. We will see, how to create partitions and buckets in the.

Data Engineering 272

Data Engineering Data Engineering Data Engineering Data Engineer 272

Analytics Vidhya

MAY 30, 2021

ArticleVideo Book This article was published as a part of the Data Science Blogathon Different components in the Hadoop Framework Introduction Hadoop is. The post HIVE – A DATA WAREHOUSE IN HADOOP FRAMEWORK appeared first on Analytics Vidhya.

Hadoop 324

Hadoop Data Warehouse Data Science Analytics 324

Analytics Vidhya

DECEMBER 1, 2020

The post 15 Basic And Highly Used Hive Queries that All Data Engineers Must know appeared first on Analytics Vidhya. Overview Get to know 15 basic hive queries including- Simple selects ? selecting columns Simple selects – selecting rows Creating new columns Hive Functions.

Data Engineering 277

Data Engineering Data Engineering Data Engineering Data Engineer 277

Analytics Vidhya

OCTOBER 9, 2022

This article was published as a part of the Data Science Blogathon. Introduction Apache Hadoop is an open-source framework designed to facilitate interaction with big data. Still, for those unfamiliar with this technology, one question arises, what is big data?

Hadoop 269

Hadoop Apache Hadoop Big Data Big Data 269

Analytics Vidhya

JULY 25, 2022

This article was published as a part of the Data Science Blogathon. Introduction Apache Sqoop is a big data engine for transferring data between Hadoop and relational database servers. Big Data Sqoop can also be […]. The post Introduction to Apache Sqoop appeared first on Analytics Vidhya.

Hadoop 353

Hadoop Big Data Big Data Data Engineering 353

Analytics Vidhya

FEBRUARY 27, 2023

Introduction The Hadoop Distributed File System (HDFS) is a Java-based file system that is Distributed, Scalable, and Portable. Due to its lack of POSIX conformance, some believe it to be data storage instead. HDFS and […] The post Top 10 Hadoop Interview Questions You Must Know appeared first on Analytics Vidhya.

Hadoop 318

Hadoop Analytics Analytics Apache Hadoop 318

Analytics Vidhya

OCTOBER 28, 2020

The post Getting Started with Apache Hive – A Must Know Tool For all Big Data and Data Engineering Professionals appeared first on Analytics Vidhya. We will learn to do some basic operations in Apache Hive. Introduction Most of.

Big Data 270

Big Data Big Data Data Engineering Data Engineering 270

Data Science Dojo

JULY 6, 2023

Data engineering tools are software applications or frameworks specifically designed to facilitate the process of managing, processing, and transforming large volumes of data. Essential data engineering tools for 2023 Top 10 data engineering tools to watch out for in 2023 1.

Data Engineering 195

Data Engineering Data Engineering Data Engineering Data Engineer 195

Analytics Vidhya

SEPTEMBER 14, 2022

Introduction Every Data Science enthusiast’s journey goes through one of the most classical data problems – Frequent Itemset Mining, also sometimes referred to as Association Rule Mining or Market Basket Analysis. The post Frequent Itemset Mining Using MapReduce on Hadoop appeared first on Analytics Vidhya.

Hadoop 288

Hadoop Data Science Analytics Analytics 288

Data Science Dojo

JULY 24, 2023

They allow data processing tasks to be distributed across multiple machines, enabling parallel processing and scalability. It involves various technologies and techniques that enable efficient data processing and retrieval. Stay tuned for an insightful exploration into the world of Big Data Engineering with Distributed Systems!

Big Data 195

Big Data Big Data Data Engineering Data Engineering 195

Analytics Vidhya

FEBRUARY 5, 2023

Big data is nothing but the vast volume of datasets measured in terabytes or petabytes or even more. Big data […] The post A Beginner’s Guide to the Basics of Big Data and Hadoop appeared first on Analytics Vidhya.

Hadoop 290

Hadoop Big Data Big Data Analytics 290

Analytics Vidhya

JUNE 30, 2022

This article was published as a part of the Data Science Blogathon. Introduction Hive is a popular data warehouse built on top of Hadoop that is used by companies like Walmart, Tiktok, and AT&T. It is an important technology for data engineers to learn and master.

Data Warehouse 271

Data Warehouse Hadoop Data Engineering Data Engineering 271

Analytics Vidhya

MAY 5, 2022

Introduction This article will discuss the Hadoop Distributed File System, its features, components, functions, and benefits. Hadoop is a powerful platform for supporting an enormous variety of data applications. Both structured and complex data can […].

Hadoop 208

Hadoop Data Science Analytics Analytics 208

Analytics Vidhya

JULY 5, 2022

This article was published as a part of the Data Science Blogathon. Introduction MapReduce is part of the Apache Hadoop ecosystem, a framework that develops large-scale data processing. Other components of Apache Hadoop include Hadoop Distributed File System (HDFS), Yarn, and Apache Pig.

Apache Hadoop 397

Apache Hadoop Hadoop Data Science Algorithm 397

Data Science Connect

JANUARY 27, 2023

Data engineering is a crucial field that plays a vital role in the data pipeline of any organization. It is the process of collecting, storing, managing, and analyzing large amounts of data, and data engineers are responsible for designing and implementing the systems and infrastructure that make this possible.

Data Engineering 130

Data Engineering Data Engineering Data Engineering Data Engineer 130

Analytics Vidhya

MAY 12, 2022

This article was published as a part of the Data Science Blogathon. Introduction Apache Flume, a part of the Hadoop ecosystem, was developed by Cloudera. Initially, it was designed to handle log data solely, but later, it was developed to process event data. The post Get to Know Apache Flume from Scratch!

Hadoop 392

Hadoop Data Science Analytics Analytics 392

Analytics Vidhya

JANUARY 11, 2023

Introduction Amazon Elastic MapReduce (EMR) is a fully managed service that makes it easy to process large amounts of data using the popular open-source framework Apache Hadoop. EMR enables you to run petabyte-scale data warehouses and analytics workloads using the Apache Spark, Presto, and Hadoop ecosystems.

Apache Hadoop 337

Apache Hadoop Hadoop Data Warehouse Analytics 337

Analytics Vidhya

JANUARY 31, 2023

It is designed to be more flexible and generic than the original Hadoop MapReduce system, making it an attractive choice for companies looking to implement Hadoop. It allows companies to process data types and run […] The post YARN for Large Scale Computing: Beginner’s Edition appeared first on Analytics Vidhya.

Hadoop 326

Hadoop Analytics Analytics Apache Hadoop 326

Analytics Vidhya

FEBRUARY 1, 2023

Introduction In this constantly growing technical era, big data is at its peak, with the need for a tool to import and export the data between RDBMS and Hadoop. Apache Sqoop stands for “SQL to Hadoop,” and is one such tool that transfers data between Hadoop(HIVE, HBASE, HDFS, etc.)

Hadoop 306

Hadoop SQL Big Data Big Data 306

Analytics Vidhya

FEBRUARY 14, 2023

Introduction Today we have an abundance of Hadoop jobs that are running in a constant plane, but we can’t schedule these jobs manually, we need some kind of scheduler to handle this flow. Apache Oozie is one such job scheduler that allows users to run, schedule, and manage Hadoop jobs in a distributed environment.

Hadoop 316

Hadoop Analytics Analytics Apache Hadoop 316

Analytics Vidhya

OCTOBER 12, 2022

Introduction Since the 1970s, relational database management systems have solved the problems of storing and maintaining large volumes of structured data. With the advent of big data, several organizations realized the benefits of big data processing and started choosing solutions like Hadoop to […].

Hadoop 353

Hadoop Big Data Big Data Data Science 353

Analytics Vidhya

MAY 18, 2022

This article was published as a part of the Data Science Blogathon. Introduction Hadoop facilitates the processing of large datasets in a distributed manner and provides the foundation on which other services and applications can be built. MapReduce and HDFS are the two main components of Hadoop.

Hadoop 291

Hadoop Data Science Analytics Analytics 291

Analytics Vidhya

MARCH 5, 2023

Introduction Microsoft Azure HDInsight(or Microsoft HDFS) is a cloud-based Hadoop Distributed File System version. A distributed file system runs on commodity hardware and manages massive data collections. It is a fully managed cloud-based environment for analyzing and processing enormous volumes of data.

Hadoop 319

Hadoop Azure Analytics Analytics 319

Dataconomy

FEBRUARY 23, 2023

Data engineers play a crucial role in managing and processing big data. They are responsible for designing, building, and maintaining the infrastructure and tools needed to manage and process large volumes of data effectively. What is data engineering?

Big Data 91

Big Data Big Data Data Engineering Data Engineering 91

Analytics Vidhya

AUGUST 1, 2022

This article was published as a part of the Data Science Blogathon. Introduction HBase is a column-oriented non-relational database management system that operates on Hadoop Distributed File System (HDFS). HBase provides a fault-tolerant manner of storing sparse data sets, which are prevalent in several big data use cases.

Hadoop 360

Hadoop Big Data Big Data Data Science 360

Analytics Vidhya

FEBRUARY 6, 2023

Introduction HDFS (Hadoop Distributed File System) is not a traditional database but a distributed file system designed to store and process big data. It is a core component of the Apache Hadoop ecosystem and allows for storing and processing large datasets across multiple commodity servers.

Big Data 269

Big Data Big Data Apache Hadoop Hadoop 269

Analytics Vidhya

OCTOBER 28, 2021

This article was published as a part of the Data Science Blogathon What is the need for Hive? The official description of Hive is- ‘Apache Hive data warehouse software project built on top of Apache Hadoop for providing data query and analysis.

Apache Hadoop 349

Apache Hadoop Data Warehouse Hadoop SQL 349

Analytics Vidhya

OCTOBER 10, 2022

Introduction Most of you would know the different approaches for building a data and analytics platform. You would have already worked on systems that used traditional warehouses or Hadoop-based data lakes. Some of you might have also read about Lakehouses. Selecting one among […].

Data Lakes 350

Data Lakes Hadoop Data Science Analytics 350

Analytics Vidhya

FEBRUARY 2, 2023

Introduction Big data processing is crucial today. Big data analytics and learning help corporations foresee client demands, provide useful recommendations, and more. Hadoop, the Open-Source Software Framework for scalable and scattered computation of massive data sets, makes it easy.

Hadoop 306

Hadoop Big Data Analytics Big Data Analytics Big Data 306

Analytics Vidhya

APRIL 29, 2022

This article was published as a part of the Data Science Blogathon. Introduction Apache Hadoop is the most used open-source framework in the industry to store and process large data efficiently. Hive is built on the top of Hadoop for providing data storage, query and processing capabilities.

Apache Hadoop 266

Apache Hadoop Hadoop SQL Data Science 266

Analytics Vidhya

AUGUST 2, 2022

This article was published as a part of the Data Science Blogathon. Introduction Apache Oozie is a distributed workflow scheduler for performing and controlling Hadoop tasks. MapReduce, Sqoop, Pig, and Hive jobs can be easily scheduled with this tool. It […].

Hadoop 335

Hadoop Data Science Analytics Analytics 335

Analytics Vidhya

OCTOBER 23, 2020

Overview Apache Hive is a must-know tool for anyone interested in data science and data engineering Learn about the different types of tables un. The post Types of Tables in Apache Hive – A Quick Overview appeared first on Analytics Vidhya.

Data Engineering 228

Data Engineering Data Engineering Data Engineering Data Engineer 228

Analytics Vidhya

AUGUST 17, 2022

This article was published as a part of the Data Science Blogathon. Introduction Impala is an open-source and native analytics database for Hadoop. Vendors such as Cloudera, Oracle, MapReduce, and Amazon have shipped Impala. If you want to learn all things Impala, you’ve come to the right place.

Hadoop 291

Hadoop Data Science Database Analytics 291

Analytics Vidhya

SEPTEMBER 20, 2022

This article was published as a part of the Data Science Blogathon. Introduction Apache Oozie is a Hadoop workflow scheduler. Users can design Directed Acyclic Graphs of workflows that can be run in parallel and sequentially in Hadoop. It is a system that manages the workflow of dependent tasks.

Hadoop 268

Hadoop Data Science Analytics Analytics 268

Towards AI

FEBRUARY 2, 2024

I hope that you have sufficient knowledge of big data and Hadoop concepts like Map, reduce, transformations, actions, lazy evaluation, and many more topics in Hadoop and Spark. Extracting day, month and year from date column: #extract year, month, and day details from the data framedf.select(year("date column").distinct().orderBy(year("date

Data Engineering 115

Data Engineering Data Engineering Data Engineering Data Engineer 115

Analytics Vidhya

AUGUST 3, 2022

This article was published as a part of the Data Science Blogathon. Introduction Zookeeper in Hadoop can be considered a centralized repository where distributed applications can put data into and retrieve data from. For clarity, Zookeeper can be […].

Hadoop 270

Hadoop Data Science Analytics Analytics 270