30+ Big Data Interview Questions
Analytics Vidhya
JANUARY 17, 2024
Introduction In the realm of Big Data, professionals are expected to navigate complex landscapes involving vast datasets, distributed systems, and specialized tools.
Integration of Python with Hadoop and Spark
Analytics Vidhya
MAY 30, 2021
ArticleVideo Book This article was published as a part of the Data Science Blogathon Introduction Big data is the collection of data that is vast. The post Integration of Python with Hadoop and Spark appeared first on Analytics Vidhya.
Hadoop 
367
367 Join 17,000+
professionals
Sign Up for our Newsletter
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
A comprehensive guide to Feature Selection using Wrapper methods in Python
Analytics Vidhya
OCTOBER 24, 2020
This article was published as a part of the Data Science Blogathon. Introduction In today’s era of Big data and IoT, we are easily. The post A comprehensive guide to Feature Selection using Wrapper methods in Python appeared first on Analytics Vidhya.
Python 398
398 
Relationship Between Facebook and Big Data
Analytics Vidhya
OCTOBER 30, 2022
The post Relationship Between Facebook and Big Data appeared first on Analytics Vidhya. Introduction Source – Unsplash You must often receive birthday notifications from Facebook, like “Amit Pathak and 4 others have their birthday today” What is so special about this notification?
Big Data 349
349 
Learn About Apache Spark Using Python
Analytics Vidhya
APRIL 12, 2022
Introduction In the last article, we discussed Apache Spark and the big data ecosystem, and we discussed the role of apache spark in data processing in big data. The post Learn About Apache Spark Using Python appeared first on Analytics Vidhya. If you haven’t read it yet, you can find it on this page.
Python 328
328 
Python vs Scala for Apache Spark – Which is Better?
Analytics Vidhya
FEBRUARY 28, 2023
Introduction Apache Spark is a powerful big data processing engine that has gained widespread popularity recently due to its ability to process massive amounts of data types quickly and efficiently. While Spark can be used with several programming languages, Python and Scala are popular for building Spark applications.
Python 347
347 
End-to-End Beginners Guide on Spark SQL in Python
Analytics Vidhya
APRIL 12, 2022
Introduction In this article, we are going to cover Spark SQL in Python. In the last article, we have already introduced Spark and its work and its role in Big data. The post End-to-End Beginners Guide on Spark SQL in Python appeared first on Analytics Vidhya. If you haven’t checked it yet, please go to this link.
SQL 337
337 
Monitoring Data Quality for Your Big Data Pipelines Made Easy
Analytics Vidhya
NOVEMBER 8, 2023
In the data-driven world […] The post Monitoring Data Quality for Your Big Data Pipelines Made Easy appeared first on Analytics Vidhya. Determine success by the precision of your charts, the equipment’s dependability, and your crew’s expertise. A single mistake, glitch, or slip-up could endanger the trip.
Data Pipeline 341
341 
All About Big Data File Formats
Analytics Vidhya
MAY 31, 2022
This article was published as a part of the Data Science Blogathon. Introduction to Big Data File Formats In the digital era, every day we generate thousands of terabytes of data. The most challenging task is to store and process this data.
Big Data 271
271 
A beginners guide to Multi-Processing in Python
Analytics Vidhya
APRIL 26, 2021
ArticleVideo Book This article was published as a part of the Data Science Blogathon. In the era of Big Data, Python has become the. The post A beginners guide to Multi-Processing in Python appeared first on Analytics Vidhya.
Python 296
296 
Introduction to Data Science: How to “Big Data” with Python
Dataconomy
OCTOBER 18, 2016
Katharine Jarmul and Data Natives are joining forces to give you an amazing chance to delve deeply into Python and how to apply it to data manipulation, and data wrangling. By the end of her workshop, Learn Python for Data Analysis, you will feel comfortable importing and running simple Python analysis on your.
Big Data 196
196 
High Performance Big Data Analysis Using NumPy, Numba & Python Asynchronous Programming
Dataconomy
JULY 31, 2017
Introduction A couple of months ago a client of mine asked me the following question: “What is the faster data structure object in Python for Big Data analysis today?” The post High Performance Big Data Analysis Using NumPy, Numba & Python Asynchronous Programming appeared first on Dataconomy.
Big Data 183
183 
PySpark for Beginners – Take your First Steps into Big Data Analytics (with Code)
Analytics Vidhya
OCTOBER 27, 2019
Overview Big Data is becoming bigger by the day, and at an unprecedented pace How do you store, process and use this amount of. The post PySpark for Beginners – Take your First Steps into Big Data Analytics (with Code) appeared first on Analytics Vidhya.
Integrating Python in Power BI: Get the best of both worlds
Analytics Vidhya
AUGUST 30, 2020
Overview A demonstration of statistical analytics by Integrating Python within Power BI Share the findings using dashboards and reports Introduction Power BI is. The post Integrating Python in Power BI: Get the best of both worlds appeared first on Analytics Vidhya.
Power BI 400
400 
MongoDB in Python Tutorial for Beginners (using PyMongo)
Analytics Vidhya
FEBRUARY 19, 2020
Overview MongoDB is a popular unstructured database that data scientists should be aware of We will discuss how you can work with a MongoDB. The post MongoDB in Python Tutorial for Beginners (using PyMongo) appeared first on Analytics Vidhya.
Python 397
397 
Big Data. Big Impact
KDnuggets
JANUARY 22, 2020
Ramapo College’s Master of Science in Data Science program will teach you to collect, synthesize, and analyze big data, become skilled in programming languages like R and Python, and leverage advanced tools to meet the demands of modern business and science.
Big Data 169
169 
Building A Machine Learning Pipeline Using Pyspark
Analytics Vidhya
JUNE 9, 2022
This article was published as a part of the Data Science Blogathon. Introduction to Pyspark Spark is an open-source framework for big data processing. It was originally written in scala and later on due to increasing demand for machine learning using big data a python API of the same was released.
Machine Learning 328
328 
Big data engineering simplified: Exploring roles of distributed systems
Data Science Dojo
JULY 24, 2023
The generation and accumulation of vast amounts of data have become a defining characteristic of our world. This data, often referred to as Big Data , encompasses information from various sources, including social media interactions, online transactions, sensor data, and more. databases), semi-structured data (e.g.,
Big Data 195
195 
SQream Announces Strategic Integration for Powerful Big Data Analytics with Dataiku
insideBIGDATA
FEBRUARY 9, 2024
SQream, the scalable GPU data analytics platform, announced a strategic integration with Dataiku, the platform for everyday AI. This collaboration brings together SQream’s best-in-class big data analytics technology with Dataiku’s flexible and scalable data science and machine learning (ML) platform.
Learn how to use PySpark in under 5 minutes (Installation + Tutorial)
KDnuggets
AUGUST 13, 2019
Apache Spark is one of the hottest and largest open source project in data processing framework with rich high-level APIs for the programming languages like Scala, Python, Java and R. It realizes the potential of bringing together both Big Data and machine learning.
Big Data 61
61 
Build and deploy a UI for your generative AI applications with AWS and Python
AWS Machine Learning Blog
NOVEMBER 6, 2024
In this post, we explore a practical solution that uses Streamlit , a Python library for building interactive data applications, and AWS services like Amazon Elastic Container Service (Amazon ECS), Amazon Cognito , and the AWS Cloud Development Kit (AWS CDK) to create a user-friendly generative AI application with authentication and deployment.
AWS 120
120 
Big Data vs. Data Science: Demystifying the Buzzwords
Pickl AI
APRIL 21, 2025
Summary: Big Data refers to the vast volumes of structured and unstructured data generated at high speed, requiring specialized tools for storage and processing. Data Science, on the other hand, uses scientific methods and algorithms to analyses this data, extract insights, and inform decisions.
Big Data 52
52 
Introduction to Apache Spark and its Datasets
Analytics Vidhya
AUGUST 17, 2022
This article was published as a part of the Data Science Blogathon. Introduction In this article, we will introduce you to the big data ecosystem and the role of Apache Spark in Big data. We will also cover the Distributed database system, the backbone of big data. In today’s world, data is the fuel.
Big Data 302
302 
A Brief Introduction to Apache HBase and it’s Architecture
Analytics Vidhya
OCTOBER 12, 2022
Introduction Since the 1970s, relational database management systems have solved the problems of storing and maintaining large volumes of structured data. With the advent of big data, several organizations realized the benefits of big data processing and started choosing solutions like Hadoop to […].
Hadoop 353
353 
From Parchment to Python: How Smart Data Evolved to What It Is Today
Dataversity
APRIL 23, 2025
Today, we navigate a landscape dominated by code, algorithms, and digital streams of data, a far cry from those early days. Yet, despite these transformative changes, the […] The post From Parchment to Python: How Smart Data Evolved to What It Is Today appeared first on DATAVERSITY.
Python 91
91 
Remote Data Science Jobs: 5 High-Demand Roles for Career Growth
Data Science Dojo
OCTOBER 31, 2024
Strong analytical skills and the ability to work with large datasets are critical, as is familiarity with data modeling and ETL processes. Additionally, knowledge of programming languages like Python or R can be beneficial for advanced analytics. Prepare to discuss your experience and problem-solving abilities with these languages.
Data Science 195
195 
Roles of Python Developer in Data Science Teams
Smart Data Collective
SEPTEMBER 5, 2022
One of the fields of professionals that are so important for data science projects are Python developers. What is the Python programming language? Why is it so important in the data science profession ? What Is Python? Python is a powerful programming language that is widely used in many different industries today.
Data Science 141
141 
An End-to-End Starter Guide on Apache Spark and RDD
Analytics Vidhya
JUNE 2, 2022
This article was published as a part of the Data Science Blogathon. Introduction In this article, we will introduce you to Apache Spark and its role in big data and the way it makes a big data ecosystem we will also explore Resilient Distributed Dataset (RDD) in spark. As we all have seen the growth of […].
Big Data 312
312 
Three R Libraries for Automated EDA
Analytics Vidhya
OCTOBER 7, 2022
Introduction With the increasing use of technology, data accumulation is faster than ever due to connected smart devices. These devices continuously collect and transmit data that can be processed, transformed, and stored for later use. This collected data, known as big data, holds valuable […].
EDA 400
400 
Top 10 Platforms to Practice Data Science Skills
Analytics Vidhya
JULY 11, 2024
Introduction Data science is one of the professions in high demand nowadays due to the growing focus on analyzing big data. Hypothesis and conclusion-making from data broadly involve technical and non-technical skills in the interdisciplinary field of data science.
Data Science 352
352 
Apache Spark Performance Optimization for Data Engineers
Analytics Vidhya
SEPTEMBER 30, 2021
This article was published as a part of the Data Science Blogathon Introduction Apache Spark is a big data processing framework that has long become one of the most popular and frequently encountered in all kinds of projects related to Big Data.
Data Engineering 318
318 
Good ETL Practices with Apache Airflow
Analytics Vidhya
NOVEMBER 30, 2021
Introduction to ETL ETL is a type of three-step data integration: Extraction, Transformation, Load are processing, used to combine data from multiple sources. It is commonly used to build Big Data. In this process, data is pulled (extracted) from a source system, to […].
ETL 382
382 
Graceful External Termination: Handling Pod Deletions in Kubernetes Data Ingestion and Streaming…
IBM Data Science in Practice
APRIL 7, 2025
Graceful External Termination: Handling Pod Deletions in Kubernetes Data Ingestion and Streaming Jobs When running big-data pipelines in Kubernetes, especially streaming jobs, its easy to overlook how these jobs deal with termination. If not handled correctly, this can lead to locks, data issues, and a negative user experience.
Python 130
130 
How To Learn Python For Data Science?
Pickl AI
NOVEMBER 4, 2024
Summary: Python for Data Science is crucial for efficiently analysing large datasets. With numerous resources available, mastering Python opens up exciting career opportunities. Introduction Python for Data Science has emerged as a pivotal tool in the data-driven world. in 2022, according to the PYPL Index.
Data Science 52
52 
Top 26 Data Science Tools for Data Scientists in 2024
Analytics Vidhya
DECEMBER 12, 2023
Introduction The field of data science is evolving rapidly, and staying ahead of the curve requires leveraging the latest and most powerful tools available. In 2024, data scientists have a plethora of options to choose from, catering to various aspects of their work, including programming, big data, AI, visualization, and more.
Data Scientist 328
328 
KDnuggets™ News 19:n36, Sep 25: The Hidden Risk of AI and Big Data; The 5 Sampling Algorithms every Data Scientist needs to know
KDnuggets
SEPTEMBER 25, 2019
Learn about unexpected risk of AI applied to Big Data; Study 5 Sampling Algorithms every Data Scientist needs to know; Read how one data scientist copes with his boring days of deploying machine learning; 5 beginner-friendly steps to learn ML with Python; and more.
Data Scientist 136
136 
Step-by-Step Guide to Becoming a Data Analyst in 2023
Analytics Vidhya
JANUARY 17, 2023
Corporations across all industries have invested significantly in big data, establishing analytics departments, particularly in telecommunications, insurance, advertising, financial services, healthcare, and technology. The post Step-by-Step Guide to Becoming a Data Analyst in 2023 appeared first on Analytics Vidhya.
Data Analyst 305
305 
30 Best Data Science Books to Read in 2023
Analytics Vidhya
FEBRUARY 28, 2023
To achieve maximum efficiency, every company strives to use various data at every stage of its operations.
Data Science 353
353 
The Lifecycle to Build a Web Application for Prediction from Scratch
Analytics Vidhya
AUGUST 31, 2020
The data science lifecycle is designed for big data issues and data science projects. Generally, the data science project consists of seven steps which. The post The Lifecycle to Build a Web Application for Prediction from Scratch appeared first on Analytics Vidhya.
Data Science 329
329 
A Comprehensive Guide to Apache Hive
Analytics Vidhya
MAY 24, 2022
This article was published as a part of the Data Science Blogathon. Introduction on Apache Hive Advanced big data tools must handle the massive amounts of structured and unstructured data generated daily. Data is not increasing only in terms of volume, but the variety and veracity of data are also growing.
Big Data 290
290 
Let's personalize your content