Analytics Vidhya

OCTOBER 19, 2021

In this tutorial, you will see the top 5 features that developers should know before implementing a solution on the Snowflake data […]. The post 5 Features Of Snowflake That Data Engineers Must Know appeared first on Analytics Vidhya.

Data Engineer 400

Data Engineer Data Engineering Data Engineering Data Engineering 400

Analytics Vidhya

MAY 30, 2024

Introduction Python is the favorite language for most data engineers due to its adaptability and abundance of libraries for various tasks such as manipulation, machine learning, and data visualization. This post looks at the top 9 Python libraries necessary for data engineers to have successful careers.

Data Engineer 317

Data Engineer Data Engineering Data Engineering Data Engineering 317

Analytics Vidhya

SEPTEMBER 1, 2022

Introduction Data engineering and data science have been one of the hottest trends in the vocational market for quite some time. To build a successful career in data engineering, the aspirants need […]. The post Crucial Data Engineer Skills for a Successful Career appeared first on Analytics Vidhya.

Data Engineer 343

Data Engineer Data Engineering Data Engineering Data Engineering 343

Analytics Vidhya

AUGUST 6, 2022

With QlikView, you can analyze and visualize data and their relationships and use these analyzes to make decisions. It Supports various data sources, including […]. The post QlikView for Data Engineers Explained with Architecture appeared first on Analytics Vidhya.

Data Engineer 383

Data Engineer Data Engineering Data Engineering Data Engineering 383

Analytics Vidhya

SEPTEMBER 26, 2022

The post Web Scrapping- Tool for Data Engineering appeared first on Analytics Vidhya. The usefulness of the topic is one that easily helps other disciplines. Web content could be required in a way that makes it less effective to visit and use a website […].

Data Engineer 364

Data Engineer Data Engineering Data Engineering Data Engineering 364

Analytics Vidhya

JUNE 25, 2023

In a data-driven world, behind-the-scenes heroes like data engineers play a crucial role in ensuring smooth data flow. A data engineer investigates the issue, identifies a glitch in the e-commerce platform’s data funnel, and swiftly implements seamless data pipelines.

Data Engineer 306

Data Engineer Data Engineering Data Engineering Data Engineering 306

Analytics Vidhya

DECEMBER 12, 2022

Introduction Dear Data Engineers, this article is a very interesting topic. Let me give some flashback; a few years ago, Mr.Someone in the discussion coined the new word how ACID and BASE properties of DATA. The post Understand the ACID and BASE in Morden Data Engineering appeared first on Analytics Vidhya.

Data Engineer 353

Data Engineer Data Engineering Data Engineering Data Engineering 353

Analytics Vidhya

JULY 22, 2022

Since its inception, BigQuery has evolved into a more economical and fully managed data warehouse that can run lightning-fast […]. The post Google BigQuery Architecture for Data Engineers appeared first on Analytics Vidhya.

Data Engineer 344

Data Engineer Data Engineering Data Engineering Data Engineering 344

Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. The goal of this domain is to collect, store, and process data efficiently and efficiently so that it can be used to support business decisions and power data-driven applications.

Data Engineer 318

Data Engineer Data Engineering Data Engineering Data Engineering 318

Analytics Vidhya

JULY 9, 2022

The data repository should […]. The post Basics of Data Modeling and Warehousing for Data Engineers appeared first on Analytics Vidhya. Even asking basic questions like “how many customers we have in some places,” or “what product do our customers in their 20s buy the most” can be a challenge.

Data Engineer 349

Data Engineer Data Engineering Data Engineering Data Engineering 349

Analytics Vidhya

JANUARY 2, 2023

And so, there is no doubt that Data Engineers use it extensively to build and manage their ETL pipelines. The post Data Engineering 101– BranchPythonOperator in Apache Airflow appeared first on Analytics Vidhya. But not all the pipelines you build in Airflow will be straightforward.

Data Engineer 306

Data Engineer Data Engineering Data Engineering Data Engineering 306

Analytics Vidhya

AUGUST 4, 2024

These powerful tools are designed to manage and query intricate data relationships effortlessly. This article discusses […] The post Neo4j vs. Amazon Neptune: Graph Databases in Data Engineering appeared first on Analytics Vidhya.

Data Engineer 305

Data Engineer Data Engineering Data Engineering Data Engineering 305

Analytics Vidhya

APRIL 3, 2023

Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers. Nevertheless, setting up a streaming data pipeline to power such dashboards may […] The post Data Engineering for Streaming Data on GCP appeared first on Analytics Vidhya.

Data Engineer 319

Data Engineer Data Engineering Data Engineering Data Engineering 319

Analytics Vidhya

OCTOBER 10, 2022

The post Data Abstraction for Data Engineering with its Different Levels appeared first on Analytics Vidhya. As mentioned earlier, when determining requirements, we collect information about different business processes and […].

Data Engineer 349

Data Engineer Data Engineering Data Engineering Data Engineering 349

Analytics Vidhya

JUNE 20, 2023

Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.

Data Engineer 305

Data Engineer Data Engineering Data Engineering Data Engineering 305

Analytics Vidhya

JUNE 24, 2023

He is an experienced data engineer with a passion for problem-solving and a drive for continuous growth. Thus, providing valuable insights into the field of data engineering. Introduction We had an amazing opportunity to learn from Mr. Pavan.

Data Engineer 290

Data Engineer Data Engineering Data Engineering Data Engineering 290

insideBIGDATA

JUNE 14, 2023

Airbyte, creators of a fast-growing open-source data integration platform, made available results of the biggest data engineering survey in the market which provides insights into the latest trends, tools, and practices in data engineering – especially adoption of tools in the modern data stack.

Data Engineer 375

Data Engineer Data Engineering Data Engineering Data Engineering 375

Analytics Vidhya

JANUARY 2, 2023

While not all of us are tech enthusiasts, we all have a fair knowledge of how Data Science works in our day-to-day lives. All of this is based on Data Science which is […]. The post Step-by-Step Roadmap to Become a Data Engineer in 2023 appeared first on Analytics Vidhya.

Data Engineer 329

Data Engineer Data Engineering Data Engineering Data Engineering 329

Analytics Vidhya

JULY 25, 2023

In the world of data, two crucial roles play a significant part in unlocking the power of information: Data Scientists and Data Engineers. But what sets these wizards of data apart? Welcome to the ultimate showdown of Data Scientist vs Data Engineer! appeared first on Analytics Vidhya.

Data Scientist 271

Data Scientist Data Engineer Data Engineering Data Engineering 271

insideBIGDATA

APRIL 6, 2023

In this contributed article, data engineer Koushik Nandiraju discusses how a predictive data and analytics platform aligned with business objectives is no longer an option but a necessity.

Predictive Analytics 367

Predictive Analytics Analytics Analytics Data Warehouse 367

Flipboard

MARCH 12, 2025

Suri Nuthalapati, Technical Leader - Data & AI at Cloudera | Founder Trida Labs | Founder Farmioc. The rise of artificial intelligence(AI) is fundamentally changing the world of data analytics and data engineering. Advanced AI systemsAI agents that autonomously act, starting to change how

Data Engineer 169

Data Engineer Data Engineering Data Engineering Data Engineering 169

Analytics Vidhya

JULY 29, 2022

Introduction We are all pretty much familiar with the common modern cloud data warehouse model, which essentially provides a platform comprising a data lake (based on a cloud storage account such as Azure Data Lake Storage Gen2) AND a data warehouse compute engine […].

Azure 398

Azure Data Warehouse Data Lakes Analytics 398

Towards AI

OCTOBER 31, 2024

Data analytics has become a key driver of commercial success in recent years. The ability to turn large data sets into actionable insights can mean the difference between a successful campaign and missed opportunities. According to Gartner’s Hype Cycle, GenAI is at the peak, showcasing its potential to transform analytics.¹

Data Quality 122

Data Quality Analytics Analytics Clean Data 122

Analytics Vidhya

JULY 18, 2022

Life would be far easier if you didn’t have to scroll through job sites and referral sites to find and apply for the data science jobs you wanted. The post Analytics Vidhya Presents JOB-A-THON – Look No Further for Your Dream Data Science Job appeared first on Analytics Vidhya.

Data Science 399

Data Science Analytics Analytics Data Engineer 399

Analytics Vidhya

DECEMBER 12, 2022

Not just the leading technology giants in India but medium and small-scale companies are also betting on data science to revolutionize how business operations are performed. Data science is the field where large datasets are collected, analyzed, […].

Analytics 342

Analytics Analytics Data Science Data Visualization 342

Analytics Vidhya

NOVEMBER 26, 2023

In this Leading with Data episode, explore the analytics landscape with Dr. Swati Jain, a seasoned leader boasting over two decades of experience. From her unforeseen foray into analytics to steering EXL Analytics’ India business, Dr. Jain imparts invaluable insights into the ever-evolving world of data science.

Analytics 320

Analytics Analytics Data Science Data Analysis 320

KDnuggets

APRIL 19, 2024

The collection includes free courses on Python, SQL, Data Analytics, Business Intelligence, Data Engineering, Machine Learning, Deep Learning, Generative AI, and MLOps.

Data Science 383

Data Science Deep Learning Deep Learning SQL 383

KDnuggets

MAY 19, 2022

Read the best books on Programming, Statistics, Data Engineering, Web Scraping, Data Analytics, Business Intelligence, Data Applications, Data Management, Big Data, and Cloud Architecture.

Data Science 400

Data Science Big Data Big Data Business Intelligence 400

Analytics Vidhya

APRIL 28, 2023

With such large-scale data production, it is essential to have a field that focuses on deriving insights from it. What is data analytics? What tools help in data analytics? How can data analytics be applied to various industries? appeared first on Analytics Vidhya.

Analytics 306

Analytics Analytics Data Mining Data Mining 306

Analytics Vidhya

JULY 9, 2022

Introduction to Apache Airflow “Apache Airflow is the most widely-adopted, open-source workflow management platform for data engineering pipelines. Most organizations today with complex data pipelines to […]. The post Airflow for Orchestrating REST API Applications appeared first on Analytics Vidhya.

Data Pipeline 388

Data Pipeline Data Engineer Data Engineering Data Engineering 388

databricks

JULY 8, 2024

We are proud to announce two new analyst reports recognizing Databricks in the data engineering and data streaming space: IDC MarketScape: Worldwide Analytic.

Data Pipeline 344

Data Pipeline Cloud Data Data Engineer Data Engineering 344

Analytics Vidhya

JUNE 6, 2022

Introduction In this article, we will discuss advanced topics in hives which are required for Data-Engineering. Whenever we design a Big-data solution and execute hive queries on clusters it is the responsibility of a developer to optimize the hive queries. Performance Tuning in […].

Data Engineer 367

Data Engineer Data Engineering Data Engineering Data Engineering 367

Analytics Vidhya

JULY 17, 2024

Whether you are a data analyst, data scientist, or data engineer, summarizing and aggregating data is essential. As a data engineer working on […] The post Conditional Aggregation in SQL appeared first on Analytics Vidhya.

SQL 317

SQL Data Analyst Data Engineer Data Engineering 317

Analytics Vidhya

JULY 25, 2022

Introduction Apache Sqoop is a big data engine for transferring data between Hadoop and relational database servers. Sqoop transfers data from RDBMS (Relational Database Management System) such as MySQL and Oracle to HDFS (Hadoop Distributed File System). Big Data Sqoop can also be […].

Hadoop 353

Hadoop Big Data Big Data Data Engineer 353

KDnuggets

NOVEMBER 28, 2023

This posts talks about what needs to be taken care of in IoV data analysis, and shows the difference between a near real-time analytic platform and an actual real-time analytic platform with a real-world example.

Analytics 340

Analytics Analytics Big Data Big Data 340

databricks

FEBRUARY 3, 2025

In the rapidly evolving landscape of data engineering and analytics, speed, scalability, and simplicity are invaluable. Serverless compute addresses these needs by eliminating.

Data Engineer 313

Data Engineer Data Engineering Data Engineering Data Engineering 313

Analytics Vidhya

AUGUST 19, 2022

Introduction AWS Glue helps Data Engineers to prepare data for other data consumers through the Extract, Transform & Load (ETL) Process. The managed service offers a simple and cost-effective method of categorizing and managing big data in an enterprise. It provides organizations with […].

AWS 370

AWS ETL Big Data Big Data 370