insideBIGDATA

APRIL 15, 2024

In this contributed article, Rob Gibbon, Product Manager at Canonical, suggests that data engineers typically know what they need to get done. If you're working on premise, it can be hard to get data-intensive solutions off the ground quickly. The problem is that their environment doesn't always make it easy.

Data Engineer 195

Data Engineer Data Engineering Data Engineering Data Engineering 195

Analytics Vidhya

JUNE 14, 2021

ArticleVideo Book This article was published as a part of the Data Science Blogathon Introduction First of all, we are surrounded by data in day-to-day. The post Data Engineering – Concepts and Importance appeared first on Analytics Vidhya.

Data Engineer 336

Data Engineer Data Engineering Data Engineering Data Engineering 336

insideBIGDATA

JUNE 14, 2023

Airbyte, creators of a fast-growing open-source data integration platform, made available results of the biggest data engineering survey in the market which provides insights into the latest trends, tools, and practices in data engineering – especially adoption of tools in the modern data stack.

Data Engineer 369

Data Engineer Data Engineering Data Engineering Data Engineering 369

insideBIGDATA

JANUARY 2, 2024

Straight from the executive suite, you’ll learn about what’s predicted to happen with AI, GenAI, LLMs, BI, data science, data engineering, and much more. From the company’s point of view 2024 should be quite a year! Enjoy these special perspectives from one of our industry’s best known movers and shakers.

Big Data 384

Big Data Big Data Data Engineer Data Engineering 384

Analytics Vidhya

AUGUST 31, 2022

This article was published as a part of the Data Science Blogathon. The need to maximize company efficiency and profitability has led the world to leverage data as a powerful tool. Data is reusable, everywhere, replicable, easily transferable, and […]. The post Why Big Data needs to become Smart Data?

Big Data 260

Big Data Big Data Data Science Analytics 260

Analytics Vidhya

NOVEMBER 11, 2020

This article was published as a part of the Data Science Blogathon. Overview With the demand for big data and machine learning, this article. The post Introduction to Spark MLlib for Big Data and Machine Learning appeared first on Analytics Vidhya.

Big Data 339

Big Data Big Data Machine Learning Machine Learning 339

Analytics Vidhya

JULY 25, 2022

This article was published as a part of the Data Science Blogathon. Introduction Apache Sqoop is a big data engine for transferring data between Hadoop and relational database servers. Big Data Sqoop can also be […].

Hadoop 342

Hadoop Big Data Big Data Data Engineer 342

Analytics Vidhya

SEPTEMBER 1, 2021

This article was published as a part of the Data Science Blogathon A data scientist’s ability to extract value from data is closely related to how well-developed a company’s data storage and processing infrastructure is.

ETL 239

ETL Data Engineer Data Engineering Data Engineering 239

Analytics Vidhya

JUNE 6, 2022

This article was published as a part of the Data Science Blogathon. Introduction In this article, we will discuss advanced topics in hives which are required for Data-Engineering. Performance Tuning in […].

Data Engineer 332

Data Engineer Data Engineering Data Engineering Data Engineering 332

Analytics Vidhya

AUGUST 19, 2022

This article was published as a part of the Data Science Blogathon. Introduction AWS Glue helps Data Engineers to prepare data for other data consumers through the Extract, Transform & Load (ETL) Process. It provides organizations with […].

AWS 358

AWS ETL Big Data Big Data 358

Analytics Vidhya

JULY 29, 2022

This article was published as a part of the Data Science Blogathon. Introduction One of the sources of Big Data is the traditional application management system or the interaction of applications with relational databases using RDBMS. Big Data storage and analysis […].

Big Data 385

Big Data Big Data Database Data Science 385

Analytics Vidhya

AUGUST 17, 2022

This article was published as a part of the Data Science Blogathon. Introduction In this article, we will introduce you to the big data ecosystem and the role of Apache Spark in Big data. We will also cover the Distributed database system, the backbone of big data. Almost […].

Big Data 260

Big Data Big Data Data Science Database 260

Smart Data Collective

SEPTEMBER 28, 2022

A data management solution can help you make better business decisions by giving you access to the right information at the right time. Data engineering services can analyze large amounts of data and identify trends that would otherwise be missed. Big data management increases the reliability of your data.

Big Data 106

Big Data Big Data Big Data Analytics Big Data Analytics 106

Analytics Vidhya

OCTOBER 12, 2022

This article was published as a part of the Data Science Blogathon. Introduction Since the 1970s, relational database management systems have solved the problems of storing and maintaining large volumes of structured data.

Hadoop 331

Hadoop Big Data Big Data Data Science 331

ODSC - Open Data Science

JANUARY 2, 2024

We couldn’t be more excited to announce two events that will be co-located with ODSC East in Boston this April: The Data Engineering Summit and the Ai X Innovation Summit. Data Engineering Summit Our second annual Data Engineering Summit will be in-person for the first time! Learn more about them below.

Data Engineer 52

Data Engineer Data Engineering Data Engineering Data Engineering 52

insideBIGDATA

APRIL 6, 2023

In this contributed article, data engineer Koushik Nandiraju discusses how a predictive data and analytics platform aligned with business objectives is no longer an option but a necessity.

Predictive Analytics 367

Predictive Analytics Analytics Analytics Data Warehouse 367

Analytics Vidhya

OCTOBER 9, 2022

This article was published as a part of the Data Science Blogathon. Introduction Apache Hadoop is an open-source framework designed to facilitate interaction with big data. Still, for those unfamiliar with this technology, one question arises, what is big data?

Hadoop 237

Hadoop Apache Hadoop Big Data Big Data 237

Analytics Vidhya

MAY 30, 2021

ArticleVideo Book This article was published as a part of the Data Science Blogathon Introduction Big data is the collection of data that is vast. The post Integration of Python with Hadoop and Spark appeared first on Analytics Vidhya.

Hadoop 367

Hadoop Python Big Data Big Data 367

Analytics Vidhya

FEBRUARY 3, 2022

This article was published as a part of the Data Science Blogathon. In this article, we shall discuss the upcoming innovations in the field of artificial intelligence, big data, machine learning and overall, Data Science Trends in 2022. Times change, technology improves and our lives get better.

Data Science 312

Data Science Natural Language Processing Deep Learning Deep Learning 312

Analytics Vidhya

JULY 22, 2022

This article was published as a part of the Data Science Blogathon. Introduction The big data industry is growing daily and needs tools to process vast volumes of data. That’s why you need to know about Apache Kafka, a publish-subscribe messaging system you can use to build distributed applications.

Apache Kafka 316

Apache Kafka Big Data Big Data Data Science 316

Analytics Vidhya

SEPTEMBER 13, 2022

This article was published as a part of the Data Science Blogathon. Introduction In the era of big data, it’s no surprise that more and more marketers are using data science in marketing to better position their brands, products, and services in today’s hyper-competitive marketplace.

Data Science 349

Data Science Big Data Big Data Analytics 349

Analytics Vidhya

AUGUST 29, 2022

This article was published as a part of the Data Science Blogathon. Introduction Apache Flink is a big data framework that allows programmers to process huge amounts of data in a very efficient and scalable way. The […].

Big Data 277

Big Data Big Data Data Science Analytics 277

Analytics Vidhya

AUGUST 1, 2022

This article was published as a part of the Data Science Blogathon. HBase provides a fault-tolerant manner of storing sparse data sets, which are prevalent in several big data use cases. It is ideal for real-time data processing or […].

Hadoop 329

Hadoop Big Data Big Data Data Science 329

Pickl AI

JULY 25, 2023

Unfolding the difference between data engineer, data scientist, and data analyst. Data engineers are essential professionals responsible for designing, constructing, and maintaining an organization’s data infrastructure. Data Visualization: Matplotlib, Seaborn, Tableau, etc.

Data Engineer 52

Data Engineer Data Engineering Data Engineering Data Engineering 52

Analytics Vidhya

AUGUST 31, 2022

This article was published as a part of the Data Science Blogathon. Introduction A data lake is a central data repository that allows us to store all of our structured and unstructured data on a large scale.

Data Lakes 264

Data Lakes Big Data Big Data Data Science 264

ODSC - Open Data Science

MAY 5, 2023

Data science and data engineering are incredibly resource intensive. By using cloud computing, you can easily address a lot of these issues, as many data science cloud options have databases on the cloud that you can access without needing to tinker with your hardware. Delta & Databricks Make This A Reality!

Data Science 52

Data Science Data Engineer Data Engineering Data Engineering 52

Analytics Vidhya

DECEMBER 26, 2022

This article was published as a part of the Data Science Blogathon. NoSQL databases are often used for big data and real-time web applications. Introduction A NoSQL database is a non-relational database that does not use the traditional table-based schema of a relational database.

Database 257

Database Big Data Big Data Data Science 257

Smart Data Collective

JULY 25, 2019

Not long ago, big data was one of the most talked about tech trends , as was artificial intelligence (AI). But, in case people need a reminder of how fast technology evolves , they only need to consider something newer — big data AI. So, big data AI can both compile information and respond to it.

Big Data 83

Big Data Big Data AI AI 83

Analytics Vidhya

SEPTEMBER 28, 2022

This article was published as a part of the Data Science Blogathon. Introduction to Data Warehouse SQL Data Warehouse is also a cloud-based data warehouse that uses Massively Parallel Processing (MPP) to run complex queries across petabytes of data rapidly. Import big […].

Data Warehouse 247

Data Warehouse Azure SQL Big Data 247

Towards AI

FEBRUARY 2, 2024

I hope that you have sufficient knowledge of big data and Hadoop concepts like Map, reduce, transformations, actions, lazy evaluation, and many more topics in Hadoop and Spark. Extracting day, month and year from date column: #extract year, month, and day details from the data framedf.select(year("date column").distinct().orderBy(year("date

Data Engineering 115

Data Engineering Data Engineer Data Engineering Data Engineering 115

Data Science Blog

JUNE 21, 2023

In this article, we will highlight the key elements when it comes to process mining architectures as well as the most common mistakes, to help organizations leverage the power of process mining while maintain cost control. By considering lightweighted data preprocessing techniques organizations can save both time and money.

Big Data 130

Big Data Big Data Data Engineering Data Engineer 130

Towards AI

JULY 24, 2023

Photo by h heyerlein on Unsplash In thе agе of big data, organizations arе incrеasingly rеliant on professionals with spеcializеd skills to unlock thе potential hiddеn within thеir vast databases.

Data Analyst 98

Data Analyst Data Scientist Data Engineer Data Engineering 98

Dataconomy

FEBRUARY 23, 2023

Data engineers play a crucial role in managing and processing big data. They are responsible for designing, building, and maintaining the infrastructure and tools needed to manage and process large volumes of data effectively. What is data engineering?

Big Data 91

Big Data Big Data Data Engineer Data Engineering 91

Becoming Human

JANUARY 23, 2023

And for searching the term you landed on multiple blogs, articles as well YouTube videos, because this is a very vast topic, or I, would say a vast Industry. I’m not saying those are incorrect or wrong even though every article has its mindset behind the term ‘ Data Science ’.

Data Science 105

Data Science Big Data Big Data Deep Learning 105

insideBIGDATA

AUGUST 22, 2023

In this contributed article, Mayank Mehra, head of product management at Modak, shares the importance of incorporating effective data observability practices to equip data and analytics leaders with essential insights into the health of their data stacks.

Data Observability 370

Data Observability Analytics Analytics Data Engineer 370

ODSC - Open Data Science

JUNE 4, 2024

To help our data scientists, data engineers, AI practitioners and data professionals of all types stay at the forefront of their fields, this day will be dedicated to hands-on training and workshops from leading experts. Friday, September 6th The final day of ODSC Europe will start strong with Keynote talks.

Data Science 75

Data Science Big Data Analytics Big Data Analytics Data Engineer 75

Women in Big Data

SEPTEMBER 11, 2023

This is the article written by one of Women in Big Data India volunteers about her internship journey at Salesforce. “Hello, I’m Tamanna Khaitan , a final-year undergraduate at Indira Gandhi Delhi Technical University for Women, pursuing Electronics and Communications. I got to work on Devops and backend.

Big Data 52

Big Data Big Data Data Engineering Data Engineer 52