Data Science Blog

NOVEMBER 15, 2023

New big data architectures and, above all, data sharing concepts such as Data Mesh are ideal for creating a common database for many data products and applications. The Event Log Data Model for Process Mining Process Mining as an analytical system can very well be imagined as an iceberg.

Data Modeling 245

Data Modeling Data Models Business Intelligence Business Intelligence 245

Analytics Vidhya

JULY 27, 2022

Introduction to Apache Flume Apache Flume is a data ingestion mechanism for gathering, aggregating, and transmitting huge amounts of streaming data from diverse sources, such as log files, events, and so on, to a centralized data storage. It has a simplistic and adaptable […].

Data Science 389

Data Science Analytics Analytics Hadoop 389

Analytics Vidhya

MAY 12, 2022

Initially, it was designed to handle log data solely, but later, it was developed to process event data. Introduction Apache Flume, a part of the Hadoop ecosystem, was developed by Cloudera. The Apache Flume tool is designed mainly for ingesting a high volume […]. The post Get to Know Apache Flume from Scratch!

Hadoop 392

Hadoop Data Science Analytics Analytics 392

Analytics Vidhya

JANUARY 30, 2023

Introduction Azure Functions is a serverless computing service provided by Azure that provides users a platform to write code without having to provision or manage infrastructure in response to a variety of events. Azure functions allow developers […] The post How to Develop Serverless Code Using Azure Functions?

Azure 327

Azure Database Analytics Analytics 327

Women in Big Data

NOVEMBER 4, 2024

After this inspiring start, mentees had the opportunity to participate in speed mentoring sessions, where they engaged with some of the best leaders in data science, AI, big data-driven solutions, sales, business development, data engineering, partnerships, and more.

Big Data 52

Big Data Big Data Data Engineering Data Engineer 52

Data Science Dojo

JANUARY 27, 2023

Obstacles for data engineers & developers  Collection and maintenance of data from different sources is itself a hectic task for data engineers and developers. Connectors are packaged as Docker images, which allows total flexibility over the technologies used to implement them.

Azure 370

Azure Data Science Data Pipeline Data Engineering 370

ODSC - Open Data Science

JANUARY 9, 2024

Now that we’re in 2024, it’s important to remember that data engineering is a critical discipline for any organization that wants to make the most of its data. These data professionals are responsible for building and maintaining the infrastructure that allows organizations to collect, store, process, and analyze data.

Data Engineering 97

Data Engineering Data Engineer Data Engineering Data Engineering 97

Analytics Vidhya

JANUARY 1, 2023

This article was published as a part of the Data Science Blogathon. convenient Introduction AWS Lambda is a serverless computing service that lets you run code in response to events while having the underlying compute resources managed for you automatically.

AWS 335

AWS Data Science Analytics Analytics 335

DataRobot Blog

JUNE 1, 2022

Join DataRobot and leading organizations June 7 and 8 at DataRobot AI Experience 2022 (AIX) , a unique virtual event that will help you rapidly unlock the power of AI for your most strategic business initiatives. Join the virtual event sessions in your local time across Asia-Pacific, EMEA, and the Americas. Join DataRobot AIX June 7–8.

Data Scientist 119

Data Scientist AI AI Machine Learning 119

Analytics Vidhya

DECEMBER 30, 2022

All these sites use some event streaming tool to monitor user activities. […]. Introduction Have you ever wondered how Instagram recommends similar kinds of reels while you are scrolling through your feed or ad recommendations for similar products that you were browsing on Amazon?

Apache Kafka 315

Apache Kafka Data Science Analytics Analytics 315

Analytics Vidhya

MARCH 7, 2023

Introduction Apache Flume is a tool/service/data ingestion mechanism for gathering, aggregating, and delivering huge amounts of streaming data from diverse sources, such as log files, events, and so on, to centralized data storage. Flume is a tool that is very dependable, distributed, and customizable.

Analytics 315

Analytics Analytics Hadoop Data Engineering 315

Data Science Dojo

OCTOBER 31, 2024

Specializing as a Data Scientist or Data Engineer Over time, you can pivot into roles focusing on machine learning and predictive modeling (Data Scientist) or building and maintaining data infrastructure (Data Engineer). This role builds a foundation for specialization.

Data Science 195

Data Science Data Scientist Machine Learning Machine Learning 195

KDnuggets

JANUARY 6, 2020

In this webinar, Jan 15 @ 12PM EST, we'll offer solutions to the common challenges data scientists and data engineers face when building a machine learning pipeline. Register now to attend live or to watch a recording afterwards.

Machine Learning 171

Machine Learning Machine Learning Data Scientist Data Engineer 171

ODSC - Open Data Science

JANUARY 2, 2024

We couldn’t be more excited to announce two events that will be co-located with ODSC East in Boston this April: The Data Engineering Summit and the Ai X Innovation Summit. These two co-located events represent an opportunity to dive even deeper into the topics and trends shaping these disciplines. Register for free today!

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

ODSC - Open Data Science

FEBRUARY 21, 2024

For the first time ever, the Data Engineering Summit will be in person! Co-located with the leading Data Science and AI Training Conference, ODSC East, this summit will gather the leading minds in Data Engineering in Boston on April 23rd and 24th. Interested in attending an ODSC event? Sign me up!

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

Pickl AI

NOVEMBER 4, 2024

Summary: The fundamentals of Data Engineering encompass essential practices like data modelling, warehousing, pipelines, and integration. Understanding these concepts enables professionals to build robust systems that facilitate effective data management and insightful analysis. What is Data Engineering?

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

ODSC - Open Data Science

JANUARY 30, 2024

Data engineering is a hot topic in the AI industry right now. And as data’s complexity and volume grow, its importance across industries will only become more noticeable. But what exactly do data engineers do? So let’s do a quick overview of the job of data engineer, and maybe you might find a new interest.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

Flipboard

MAY 16, 2025

While speaking at AIMs event DES 2025, Manjunatha G, engineering and site leader at the 3M Global Technology Centre, laid out a practical path to integrate AI agents into data engineering workflows.

Data Pipeline 73

Data Pipeline Data Engineer Data Engineering Data Engineering 73

ODSC - Open Data Science

JANUARY 18, 2024

Data engineering is a rapidly growing field, and there is a high demand for skilled data engineers. If you are a data scientist, you may be wondering if you can transition into data engineering. In this blog post, we will discuss how you can become a data engineer if you are a data scientist.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

ODSC - Open Data Science

FEBRUARY 6, 2024

Data engineering has become an integral part of the modern tech landscape, driving advancements and efficiencies across industries. So let’s explore the world of open-source tools for data engineers, shedding light on how these resources are shaping the future of data handling, processing, and visualization.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

Data Science Dojo

NOVEMBER 15, 2023

Here are nine of the top AI conferences happening in North America in 2023 and 2024 that you must attend: Top AI events and conferences in North America attend in 2023 Big Data and AI TORONTO 2023: Big Data and AI Toronto is the premier event for data professionals in Canada.

Machine Learning 195

Machine Learning Machine Learning AI AI 195

Hacker News

JULY 18, 2024

ABOUT EVENTUAL Eventual is a data platform that helps data scientists and engineers build data applications across ETL, analytics and ML/AI. OUR PRODUCT IS OPEN-SOURCE AND USED AT ENTERPRISE SCALE Our distributed data engine Daft [link] is open-sourced and runs on 800k CPU cores daily.

ML 130

ML ML Python ETL 130

ODSC - Open Data Science

FEBRUARY 15, 2024

We couldn’t be more excited to announce the first sessions for our second annual Data Engineering Summit , co-located with ODSC East this April. Join us for 2 days of talks and panels from leading experts and data engineering pioneers. Is Gen AI A Data Engineering or Software Engineering Problem?

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

ODSC - Open Data Science

OCTOBER 17, 2024

Come for the hands-on AI deep dives, but stay for the immersive events! Data Engineering Summit The Data Engineering Summit , co-located with ODSC West, is your ticket to optimizing efficiency, enhancing scalability, and successfully tackling the toughest data challenges. Check them out below!

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

ODSC - Open Data Science

MARCH 22, 2024

Be sure to check out his talk, “ Building Data Contracts with Open Source Tools ,” there! Data engineering is a critical function in all industries. However, data engineering grows exponentially as the company grows, acquires, or merges with others. He is passionate about software engineering and all things data.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

phData

AUGUST 11, 2023

Data engineering in healthcare is taking a giant leap forward with rapid industrial development. However, data collection and analysis have been commonplace in the healthcare sector for ages. Data Engineering in day-to-day hospital administration can help with better decision-making and patient diagnosis/prognosis.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

Mlearning.ai

MAY 16, 2023

Data engineering is a rapidly growing field that designs and develops systems that process and manage large amounts of data. There are various architectural design patterns in data engineering that are used to solve different data-related problems.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

phData

APRIL 25, 2024

Welcome to April’s employee series, Beyond the Data! We’re featuring Data Engineer Franco Borgiani from our LATAM office. I was born there but left when I decided to pursue a career as a Software Engineer. What do you actually do on any given day as a Data Engineer? As a Data Engineer, my daily tasks vary.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

ODSC - Open Data Science

MAY 5, 2023

Data science and data engineering are incredibly resource intensive. By using cloud computing, you can easily address a lot of these issues, as many data science cloud options have databases on the cloud that you can access without needing to tinker with your hardware.

Data Science 52

Data Science Data Engineering Data Engineer Data Engineering 52

ODSC - Open Data Science

MARCH 17, 2025

Well, with the Open Pass, youll also enjoy a full series of networking events; the perfect way to unwind after a day after a full day of learning. At these networking events, youll be able to meet some of your fellow data practitioners who have come from all across the country. Theyre running out quickly, so dont missout!

Data Science 52

Data Science Computer Science Computer Science Machine Learning 52

AWS Machine Learning Blog

APRIL 3, 2025

Lambda enables serverless, event-driven data processing tasks, allowing for real-time transformations and calculations as data arrives. Step Functions complements this by orchestrating complex workflows, coordinating multiple Lambda functions, and managing error handling for sophisticated data processing pipelines.

AWS 100

AWS Data Governance Data Silos SQL 100

phData

SEPTEMBER 19, 2023

Welcome to Beyond the Data, a curious series that investigates the people behind the talent of phData. In this article, we’re featuring Deepa Ganiger, who serves as a Solution Architect/Lead Data Engineer at phData. On the weekends, I volunteer and take part in community events. Describing Deepa in just a few paragraphs?

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

Data Science Blog

JUNE 27, 2023

In this representation, there is a separate store for events within the speed layer and another store for data loaded during batch processing. The serving layer acts as a mediator, enabling subsequent applications to access the data. This architectural concept relies on event streaming as the core element of data delivery.

Big Data 130

Big Data Big Data Apache Kafka Database 130

ODSC - Open Data Science

JANUARY 11, 2024

Getting Started with AI in High-Risk Industries, How to Become a Data Engineer, and Query-Driven Data Modeling How To Get Started With Building AI in High-Risk Industries This guide will get you started building AI in your organization with ease, axing unnecessary jargon and fluff, so you can start today.

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

ODSC - Open Data Science

FEBRUARY 3, 2023

The Ascent of ChatGPT, Data Engineering Summit Highlights, and Ten Industries Generative AI is Disrupting The Ascent of ChatGPT Since its release in November 2022 by OpenAI, ChatGPT has taken the world by storm. Learn more about how you can volunteer for either the in-person or virtual team and get a free ticket to the event.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

AWS Machine Learning Blog

JANUARY 10, 2024

Specialist Data Engineering at Merck, and Prabakaran Mathaiyan, Sr. ML Engineer at Tiger Analytics. EventBridge monitors status change events to automatically take actions with simple rules. EventBridge monitors SageMaker for the model registration event and triggers an event rule that invokes a Lambda function.

ML 135

ML ML AWS Machine Learning 135

Data Science 101

FEBRUARY 15, 2020

Azure Cognitive Services Named Entity Recognition gets some new types Persontype, product, event, organization, date are just some of them Amazon Aurora PostgreSQL Supports Machine Learning Aurora PostgreSQL can now use SQL to call ML models created with SageMaker. Google Announces BigQuery Data Challenge. Training and Courses.

Cloud Data 112

Cloud Data Data Science Deep Learning Deep Learning 112