Hacker News

NOVEMBER 19, 2024

Here are a few of the things that you might do as an AI Engineer at TigerEye: - Design, develop, and validate statistical models to explain past behavior and to predict future behavior of our customers’ sales teams - Own training, integration, deployment, versioning, and monitoring of ML components - Improve TigerEye’s existing metrics collection and (..)

Computer Science 162

Computer Science Computer Science ML ML 162

Pickl AI

OCTOBER 6, 2024

Summary: This blog explores the key differences between ETL and ELT, detailing their processes, advantages, and disadvantages. This blog explores the fundamental concepts of ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform), two pivotal methods in modern data architectures. What is ETL?

ETL 52

ETL Data Warehouse Data Quality Data Lakes 52

Data Science Blog

SEPTEMBER 19, 2023

With the evolution of cloud computing, many organizations are now migrating their Data Warehouse Systems to the cloud for better scalability, flexibility, and cost-efficiency. So why using IaC for Cloud Data Infrastructures? Infrastructure as Code (IaC) can be a game-changer in this scenario.

Data Warehouse 259

Data Warehouse Azure SQL Database 259

Smart Data Collective

AUGUST 17, 2022

But keep in mind one thing which is you have to either replicate the topics in your cloud cluster or you will have to develop a custom connector to read and copy back and forth from the cloud to the application. A three-step ETL framework job should do the trick. Step 3: Create an ETL job and save that data to a data lake.

Apache Kafka 130

Apache Kafka ETL Data Lakes AWS 130

Data Science Dojo

FEBRUARY 20, 2023

Best tools and platforms for MLOPs – Data Science Dojo Google Cloud Platform Google Cloud Platform is a comprehensive offering of cloud computing services. It offers a range of products, including Google Cloud Storage, Google Cloud Deployment Manager, Google Cloud Functions, and others.

Machine Learning 195

Machine Learning Machine Learning AWS Azure 195

Smart Data Collective

MAY 20, 2019

The popular tools, on the other hand, include Power BI, ETL, IBM Db2, and Teradata. Cloud Computing and Related Mechanics. Big data, advanced analytics, machine learning, none of these technologies would exist without cloud computing and the resulting infrastructure.

Analytics 111

Analytics Analytics Data Analyst Machine Learning 111

Data Science Blog

JUNE 21, 2023

Cloud-Based infrastructure with process mining? Depending on the data strategy of one organization, one cost-effective approach to process mining could be to leverage cloud computing resources. But costs won’t decrease only migrating from on-premises to cloud and vice versa.

Big Data 130

Big Data Big Data Data Engineering Data Engineering 130

Alation

JANUARY 17, 2023

Reverse ETL tools. The rise of cloud computing and cloud data warehousing has catalyzed the growth of the modern data stack. The rise of cloud computing and cloud data warehousing has catalyzed the growth of the modern data stack. A Note on the Shift from ETL to ELT. Data orchestration tools.

Data Warehouse 52

Data Warehouse ETL Tableau Cloud Data 52

Applied Data Science

AUGUST 2, 2021

They build production-ready systems using best-practice containerisation technologies, ETL tools and APIs. They are skilled at deploying to any cloud or on-premises infrastructure. Data engineers are the glue that binds the products of data scientists into a coherent and robust data pipeline.

Data Science 130

Data Science Data Scientist ML ML 130

phData

FEBRUARY 21, 2025

In this post, we will be particularly interested in the impact that cloud computing left on the modern data warehouse. In the cloud, the physical distance between the data source and the cloud data warehouse region can impact latency. Data integrations and pipelines can also impact latency.

Data Warehouse 52

Data Warehouse Cloud Data ETL Cloud Computing 52

Data Science Dojo

JULY 3, 2024

Data Engineering : Building and maintaining data pipelines, ETL (Extract, Transform, Load) processes, and data warehousing. Cloud Computing : Utilizing cloud services for data storage and processing, often covering platforms such as AWS, Azure, and Google Cloud.

Data Science 150

Data Science Machine Learning Machine Learning Data Visualization 150

ODSC - Open Data Science

JUNE 19, 2023

This entails the use of other technologies such as distributed computing, edge computing, and cloud computing. When it comes to data integration, RTOS can work with systems that employ data warehousing, API management, and ETL technologies. Moreover, RTOS is built to be scalable and flexible.

Big Data 96

Big Data Big Data Artificial Intelligence Artificial Intelligence 96

IBM Journey to AI blog

JUNE 29, 2023

These tools enable the extraction, transformation, and loading (ETL) of data from various sources. Cloud-based solutions Cloud computing offers scalability, flexibility, and accessibility, making it an ideal choice for integrated business planning.

Analytics 94

Analytics Analytics Business Intelligence Business Intelligence 94

Precisely

MARCH 9, 2023

As cloud computing platforms make it possible to perform advanced analytics on ever larger and more diverse data sets, new and innovative approaches have emerged for storing, preprocessing, and analyzing information. Hadoop, Snowflake, Databricks and other products have rapidly gained adoption.

Data Lakes 59

Data Lakes Data Warehouse Hadoop Big Data 59

Dataconomy

FEBRUARY 23, 2023

This involves working with various tools and technologies, such as ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes, to move data from its source to its destination. Cloud computing: Cloud computing provides a scalable and cost-effective solution for managing and processing large volumes of data.

Big Data 91

Big Data Big Data Data Engineering Data Engineering 91

Precisely

APRIL 6, 2023

Twenty years ago, top-performing organizations were using “extract, transform, load” (ETL) processes to normalize and aggregate data for business analytics. Cloud computing technology improves the speed and scale at which organizations can process data. Real-time data is the goal.

Data Quality 52

Data Quality Data Pipeline Analytics Analytics 52

Pickl AI

APRIL 6, 2023

Answer : Microsoft Azure is a cloud computing platform and service that Microsoft provides. This includes Database System Management (SQL or Non-SQL), Data Warehousing, Machine Learning, programming basics, and ETL. Following the potential questions in general that you might get asked: 1. What is Microsoft Azure?

Azure 52

Azure Data Engineering Data Engineering Data Engineer 52

Pickl AI

NOVEMBER 4, 2024

Key components of data warehousing include: ETL Processes: ETL stands for Extract, Transform, Load. ETL is vital for ensuring data quality and integrity. These tools help organisations harness the power of cloud computing for Data Engineering solutions.

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

phData

JUNE 27, 2023

With the rise of cloud computing, the MDS has evolved even further to include cloud-based storage and tools for analysis. Having a singular ELT/ETL solution is a firm requirement Dbt does not handle ingestion well, it works best when paired with another ingestion tool such as Fivetran.

SQL 52

SQL Database Data Quality Data Warehouse 52

The MLOps Blog

DECEMBER 7, 2022

The inherent cost of cloud computing : To illustrate the point, Argentina’s minimum wage is currently around 200 dollars per month. And that’s when what usually happens, happened: We came for the ML models, we stayed for the ETLs. But even when the ETLs were well thought out, they were a bit “outdated” in their approach.

ML 52

ML ML AWS ETL 52

Towards AI

FEBRUARY 11, 2025

Consider these common scenarios: A perfect validation script cant fix inconsistent data entry practices The most robust ETL pipeline cant resolve disagreements about business rules Real-time quality monitoring cant replace clear data ownership.

Data Quality 54

Data Quality Data Engineering Data Engineering Data Engineer 54

Dataconomy

FEBRUARY 12, 2024

The acronym ETL—Extract, Transform, Load—has long been the linchpin of modern data management, orchestrating the movement and manipulation of data across systems and databases. However, the exponential growth in data volume, velocity, and variety is challenging the traditional paradigms of ETL, ushering in a transformative era.

ETL 195

ETL Data Governance Machine Learning Machine Learning 195

Flipboard

FEBRUARY 24, 2025

In this post, we demonstrate how to use Amazon OpenSearch Service with purpose-built document ETL tools, Aryn DocParse and Sycamore, to quickly build a RAG application that relies on complex documents. We use over 75 PDF reports from the National Transportation Safety Board (NTSB) about aircraft incidents.

ETL 142

ETL Cloud Computing Machine Learning Machine Learning 142

AWS Machine Learning Blog

JANUARY 7, 2025

His work is focused on the implementation of efficient ETL data analytics pipelines, and solving business problems via automation, experimenting and innovating using AWS services with a code-first approach using AWS CDK. Martin Gregory is a Senior Market Data Technician at Parameta Solutions with over 25 years of experience.

AWS 104

AWS AI AI ML 104