Building a Scalable ETL with SQL + Python
KDnuggets
APRIL 21, 2022
This post will look at building a modular ETL pipeline that transforms data with SQL and visualizes it with Python and R.
ETL Python SQL 
5 Error Handling Patterns in Python (Beyond Try-Except)
KDnuggets
JUNE 6, 2025
Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter 5 Error Handling Patterns in Python (Beyond Try-Except) Stop letting errors crash your app.
Python 
165
165 Join 17,000+
professionals
Sign Up for our Newsletter
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
KDnuggets News, April 27: A Brief Introduction to Papers With Code; Machine Learning Books You Need To Read In 2022
KDnuggets
APRIL 27, 2022
A Brief Introduction to Papers With Code; Machine Learning Books You Need To Read In 2022; Building a Scalable ETL with SQL + Python; 7 Steps to Mastering SQL for Data Science; Top Data Science Projects to Build Your Skills.
Machine Learning 270
270 
Remote Data Science Jobs: 5 High-Demand Roles for Career Growth
Data Science Dojo
OCTOBER 31, 2024
Key Skills Proficiency in SQL is essential, along with experience in data visualization tools such as Tableau or Power BI. Strong analytical skills and the ability to work with large datasets are critical, as is familiarity with data modeling and ETL processes. Familiarity with machine learning, algorithms, and statistical modeling.
Data Science 195
195 
Run the Full DeepSeek-R1-0528 Model Locally
KDnuggets
JUNE 9, 2025
By subscribing you accept KDnuggets Privacy Policy Leave this field empty if youre human: Latest Posts Run the Full DeepSeek-R1-0528 Model Locally 7 Cool Python Projects to Automate the Boring Stuff 5 Error Handling Patterns in Python (Beyond Try-Except) 10 Awesome OCR Models for 2025 WTF is GRPO?!?
Bringing Declarative Pipelines to the Apache Spark™ Open Source Project
databricks
JUNE 12, 2025
Over the past decade, we’ve seen Apache Spark evolve from a powerful general-purpose compute engine into a critical layer of the Open Lakehouse Architecture - with Spark SQL, Structured Streaming, open table formats, and unified governance serving as pillars for modern data platforms. With the recent release of Apache Spark 4.0,
SQL 130
130 
Why using Infrastructure as Code for developing Cloud-based Data Warehouse Systems?
Data Science Blog
SEPTEMBER 19, 2023
This brings reliability to data ETL (Extract, Transform, Load) processes, query performances, and other critical data operations. using for loops in Python). The following Terraform script will create an Azure Resource Group, a SQL Server, and a SQL Database. So why using IaC for Cloud Data Infrastructures?
Data Warehouse 259
259 
Streamlining ETL data processing at Talent.com with Amazon SageMaker
AWS Machine Learning Blog
DECEMBER 14, 2023
Our pipeline belongs to the general ETL (extract, transform, and load) process family that combines data from multiple sources into a large, central repository. The system includes feature engineering, deep learning model architecture design, hyperparameter optimization, and model evaluation, where all modules are run using Python.
ETL 121
121 
Explore data with ease: Use SQL and Text-to-SQL in Amazon SageMaker Studio JupyterLab notebooks
AWS Machine Learning Blog
APRIL 16, 2024
They then use SQL to explore, analyze, visualize, and integrate data from various sources before using it in their ML training and inference. Previously, data scientists often found themselves juggling multiple tools to support SQL in their workflow, which hindered productivity.
SQL 130
130 
TigerEye (YC S22) Is Hiring a Full Stack Engineer
Hacker News
NOVEMBER 19, 2024
Here are a few of the things that you might do as an AI Engineer at TigerEye: - Design, develop, and validate statistical models to explain past behavior and to predict future behavior of our customers’ sales teams - Own training, integration, deployment, versioning, and monitoring of ML components - Improve TigerEye’s existing metrics collection and (..)
Computer Science 146
146 
5 Reasons Why SQL is Still the Most Accessible Language for New Data Scientists
ODSC - Open Data Science
APRIL 6, 2023
For budding data scientists and data analysts, there are mountains of information about why you should learn R over Python and the other way around. Though both are great to learn, what gets left out of the conversation is a simple yet powerful programming language that everyone in the data science world can agree on, SQL.
SQL 98
98 
Data engineer
Dataconomy
JUNE 12, 2025
Technical skills Proficiency in programming languages: Familiarity with languages like C#, Java, Python, R, Ruby, Scala, and SQL is essential for building data solutions. Familiarity with ETL tools and data warehousing concepts: Knowledge of tools designed to extract, transform, and load data is crucial.
Navigate your way to success – Top 10 data science careers to pursue in 2023
Data Science Dojo
MAY 10, 2023
Here, we outline the essential skills and qualifications that pave way for data science careers: Proficiency in Programming Languages – Mastery of programming languages such as Python, R, and SQL forms the foundation of a data scientist’s toolkit.
Data Science 368
368 
Essential data engineering tools for 2023: Empowering for management and analysis
Data Science Dojo
JULY 6, 2023
These tools provide data engineers with the necessary capabilities to efficiently extract, transform, and load (ETL) data, build data pipelines, and prepare data for analysis and consumption by other applications. dbt focuses on transforming raw data into analytics-ready tables using SQL-based transformations.
Data Engineering 195
195 
An integrated experience for all your data and AI with Amazon SageMaker Unified Studio (preview)
DECEMBER 11, 2024
Data processing and SQL analytics Analyze, prepare, and integrate data for analytics and AI using Amazon Athena, Amazon EMR, AWS Glue, and Amazon Redshift. With SageMaker Unified Studio notebooks, you can use Python or Spark to interactively explore and visualize data, prepare data for analytics and ML, and train ML models.
SQL 159
159 
ETL Process Explained: Essential Steps for Effective Data Management
Pickl AI
OCTOBER 17, 2024
Summary: The ETL process, which consists of data extraction, transformation, and loading, is vital for effective data management. Introduction The ETL process is crucial in modern data management. What is ETL? ETL stands for Extract, Transform, Load.
ETL 52
52 
Data Science Career Paths: Analyst, Scientist, Engineer – What’s Right for You?
How to Learn Machine Learning
APRIL 26, 2025
The processes of SQL, Python scripts, and web scraping libraries such as BeautifulSoup or Scrapy are used for carrying out the data collection. Tools like Python (with pandas and NumPy), R, and ETL platforms like Apache NiFi or Talend are used for data preparation before analysis.
Data Science 52
52 
What is Open Database Connectivity (ODBC) and Why Is It Important?
Pickl AI
NOVEMBER 4, 2024
Coming to APIs again, discover how to use ChatGPT APIs in Python by clicking on the link. Each database type requires its specific driver, which interprets the application’s SQL queries and translates them into a format the database can understand. INSERT : Add new records to a table. UPDATE : Modify existing records in a table.
Database 52
52 
A Guide to Choose the Best Data Science Bootcamp
Data Science Dojo
JULY 3, 2024
They cover a wide range of topics, ranging from Python, R, and statistics to machine learning and data visualization. Here’s a list of key skills that are typically covered in a good data science bootcamp: Programming Languages : Python : Widely used for its simplicity and extensive libraries for data analysis and machine learning.
Data Science 150
150 
How Formula 1® uses generative AI to accelerate race-day issue resolution
AWS Machine Learning Blog
FEBRUARY 18, 2025
The assistant is connected to internal and external systems, with the capability to query various sources such as SQL databases, Amazon CloudWatch logs, and third-party tools to check the live system health status. Creating ETL pipelines to transform log data Preparing your data to provide quality results is the first step in an AI project.
AWS 76
76 
How SnapLogic built a text-to-pipeline application with Amazon Bedrock to translate business intent into action
NOVEMBER 24, 2023
This use case highlights how large language models (LLMs) are able to become a translator between human languages (English, Spanish, Arabic, and more) and machine interpretable languages (Python, Java, Scala, SQL, and so on) along with sophisticated internal reasoning. Room for improvement!
Database 157
157 
IBM watsonx Platform: Compliance obligations to controls mapping
IBM Journey to AI blog
OCTOBER 30, 2024
Notably, the prompt text, model references, and prompt engineering parameters are meticulously formatted as Python code within notebooks, allowing for seamless programmable interaction. This approach enables centralized access and sharing while minimizing extract, transform and load (ETL) processes and data duplication.
Launch HN: Chonkie (YC X25) – Open-Source Library for Advanced Chunking
Hacker News
JUNE 9, 2025
Python: https://github.com/chonkie-inc/chonkie TypeScript: https://github.com/chonkie-inc/chonkie-ts Here's a video showing our code chunker: https://youtu.be/Xclkh6bU1P0. 200k+ tokens) with many SQL snippets, query results and database metadata (e.g. table and column info).
Database 85
85 
Difference Between JDBC and ODBC in Database Connectivity
Pickl AI
NOVEMBER 5, 2024
It allows developers to easily connect to databases, execute SQL queries, and retrieve data. It operates as an intermediary, translating Java calls into SQL commands the database understands. ODBC uses standard SQL syntax, enabling different applications to communicate with databases regardless of the programming language.
Database 52
52 
The Best Data Management Tools For Small Businesses
Smart Data Collective
APRIL 29, 2020
Extraction, Transform, Load (ETL). Redshift is the product for data warehousing, and Athena provides SQL data analytics. It has useful features, such as an in-browser SQL editor for queries and data analysis, various data connectors for easy data ingestion, and automated data prepossessing and ingestion. Master data management.
Top ETL Tools: Unveiling the Best Solutions for Data Integration
Pickl AI
JUNE 7, 2024
Summary: Choosing the right ETL tool is crucial for seamless data integration. At the heart of this process lie ETL Tools—Extract, Transform, Load—a trio that extracts data, tweaks it, and loads it into a destination. Choosing the right ETL tool is crucial for smooth data management. What is ETL?
ETL 40
40 
Harmonize data using AWS Glue and AWS Lake Formation FindMatches ML to build a customer 360 view
JUNE 26, 2023
Transform raw insurance data into CSV format acceptable to Neptune Bulk Loader , using an AWS Glue extract, transform, and load (ETL) job. Run an AWS Glue ETL job to merge the raw property and auto insurance data into one dataset and catalog the merged dataset. We use Python scripts to analyze the data in a Jupyter notebook.
AWS 123
123 
Best Data Engineering Tools Every Engineer Should Know
Pickl AI
MARCH 19, 2025
Tools like Python, SQL, Apache Spark, and Snowflake help engineers automate workflows and improve efficiency. Python, SQL, and Apache Spark are essential for data engineering workflows. Python Python is one of the most popular programming languages for data engineering.
Importing Data in Python Cheat Sheet with Comprehensive Tutorial
Pickl AI
NOVEMBER 14, 2023
Looking for an effective and handy Python code repository in the form of Importing Data in Python Cheat Sheet? Your journey ends here where you will learn the essential handy tips quickly and efficiently with proper explanations which will make any type of data importing journey into the Python platform super easy.
Python 52
52 
Best Practices When Developing Matillion Jobs
phData
SEPTEMBER 2, 2024
In this blog, we will cover the best practices for developing jobs in Matillion, an ETL/ELT tool built specifically for cloud database platforms. Use of Python Component The Python component, including using Jython to connect to various databases, should not be used for resource-intensive data processing.
ETL 52
52 
Use mobility data to derive insights using Amazon SageMaker geospatial capabilities
AWS Machine Learning Blog
JANUARY 17, 2024
The GPU-powered interactive visualizer and Python notebooks provide a seamless way to explore millions of data points in a single window and share insights and results. As part of the initial ETL, this raw data can be loaded onto tables using AWS Glue. Sources and schema There are few sources of mobility data.
Clustering 136
136 
Azure service cloud summarized: Part I
Mlearning.ai
APRIL 24, 2023
The Coursera class is direct to the point and gives concrete instructions about how to use the Azure Portal interface, Databricks, and the Python SDK; if you know nothing about Azure and need to use the service platform right away I highly recommend this course.
Azure 52
52 
The Data Dilemma: Exploring the Key Differences Between Data Science and Data Engineering
Pickl AI
JULY 25, 2023
They create data pipelines, ETL processes, and databases to facilitate smooth data flow and storage. With expertise in programming languages like Python , Java , SQL, and knowledge of big data technologies like Hadoop and Spark, data engineers optimize pipelines for data scientists and analysts to access valuable insights efficiently.
What Are Snowflake’s Best Features for Data Transformation?
phData
AUGUST 8, 2024
Putting the T for Transformation in ELT (ETL) is essential to any data pipeline. They let you create virtual tables from the results of an SQL query. Stored Procedures In any data warehousing solution, stored procedures encapsulate SQL logic into repeatable routines, but Snowflake has some tricks up its sleeve.
SQL 52
52 
Optimizing Matillion Workflows: A Guide to Visual Design and Best Practices
phData
APRIL 28, 2025
A Matillion pipeline is a collection of jobs that extract, load, and transform (ETL/ELT) data from various sources into a target system, such as a cloud data warehouse like Snowflake. Intuitive Workflow Design Workflows should be easy to follow and visually organized, much like clean, well-structured SQL or Python code.
AI 52
52 
How Fivetran and dbt Help With ELT
phData
AUGUST 9, 2023
This is unlike the more traditional ETL method, where data is transformed before loading into the data warehouse. By bringing raw data into the data warehouse and then transforming it there, ELT provides more flexibility compared to ETL’s fixed pipelines. ETL systems just couldn’t handle the massive flows of raw data.
The 2021 Executive Guide To Data Science and AI
Applied Data Science
AUGUST 2, 2021
The most common data science languages are Python and R — SQL is also a must have skill for acquiring and manipulating data. They build production-ready systems using best-practice containerisation technologies, ETL tools and APIs. The Data Engineer Not everyone working on a data science project is a data scientist.
Data Science 130
130 
How Does Snowpark Work?
phData
FEBRUARY 7, 2024
Snowpark is the set of libraries and runtimes in Snowflake that securely deploy and process non-SQL code, including Python, Java, and Scala. On the server side, runtimes include Python, Java, and Scala in the warehouse model or Snowpark Container Services (public preview). filter(col("id") == 1).select(col("name"),
Python 52
52 
Top Data Analytics Skills and Platforms for 2023
ODSC - Open Data Science
APRIL 3, 2023
Data Wrangling: Data Quality, ETL, Databases, Big Data The modern data analyst is expected to be able to source and retrieve their own data for analysis. Competence in data quality, databases, and ETL (Extract, Transform, Load) are essential. SQL excels with big data and statistics, making it important in order to query databases.
Analytics 52
52 
Software Engineering Patterns for Machine Learning
The MLOps Blog
SEPTEMBER 7, 2023
From writing code for doing exploratory analysis, experimentation code for modeling, ETLs for creating training datasets, Airflow (or similar) code to generate DAGs, REST APIs, streaming jobs, monitoring jobs, etc. Implementing these practices can enhance the efficiency and consistency of ETL workflows.
6 Data And Analytics Trends To Prepare For In 2020
Smart Data Collective
MAY 20, 2019
For frameworks and languages, there’s SAS, Python, R, Apache Hadoop and many others. The popular tools, on the other hand, include Power BI, ETL, IBM Db2, and Teradata. SQL programming skills, specific tool experience — Tableau for example — and problem-solving are just a handful of examples.
Analytics 111
111 
Who is a BI Developer: Role, Responsibilities & Skills
Pickl AI
JULY 3, 2023
Here are steps you can follow to pursue a career as a BI Developer: Acquire a solid foundation in data and analytics: Start by building a strong understanding of data concepts, relational databases, SQL (Structured Query Language), and data modeling.
Build an automated insight extraction framework for customer feedback analysis with Amazon Bedrock and Amazon QuickSight
AWS Machine Learning Blog
JUNE 25, 2024
In addition, the generative business intelligence (BI) capabilities of QuickSight allow you to ask questions about customer feedback using natural language, without the need to write SQL queries or learn a BI tool. Prompt engineering To invoke Amazon Bedrock, you can follow our code sample that uses the Python SDK.
AWS 131
131 
A Comprehensive Guide to Business Intelligence Analysts
Pickl AI
MARCH 3, 2025
Key skills include SQL, data visualization, and business acumen. Essential skills include SQL, data visualization, and strong analytical abilities. Technical Skill Development Master SQL for database querying and manipulation. Learn programming languages like Python or R for advanced Data Analysis and automation.
Let's personalize your content