Hacker News

NOVEMBER 19, 2024

Here are a few of the things that you might do as an AI Engineer at TigerEye: - Design, develop, and validate statistical models to explain past behavior and to predict future behavior of our customers’ sales teams - Own training, integration, deployment, versioning, and monitoring of ML components - Improve TigerEye’s existing metrics collection and (..)

Computer Science 109

Computer Science Computer Science ETL ML 109

Data Science Blog

JULY 20, 2024

Die Bedeutung effizienter und zuverlässiger Datenpipelines in den Bereichen Data Science und Data Engineering ist enorm. Automatisierung: Erstellt SQL-Code, DACPAC-Dateien, SSIS-Pakete, Data Factory-ARM-Vorlagen und XMLA-Dateien. Data Lakes: Unterstützt MS Azure Blob Storage.

Azure 130

Azure Power BI SQL Data Lakes 130

Pickl AI

JUNE 7, 2024

Summary: Choosing the right ETL tool is crucial for seamless data integration. Top contenders like Apache Airflow and AWS Glue offer unique features, empowering businesses with efficient workflows, high data quality, and informed decision-making capabilities. Choosing the right ETL tool is crucial for smooth data management.

ETL 40

ETL Data Quality Data Pipeline Data Warehouse 40

Pickl AI

APRIL 6, 2023

Accordingly, one of the most demanding roles is that of Azure Data Engineer Jobs that you might be interested in. The following blog will help you know about the Azure Data Engineering Job Description, salary, and certification course. How to Become an Azure Data Engineer?

Azure 52

Azure Data Engineering Data Engineer Data Engineering 52

ODSC - Open Data Science

JANUARY 18, 2024

Data engineering is a rapidly growing field, and there is a high demand for skilled data engineers. If you are a data scientist, you may be wondering if you can transition into data engineering. In this blog post, we will discuss how you can become a data engineer if you are a data scientist.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

Pickl AI

JULY 25, 2023

Unfolding the difference between data engineer, data scientist, and data analyst. Data engineers are essential professionals responsible for designing, constructing, and maintaining an organization’s data infrastructure. Data Visualization: Matplotlib, Seaborn, Tableau, etc.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

Data Science Blog

SEPTEMBER 19, 2023

So why using IaC for Cloud Data Infrastructures? For Data Warehouse Systems that often require powerful (and expensive) computing resources, this level of control can translate into significant cost savings. This brings reliability to data ETL (Extract, Transform, Load) processes, query performances, and other critical data operations.

Data Warehouse 259

Data Warehouse Azure SQL Database 259

DataSeries

AUGUST 15, 2024

Enrich data engineering skills by building problem-solving ability with real-world projects, teaming with peers, participating in coding challenges, and more. Globally several organizations are hiring data engineers to extract, process and analyze information, which is available in the vast volumes of data sets.

Data Engineering 40

Data Engineering Data Engineer Data Engineering Data Engineering 40

phData

AUGUST 17, 2023

In the rapidly evolving landscape of data engineering, Snowflake Data Cloud has emerged as a leading cloud-based data warehousing solution, providing powerful capabilities for storing, processing, and analyzing vast amounts of data.

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

phData

JULY 12, 2023

Unlike traditional methods that rely on complex SQL queries for orchestration, Matillion Jobs provides a more streamlined approach. By converting SQL scripts into Matillion Jobs , users can take advantage of the platform’s advanced features for job orchestration, scheduling, and sharing. What is Matillion ETL?

SQL 52

SQL ETL Database Data Pipeline 52

phData

APRIL 21, 2023

Unlike traditional methods that rely on complex SQL queries for orchestration, Matillion Jobs provide a more streamlined approach. By converting SQL scripts into Matillion Jobs , users can take advantage of the platform’s advanced features for job orchestration, scheduling, and sharing. In our case, this table is “orders.”

SQL 52

SQL ETL Database Data Pipeline 52

Analytics Vidhya

FEBRUARY 20, 2023

Introduction Azure data factory (ADF) is a cloud-based data ingestion and ETL (Extract, Transform, Load) tool. The data-driven workflow in ADF orchestrates and automates data movement and data transformation.

Azure 249

Azure ETL Analytics Analytics 249

Analytics Vidhya

AUGUST 29, 2022

This article was published as a part of the Data Science Blogathon. Introduction Data scientists, engineers, and BI analysts often need to analyze, process, or query different data sources. The post Understand Apache Drill and its Working appeared first on Analytics Vidhya.

ETL 262

ETL Data Scientist Data Science Analytics 262

Data Science Dojo

MAY 10, 2023

Data Engineer Data engineers are responsible for building, maintaining, and optimizing data infrastructures. They require strong programming skills, expertise in data processing, and knowledge of database management.

Data Science 368

Data Science Database Administration Data Scientist Machine Learning 368

Data Science Dojo

OCTOBER 31, 2024

Their role is crucial in understanding the underlying data structures and how to leverage them for insights. Key Skills Proficiency in SQL is essential, along with experience in data visualization tools such as Tableau or Power BI. Programming Questions Data science roles typically require knowledge of Python, SQL, R, or Hadoop.

Data Science 195

Data Science Data Scientist Machine Learning Machine Learning 195

phData

SEPTEMBER 2, 2024

Best practices are a pivotal part of any software development, and data engineering is no exception. This ensures the data pipelines we create are robust, durable, and secure, providing the desired data to the organization effectively and consistently. What Are Matillion Jobs and Why Do They Matter?

ETL 52

ETL Data Warehouse SQL Database 52

phData

MARCH 8, 2023

It allows organizations to easily connect their disparate data sources without having to manage any infrastructure. Fivetran’s automated data movement platform simplifies the ETL (extract, transform, load) process by automating most of the time-consuming tasks of ETL that data engineers would typically do.

Data Warehouse 96

Data Warehouse Data Engineering Data Engineering Data Engineering 96

Data Science Dojo

JULY 3, 2024

Data Processing and Analysis : Techniques for data cleaning, manipulation, and analysis using libraries such as Pandas and Numpy in Python. Databases and SQL : Managing and querying relational databases using SQL, as well as working with NoSQL databases like MongoDB.

Data Science 195

Data Science Machine Learning Machine Learning Data Visualization 195

The MLOps Blog

SEPTEMBER 7, 2023

Data Scientists and ML Engineers typically write lots and lots of code. From writing code for doing exploratory analysis, experimentation code for modeling, ETLs for creating training datasets, Airflow (or similar) code to generate DAGs, REST APIs, streaming jobs, monitoring jobs, etc.

Machine Learning 52

Machine Learning Machine Learning ETL ML 52

phData

MARCH 1, 2024

There’s no need for developers or analysts to manually adjust table schemas or modify ETL (Extract, Transform, Load) processes whenever the source data structure changes. Time Efficiency – The automated schema detection and evolution features contribute to faster data availability.

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

AWS Machine Learning Blog

NOVEMBER 20, 2024

Repeat the steps to add another Aurora MySQL data source, called aggregated_sales , for the same database but with the following details in the Sync scope This data source will be used by Amazon Q for answering questions on aggregated sales. Data Engineer at Amazon Ads. For IAM role , choose Create a new service role.

Database 83

Database AWS SQL ETL 83

phData

JUNE 14, 2023

In recent years, data engineering teams working with the Snowflake Data Cloud platform have embraced the continuous integration/continuous delivery (CI/CD) software development process to develop data products and manage ETL/ELT workloads more efficiently. What Are the Benefits of CI/CD Pipeline For Snowflake?

Data Pipeline 52

Data Pipeline Database SQL Data Engineering 52

phData

FEBRUARY 7, 2024

Snowpark is the set of libraries and runtimes in Snowflake that securely deploy and process non-SQL code, including Python, Java, and Scala. A DataFrame is like a query that must be evaluated to retrieve data. An action causes the DataFrame to be evaluated and sends the corresponding SQL statement to the server for execution.

Python 52

Python ML ML SQL 52

ODSC - Open Data Science

APRIL 3, 2023

Skills like effective verbal and written communication will help back up the numbers, while data visualization (specific frameworks in the next section) can help you tell a complete story. Data Wrangling: Data Quality, ETL, Databases, Big Data The modern data analyst is expected to be able to source and retrieve their own data for analysis.

Analytics 52

Analytics Analytics Data Analyst Data Science 52

phData

JULY 17, 2023

This is where Fivetran and the Modern Data Stack come in. Fivetran is a fully-automated, zero-maintenance data pipeline tool that automates the ETL process from data sources to your cloud warehouse. Centralize Many Different Data Sources Into a Single Cloud-Based Target (i.e. What is Fivetran?

Data Warehouse 52

Data Warehouse Data Pipeline Cloud Data ETL 52

IBM Journey to AI blog

MARCH 14, 2024

Db2 Warehouse fully supports open formats such as Parquet, Avro, ORC and Iceberg table format to share data and extract new insights across teams without duplication or additional extract, transform, load (ETL). This allows you to scale all analytics and AI workloads across the enterprise with trusted data. 

AWS 97

AWS Database AI AI 97

Applied Data Science

AUGUST 2, 2021

Team Building the right data science team is complex. With a range of role types available, how do you find the perfect balance of Data Scientists , Data Engineers and Data Analysts to include in your team? The Data Engineer Not everyone working on a data science project is a data scientist.

Data Science 130

Data Science Data Scientist Data Analyst Machine Learning 130

Alation

APRIL 4, 2023

Profiling delivers a birds-eye view of the statistics of the data, such as minimum, maximum, median, and null values. This empowers users to judge data’s quality and fitness for purpose quickly. This expanded connector to Databricks Unity Catalog does just that, delivering to joint customers a comprehensive view of all cloud data.

DataOps 52

DataOps Data Engineering Data Engineering Data Engineering 52

Alation

JANUARY 17, 2023

It is known to have benefits in handling data due to its robustness, speed, and scalability. A typical modern data stack consists of the following: A data warehouse. Data ingestion/integration services. Reverse ETL tools. Data orchestration tools. A Note on the Shift from ETL to ELT. Data scientists.

Data Warehouse 52

Data Warehouse ETL Tableau Cloud Data 52

AWS Machine Learning Blog

OCTOBER 9, 2024

To start using CloudWatch anomaly detection, you first must ingest data into CloudWatch and then enable anomaly detection on the log group. Using Amazon Redshift ML for anomaly detection Amazon Redshift ML makes it easy to create, train, and apply machine learning models using familiar SQL commands in Amazon Redshift data warehouses.

AWS 83

AWS ML ML Data Quality 83

phData

SEPTEMBER 4, 2024

ThoughtSpot is a cloud-based AI-powered analytics platform that uses natural language processing (NLP) or natural language query (NLQ) to quickly query results and generate visualizations without the user needing to know any SQL or table relations. Suppose your business requires more robust capabilities across your technology stack.

Analytics 52

Analytics Analytics SQL ETL 52

phData

AUGUST 10, 2023

That said, dbt provides the ability to generate data vault models and also allows you to write your data transformations using SQL and code-reusable macros powered by Jinja2 to run your data pipelines in a clean and efficient way. The most important reason for using DBT in Data Vault 2.0

SQL 52

SQL Data Observability Data Quality Data Pipeline 52

phData

JANUARY 4, 2023

With Snowflake, organizations can be data consumers, data providers, or both. Complete SQL Database No need to learn new tools as Snowflake supports the tools millions of business users already know how to use today. Ready to Get Started in the Migration to Snowflake?

Data Warehouse 52

Data Warehouse SQL Database ETL 52

IBM Journey to AI blog

JANUARY 5, 2023

The first generation of data architectures represented by enterprise data warehouse and business intelligence platforms were characterized by thousands of ETL jobs, tables, and reports that only a small group of specialized data engineers understood, resulting in an under-realized positive impact on the business.

Data Quality 52

Data Quality Data Lakes Data Warehouse Business Intelligence 52

phData

FEBRUARY 9, 2023

Before we dive in, it’s important to note that there are multiple ways to migrate data from Redshift tables to Snowflake. One popular route is leveraging third-party ETL tools like Fivetran to ensure a smooth and successful migration. For this blog, we’ll look at how to do this by using the Redshift unload command, Snowpipe, and Spark.

AWS 59

AWS ETL Data Preparation SQL 59

Alation

MAY 24, 2022

Prime examples of this in the data catalog include: Trust Flags — Allow the data community to endorse, warn, and deprecate data to signal whether data can or can’t be used. Data Profiling — Statistics such as min, max, mean, and null can be applied to certain columns to understand its shape.

Data Quality 98

Data Quality Data Governance ETL Data Observability 98

Tableau

AUGUST 21, 2023

Finally, Tableau allows you to create custom territories using Tableau groups and overlay data with demographic information, giving you a comprehensive view of your data. ODBC is an industry standard that allows various software to access data.

Tableau 97

Tableau Database Analytics Analytics 97