ODSC - Open Data Science

FEBRUARY 6, 2024

Data engineering has become an integral part of the modern tech landscape, driving advancements and efficiencies across industries. So let’s explore the world of open-source tools for data engineers, shedding light on how these resources are shaping the future of data handling, processing, and visualization.

Data Engineer 52

Data Engineer Data Engineering Data Engineering Data Engineering 52

Data Science Blog

NOVEMBER 15, 2023

The creation of this data model requires the data connection to the source system (e.g. SAP ERP), the extraction of the data and, above all, the data modeling for the event log. DATANOMIQ Data Mesh Cloud Architecture – This image is animated! Click to enlarge!

Data Models 245

Data Models Data Modeling Business Intelligence Business Intelligence 245

phData

MARCH 20, 2023

Length of Interview: 30 – 45 minutes Interview 2: Leadership In this interview, you will meet with the Director of the Solutions Engineering team. The discussion points in this interview will include a review of your current experience as it relates to cloud data engineering and solution engineering.

Cloud Data 98

Cloud Data Azure Data Engineering Data Engineer 98

AWS Machine Learning Blog

DECEMBER 1, 2023

This is where the AWS suite of low-code and no-code ML services becomes an essential tool. As a strategic systems integrator with deep ML experience, Deloitte utilizes the no-code and low-code ML tools from AWS to efficiently build and deploy ML models for Deloitte’s clients and for internal assets.

Machine Learning 103

Machine Learning Machine Learning Data Preparation ML 103

Data Science Dojo

JULY 3, 2024

Big Data Technologies : Handling and processing large datasets using tools like Hadoop, Spark, and cloud platforms such as AWS and Google Cloud. Data Processing and Analysis : Techniques for data cleaning, manipulation, and analysis using libraries such as Pandas and Numpy in Python.

Data Science 150

Data Science Machine Learning Machine Learning Data Visualization 150

phData

FEBRUARY 21, 2025

The Cloud represents an iteration beyond the on-prem data warehouse, where computing resources are delivered over the Internet and are managed by a third-party provider. Examples include: Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). Data integrations and pipelines can also impact latency.

Data Warehouse 52

Data Warehouse Cloud Data ETL Cloud Computing 52

phData

NOVEMBER 8, 2024

Data Versioning and Time Travel Open Table Formats empower users with time travel capabilities, allowing them to access previous dataset versions. Versioning also ensures a safer experimentation environment, where data scientists can test new models or hypotheses on historical data snapshots without impacting live data.

Data Lakes 52

Data Lakes Data Warehouse Database Azure 52

phData

JANUARY 5, 2023

Python is the top programming language used by data engineers in almost every industry. Python has proven proficient in setting up pipelines, maintaining data flows, and transforming data with its simple syntax and proficiency in automation. Truly a must-have tool in your data engineering arsenal!

Python 52

Python Data Engineering Data Engineer Data Engineering 52

IBM Journey to AI blog

MAY 15, 2024

There are several styles of data integration. Data engineers build data pipelines, which are called data integration tasks or jobs, as incremental steps to perform data operations and orchestrate these data pipelines in an overall workflow.

Data Pipeline 85

Data Pipeline ETL SQL Database 85

Data Science Blog

SEPTEMBER 3, 2024

Reduzierte Personalkosten , sind oft dann gegeben, wenn interne Data Engineers verfügbar sind, die die Datenmodelle intern entwickeln. Höhere Data Readiness , denn für eine zentrale Datenplattform lohn es sich eher, Daten aus weniger genutzten Quellen anzuschließen. Müssen Rohdatentabellen in die Analyse-Tools wie z.

Data Science 130

Data Science Power BI Azure Data Warehouse 130

phData

FEBRUARY 14, 2023

Matillion Matillion is a complete ETL tool that integrates with an extensive list of pre-built data source connectors, loads data into cloud data environments such as Snowflake, and then performs transformations to make data consumable by analytics tools such as Tableau and PowerBI.

Data Warehouse 59

Data Warehouse Azure AWS Database 59

IBM Journey to AI blog

JULY 17, 2023

Within watsonx.ai, users can take advantage of open-source frameworks like PyTorch, TensorFlow and scikit-learn alongside IBM’s entire machine learning and data science toolkit and its ecosystem tools for code-based and visual data science capabilities. Savings may vary depending on configurations, workloads and vendor.

AI 74

AI AI Machine Learning Machine Learning 74

phData

AUGUST 6, 2024

Organizations must ensure their data pipelines are well designed and implemented to achieve this, especially as their engagement with cloud data platforms such as the Snowflake Data Cloud grows. For customers in Snowflake, Snowpark is a powerful tool for building these effective and scalable data pipelines.

Data Pipeline 52

Data Pipeline Python Data Engineering Data Engineer 52

Alation

MARCH 1, 2022

Through Impact Analysis, users can determine if a problem occurred with data upstream, and locate the impacted data downstream. With robust data lineage, data engineers can find and fix issues fast and prevent them from recurring. Similarly, analysts gain a clear view of how data is created.

Data Warehouse 52

Data Warehouse Data Lakes Cloud Data Database 52

phData

JUNE 14, 2023

In recent years, data engineering teams working with the Snowflake Data Cloud platform have embraced the continuous integration/continuous delivery (CI/CD) software development process to develop data products and manage ETL/ELT workloads more efficiently. What Are the Benefits of CI/CD Pipeline For Snowflake?

Data Pipeline 52

Data Pipeline Database SQL Data Engineering 52

phData

SEPTEMBER 2, 2024

Best practices are a pivotal part of any software development, and data engineering is no exception. This ensures the data pipelines we create are robust, durable, and secure, providing the desired data to the organization effectively and consistently. What Are Matillion Jobs and Why Do They Matter?

ETL 52

ETL Data Warehouse SQL Database 52

phData

APRIL 4, 2023

Fivetran today announced support for Amazon Simple Storage Service (Amazon S3) with Apache Iceberg data lake format. Amazon S3 is an object storage service from Amazon Web Services (AWS) that offers industry-leading scalability, data availability, security, and performance.

Data Lakes 52

Data Lakes Data Warehouse Cloud Data AWS 52

Alation

MARCH 22, 2022

Accenture calls it the Intelligent Data Foundation (IDF), and it’s used by dozens of enterprises with very complex data landscapes and analytic requirements. Simply put, IDF standardizes data engineering processes. IDF works natively on cloud platforms like AWS. Take a look at figure 1 below.

DataOps 52

DataOps Data Pipeline Data Engineering Data Engineer 52

The MLOps Blog

MAY 17, 2023

This article explores the importance of ETL pipelines in machine learning, a hands-on example of building ETL pipelines with a popular tool, and suggests the best ways for data engineers to enhance and sustain their pipelines. Before delving into the technical details, let’s review some fundamental concepts.

ETL 59

ETL Data Pipeline ML ML 59

phData

SEPTEMBER 27, 2024

Modern business operations rely heavily on data engineering and transformation processes to turn raw data into valuable insights. Matillion, a robust ELT (Extract, Load, Transform) platform, simplifies data integration and transformation complexities with a no-code or high-code experience. What is a Matillion Job?

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

phData

MAY 25, 2023

One big issue that contributes to this resistance is that although Snowflake is a great cloud data warehousing platform, Microsoft has a data warehousing tool of its own called Synapse. Gateways are being used as another layer of security between Snowflake or cloud data source and Power BI users.

Power BI 84

Power BI Analytics Analytics Azure 84

Alation

FEBRUARY 8, 2022

While data fabric takes a product-and-tech-centric approach, data mesh takes a completely different perspective. Data mesh inverts the common model of having a centralized team (such as a data engineering team), who manage and transform data for wider consumption. But why is such an inversion needed?

Data Governance 52

Data Governance Data Engineering Data Engineer Data Engineering 52

phData

DECEMBER 4, 2023

However, if there’s one thing we’ve learned from years of successful cloud data implementations here at phData, it’s the importance of: Defining and implementing processes Building automation, and Performing configuration …even before you create the first user account.

Clustering 98

Clustering Database SQL Data Pipeline 98

phData

OCTOBER 28, 2024

Modern low-code/no-code ETL tools allow data engineers and analysts to build pipelines seamlessly using a drag-and-drop and configure approach with minimal coding. Matillion ETL for Snowflake is an ELT/ETL tool that allows for the ingestion, transformation, and building of analytics for data in the Snowflake AI Data Cloud.

Python 52

Python ETL AWS Database 52

Flipboard

DECEMBER 18, 2023

Amazon Redshift powers data-driven decisions for tens of thousands of customers every day with a fully managed, AI-powered cloud data warehouse, delivering the best price-performance for your analytics workloads. Learn more about the AWS zero-ETL future with newly launched AWS databases integrations with Amazon Redshift.

AWS 139

AWS Data Warehouse ETL SQL 139

AWS Machine Learning Blog

APRIL 3, 2025

At the heart of this transformation is the OMRON Data & Analytics Platform (ODAP), an innovative initiative designed to revolutionize how the company harnesses its data assets. The robust security features provided by Amazon S3, including encryption and durability, were used to provide data protection.

AWS 71

AWS Data Governance Data Silos SQL 71

Data Science Blog

JULY 23, 2023

Azure’s strong focus on security, compliance, and global presence, along with hybrid cloud capabilities and cost management tools, make it an ideal choice for industrial firms seeking to modernize, innovate, and improve efficiency.

Data Science 246

Data Science Azure Power BI Business Intelligence 246

phData

JULY 18, 2023

With the birth of cloud data warehouses, data applications, and generative AI , processing large volumes of data faster and cheaper is more approachable and desired than ever. First up, let’s dive into the foundation of every Modern Data Stack, a cloud-based data warehouse.

Data Warehouse 52

Data Warehouse Analytics Analytics SQL 52

AWS Machine Learning Blog

NOVEMBER 1, 2024

AWS can play a key role in enabling fast implementation of these decentralized clinical trials. By exploring these AWS powered alternatives, we aim to demonstrate how organizations can drive progress towards more environmentally friendly clinical research practices.

AWS 108

AWS Data Lakes Machine Learning Machine Learning 108

AWS Machine Learning Blog

OCTOBER 28, 2024

The workflow includes the following steps: Within the SageMaker Canvas interface, the user composes a SQL query to run against the GCP BigQuery data warehouse. Athena uses the Athena Google BigQuery connector , which uses a pre-built AWS Lambda function to enable Athena federated query capabilities.

Machine Learning 97

Machine Learning Machine Learning ML ML 97