Data Science 101

NOVEMBER 7, 2019

Microsoft just held one of its largest conferences of the year, and a few major announcements were made which pertain to the cloud data science world. Azure Synapse Analytics can be seen as a merge of Azure SQL Data Warehouse and Azure Data Lake. Here they are in my order of importance (based upon my opinion).

Data Science 106

Data Science Azure SQL Machine Learning 106

Precisely

JANUARY 25, 2024

There are three potential approaches to mainframe modernization: Data Replication creates a duplicate copy of mainframe data in a cloud data warehouse or data lake, enabling high-performance analytics virtually in real time, without negatively impacting mainframe performance.

Data Governance 69

Data Governance Database Cloud Data Data Lakes 69

phData

AUGUST 9, 2023

This makes ELT aligned with modern data practices and helps explain why it has become the dominant pattern, replacing the once-standard ETL approach. The Story of ELT In the early days of data warehousing, ETL was the standard for data processing. Thus, the early data lakes began following more of the EL-style flow.

ETL 52

ETL Data Warehouse Cloud Data Big Data 52

Flipboard

NOVEMBER 24, 2023

JuMa is tightly integrated with a range of BMW Central IT services, including identity and access management, roles and rights management, BMW Cloud Data Hub (BMW’s data lake on AWS) and on-premises databases.

ML 151

ML ML AWS AI 151

Smart Data Collective

OCTOBER 27, 2021

Google BigQuery is a serverless and cost-effective multi-cloud data warehouse. Accessing data stored on Google BigQuery is secured with default and customer-managed encryption keys, and you can easily share any business intelligence insight derived from such data with teams and members of your organization with a few clicks.

Analytics 118

Analytics Analytics Data Warehouse Business Intelligence 118

Alation

MARCH 1, 2022

Lineage helps them identify the source of bad data to fix the problem fast. Manual lineage will give ARC a fuller picture of how data was created between AWS S3 data lake, Snowflake cloud data warehouse and Tableau (and how it can be fixed). Time is money,” said Leonard Kwok, Senior Data Analyst, ARC.

Data Warehouse 52

Data Warehouse Data Lakes Cloud Data Database 52

Alation

DECEMBER 14, 2021

It’s only been 15 years since AWS took the first steps to the cloud with S3 and EC2, which launched in 2006. With the database services launched soon after, developers had all the tools they needed to create applications without having to create the infrastructure to run them. What about other data sources? In Conclusion.

AWS 52

AWS Data Lakes Data Warehouse Machine Learning 52

phData

FEBRUARY 14, 2023

Data integration is essentially the Extract and Load portion of the Extract, Load, and Transform (ELT) process. Data ingestion involves connecting your data sources, including databases, flat files, streaming data, etc, to your data warehouse. Snowflake provides native ways for data ingestion.

Data Warehouse 59

Data Warehouse Azure AWS Database 59

Dataconomy

SEPTEMBER 4, 2023

Cloud-based business intelligence (BI): Cloud-based BI tools enable organizations to access and analyze data from cloud-based sources and on-premises databases. These tools offer the flexibility of accessing insights from anywhere, and they often integrate with other cloud analytics solutions.

Analytics 203

Analytics Analytics Big Data Analytics Big Data Analytics 203

phData

FEBRUARY 5, 2024

Creating the databases, schemas, roles, and access grants that comprise a data system information architecture can be time-consuming and error-prone. Luckily phData has created a template-driven Provision Tool that automates onboarding users and projects to Snowflake, allowing your data teams to start producing real value immediately.

SQL 52

SQL Database Data Quality Data Warehouse 52

ODSC - Open Data Science

JULY 11, 2023

Co-location data centers: These are data centers that are owned and operated by third-party providers and are used to house the IT equipment of multiple organizations. What are the similarities and differences between data centers, data lake houses, and data lakes? Not a cloud computer?

Data Lakes 40

Data Lakes AI AI Cloud Computing 40

Flipboard

AUGUST 17, 2023

Amazon Redshift is the most popular cloud data warehouse that is used by tens of thousands of customers to analyze exabytes of data every day. AWS Glue is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, ML, and application development.

ML 123

ML ML AWS Data Warehouse 123

Mlearning.ai

FEBRUARY 16, 2023

Thus, the solution allows for scaling data workloads independently from one another and seamlessly handling data warehousing, data lakes , data sharing, and engineering. Snowflake Database Pros Extensive Storage Opportunities Snowflake provides affordability, scalability, and a user-friendly interface.

Data Warehouse 52

Data Warehouse Business Intelligence Business Intelligence Database 52

Precisely

JANUARY 19, 2023

Today, mainframes deliver scalability and global access, and they’re still a key element of the infrastructure that makes private clouds possible at many organizations. In 2023, expect to see broader adoption of streaming data pipelines that bring mainframe data to the cloud, offering a powerful tool for “modernizing in place.”

AWS 52

AWS Cloud Computing Data Pipeline Big Data 52

Alation

JUNE 21, 2022

Why start with a data source and build a visualization, if you can just find a visualization that already exists, complete with metadata about it? Data scientists went beyond database tables to data lakes and cloud data stores. Data scientists want to catalog not just information sources, but models.

DataOps 52

DataOps Data Scientist Data Quality Data Pipeline 52

Alation

JUNE 9, 2021

The explosion in data and database types is a major pain point of the modern data consumer. What is Data Search & Discovery? According to IDC , more than 59 zettabytes (59,000,000,000,000,000,000,000 bytes) of data was created, captured, and consumed in the world in 2020. Today they have too much.

Data Governance 59

Data Governance Database Cloud Data Machine Learning 59

phData

SEPTEMBER 20, 2023

A data mesh is a conceptual architectural approach for managing data in large organizations. Traditional data management approaches often involve centralizing data in a data warehouse or data lake, leading to challenges like data silos, data ownership issues, and data access and processing bottlenecks.

Data Silos 52

Data Silos Database Data Quality Data Engineering 52

IBM Journey to AI blog

JULY 17, 2023

Through workload optimization across multiple query engines and storage tiers, organizations can reduce data warehouse costs by up to 50 percent. 1 Watsonx.data offers built-in governance and automation to get to trusted insights within minutes, and integrations with existing databases and tools to simplify setup and user experience.

AI 75

AI AI Machine Learning Machine Learning 75

Alation

APRIL 14, 2022

This two-part series will explore how data discovery, fragmented data governance , ongoing data drift, and the need for ML explainability can all be overcome with a data catalog for accurate data and metadata record keeping. The Cloud Data Migration Challenge. Data pipeline orchestration.

Data Governance 59

Data Governance ML ML Cloud Data 59

phData

DECEMBER 4, 2023

However, if there’s one thing we’ve learned from years of successful cloud data implementations here at phData, it’s the importance of: Defining and implementing processes Building automation, and Performing configuration …even before you create the first user account. This includes users, roles, schemas, databases, and warehouses.

Clustering 98

Clustering Database SQL Data Pipeline 98

phData

MAY 6, 2024

Another benefit of deterministic matching is that the process to build these identities is relatively simple, and tools your teams might already use, like SQL and dbt , can efficiently manage this process within your cloud data warehouse. Store this data in a customer data platform or data lake.

Data Lakes 52

Data Lakes Data Warehouse SQL Data Quality 52

Alation

MARCH 29, 2021

Alation helps connects to any source Alation helps connect to virtually any data source through pre-built connectors. Alation crawls and indexes data assets stored across disparate repositories, including cloud data lakes, databases, Hadoop files, and data visualization tools.

Machine Learning 59

Machine Learning Machine Learning Data Lakes Hadoop 59

phData

SEPTEMBER 27, 2024

. “ This sounds great in theory, but how does it work in practice with customer data or something like a ‘composable CDP’? Well, implementing transitional modeling does require a shift in how we think about and work with customer data. It often involves specialized databases designed to handle this kind of atomic, temporal data.

Data Models 52

Data Models Data Modeling Apache Kafka Data Lakes 52

Alation

MARCH 22, 2022

Alation’s governance capabilities include automated classification, profiling, data quality, lineage, stewardship, and deep policy integration with leading cloud-native databases like Snowflake. This produces end-to-end lineage so business and technology users alike can understand the state of a data lake and/or lake house.

DataOps 52

DataOps Data Pipeline Data Engineering Data Engineer 52

The MLOps Blog

MAY 17, 2023

Tool Cloudbased Pre-Built Connectors Serverless Pre-Built Transformation Options API Support Fully Managed Hevo Data AWS Glue GCP Cloud Data Fusion Apache Spark Talend Apache Airflow You may also like Comparing Tools For Data Processing Pipelines How to build an ML ETL pipeline?

ETL 59

ETL Data Pipeline ML ML 59

Dataversity

FEBRUARY 8, 2022

Every generation of data infrastructure technology has promised more speed and agility, or better standardization, centralization, and control.

Data Lakes 52

Data Lakes Data Warehouse Database Cloud Data 52

Dataversity

MAY 17, 2021

There are advantages and disadvantages to both ETL and ELT. To understand which method is a better fit, it’s important to understand what it means when one letter comes before the other. The post Understanding the ETL vs. ELT Alphabet Soup and When to Use Each appeared first on DATAVERSITY.

ETL 52

ETL Data Lakes Data Warehouse Database 52

Snorkel AI

MAY 26, 2023

And so data scientists might be leveraging one compute service and might be leveraging an extracted CSV for their experimentation. And then the production teams might be leveraging a totally different single source of truth or data warehouse or data lake and totally different compute infrastructure for deploying models into production.

SQL 52

SQL ML ML Python 52

Snorkel AI

MAY 26, 2023

And so data scientists might be leveraging one compute service and might be leveraging an extracted CSV for their experimentation. And then the production teams might be leveraging a totally different single source of truth or data warehouse or data lake and totally different compute infrastructure for deploying models into production.

SQL 52

SQL ML ML Python 52

phData

NOVEMBER 9, 2023

Snowflake’s Data Cloud has emerged as a leader in cloud data warehousing. As a fundamental piece of the modern data stack , Snowflake is helping thousands of businesses store, transform, and derive insights from their data easier, faster, and more efficiently than ever before. What is a Data Lake?

Data Warehouse 98

Data Warehouse Data Lakes Clustering Cloud Data 98

Flipboard

DECEMBER 18, 2023

Amazon Redshift powers data-driven decisions for tens of thousands of customers every day with a fully managed, AI-powered cloud data warehouse, delivering the best price-performance for your analytics workloads. Learn more about the AWS zero-ETL future with newly launched AWS databases integrations with Amazon Redshift.

AWS 137

AWS Data Warehouse ETL SQL 137

Alation

APRIL 19, 2022

Data producers and consumers alike are working from home and hybrid locations more often. And in an increasingly remote workforce, people need to access data systems easily to do their jobs. This might mean that they’re accessing a database from a smartphone, computer, or tablet. Today, data dwells everywhere.

Data Governance 52

Data Governance Cloud Data Database Data Silos 52