ODSC - Open Data Science

SEPTEMBER 27, 2023

In the ever-evolving world of big data, managing vast amounts of information efficiently has become a critical challenge for businesses across the globe. As data lakes gain prominence as a preferred solution for storing and processing enormous datasets, the need for effective data version control mechanisms becomes increasingly evident.

Data Lakes 98

Data Lakes Data Warehouse Database Big Data 98

Data Science 101

NOVEMBER 7, 2019

Microsoft just held one of its largest conferences of the year, and a few major announcements were made which pertain to the cloud data science world. Azure Synapse Analytics can be seen as a merge of Azure SQL Data Warehouse and Azure Data Lake. Those are the big data science announcements of the week.

Data Science 106

Data Science Azure SQL Machine Learning 106

Data Science Dojo

SEPTEMBER 11, 2024

With this full-fledged solution, you don’t have to spend all your time and effort combining different services or duplicating data. Overview of One Lake Fabric features a lake-centric architecture, with a central repository known as OneLake. Now, we can save the data as delta tables to use later for sales analytics.

Power BI 337

Power BI Data Pipeline Data Warehouse Data Engineering 337

AWS Machine Learning Blog

JUNE 20, 2024

Our goal was to improve the user experience of an existing application used to explore the counters and insights data. The data is stored in a data lake and retrieved by SQL using Amazon Athena. The following figure shows a search query that was translated to SQL and run. The challenge is to assure quality.

SQL 120

SQL Database AWS Machine Learning 120

AWS Machine Learning Blog

FEBRUARY 28, 2024

Structured Query Language (SQL) is a complex language that requires an understanding of databases and metadata. Today, generative AI can enable people without SQL knowledge. This generative AI task is called text-to-SQL, which generates SQL queries from natural language processing (NLP) and converts text into semantically correct SQL.

SQL 143

SQL AWS Database ML 143

ODSC - Open Data Science

FEBRUARY 2, 2023

Here’s what we found for both skills and platforms that are in demand for data scientist jobs. Data Science Skills and Competencies Aside from knowing particular frameworks and languages, there are various topics and competencies that any data scientist should know. Joking aside, this does infer particular skills.

Data Science 98

Data Science Data Scientist Computer Science Computer Science 98

IBM Journey to AI blog

JULY 5, 2023

Data Lakes have been around for well over a decade now, supporting the analytic operations of some of the largest world corporations. Such data volumes are not easy to move, migrate or modernize. The challenges of a monolithic data lake architecture Data lakes are, at a high level, single repositories of data at scale.

Data Lakes 78

Data Lakes Data Warehouse Data Governance Analytics 78

How to Learn Machine Learning

MAY 2, 2025

Best Practices for Azure Machine Learning Projects To get the most out of Azure Machine Learning, consider these best practices: Data Management Use Azure Data Stores : Connect to various data sources including Azure Blob Storage, Azure Data Lake, and Azure SQL Database for efficient data access.

Azure 52

Azure Machine Learning Machine Learning Data Science 52

IBM Journey to AI blog

SEPTEMBER 19, 2023

Though you may encounter the terms “data science” and “data analytics” being used interchangeably in conversations or online, they refer to two distinctly different concepts. Meanwhile, data analytics is the act of examining datasets to extract value and find answers to specific questions.

Data Science 94

Data Science Analytics Analytics Data Scientist 94

Dataconomy

JUNE 1, 2023

Unified data storage : Fabric’s centralized data lake, Microsoft OneLake, eliminates data silos and provides a unified storage system, simplifying data access and retrieval. OneLake is designed to store a single copy of data in a unified location, leveraging the open-source Apache Parquet format.

Power BI 194

Power BI Data Lakes Azure Data Silos 194

Tableau

JUNE 8, 2021

In many of the conversations we have with IT and business leaders, there is a sense of frustration about the speed of time-to-value for big data and data science projects. We often hear that organizations have invested in data science capabilities but are struggling to operationalize their machine learning models.

Tableau 102

Tableau Data Lakes Data Warehouse SQL 102

AWS Machine Learning Blog

APRIL 3, 2025

Amazon AppFlow was used to facilitate the smooth and secure transfer of data from various sources into ODAP. Additionally, Amazon Simple Storage Service (Amazon S3) served as the central data lake, providing a scalable and cost-effective storage solution for the diverse data types collected from different systems.

AWS 94

AWS Data Governance Data Silos SQL 94

Pickl AI

NOVEMBER 15, 2023

Discover the nuanced dissimilarities between Data Lakes and Data Warehouses. Data management in the digital age has become a crucial aspect of businesses, and two prominent concepts in this realm are Data Lakes and Data Warehouses. It acts as a repository for storing all the data.

Data Lakes 52

Data Lakes Data Warehouse Database ETL 52

phData

SEPTEMBER 19, 2023

With the amount of data companies are using growing to unprecedented levels, organizations are grappling with the challenge of efficiently managing and deriving insights from these vast volumes of structured and unstructured data. What is a Data Lake? Consistency of data throughout the data lake.

Data Lakes 52

Data Lakes Data Modeling Data Models Data Warehouse 52

ODSC - Open Data Science

SEPTEMBER 29, 2023

Data management problems can also lead to data silos; disparate collections of databases that don’t communicate with each other, leading to flawed analysis based on incomplete or incorrect datasets. One way to address this is to implement a data lake: a large and complex database of diverse datasets all stored in their original format.

Data Lakes 52

Data Lakes Clustering Big Data Big Data 52

ODSC - Open Data Science

SEPTEMBER 12, 2023

Most data scientists are familiar with the concept of time series data and work with it often. The time series database (TSDB) , however, is still an underutilized tool in the data science community. Typically, time series analysis is performed either on CSV files or data lakes.

Data Scientist 97

Data Scientist Database Data Lakes Data Science 97

Data Science Dojo

JULY 6, 2023

It offers extensibility and integration with various data engineering tools. dbt (Data Build Tool): dbt is an open-source data transformation and modeling tool. It allows data engineers to build, test, and maintain data pipelines in a version-controlled manner.

Data Engineering 195

Data Engineering Data Engineer Data Engineering Data Engineering 195

Flipboard

DECEMBER 6, 2023

In this post, we discuss a Q&A bot use case that Q4 has implemented, the challenges that numerical and structured datasets presented, and how Q4 concluded that using SQL may be a viable solution. This would have required a dedicated cross-disciplinary team with expertise in data science, machine learning, and domain knowledge.

SQL 168

SQL Database AWS Machine Learning 168

ODSC - Open Data Science

OCTOBER 13, 2024

Be sure to check out her talk, “ Don’t Go Over the Deep End: Building an Effective OSS Management Layer for Your Data Lake ,” there! Managing a data lake can often feel like being lost at sea — especially when dealing with both structured and unstructured data.

Data Lakes 40

Data Lakes Database Data Pipeline SQL 40

DagsHub

DECEMBER 11, 2023

The following points illustrates some of the main reasons why data versioning is crucial to the success of any data science and machine learning project: Storage space One of the reasons of versioning data is to be able to keep track of multiple versions of the same data which obviously need to be stored as well.

Machine Learning 98

Machine Learning Machine Learning Data Lakes Data Science 98

ODSC - Open Data Science

FEBRUARY 24, 2023

With Great Expectations , data teams can express what they “expect” from their data using simple assertions. Great Expectations provides support for different data backends such as flat file formats, SQL databases, Pandas dataframes and Sparks, and comes with built-in notification and data documentation functionality.

Exploratory Data Analysis 126

Exploratory Data Analysis Data Visualization Data Analysis Data Analysis 126

AWS Machine Learning Blog

SEPTEMBER 27, 2024

Amazon Simple Storage Service (Amazon S3) stores the model artifacts and creates a data lake to host the inference output, document analysis output, and other datasets in CSV format. SageMaker JumpStart provided deployable models that could be trained for object detection use cases with minimal data science knowledge and overhead.

AWS 124

AWS Data Lakes ML ML 124

IBM Journey to AI blog

SEPTEMBER 25, 2023

Every day, millions of riders use the Uber app, unwittingly contributing to a complex web of data-driven decisions. This blog takes you on a journey into the world of Uber’s analytics and the critical role that Presto, the open source SQL query engine, plays in driving their success. What is Presto?

Data Lakes 102

Data Lakes Analytics Analytics Clustering 102

Tableau

JUNE 8, 2021

In many of the conversations we have with IT and business leaders, there is a sense of frustration about the speed of time-to-value for big data and data science projects. We often hear that organizations have invested in data science capabilities but are struggling to operationalize their machine learning models.

Tableau 52

Tableau Data Lakes Data Warehouse SQL 52

ODSC - Open Data Science

JANUARY 18, 2024

These professionals will work with their colleagues to ensure that data is accessible, with proper access. So let’s go through each step one by one, and help you build a roadmap toward becoming a data engineer. Identify your existing data science strengths. Stay on top of data engineering trends. Get more training!

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

phData

SEPTEMBER 20, 2023

Snowpark is the set of libraries and runtimes in Snowflake that securely deploy and process non-SQL code, including Python , Java, and Scala. As a declarative language, SQL is very powerful in allowing users from all backgrounds to ask questions about data. What is Snowflake’s Snowpark? Why Does Snowpark Matter?

SQL 98

SQL Python Data Lakes Machine Learning 98

AWS Machine Learning Blog

JUNE 5, 2023

Many teams are turning to Athena to enable interactive querying and analyze their data in the respective data stores without creating multiple data copies. Athena allows applications to use standard SQL to query massive amounts of data on an S3 data lake. Create a data lake with Lake Formation.

Machine Learning 98

Machine Learning Machine Learning AWS Data Lakes 98

Dataversity

MARCH 12, 2021

blog series, we experiment with the most interesting blends of data and tools. Whether it’s mixing traditional sources with modern data lakes, open-source devops on the cloud with protected internal legacy tools, SQL with noSQL, web-wisdom-of-the-crowd with in-house handwritten notes, or IoT […].

Data Lakes 52

Data Lakes SQL Database Data Science 52

Dataversity

MAY 7, 2021

blog series, we experiment with the most interesting blends of data and tools. Whether it’s mixing traditional sources with modern data lakes, open-source DevOps on the cloud with protected internal legacy tools, SQL with noSQL, web-wisdom-of-the-crowd with in-house handwritten notes, or IoT […].

Data Lakes 52

Data Lakes SQL Database Data Science 52

Alation

OCTOBER 7, 2021

In this essay, Jason reflects on the value of thinking spatially about data, showing how his experience as a graduate student influences his role as a data scientist today. The popularity of location data and GIS-styled analyses has amplified a common cry in GIS- turned-data-science circles: “Spatial isn’t special!”

Data Scientist 98

Data Scientist Data Lakes Data Science SQL 98

Dataversity

FEBRUARY 2, 2022

blog series, we experiment with the most interesting blends of data and tools. In the “Will They Blend?”

Data Lakes 52

Data Lakes SQL Analytics Analytics 52

Dataversity

JULY 9, 2021

blog series, we experiment with the most interesting blends of data and tools. Whether it’s mixing traditional sources with modern data lakes, open-source DevOps on the cloud with protected internal legacy tools, SQL with NoSQL, web-wisdom-of-the-crowd with in-house handwritten notes, or IoT […].

Azure 52

Azure Data Lakes SQL ML 52

Snorkel AI

MAY 26, 2023

[link] Ahmad Khan, head of artificial intelligence and machine learning strategy at Snowflake gave a presentation entitled “Scalable SQL + Python ML Pipelines in the Cloud” about his company’s Snowpark service at Snorkel AI’s Future of Data-Centric AI virtual conference in August 2022. Welcome everybody.

SQL 52

SQL ML ML Python 52

Snorkel AI

MAY 26, 2023

[link] Ahmad Khan, head of artificial intelligence and machine learning strategy at Snowflake gave a presentation entitled “Scalable SQL + Python ML Pipelines in the Cloud” about his company’s Snowpark service at Snorkel AI’s Future of Data-Centric AI virtual conference in August 2022. Welcome everybody.

SQL 52

SQL ML ML Python 52

AWS Machine Learning Blog

AUGUST 21, 2023

You can streamline the process of feature engineering and data preparation with SageMaker Data Wrangler and finish each stage of the data preparation workflow (including data selection, purification, exploration, visualization, and processing at scale) within a single visual interface.

AWS 98

AWS Data Lakes Clustering Data Preparation 98

ODSC - Open Data Science

JULY 27, 2023

Choosing a Data Lake Format: What to Actually Look For The differences between many data lake products today might not matter as much as you think. When choosing a data lake, here’s something else to consider. Use this guide to get started with your prompt engineering skills! Register now for 60% off.

Data Lakes 52

Data Lakes SQL AI AI 52

ODSC - Open Data Science

APRIL 3, 2023

As the sibling of data science, data analytics is still a hot field that garners significant interest. Companies have plenty of data at their disposal and are looking for people who can make sense of it and make deductions quickly and efficiently. Cloud Services: Google Cloud Platform, AWS, Azure.

Analytics 52

Analytics Analytics Data Analyst Data Science 52