ODSC - Open Data Science

JANUARY 30, 2024

Data engineering is a hot topic in the AI industry right now. And as data’s complexity and volume grow, its importance across industries will only become more noticeable. But what exactly do data engineers do? So let’s do a quick overview of the job of data engineer, and maybe you might find a new interest.

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

Pickl AI

AUGUST 1, 2023

Aspiring and experienced Data Engineers alike can benefit from a curated list of books covering essential concepts and practical techniques. These 10 Best Data Engineering Books for beginners encompass a range of topics, from foundational principles to advanced data processing methods. What is Data Engineering?

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

ODSC - Open Data Science

FEBRUARY 15, 2024

We couldn’t be more excited to announce the first sessions for our second annual Data Engineering Summit , co-located with ODSC East this April. Join us for 2 days of talks and panels from leading experts and data engineering pioneers. In the meantime, check out our first group of sessions.

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

ODSC - Open Data Science

FEBRUARY 15, 2024

The Future of the Single Source of Truth is an Open Data Lake Organizations that strive for high-performance data systems are increasingly turning towards the ELT (Extract, Load, Transform) model using an open data lake.

Data Lakes 52

Data Lakes Data Visualization Machine Learning Machine Learning 52

ODSC - Open Data Science

JANUARY 18, 2024

Data engineering is a rapidly growing field, and there is a high demand for skilled data engineers. If you are a data scientist, you may be wondering if you can transition into data engineering. In this blog post, we will discuss how you can become a data engineer if you are a data scientist.

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

phData

SEPTEMBER 20, 2023

Snowpark is the set of libraries and runtimes in Snowflake that securely deploy and process non-SQL code, including Python , Java, and Scala. On the server side, runtimes include Python, Java, and Scala in the warehouse model or Snowpark Container Services (private preview). Why is Snowpark Exciting to us?

SQL 98

SQL Python Data Lakes Machine Learning 98

ODSC - Open Data Science

FEBRUARY 2, 2023

This doesn’t mean anything too complicated, but could range from basic Excel work to more advanced reporting to be used for data visualization later on. Computer Science and Computer Engineering Similar to knowing statistics and math, a data scientist should know the fundamentals of computer science as well.

Data Science 98

Data Science Data Scientist Computer Science Computer Science 98

AWS Machine Learning Blog

JUNE 18, 2024

The solution harnesses the capabilities of generative AI, specifically Large Language Models (LLMs), to address the challenges posed by diverse sensor data and automatically generate Python functions based on various data formats. The solution only invokes the LLM for new device data file type (code has not yet been generated).

AWS 113

AWS Python AI AI 113

Alation

JANUARY 26, 2023

Governance can — and should — be the responsibility of every data user, though how that’s achieved will depend on the role within the organization. This article will focus on how data engineers can improve their approach to data governance. How can data engineers address these challenges directly?

Data Governance 52

Data Governance Data Engineering Data Engineering Data Engineering 52

AWS Machine Learning Blog

FEBRUARY 21, 2025

This setup uses the AWS SDK for Python (Boto3) to interact with AWS services. He specializes in large language models, cloud infrastructure, and scalable data systems, focusing on building intelligent solutions that enhance automation and data accessibility across Amazons operations.

AWS 104

AWS Natural Language Processing Machine Learning Machine Learning 104

Mlearning.ai

JULY 8, 2023

EL stands for extract and load, and its primary goal is to just move the data from one place to another where the destination is usually a Data Warehouse or a Data Lake. The most fundamental difference between ELT and ETL is that the former first loads the data into the target storage and, then, processes them.

ETL 52

ETL Azure Python Internet of Things 52

ODSC - Open Data Science

DECEMBER 19, 2023

Data Engineer Data engineers are responsible for the end-to-end process of collecting, storing, and processing data. They use their knowledge of data warehousing, data lakes, and big data technologies to build and maintain data pipelines.

Data Scientist 96

Data Scientist Machine Learning Machine Learning Computer Science 96

Snorkel AI

MAY 26, 2023

[link] Ahmad Khan, head of artificial intelligence and machine learning strategy at Snowflake gave a presentation entitled “Scalable SQL + Python ML Pipelines in the Cloud” about his company’s Snowpark service at Snorkel AI’s Future of Data-Centric AI virtual conference in August 2022. Welcome everybody.

SQL 52

SQL ML ML Python 52

Snorkel AI

MAY 26, 2023

[link] Ahmad Khan, head of artificial intelligence and machine learning strategy at Snowflake gave a presentation entitled “Scalable SQL + Python ML Pipelines in the Cloud” about his company’s Snowpark service at Snorkel AI’s Future of Data-Centric AI virtual conference in August 2022. Welcome everybody.

SQL 52

SQL ML ML Python 52

IBM Journey to AI blog

SEPTEMBER 19, 2023

To pursue a data science career, you need a deep understanding and expansive knowledge of machine learning and AI. Your skill set should include the ability to write in the programming languages Python, SAS, R and Scala. And you should have experience working with big data platforms such as Hadoop or Apache Spark.

Data Science 94

Data Science Analytics Analytics Data Scientist 94

The MLOps Blog

JUNE 27, 2023

Alignment to other tools in the organization’s tech stack Consider how well the MLOps tool integrates with your existing tools and workflows, such as data sources, data engineering platforms, code repositories, CI/CD pipelines, monitoring systems, etc. This provides end-to-end support for data engineering and MLOps workflows.

Machine Learning 134

Machine Learning Machine Learning ML ML 134

The MLOps Blog

JANUARY 23, 2023

However, there are some key differences that we need to consider: Size and complexity of the data In machine learning, we are often working with much larger data. Basically, every machine learning project needs data. Given the range of tools and data types, a separate data versioning logic will be necessary.

ML 52

ML ML Data Lakes Machine Learning 52

Flipboard

NOVEMBER 24, 2023

JuMa is a service of BMW Group’s AI platform for its data analysts, ML engineers, and data scientists that provides a user-friendly workspace with an integrated development environment (IDE). It is powered by Amazon SageMaker Studio and provides JupyterLab for Python and Posit Workbench for R.

ML 153

ML ML AWS AI 153

ODSC - Open Data Science

NOVEMBER 19, 2024

Mustafa Hajij introduced TopoX, a comprehensive Python suite for topological deep learning. This session demonstrated how to leverage these tools using Python and PyTorch, offering attendees practical techniques to apply in their research and projects. Introduction to Containers for Data Science / Data Engineering with Michael A.

Deep Learning 52

Deep Learning Deep Learning Data Science AI 52

AWS Machine Learning Blog

JUNE 20, 2024

Our goal was to improve the user experience of an existing application used to explore the counters and insights data. The data is stored in a data lake and retrieved by SQL using Amazon Athena. You can experiment with and evaluate top FMs for your use case and customize them with your data.

SQL 95

SQL Database AWS Machine Learning 95

ODSC - Open Data Science

OCTOBER 20, 2023

You’ll use MLRun, Langchain, and Milvus for this exercise and cover topics like the integration of AI/ML applications, leveraging Python SDKs, as well as building, testing, and tuning your work. In this session, we’ll demonstrate how you can fine-tune a Gen AI model, build a Gen AI application, and deploy it in 20 minutes.

AI 52

AI AI Data Science Machine Learning 52

IBM Journey to AI blog

JULY 17, 2023

Within watsonx.ai, users can take advantage of open-source frameworks like PyTorch, TensorFlow and scikit-learn alongside IBM’s entire machine learning and data science toolkit and its ecosystem tools for code-based and visual data science capabilities.

AI 74

AI AI Machine Learning Machine Learning 74

ODSC - Open Data Science

NOVEMBER 18, 2024

Introduction to Containers for Data Science/Data Engineering Michael A Fudge | Professor of Practice, MSIS Program Director | Syracuse University’s iSchool In this hands-on session, you’ll learn how to leverage the benefits of containers for DS and data engineering workflows.

Deep Learning 40

Deep Learning Deep Learning Database Data Science 40

DagsHub

OCTOBER 23, 2024

To combine the collected data, you can integrate different data producers into a data lake as a repository. A central repository for unstructured data is beneficial for tasks like analytics and data virtualization. Data Cleaning The next step is to clean the data after ingesting it into the data lake.

Machine Learning 52

Machine Learning Machine Learning Data Lakes AI 52

ODSC - Open Data Science

MARCH 11, 2024

This track will focus on helping you build skills in text mining, data storytelling, data mining, and predictive analytics through use cases highlighting the latest techniques and processes to collect, clean, and analyze growing volumes of structured data.

Data Science 52

Data Science Deep Learning Deep Learning Machine Learning 52

ODSC - Open Data Science

APRIL 3, 2023

Data analysts often must go out and find their data, process it, clean it, and get it ready for analysis. This pushes into Big Data as well, as many companies now have significant amounts of data and large data lakes that need analyzing.

Analytics 52

Analytics Analytics Data Analyst Data Science 52

phData

FEBRUARY 24, 2023

Organizations can unite their siloed data and securely share governed data while executing diverse analytic workloads. Snowflake’s engine provides a solution for data warehousing, data lakes, data engineering, data science, data application development, and data sharing.

Analytics 52

Analytics Analytics Database Python 52

ODSC - Open Data Science

MARCH 30, 2023

Here are 5 reasons why it’s important to always keep learning in data science and AI. Learn about data structures, control structures, functions, models, file handling, and other basics of coding with Python in this upcoming programming primer, included in the Mini-Bootcamp Pass. Register now for 40% off.

Azure 52

Azure ML ML Data Models 52

Pickl AI

SEPTEMBER 5, 2024

Storage Solutions: Secure and scalable storage options like Azure Blob Storage and Azure Data Lake Storage. Key features and benefits of Azure for Data Science include: Scalability: Easily scale resources up or down based on demand, ideal for handling large datasets and complex computations.

Azure 52

Azure Data Scientist Data Science Machine Learning 52

AWS Machine Learning Blog

FEBRUARY 7, 2025

Data Governance Account This account hosts data governance services for data lake, central feature store, and fine-grained data access. The SageMaker Project Portfolio has SageMaker projects that data scientists and ML engineers can use to accelerate model training and deployment.

ML 62

ML ML Data Scientist AWS 62

Alation

MARCH 1, 2022

Through Impact Analysis, users can determine if a problem occurred with data upstream, and locate the impacted data downstream. With robust data lineage, data engineers can find and fix issues fast and prevent them from recurring. Similarly, analysts gain a clear view of how data is created.

Data Warehouse 52

Data Warehouse Data Lakes Cloud Data Database 52

Alation

JULY 19, 2022

This creates a second layer of governance to ensure the data scientist is using the right data in ways that are permitted. Explore the Data. Though most data scientists will ultimately want to plot the data directly in a Python or R notebook to play around with it, data catalogs give them a jump start on the exploration phase.

Data Scientist 59

Data Scientist Data Quality Data Science Data Analyst 59

ODSC - Open Data Science

APRIL 24, 2023

To cluster the data we have to calculate distances between IPs — The number of all possible IP pairs is very large, and we had to solve the scale problem. Data Processing and Clustering Our data is stored in a Data Lake and we used PrestoDB as a query engine.

Clustering 52

Clustering SQL Algorithm Data Science 52

phData

FEBRUARY 14, 2023

Qlik Replicate Qlik Replicate is a data integration tool that supports a wide range of source and target endpoints with configuration and automation capabilities that can give your organization easy, high-performance access to the latest and most accurate data. This allows users to utilize Python to customize transformations.

Data Warehouse 59

Data Warehouse Azure AWS Database 59

The MLOps Blog

MAY 17, 2023

This article explores the importance of ETL pipelines in machine learning, a hands-on example of building ETL pipelines with a popular tool, and suggests the best ways for data engineers to enhance and sustain their pipelines. We also need data profiling i.e. data discovery, to understand if the data is appropriate for ETL.

ETL 59

ETL Data Pipeline ML ML 59

Alation

NOVEMBER 28, 2022

But refreshing this analysis with the latest data was impossible… unless you were proficient in SQL or Python. We wanted to make it easy for anyone to pull data and self service without the technical know-how of the underlying database or data lake. Sathish and I met in 2004 when we were working for Oracle.

Data Governance 52

Data Governance Database Data Quality Data Lakes 52

The MLOps Blog

MARCH 21, 2023

.” — Conor Murphy , Lead Data Scientist at Databricks, in “Survey of Production ML Tech Stacks” at the Data+AI Summit 2022 Your team should be motivated by MLOps to show everything that goes into making a machine learning model, from getting the data to deploying and monitoring the model. Allegro.io

Machine Learning 69

Machine Learning Machine Learning Data Scientist ML 69