AWS Machine Learning Blog

MARCH 3, 2025

SageMaker HyperPod recipes help data scientists and developers of all skill sets to get started training and fine-tuning popular publicly available generative AI models in minutes with state-of-the-art training performance. The launcher will interface with your cluster with Slurm or Kubernetes native constructs.

Clustering 81

Clustering AWS ML ML 81

Data Science Dojo

FEBRUARY 20, 2023

It allows data scientists to build models that can automate specific tasks. It provides a large cluster of clusters on a single machine. Spark is a general-purpose distributed data processing engine that can handle large volumes of data for applications like data analysis, fraud detection, and machine learning.

Machine Learning 195

Machine Learning Machine Learning AWS Azure 195

AWS Machine Learning Blog

MARCH 15, 2023

Tens of thousands of AWS customers use AWS machine learning (ML) services to accelerate their ML development with fully managed infrastructure and tools. We demonstrate how two different personas, a data scientist and an MLOps engineer, can collaborate to lift and shift hundreds of legacy models.

AWS 134

AWS Machine Learning Machine Learning Data Scientist 134

AWS Machine Learning Blog

APRIL 22, 2024

Amazon SageMaker HyperPod is purpose-built to accelerate foundation model (FM) training, removing the undifferentiated heavy lifting involved in managing and optimizing a large training compute cluster. In this solution, HyperPod cluster instances use the LDAPS protocol to connect to the AWS Managed Microsoft AD via an NLB.

Clustering 104

Clustering AWS Machine Learning Machine Learning 104

AWS Machine Learning Blog

SEPTEMBER 18, 2024

The compute clusters used in these scenarios are composed of more than thousands of AI accelerators such as GPUs or AWS Trainium and AWS Inferentia , custom machine learning (ML) chips designed by Amazon Web Services (AWS) to accelerate deep learning workloads in the cloud.

Clustering 104

Clustering AWS ML ML 104

AWS Machine Learning Blog

APRIL 29, 2024

This is a guest post co-authored with Ville Tuulos (Co-founder and CEO) and Eddie Mattia (Data Scientist) of Outerbounds. For AWS and Outerbounds customers, the goal is to build a differentiated machine learning and artificial intelligence (ML/AI) system and reliably improve it over time.

AWS 117

AWS ML ML Python 117

AWS Machine Learning Blog

DECEMBER 4, 2024

Orchestrate with Tecton-managed EMR clusters – After features are deployed, Tecton automatically creates the scheduling, provisioning, and orchestration needed for pipelines that can run on Amazon EMR compute engines. You can view and create EMR clusters directly through the SageMaker notebook.

ML 86

ML ML AWS AI 86

AWS Machine Learning Blog

NOVEMBER 27, 2024

Launching a machine learning (ML) training cluster with Amazon SageMaker training jobs is a seamless process that begins with a straightforward API call, AWS Command Line Interface (AWS CLI) command, or AWS SDK interaction. Surya Kari is a Senior Generative AI Data Scientist at AWS.

AWS 99

AWS Clustering ML ML 99

AWS Machine Learning Blog

SEPTEMBER 4, 2024

A challenge for DevOps engineers is the additional complexity that comes from using Kubernetes to manage the deployment stage while resorting to other tools (such as the AWS SDK or AWS CloudFormation ) to manage the model building pipeline. kubectl for working with Kubernetes clusters. eksctl for working with EKS clusters.

AWS 95

AWS Clustering ML ML 95

AWS Machine Learning Blog

MAY 1, 2024

Llama2 by Meta is an example of an LLM offered by AWS. To learn more about Llama 2 on AWS, refer to Llama 2 foundation models from Meta are now available in Amazon SageMaker JumpStart. Virginia) and US West (Oregon) AWS Regions, and most recently announced general availability in the US East (Ohio) Region.

AWS 107

AWS ML ML Clustering 107

AWS Machine Learning Blog

AUGUST 21, 2023

You can streamline the process of feature engineering and data preparation with SageMaker Data Wrangler and finish each stage of the data preparation workflow (including data selection, purification, exploration, visualization, and processing at scale) within a single visual interface.

AWS 92

AWS Data Lakes Clustering Data Preparation 92

AWS Machine Learning Blog

NOVEMBER 14, 2024

We recently announced the general availability of cross-account sharing of Amazon SageMaker Model Registry using AWS Resource Access Manager (AWS RAM) , making it easier to securely share and discover machine learning (ML) models across your AWS accounts.

AWS 87

AWS ML ML Machine Learning 87

AWS Machine Learning Blog

SEPTEMBER 3, 2024

Seamless integration with SageMaker – As a built-in feature of the SageMaker platform, the EMR Serverless integration provides a unified and intuitive experience for data scientists and engineers. This same interface is also used for provisioning EMR clusters. The following diagram illustrates this solution.

AWS 106

AWS Clustering Big Data Big Data 106

AWS Machine Learning Blog

DECEMBER 13, 2024

Multiple users such as ML researchers, software engineers, data scientists, and cluster administrators can work concurrently on the same cluster, each managing their own jobs and files without interfering with others. This blog post specifically applies to HyperPod clusters using Slurm as the orchestrator.

Clustering 91

Clustering AWS ML ML 91

AWS Machine Learning Blog

MAY 1, 2024

Our unstructured data comes from the Amazon EC2 User Guide for Linux Instances and Amazon EC2 Instance Types documentation, and the structured data is derived from the EC2 Instance On-Demand Pricing for the US East (N. Virginia) AWS Region. file for deploying the solution using the AWS CDK. xlarge instance to $24.78

AWS 120

AWS Machine Learning Machine Learning SQL 120

AWS Machine Learning Blog

DECEMBER 29, 2022

However, working with data in the cloud can present challenges, such as the need to remove organizational data silos, maintain security and compliance, and reduce complexity by standardizing tooling. AWS offers tools such as RStudio on SageMaker and Amazon Redshift to help tackle these challenges. 1 Public subnet. 1 NAT gateway.

AWS 120

AWS Machine Learning Machine Learning Natural Language Processing 120

AWS Machine Learning Blog

MARCH 30, 2023

The match-related data is collected and ingested using DFL’s DataHub. Metadata of the match is processed within the AWS Lambda function MetaDataIngestion , while positional data is ingested using the AWS Fargate container called MatchLink. Tareq Haschemi is a consultant within AWS Professional Services.

AWS 83

AWS Machine Learning Machine Learning Apache Kafka 83

AWS Machine Learning Blog

OCTOBER 2, 2024

In this post, we share AWS guidance that we have learned and developed as part of real-world projects into practical guides oriented towards the AWS Well-Architected Framework , which is used to build production infrastructure and applications on AWS. To learn more, see Log Amazon Bedrock API calls using AWS CloudTrail.

AWS 113

AWS AI AI ML 113

AWS Machine Learning Blog

JANUARY 17, 2024

In this first post, we introduce mobility data, its sources, and a typical schema of this data. We then discuss the various use cases and explore how you can use AWS services to clean the data, how machine learning (ML) can aid in this effort, and how you can make ethical use of the data in generating visuals and insights.

Clustering 122

Clustering AWS ML ML 122

Flipboard

AUGUST 17, 2023

Amazon Redshift uses SQL to analyze structured and semi-structured data across data warehouses, operational databases, and data lakes, using AWS-designed hardware and ML to deliver the best price-performance at any scale. To do this, we provide an AWS CloudFormation template to create a stack that contains the resources.

ML 123

ML ML AWS Data Warehouse 123

AWS Machine Learning Blog

DECEMBER 18, 2023

By using these capabilities, businesses can efficiently store, manage, and analyze time-series data, enabling data-driven decisions and gaining a competitive edge. sales-train-data is used to store data extracted from MongoDB Atlas, while sales-forecast-output contains predictions from Canvas. Note we have two folders.

Clustering 119

Clustering AWS Database ML 119

AWS Machine Learning Blog

FEBRUARY 5, 2025

The listing indexer AWS Lambda function continuously polls the queue and processes incoming listing updates. With Amazon OpenSearch Service, you get a fully managed solution that makes it simple to deploy, scale, and operate OpenSearch in the AWS Cloud. For data handling, 24 data nodes (r6gd.2xlarge.search

K-nearest Neighbors 75

K-nearest Neighbors Machine Learning Machine Learning Database 75

AWS Machine Learning Blog

SEPTEMBER 26, 2024

During the iterative research and development phase, data scientists and researchers need to run multiple experiments with different versions of algorithms and scale to larger models. However, building large distributed training clusters is a complex and time-intensive process that requires in-depth expertise.

Clustering 112

Clustering Algorithm ML ML 112

AWS Machine Learning Blog

APRIL 5, 2024

SageMaker geospatial capabilities make it straightforward for data scientists and machine learning (ML) engineers to build, train, and deploy models using geospatial data. Now, with the specialized geospatial container in SageMaker, managing and running clusters for geospatial processing has become more straightforward.

Clustering 122

Clustering ML ML AWS 122

AWS Machine Learning Blog

APRIL 17, 2023

In conjunction with tools like RStudio on SageMaker, users are analyzing, transforming, and preparing large amounts of data as part of the data science and ML workflow. Data scientists and data engineers use Apache Spark, Hive, and Presto running on Amazon EMR for large-scale data processing.

Clustering 72

Clustering AWS Machine Learning Machine Learning 72

AWS Machine Learning Blog

NOVEMBER 16, 2023

This is a joint blog with AWS and Philips. Since 2014, the company has been offering customers its Philips HealthSuite Platform, which orchestrates dozens of AWS services that healthcare and life sciences companies use to improve patient care.

ML 106

ML ML AWS AI 106

Flipboard

MARCH 13, 2023

There are two main purposes for building this pipeline: support the data scientists for late-stage model development, and surface model predictions in the product by serving models in high volume and in real-time production traffic. Amazon SNS is fully managed pub/sub service for A2A and A2P messaging.

ML 123

ML ML AWS Data Scientist 123

AWS Machine Learning Blog

JANUARY 13, 2023

Because they’re in a highly regulated domain, HCLS partners and customers seek privacy-preserving mechanisms to manage and analyze large-scale, distributed, and sensitive data. To mitigate these challenges, we propose a federated learning (FL) framework, based on open-source FedML on AWS, which enables analyzing sensitive HCLS data.

AWS 81

AWS Analytics Analytics Machine Learning 81

AWS Machine Learning Blog

OCTOBER 16, 2023

Infrastructure and development challenges Veriff’s backend architecture is based on a microservices pattern, with services running on different Kubernetes clusters hosted on AWS infrastructure. These APIs required production-grade code, which made it challenging for data scientists to productionize models.

Data Scientist 101

Data Scientist ML ML AWS 101

O'Reilly Media

OCTOBER 19, 2021

ML use cases rarely dictate the master data management solution, so the ML stack needs to integrate with existing data warehouses. Prior to the cloud, setting up and operating a cluster that can handle workloads like this would have been a major technical challenge. Data Science Layers. Software Development Layers.

ML 145

ML ML Data Scientist AWS 145

AWS Machine Learning Blog

APRIL 16, 2024

Amazon SageMaker Studio provides a fully managed solution for data scientists to interactively build, train, and deploy machine learning (ML) models. In the process of working on their ML tasks, data scientists typically start their workflow by discovering relevant data sources and connecting to them.

SQL 108

SQL AWS Database Data Scientist 108

AWS Machine Learning Blog

MARCH 10, 2023

Aggregating and preparing large amounts of data is a critical part of ML workflow. Data scientists and data engineers use Apache Spark, Apache Hive, and Presto running on Amazon EMR for large-scale data processing. For each option, we deploy a unique stack of AWS CloudFormation templates.

Clustering 87

Clustering AWS ML ML 87

AWS Machine Learning Blog

APRIL 25, 2024

We provide a comprehensive guide on how to deploy speaker segmentation and clustering solutions using SageMaker on the AWS Cloud. Solution overview Amazon Transcribe is the go-to service for speaker diarization in AWS. Make sure the AWS account has a service quota for hosting a SageMaker endpoint for an ml.g5.2xlarge instance.

AWS 113

AWS ML ML Python 113

AWS Machine Learning Blog

JANUARY 6, 2023

Many AWS media and entertainment customers license IMDb data through AWS Data Exchange to improve content discovery and increase customer engagement and retention. In Part 1 , we discussed the applications of GNNs and how to transform and prepare our IMDb data into a knowledge graph (KG). Background. Prerequisites.

AWS 94

AWS ML ML Machine Learning 94

AWS Machine Learning Blog

JULY 6, 2023

Many organizations choose SageMaker as their ML platform because it provides a common set of tools for developers and data scientists. We also deep dive into the most common architectures and AWS resources to facilitate these integrations. In some cases, an ISV may deploy their software in the customer AWS account.

ML 88

ML ML AWS Data Scientist 88

AWS Machine Learning Blog

FEBRUARY 12, 2024

One of the several challenges faced was adapting the existing on-premises pipeline solution for use on AWS. The solution involved two key components: Modifying and extending existing code – The first part of our solution involved the modification and extension of our existing code to make it compatible with AWS infrastructure.

ML 128

ML ML AWS Machine Learning 128

O'Reilly Media

JANUARY 17, 2023

Given that, what would you say is the job of a data scientist (or ML engineer, or any other such title)? A common task for a data scientist is to build a predictive model. You know the drill: pull some data, carve it up into features, feed it into one of scikit-learn’s various algorithms. Building Models.

ML 145

ML ML Data Scientist Machine Learning 145