AWS Machine Learning Blog

MARCH 10, 2025

We walk through the journey Octus took from managing multiple cloud providers and costly GPU instances to implementing a streamlined, cost-effective solution using AWS services including Amazon Bedrock, AWS Fargate , and Amazon OpenSearch Service. Along the way, it also simplified operations as Octus is an AWS shop more generally.

AWS 105

AWS Database AI AI 105

AWS Machine Learning Blog

MARCH 3, 2025

SageMaker HyperPod recipes help data scientists and developers of all skill sets to get started training and fine-tuning popular publicly available generative AI models in minutes with state-of-the-art training performance. The launcher will interface with your cluster with Slurm or Kubernetes native constructs.

Clustering 91

Clustering AWS ML ML 91

AWS Machine Learning Blog

SEPTEMBER 18, 2024

The compute clusters used in these scenarios are composed of more than thousands of AI accelerators such as GPUs or AWS Trainium and AWS Inferentia , custom machine learning (ML) chips designed by Amazon Web Services (AWS) to accelerate deep learning workloads in the cloud.

Clustering 115

Clustering AWS ML ML 115

AWS Machine Learning Blog

MARCH 15, 2023

Tens of thousands of AWS customers use AWS machine learning (ML) services to accelerate their ML development with fully managed infrastructure and tools. We demonstrate how two different personas, a data scientist and an MLOps engineer, can collaborate to lift and shift hundreds of legacy models.

Machine Learning 143

Machine Learning AWS Machine Learning Data Scientist 143

AWS Machine Learning Blog

DECEMBER 4, 2024

Orchestrate with Tecton-managed EMR clusters – After features are deployed, Tecton automatically creates the scheduling, provisioning, and orchestration needed for pipelines that can run on Amazon EMR compute engines. You can view and create EMR clusters directly through the SageMaker notebook.

ML 98

ML ML AWS AI 98

AWS Machine Learning Blog

APRIL 22, 2024

Amazon SageMaker HyperPod is purpose-built to accelerate foundation model (FM) training, removing the undifferentiated heavy lifting involved in managing and optimizing a large training compute cluster. In this solution, HyperPod cluster instances use the LDAPS protocol to connect to the AWS Managed Microsoft AD via an NLB.

Clustering 117

Clustering AWS Machine Learning Machine Learning 117

AWS Machine Learning Blog

NOVEMBER 27, 2024

Launching a machine learning (ML) training cluster with Amazon SageMaker training jobs is a seamless process that begins with a straightforward API call, AWS Command Line Interface (AWS CLI) command, or AWS SDK interaction. Surya Kari is a Senior Generative AI Data Scientist at AWS.

AWS 109

AWS Clustering ML ML 109

Data Science Dojo

FEBRUARY 20, 2023

It allows data scientists to build models that can automate specific tasks. It provides a large cluster of clusters on a single machine. Spark is a general-purpose distributed data processing engine that can handle large volumes of data for applications like data analysis, fraud detection, and machine learning.

Machine Learning 195

Machine Learning Machine Learning AWS Azure 195

AWS Machine Learning Blog

SEPTEMBER 4, 2024

A challenge for DevOps engineers is the additional complexity that comes from using Kubernetes to manage the deployment stage while resorting to other tools (such as the AWS SDK or AWS CloudFormation ) to manage the model building pipeline. kubectl for working with Kubernetes clusters. eksctl for working with EKS clusters.

AWS 106

AWS Clustering ML ML 106

AWS Machine Learning Blog

APRIL 29, 2024

This is a guest post co-authored with Ville Tuulos (Co-founder and CEO) and Eddie Mattia (Data Scientist) of Outerbounds. For AWS and Outerbounds customers, the goal is to build a differentiated machine learning and artificial intelligence (ML/AI) system and reliably improve it over time.

AWS 124

AWS ML ML Python 124

AWS Machine Learning Blog

NOVEMBER 14, 2024

We recently announced the general availability of cross-account sharing of Amazon SageMaker Model Registry using AWS Resource Access Manager (AWS RAM) , making it easier to securely share and discover machine learning (ML) models across your AWS accounts.

AWS 101

AWS ML ML Machine Learning 101

AWS Machine Learning Blog

MAY 1, 2024

Llama2 by Meta is an example of an LLM offered by AWS. To learn more about Llama 2 on AWS, refer to Llama 2 foundation models from Meta are now available in Amazon SageMaker JumpStart. Virginia) and US West (Oregon) AWS Regions, and most recently announced general availability in the US East (Ohio) Region.

AWS 120

AWS ML ML Clustering 120

AWS Machine Learning Blog

SEPTEMBER 18, 2024

In addition to its groundbreaking AI innovations, Zeta Global has harnessed Amazon Elastic Container Service (Amazon ECS) with AWS Fargate to deploy a multitude of smaller models efficiently. These include dbt pipelines, data gathering jobs, training, evaluation, and batch inference jobs for smaller models.

AWS 120

AWS Machine Learning Machine Learning ML 120

AWS Machine Learning Blog

AUGUST 21, 2023

You can streamline the process of feature engineering and data preparation with SageMaker Data Wrangler and finish each stage of the data preparation workflow (including data selection, purification, exploration, visualization, and processing at scale) within a single visual interface.

AWS 98

AWS Data Lakes Clustering Data Preparation 98

AWS Machine Learning Blog

SEPTEMBER 3, 2024

Seamless integration with SageMaker – As a built-in feature of the SageMaker platform, the EMR Serverless integration provides a unified and intuitive experience for data scientists and engineers. This same interface is also used for provisioning EMR clusters. The following diagram illustrates this solution.

AWS 116

AWS Clustering Big Data Big Data 116

AWS Machine Learning Blog

JANUARY 17, 2024

Today, we’re excited to announce the availability of Llama 2 inference and fine-tuning support on AWS Trainium and AWS Inferentia instances in Amazon SageMaker JumpStart. In this post, we demonstrate how to deploy and fine-tune Llama 2 on Trainium and AWS Inferentia instances in SageMaker JumpStart.

AWS 129

AWS Python Machine Learning Machine Learning 129

AWS Machine Learning Blog

DECEMBER 13, 2024

Multiple users such as ML researchers, software engineers, data scientists, and cluster administrators can work concurrently on the same cluster, each managing their own jobs and files without interfering with others. This blog post specifically applies to HyperPod clusters using Slurm as the orchestrator.

Clustering 100

Clustering AWS ML ML 100

AWS Machine Learning Blog

MARCH 30, 2023

The match-related data is collected and ingested using DFL’s DataHub. Metadata of the match is processed within the AWS Lambda function MetaDataIngestion , while positional data is ingested using the AWS Fargate container called MatchLink. Tareq Haschemi is a consultant within AWS Professional Services.

AWS 97

AWS Machine Learning Machine Learning Apache Kafka 97

AWS Machine Learning Blog

DECEMBER 29, 2022

However, working with data in the cloud can present challenges, such as the need to remove organizational data silos, maintain security and compliance, and reduce complexity by standardizing tooling. AWS offers tools such as RStudio on SageMaker and Amazon Redshift to help tackle these challenges. 1 Public subnet. 1 NAT gateway.

AWS 135

AWS Machine Learning Machine Learning Natural Language Processing 135

AWS Machine Learning Blog

OCTOBER 2, 2024

In this post, we share AWS guidance that we have learned and developed as part of real-world projects into practical guides oriented towards the AWS Well-Architected Framework , which is used to build production infrastructure and applications on AWS. To learn more, see Log Amazon Bedrock API calls using AWS CloudTrail.

AWS 126

AWS AI AI ML 126

AWS Machine Learning Blog

JANUARY 17, 2024

In this first post, we introduce mobility data, its sources, and a typical schema of this data. We then discuss the various use cases and explore how you can use AWS services to clean the data, how machine learning (ML) can aid in this effort, and how you can make ethical use of the data in generating visuals and insights.

Clustering 132

Clustering AWS ML ML 132

AWS Machine Learning Blog

MAY 1, 2024

Our unstructured data comes from the Amazon EC2 User Guide for Linux Instances and Amazon EC2 Instance Types documentation, and the structured data is derived from the EC2 Instance On-Demand Pricing for the US East (N. Virginia) AWS Region. file for deploying the solution using the AWS CDK. xlarge instance to $24.78

AWS 129

AWS Machine Learning Machine Learning SQL 129

AWS Machine Learning Blog

APRIL 17, 2023

In conjunction with tools like RStudio on SageMaker, users are analyzing, transforming, and preparing large amounts of data as part of the data science and ML workflow. Data scientists and data engineers use Apache Spark, Hive, and Presto running on Amazon EMR for large-scale data processing.

Clustering 87

Clustering AWS Machine Learning Machine Learning 87

AWS Machine Learning Blog

AUGUST 8, 2024

Webex works with the world’s leading business and productivity apps—including AWS. The following diagram illustrates the WxAI architecture on AWS. The topic clustering model achieves this by clustering all the individually extracted call drivers from a large batch of calls into different topic clusters.

AWS 124

AWS AI AI Clustering 124

AWS Machine Learning Blog

OCTOBER 23, 2024

Agent Creator Creating enterprise-grade, LLM-powered applications and integrations that meet security, governance, and compliance requirements has traditionally demanded the expertise of programmers and data scientists. Data plane The data plane is where the actual data processing and integration take place.

AI 85

AI AI Database AWS 85

AWS Machine Learning Blog

JANUARY 13, 2023

Because they’re in a highly regulated domain, HCLS partners and customers seek privacy-preserving mechanisms to manage and analyze large-scale, distributed, and sensitive data. To mitigate these challenges, we propose a federated learning (FL) framework, based on open-source FedML on AWS, which enables analyzing sensitive HCLS data.

AWS 95

AWS Analytics Analytics Machine Learning 95

Flipboard

AUGUST 17, 2023

Amazon Redshift uses SQL to analyze structured and semi-structured data across data warehouses, operational databases, and data lakes, using AWS-designed hardware and ML to deliver the best price-performance at any scale. To do this, we provide an AWS CloudFormation template to create a stack that contains the resources.

ML 123

ML ML AWS Data Warehouse 123

AWS Machine Learning Blog

FEBRUARY 5, 2025

The listing indexer AWS Lambda function continuously polls the queue and processes incoming listing updates. With Amazon OpenSearch Service, you get a fully managed solution that makes it simple to deploy, scale, and operate OpenSearch in the AWS Cloud. For data handling, 24 data nodes (r6gd.2xlarge.search

K-nearest Neighbors 80

K-nearest Neighbors Machine Learning Machine Learning Database 80

AWS Machine Learning Blog

NOVEMBER 16, 2023

This is a joint blog with AWS and Philips. Since 2014, the company has been offering customers its Philips HealthSuite Platform, which orchestrates dozens of AWS services that healthcare and life sciences companies use to improve patient care.

ML 121

ML ML AWS AI 121

AWS Machine Learning Blog

OCTOBER 20, 2023

We used AWS machine learning (ML) services like Amazon SageMaker to develop a powerful generalized feed ranker (GFR). In the following sections, we discuss each component and the AWS services used in more detail. The following diagram illustrates the GFR architecture.

AWS 120

AWS Data Scientist ML ML 120

AWS Machine Learning Blog

SEPTEMBER 26, 2024

During the iterative research and development phase, data scientists and researchers need to run multiple experiments with different versions of algorithms and scale to larger models. However, building large distributed training clusters is a complex and time-intensive process that requires in-depth expertise.

Clustering 119

Clustering Algorithm ML ML 119

AWS Machine Learning Blog

APRIL 5, 2024

SageMaker geospatial capabilities make it straightforward for data scientists and machine learning (ML) engineers to build, train, and deploy models using geospatial data. Now, with the specialized geospatial container in SageMaker, managing and running clusters for geospatial processing has become more straightforward.

Clustering 132

Clustering ML ML AWS 132

AWS Machine Learning Blog

DECEMBER 18, 2023

By using these capabilities, businesses can efficiently store, manage, and analyze time-series data, enabling data-driven decisions and gaining a competitive edge. sales-train-data is used to store data extracted from MongoDB Atlas, while sales-forecast-output contains predictions from Canvas. Note we have two folders.

Clustering 121

Clustering AWS Database ML 121

AWS Machine Learning Blog

OCTOBER 16, 2023

Infrastructure and development challenges Veriff’s backend architecture is based on a microservices pattern, with services running on different Kubernetes clusters hosted on AWS infrastructure. These APIs required production-grade code, which made it challenging for data scientists to productionize models.

Data Scientist 113

Data Scientist ML ML AWS 113

AWS Machine Learning Blog

MARCH 10, 2023

Aggregating and preparing large amounts of data is a critical part of ML workflow. Data scientists and data engineers use Apache Spark, Apache Hive, and Presto running on Amazon EMR for large-scale data processing. For each option, we deploy a unique stack of AWS CloudFormation templates.

Clustering 99

Clustering AWS ML ML 99

AWS Machine Learning Blog

JANUARY 26, 2023

Data scientists train multiple ML algorithms to examine millions of consumer data records, identify anomalies, and evaluate if a person is eligible for credit. This is a common problem that data scientists face when training their models. About the Authors Tristan Miller is a Lead Data Scientist at Best Egg.

ML 98

ML ML Data Scientist AWS 98

AWS Machine Learning Blog

JULY 6, 2023

Many organizations choose SageMaker as their ML platform because it provides a common set of tools for developers and data scientists. We also deep dive into the most common architectures and AWS resources to facilitate these integrations. In some cases, an ISV may deploy their software in the customer AWS account.

ML 101

ML ML AWS Data Scientist 101