AWS Machine Learning Blog

FEBRUARY 21, 2025

Communication between the two systems was established through Kerberized Apache Livy (HTTPS) connections over AWS PrivateLink. Responsibility for maintenance and troubleshooting: Rockets DevOps/Technology team was responsible for all upgrades, scaling, and troubleshooting of the Hadoop cluster, which was installed on bare EC2 instances.

Data Science 73

Data Science AWS Hadoop Data Scientist 73

AWS Machine Learning Blog

SEPTEMBER 18, 2024

The compute clusters used in these scenarios are composed of more than thousands of AI accelerators such as GPUs or AWS Trainium and AWS Inferentia , custom machine learning (ML) chips designed by Amazon Web Services (AWS) to accelerate deep learning workloads in the cloud.

Clustering 115

Clustering AWS ML ML 115

AWS Machine Learning Blog

JULY 31, 2024

Amazon Web Services (AWS) is committed to supporting the development of cutting-edge generative artificial intelligence (AI) technologies by companies and organizations across the globe. Let’s dive in and explore how these organizations are transforming what’s possible with generative AI on AWS.

AWS 127

AWS AI AI Clustering 127

AWS Machine Learning Blog

NOVEMBER 27, 2024

Launching a machine learning (ML) training cluster with Amazon SageMaker training jobs is a seamless process that begins with a straightforward API call, AWS Command Line Interface (AWS CLI) command, or AWS SDK interaction. Surya Kari is a Senior Generative AI Data Scientist at AWS.

AWS 109

AWS Clustering ML ML 109

AWS Machine Learning Blog

FEBRUARY 13, 2025

Amazon Bedrock offers a serverless experience, so you can get started quickly, privately customize FMs with your own data, and integrate and deploy them into your applications using Amazon Web Services (AWS) services without having to manage infrastructure. AWS Lambda The API is a Fastify application written in TypeScript.

AWS 104

AWS K-nearest Neighbors Clustering Algorithm 104

Smart Data Collective

FEBRUARY 20, 2022

AWS (Amazon Web Services), the comprehensive and evolving cloud computing platform provided by Amazon, is comprised of infrastructure as a service (IaaS), platform as a service (PaaS) and packaged software as a service (SaaS). With its wide array of tools and convenience, AWS has already become a popular choice for many SaaS companies.

AWS 138

AWS Cloud Computing Data Lakes Database 138

AWS Machine Learning Blog

APRIL 29, 2024

Close collaboration with AWS Trainium has also played a major role in making the Arcee platform extremely performant, not only accelerating model training but also reducing overall costs and enforcing compliance and data integrity in the secure AWS environment. Our cluster consisted of 16 nodes, each equipped with a trn1n.32xlarge

AWS 124

AWS Clustering ML ML 124

AWS Machine Learning Blog

APRIL 4, 2023

AWS provides various services catered to time series data that are low code/no code, which both machine learning (ML) and non-ML practitioners can use for building ML solutions. In this post, we seek to separate a time series dataset into individual clusters that exhibit a higher degree of similarity between its data points and reduce noise.

Clustering 100

Clustering ML ML AWS 100

AWS Machine Learning Blog

MAY 1, 2024

Large language models (LLMs) are making a significant impact in the realm of artificial intelligence (AI). Llama2 by Meta is an example of an LLM offered by AWS. To learn more about Llama 2 on AWS, refer to Llama 2 foundation models from Meta are now available in Amazon SageMaker JumpStart.

AWS 119

AWS ML ML Clustering 119

AWS Machine Learning Blog

APRIL 29, 2024

For AWS and Outerbounds customers, the goal is to build a differentiated machine learning and artificial intelligence (ML/AI) system and reliably improve it over time. First, the AWS Trainium accelerator provides a high-performance, cost-effective, and readily available solution for training and fine-tuning large models.

AWS 125

AWS ML ML Python 125

AWS Machine Learning Blog

DECEMBER 4, 2024

Orchestrate with Tecton-managed EMR clusters – After features are deployed, Tecton automatically creates the scheduling, provisioning, and orchestration needed for pipelines that can run on Amazon EMR compute engines. You can view and create EMR clusters directly through the SageMaker notebook.

ML 96

ML ML AWS AI 96

AWS Machine Learning Blog

JUNE 25, 2024

Amazon Web Services is excited to announce the launch of the AWS Neuron Monitor container , an innovative tool designed to enhance the monitoring capabilities of AWS Inferentia and AWS Trainium chips on Amazon Elastic Kubernetes Service (Amazon EKS). The Container Insights dashboard also shows cluster status and alarms.

AWS 93

AWS ML ML Clustering 93

AWS Machine Learning Blog

MAY 30, 2024

CBRE is unlocking the potential of artificial intelligence (AI) to realize value across the entire commercial real estate lifecycle—from guiding investment decisions to managing buildings. AWS Prototyping developed an AWS Cloud Development Kit (AWS CDK) stack for deployment following AWS best practices.

AWS 130

AWS SQL Database AI 130

AWS Machine Learning Blog

NOVEMBER 14, 2024

We recently announced the general availability of cross-account sharing of Amazon SageMaker Model Registry using AWS Resource Access Manager (AWS RAM) , making it easier to securely share and discover machine learning (ML) models across your AWS accounts.

AWS 98

AWS ML ML Machine Learning 98

AWS Machine Learning Blog

MARCH 15, 2023

Tens of thousands of AWS customers use AWS machine learning (ML) services to accelerate their ML development with fully managed infrastructure and tools. The data scientist is responsible for moving the code into SageMaker, either manually or by cloning it from a code repository such as AWS CodeCommit.

Machine Learning 142

Machine Learning Machine Learning AWS Data Scientist 142

AWS Machine Learning Blog

OCTOBER 5, 2023

In this post, we walk through how to fine-tune Llama 2 on AWS Trainium , a purpose-built accelerator for LLM training, to reduce training times and costs. We review the fine-tuning scripts provided by the AWS Neuron SDK (using NeMo Megatron-LM), the various configurations we used, and the throughput results we saw.

AWS 125

AWS Machine Learning Machine Learning Deep Learning 125

AWS Machine Learning Blog

MAY 30, 2024

Generative artificial intelligence (AI) has unlocked fresh opportunities for these use cases. In this post, we introduce the Media Analysis and Policy Evaluation solution, which uses AWS AI and generative AI services to provide a framework to streamline video extraction and evaluation processes.

AWS 131

AWS AI AI ML 131

Data Science Dojo

FEBRUARY 20, 2023

It is used by businesses across industries for a wide range of applications, including fraud prevention, marketing automation, customer service, artificial intelligence (AI), chatbots, virtual assistants, and recommendations. It provides a large cluster of clusters on a single machine.

Machine Learning 195

Machine Learning Machine Learning AWS Azure 195

Dataconomy

MARCH 13, 2023

How to create an artificial intelligence? The creation of artificial intelligence (AI) has long been a dream of scientists, engineers, and innovators. Understanding artificial intelligence Before diving into the process of creating AI, it is important to understand the key concepts and types of AI.

Artificial Intelligence 98

Artificial Intelligence Artificial Intelligence Natural Language Processing Algorithm 98

AWS Machine Learning Blog

DECEMBER 12, 2023

In this post, we’ll summarize training procedure of GPT NeoX on AWS Trainium , a purpose-built machine learning (ML) accelerator optimized for deep learning training. M tokens/$) trained such models with AWS Trainium without losing any model quality. We’ll outline how we cost-effectively (3.2 billion in Pythia.

AWS 124

AWS Machine Learning Machine Learning Deep Learning 124

AWS Machine Learning Blog

JUNE 11, 2024

Each of these products are infused with artificial intelligence (AI) capabilities to deliver exceptional customer experience. During this journey, we collaborated with our AWS technical account manager and the Graviton software engineering teams.

Machine Learning 108

Machine Learning Machine Learning AWS Natural Language Processing 108

AWS Machine Learning Blog

NOVEMBER 30, 2023

Building foundation models (FMs) requires building, maintaining, and optimizing large clusters to train models with tens to hundreds of billions of parameters on vast amounts of data. SageMaker HyperPod integrates the Slurm Workload Manager for cluster and training job orchestration.

Clustering 123

Clustering AWS Machine Learning Machine Learning 123

Flipboard

NOVEMBER 17, 2023

Set up a MongoDB cluster To create a free tier MongoDB Atlas cluster, follow the instructions in Create a Cluster. This type of data is often used in ML and artificial intelligence applications. Specify the AWS Lambda function that will interact with MongoDB Atlas and the LLM to provide responses.

K-nearest Neighbors 170

K-nearest Neighbors AWS Clustering Database 170

AWS Machine Learning Blog

MAY 23, 2023

In April 2023, AWS unveiled Amazon Bedrock , which provides a way to build generative AI-powered apps via pre-trained models from startups including AI21 Labs , Anthropic , and Stability AI. Amazon Bedrock also offers access to Titan foundation models, a family of models trained in-house by AWS. Deploy the AWS CDK application.

AWS 109

AWS ML AI ML 109

Flipboard

FEBRUARY 16, 2023

Modern model pre-training often calls for larger cluster deployment to reduce time and cost. In October 2022, we launched Amazon EC2 Trn1 Instances , powered by AWS Trainium , which is the second generation machine learning accelerator designed by AWS. The following diagram shows an example.

Clustering 131

Clustering AWS Deep Learning Deep Learning 131

AWS Machine Learning Blog

APRIL 7, 2023

To reduce the barrier to entry of ML at the edge, we wanted to demonstrate an example of deploying a pre-trained model from Amazon SageMaker to AWS Wavelength , all in less than 100 lines of code. In this post, we demonstrate how to deploy a SageMaker model to AWS Wavelength to reduce model inference latency for 5G network-based applications.

AWS 99

AWS Clustering ML ML 99

Data Science 101

NOVEMBER 11, 2019

Call for Research Proposals Amazon is seeking proposals impact research in the Artificial Intelligence and Machine Learning areas. If you are at a University or non-profit, you can ask for cash and/or AWS credits. AWS Parallel Cluster for Machine Learning AWS Parallel Cluster is an open-source cluster management tool.

Cloud Data 95

Cloud Data Data Science Azure Clustering 95

IBM Journey to AI blog

JANUARY 16, 2024

Businesses are increasingly embracing data-intensive workloads, including high-performance computing, artificial intelligence (AI) and machine learning (ML). To aid in building more sustainable IT estates, IBM has partnered up with Amazon Web Services (AWS) to facilitate sustainable cloud modernization journeys.

AWS 109

AWS Clustering Database Artificial Intelligence 109

IBM Journey to AI blog

AUGUST 30, 2023

It is an enterprise cloud-based asset management platform that leverages artificial intelligence (AI) , the Internet of Things (IoT) and analytics to help optimize equipment performance, extend asset lifecycles and reduce operational downtime and costs. Why ROSA for Maximo in AWS?

AWS 84

AWS Clustering Internet of Things Artificial Intelligence 84

AWS Machine Learning Blog

AUGUST 21, 2023

Data is frequently kept in data lakes that can be managed by AWS Lake Formation , giving you the ability to implement fine-grained access control using a straightforward grant or revoke procedure. You can use the provided AWS CloudFormation stack to set up the architectural components for this solution.

AWS 97

AWS Data Lakes Clustering Data Preparation 97

AWS Machine Learning Blog

SEPTEMBER 3, 2024

From deriving insights to powering generative artificial intelligence (AI) -driven applications, the ability to efficiently process and analyze large datasets is a vital capability. This same interface is also used for provisioning EMR clusters. This same interface is also used for provisioning EMR clusters.

AWS 116

AWS Clustering Big Data Big Data 116

AWS Machine Learning Blog

MAY 31, 2024

In this blog post and open source project , we show you how you can pre-train a genomics language model, HyenaDNA , using your genomic data in the AWS Cloud. Amazon SageMaker Amazon SageMaker is a fully managed ML service offered by AWS, designed to reduce the time and cost associated with training and tuning ML models at scale.

AWS 110

AWS ML ML Machine Learning 110

AWS Machine Learning Blog

APRIL 19, 2024

Our advanced generative and predictive artificial intelligence (AI) tools enable us to search the vast space of possible drug molecules faster and more effectively. The architecture deploys a simple service in a Kubernetes pod within an EKS cluster. The following diagram illustrates the solution architecture.

Clustering 125

Clustering AI AI AWS 125

Pickl AI

JANUARY 12, 2025

Besides, there is a balance between the precision of traditional data analysis and the innovative potential of explainable artificial intelligence. Machine learning allows an explainable artificial intelligence system to learn and change to achieve improved performance in highly dynamic and complex settings.

Artificial Intelligence 52

Artificial Intelligence Artificial Intelligence AI AI 52

AWS Machine Learning Blog

MAY 23, 2024

By distributing experts across workers, expert parallelism addresses the high memory requirements of loading all experts on a single device and enables MoE training on a larger cluster. The following figure offers a simplified look at how expert parallelism works on a multi-GPU cluster.

Clustering 131

Clustering AWS Deep Learning Deep Learning 131

AWS Machine Learning Blog

OCTOBER 2, 2024

In this post, we share AWS guidance that we have learned and developed as part of real-world projects into practical guides oriented towards the AWS Well-Architected Framework , which is used to build production infrastructure and applications on AWS. To learn more, see Log Amazon Bedrock API calls using AWS CloudTrail.

AWS 124

AWS AI AI ML 124

Pickl AI

JULY 12, 2024

Summary: This guide explores Artificial Intelligence Using Python, from essential libraries like NumPy and Pandas to advanced techniques in machine learning and deep learning. It equips you to build and deploy intelligent systems confidently and efficiently.

Artificial Intelligence 52

Artificial Intelligence Artificial Intelligence Python Natural Language Processing 52