Hacker News

JUNE 12, 2024

Meta is currently operating many data centers with GPU training clusters across the world. A year ago, however, as the industry reached a critical inflection point due to the rise of artificial intelligence (AI), we recognized that to lead in the generative AI space we’d need to transform our fleet.

Clustering 179

Clustering AI AI Artificial Intelligence 179

AWS Machine Learning Blog

JULY 25, 2024

By accelerating the speed of issue detection and remediation, it increases the reliability of your ML training and reduces the wasted time and cost due to hardware failure. Choose Clusters in the navigation pane, open the trainium-inferentia cluster, choose Node groups, and locate your node group. # install.sh

Clustering 109

Clustering AWS ML ML 109

AWS Machine Learning Blog

SEPTEMBER 18, 2024

The compute clusters used in these scenarios are composed of more than thousands of AI accelerators such as GPUs or AWS Trainium and AWS Inferentia , custom machine learning (ML) chips designed by Amazon Web Services (AWS) to accelerate deep learning workloads in the cloud.

Clustering 107

Clustering AWS ML ML 107

AWS Machine Learning Blog

NOVEMBER 19, 2024

At the time, I knew little about AI or machine learning (ML). But AWS DeepRacer instantly captured my interest with its promise that even inexperienced developers could get involved in AI and ML. Panic set in as we realized we would be competing on stage in front of thousands of people while knowing little about ML.

AWS 104

AWS ML ML AI 104

Hacker News

MARCH 12, 2024

Marking a major investment in Meta’s AI future, we are announcing two 24k GPU clusters. We are sharing details on the hardware, network, storage, design, performance, and software that help us extract high throughput and reliability for various AI workloads. We use this cluster design for Llama 3 training.

Clustering 182

Clustering AI AI ML 182

Dataconomy

MAY 22, 2023

TensorFlow provides high-level APIs, such as tf.distribute, to distribute training across multiple devices, machines, or clusters. PyTorch: PyTorch , developed by Facebook’s AI Research lab, is another popular distributed learning framework.

ML 233

ML ML Machine Learning Machine Learning 233

AWS Machine Learning Blog

APRIL 22, 2024

Amazon SageMaker HyperPod is purpose-built to accelerate foundation model (FM) training, removing the undifferentiated heavy lifting involved in managing and optimizing a large training compute cluster. In this solution, HyperPod cluster instances use the LDAPS protocol to connect to the AWS Managed Microsoft AD via an NLB.

Clustering 106

Clustering AWS Machine Learning Machine Learning 106

Hacker News

JANUARY 25, 2024

He writes about ML/AI/crypto/data, leadership, and building tech teams. Adam Drake is an advisor to scale-up tech companies.

Hadoop 120

Hadoop Clustering ML ML 120

AWS Machine Learning Blog

APRIL 4, 2023

AWS provides various services catered to time series data that are low code/no code, which both machine learning (ML) and non-ML practitioners can use for building ML solutions. We use the Time Series Clustering using TSFresh + KMeans notebook, which is available on our GitHub repo.

Clustering 90

Clustering ML ML AWS 90

Hacker News

AUGUST 5, 2024

AI networks play an important role in interconnecting tens of thousands of GPUs together, forming the foundational infrastructure for training, enabling large models with hundreds of billions of parameters such as LLAMA 3.1 The growing prevalence of AI has introduced a new era of communication demands.

AI 175

AI AI Clustering Natural Language Processing 175

Hacker News

OCTOBER 15, 2024

At Open Compute Project Summit (OCP) 2024, we’re sharing details about our next-generation network fabric for our AI training clusters. Now, through OCP, we’re bringing new open advanced network technologies to our data centers, and the wider industry, for advanced AI applications.

Clustering 172

Clustering AI AI ML 172

AWS Machine Learning Blog

NOVEMBER 19, 2024

This year, generative AI and machine learning (ML) will again be in focus, with exciting keynote announcements and a variety of sessions showcasing insights from AWS experts, customer stories, and hands-on experiences with AWS services. Fifth, we’ll showcase various generative AI use cases across industries.

AWS 89

AWS ML ML AI 89

AWS Machine Learning Blog

FEBRUARY 12, 2024

Sharing in-house resources with other internal teams, the Ranking team machine learning (ML) scientists often encountered long wait times to access resources for model training and experimentation – challenging their ability to rapidly experiment and innovate. If it shows online improvement, it can be deployed to all the users.

ML 126

ML ML AWS Machine Learning 126

ML @ CMU

NOVEMBER 7, 2024

In close collaboration with the UN and local NGOs, we co-develop an interpretable predictive tool for landmine contamination to identify hazardous clusters under geographic and budget constraints, experimentally reducing false alarms and clearance time by half. RELand consistently outperforms the benchmark models on all relevant metrics.

Clustering 153

Clustering Cross Validation Machine Learning Machine Learning 153

Flipboard

AUGUST 17, 2023

Many practitioners are extending these Redshift datasets at scale for machine learning (ML) using Amazon SageMaker , a fully managed ML service, with requirements to develop features offline in a code way or low-code/no-code way, store featured data from Amazon Redshift, and make this happen at scale in a production environment.

ML 123

ML ML AWS Data Warehouse 123

Data Science Dojo

FEBRUARY 20, 2023

Machine learning (ML) is the technology that automates tasks and provides insights. It comes in many forms, with a range of tools and platforms designed to make working with ML more efficient. It provides a large cluster of clusters on a single machine. It is also useful for training models on smaller datasets.

Machine Learning 195

Machine Learning Machine Learning AWS Azure 195

ODSC - Open Data Science

FEBRUARY 23, 2023

5 Must-Know Pillars of a Data Science and AI Foundation A data science and AI foundation needs to be built up properly before diving in head-first. By knowing these core skills, like math and AI literacy, you’ll start off your career on a high note. How can AI be used to help with therapy on-demand for those who need help?

Clustering 98

Clustering Data Science Machine Learning Machine Learning 98

Google Research AI blog

MAY 25, 2023

Posted by Vincent Cohen-Addad and Alessandro Epasto, Research Scientists, Google Research, Graph Mining team Clustering is a central problem in unsupervised machine learning (ML) with many applications across domains in both industry and academic research more broadly. When clustering is applied to personal data (e.g.,

Clustering 89

Clustering Algorithm Machine Learning Machine Learning 89

Data Science Dojo

JULY 17, 2023

In recent years, there has been a growing interest in the use of artificial intelligence (AI) for data analysis. AI tools can automate many of the tasks involved in data analysis, and they can also help businesses to discover new insights from their data. Top 10 AI tools for data analysis AI Tools for Data Analysis 1.

Data Analysis 195

Data Analysis Data Analysis Tableau Machine Learning 195

AWS Machine Learning Blog

APRIL 19, 2024

Iambic Therapeutics is a drug discovery startup with a mission to create innovative AI-driven technologies to bring better medicines to cancer patients, faster. Our advanced generative and predictive artificial intelligence (AI) tools enable us to search the vast space of possible drug molecules faster and more effectively.

Clustering 119

Clustering AI AI AWS 119

AWS Machine Learning Blog

JUNE 25, 2024

This solution simplifies the integration of advanced monitoring tools such as Prometheus and Grafana, enabling you to set up and manage your machine learning (ML) workflows with AWS AI Chips. By deploying the Neuron Monitor DaemonSet across EKS nodes, developers can collect and analyze performance metrics from ML workload pods.

AWS 86

AWS ML ML Clustering 86

Smart Data Collective

NOVEMBER 1, 2020

Machines, artificial intelligence (AI), and unsupervised learning are reshaping the way businesses vie for a place under the sun. Unsupervised ML: The Basics. Unlike supervised ML, we do not manage the unsupervised model. Unsupervised ML uses algorithms that draw conclusions on unlabeled datasets.

Machine Learning 138

Machine Learning Machine Learning Clustering Data Mining 138

Towards AI

FEBRUARY 21, 2024

Author(s): Alessandro Amenta Originally published on Towards AI. Image generated with DALL-E 3 In the fast-paced world of Machine Learning (ML) research, keeping up with the latest findings is crucial and exciting, but let’s be honest — it’s also a challenge. What’s the next big thing in ML?

ML 93

ML ML Machine Learning Machine Learning 93

Flipboard

NOVEMBER 17, 2023

Generative AI models have the potential to revolutionize enterprise operations, but businesses must carefully consider how to harness their power while overcoming challenges such as safeguarding data and ensuring the quality of AI-generated content. This type of data is often used in ML and artificial intelligence applications.

K-nearest Neighbors 170

K-nearest Neighbors AWS Clustering Database 170

Snorkel AI

APRIL 27, 2023

Here at Snorkel AI, we devote our time to building and maintaining our machine-learning development platform, Snorkel Flow. Spark, Dask, and any other workflow executors used for experimentation can grow along with the size of the cluster. Reproducibility is important in ML. Reproducibility is important in ML.

Machine Learning 97

Machine Learning Machine Learning Clustering ML 97

Snorkel AI

APRIL 27, 2023

Here at Snorkel AI, we devote our time to building and maintaining our machine-learning development platform, Snorkel Flow. Spark, Dask, and any other workflow executors used for experimentation can grow along with the size of the cluster. Reproducibility is important in ML. Reproducibility is important in ML.

Machine Learning 96

Machine Learning Machine Learning Clustering ML 96

AWS Machine Learning Blog

OCTOBER 16, 2024

Amazon SageMaker supports geospatial machine learning (ML) capabilities, allowing data scientists and ML engineers to build, train, and deploy ML models using geospatial data. We use the purpose-built geospatial container with SageMaker Processing jobs for a simplified, managed experience to create and run a cluster.

ML 103

ML ML Clustering Machine Learning 103

AWS Machine Learning Blog

APRIL 1, 2024

Machine learning (ML) research has proven that large language models (LLMs) trained with significantly large datasets result in better model quality. Distributed model training requires a cluster of worker nodes that can scale. The following figure shows how FSDP works for two data parallel processes.

Clustering 122

Clustering AWS ML ML 122

Flipboard

JULY 10, 2023

Over time, many enterprises have built an on-premises cluster of servers, accumulating data, and then procuring more servers and storage. They often …

ML 94

ML ML Clustering AI 94

Data Science Dojo

MAY 10, 2024

Their impact on ML tasks has made them a cornerstone of AI advancements. It allows ML models to work with data but in a limited manner. SOMs work to bring down the information into a 2-dimensional map where similar data points form clusters, providing a starting point for advanced embeddings.

Supervised Learning 259

Supervised Learning Clustering ML ML 259

AWS Machine Learning Blog

OCTOBER 24, 2024

Machine learning (ML) helps organizations to increase revenue, drive business growth, and reduce costs by optimizing core business functions such as supply and demand forecasting, customer churn prediction, credit risk scoring, pricing, predicting late shipments, and many others. For this post we’ll use a provisioned Amazon Redshift cluster.

Data Warehouse 106

Data Warehouse Machine Learning Machine Learning Cloud Data 106

AWS Machine Learning Blog

DECEMBER 4, 2024

Businesses are under pressure to show return on investment (ROI) from AI use cases, whether predictive machine learning (ML) or generative AI. Only 54% of ML prototypes make it to production, and only 5% of generative AI use cases make it to production. Using SageMaker, you can build, train and deploy ML models.

ML 83

ML ML AWS AI 83

phData

JANUARY 30, 2024

Snowpark ML is transforming the way that organizations implement AI solutions. Snowpark allows ML models and code to run on Snowflake warehouses. By “bringing the code to the data,” we’ve seen ML applications run anywhere from 4-100x faster than other architectures.

ML 98

ML ML Python Database 98

Towards AI

JULY 17, 2023

Last Updated on July 18, 2023 by Editorial Team Author(s): Muttineni Sai Rohith Originally published on Towards AI. Pyspark MLlib | Classification using Pyspark ML In the previous sections, we discussed about RDD, Dataframes, and Pyspark concepts. In this article, we will discuss about Pyspark MLlib and Spark ML.

ML 74

ML ML Decision Trees Machine Learning 74

Flipboard

JUNE 20, 2023

Machine learning (ML) engineers have traditionally focused on striking a balance between model training and deployment cost vs. performance. This is important because training ML models and then using the trained models to make predictions (inference) can be highly energy-intensive tasks.

AWS 123

AWS Machine Learning Machine Learning Deep Learning 123

AWS Machine Learning Blog

DECEMBER 24, 2024

To reduce costs while continuing to use the power of AI , many companies have shifted to fine tuning LLMs on their domain-specific data using Parameter-Efficient Fine Tuning (PEFT). Manually managing such complexity can often be counter-productive and take away valuable resources from your businesses AI development.

AWS 97

AWS Clustering Deep Learning Deep Learning 97

IBM Journey to AI blog

DECEMBER 20, 2023

Machine learning (ML) technologies can drive decision-making in virtually all industries, from healthcare to human resources to finance and in myriad use cases, like computer vision , large language models (LLMs), speech recognition, self-driving cars and more. However, the growing influence of ML isn’t without complications.

Machine Learning 92

Machine Learning Machine Learning Supervised Learning Clustering 92