Hacker News

NOVEMBER 13, 2023

Benchmarks & Tips for Big Data, Hadoop, AWS, Google Cloud, PostgreSQL, Spark, Python & More.

Hadoop 178

Hadoop Big Data Big Data AWS 178

Hacker News

NOVEMBER 22, 2024

Benchmarks & Tips for Big Data, Hadoop, AWS, Google Cloud, PostgreSQL, Spark, Python & More.

Hadoop 160

Hadoop Big Data Big Data AWS 160

Hacker News

SEPTEMBER 25, 2024

Benchmarks & Tips for Big Data, Hadoop, AWS, Google Cloud, PostgreSQL, Spark, Python & More.

Hadoop 156

Hadoop Big Data Big Data AWS 156

Hacker News

DECEMBER 1, 2024

Benchmarks & Tips for Big Data, Hadoop, AWS, Google Cloud, PostgreSQL, Spark, Python & More.

Hadoop 140

Hadoop Big Data Big Data AWS 140

Hacker News

SEPTEMBER 15, 2024

Benchmarks & Tips for Big Data, Hadoop, AWS, Google Cloud, PostgreSQL, Spark, Python & More.

Hadoop 133

Hadoop Big Data Big Data AWS 133

Hacker News

NOVEMBER 2, 2024

Benchmarks & Tips for Big Data, Hadoop, AWS, Google Cloud, PostgreSQL, Spark, Python & More.

Hadoop 128

Hadoop Big Data Big Data AWS 128

Hacker News

MARCH 12, 2025

Benchmarks & Tips for Big Data, Hadoop, AWS, Google Cloud, PostgreSQL, Spark, Python & More.

Hadoop 109

Hadoop Big Data Big Data AWS 109

Hacker News

MARCH 15, 2024

Benchmarks & Tips for Big Data, Hadoop, AWS, Google Cloud, PostgreSQL, Spark, Python & More.

Hadoop 132

Hadoop Big Data Big Data AWS 132

Data Science Dojo

JULY 6, 2023

Amazon Redshift: Amazon Redshift is a cloud-based data warehousing service provided by Amazon Web Services (AWS). It integrates seamlessly with other AWS services and supports various data integration and transformation workflows. Apache Hadoop An open-source framework for distributed storage and processing of large datasets.

Data Engineering 195

Data Engineering Data Engineering Data Engineering Data Engineer 195

Data Science Dojo

OCTOBER 31, 2024

Additionally, knowledge of cloud platforms (AWS, Google Cloud) and experience with deployment tools (Docker, Kubernetes) are highly valuable. Programming Questions Data science roles typically require knowledge of Python, SQL, R, or Hadoop. Prepare to discuss your experience and problem-solving abilities with these languages.

Data Science 195

Data Science Data Scientist Machine Learning Machine Learning 195

Smart Data Collective

SEPTEMBER 8, 2021

Extract : In this step, data is extracted from a vast array of sources present in different formats such as Flat Files, Hadoop Files, XML, JSON, etc. Here are few best Open-Source ETL tools on the market: Hadoop : Hadoop distinguishes itself as a general-purpose Distributed Computing platform.

ETL 126

ETL Hadoop Data Warehouse Data Pipeline 126

Data Science 101

MARCH 7, 2020

Azure HDInsight now supports Apache analytics projects This announcement includes Spark, Hadoop, and Kafka. AWS DeepRacer 2020 Season is underway This looks to be a fun project. The service is awesome but used to be a bit spendy to try out. The frameworks in Azure will now have better security, performance, and monitoring.

Cloud Data 118

Cloud Data Data Science Azure Hadoop 118

phData

APRIL 26, 2024

One common scenario that we’ve helped many clients with involves migrating data from Hive tables in a Hadoop environment to the Snowflake Data Cloud. You can easily set an EMR cluster on an AWS account using the following simple steps: Sign in to AWS Management Console and navigate to the EMR service. ap-southeast-2.compute.amazonaws.com

Hadoop 52

Hadoop Clustering AWS Database 52

Flipboard

NOVEMBER 17, 2023

Specify the AWS Lambda function that will interact with MongoDB Atlas and the LLM to provide responses. As always, AWS welcomes feedback. About the authors Igor Alekseev is a Senior Partner Solution Architect at AWS in Data and Analytics domain. Choose Build and after the build is successful, choose Test.

K-nearest Neighbors 170

K-nearest Neighbors AWS Clustering Database 170

AWS Machine Learning Blog

MAY 16, 2024

The main AWS services used are SageMaker, Amazon EMR , AWS CodeBuild , Amazon Simple Storage Service (Amazon S3), Amazon EventBridge , AWS Lambda , and Amazon API Gateway. With Amazon EMR, which provides fully managed environments like Apache Hadoop and Spark, we were able to process data faster.

AWS 122

AWS ML ML Deep Learning 122

AWS Machine Learning Blog

APRIL 19, 2024

We used AWS services including Amazon Bedrock , Amazon SageMaker , and Amazon OpenSearch Serverless in this solution. In this series, we use the slide deck Train and deploy Stable Diffusion using AWS Trainium & AWS Inferentia from the AWS Summit in Toronto, June 2023 to demonstrate the solution. I need numbers."

AWS 126

AWS ML ML Database 126

AWS Machine Learning Blog

FEBRUARY 13, 2024

Model training was accelerated by 50% through the use of the SMDDP library, which includes optimized communication algorithms designed specifically for AWS infrastructure. For SageMaker distributed training, the instances need to be in the same AWS Region and Availability Zone. days in AWS vs. 9 days on their legacy platform).

AWS 128

AWS AI AI ML 128

Data Science Connect

JANUARY 27, 2023

Familiarize yourself with essential data technologies: Data engineers often work with large, complex data sets, and it’s important to be familiar with technologies like Hadoop, Spark, and Hive that can help you process and analyze this data.

Data Engineering 130

Data Engineering Data Engineering Data Engineering Data Engineer 130

Data Science Blog

MAY 15, 2023

Spark ist direkt auf mehreren Cloud-Plattformen verfügbar, darunter AWS, Azure und Google Cloud Platform.Apacke Spark ist jedoch mehr als nur ein Tool, es ist die Grundbasis für die meisten anderen Tools. Delta Lake baut auf Apache Spark auf und ist auf mehreren Cloud-Plattformen verfügbar, darunter AWS, Azure und Google Cloud Platform.

Data Warehouse 147

Data Warehouse Data Lakes Azure AWS 147

Data Science Dojo

JULY 24, 2023

Hadoop Distributed File System (HDFS) : HDFS is a distributed file system designed to store vast amounts of data across multiple nodes in a Hadoop cluster. Amazon S3: Amazon Simple Storage Service (S3) is a scalable object storage service provided by Amazon Web Services (AWS).

Big Data 195

Big Data Big Data Data Engineering Data Engineering 195

AWS Machine Learning Blog

DECEMBER 18, 2023

Try out MongoDB Atlas Try out MongoDB Atlas Time Series Try out Amazon SageMaker Canvas Try out MongoDB Charts About the authors Igor Alekseev is a Senior Partner Solution Architect at AWS in Data and Analytics domain. In his role Igor is working with strategic partners helping them build complex, AWS-optimized architectures.

Clustering 123

Clustering AWS Database ML 123

phData

MARCH 7, 2023

#The S3 Event Handler #TODO: Edit the AWS region #gg.eventhandler.s3.region= region= #TODO: Edit the AWS S3 bucket #gg.eventhandler.s3.bucketMappingTemplate= bucketMappingTemplate= #TODO:Set the classpath to include AWS Java SDK and Snowflake JDBC driver. jar #TODO:Set the AWS access key and secret key. gg.classpath=./snowflake-jdbc-3.13.7.jar:hadoop-3.2.1/share/hadoop/common/*:hadoop-3.2.1/share/hadoop/common/lib/*:hadoop-3.2.1/share/hadoop/hdfs/*:hadoop-3.2.1/share/

Hadoop 59

Hadoop Database Data Warehouse AWS 59

O'Reilly Media

SEPTEMBER 15, 2021

Cloud certifications, specifically in AWS and Microsoft Azure, were most strongly associated with salary increases. As we’ll see later, cloud certifications (specifically in AWS and Microsoft Azure) were the most popular and appeared to have the largest effect on salaries. The top certification was for AWS (3.9%

AI 145

AI AI Azure AWS 145

Analytics Vidhya

MARCH 8, 2023

The Biggest Data Science Blogathon is now live! Knowledge is power. Sharing knowledge is the key to unlocking that power.”― Martin Uzochukwu Ugwu Analytics Vidhya is back with the largest data-sharing knowledge competition- The Data Science Blogathon.

Data Science 345

Data Science Analytics Analytics Apache Hadoop 345

Analytics Vidhya

JANUARY 2, 2023

Introduction You must have noticed the personalization happening in the digital world, from personalized Youtube videos to canny ad recommendations on Instagram. While not all of us are tech enthusiasts, we all have a fair knowledge of how Data Science works in our day-to-day lives. All of this is based on Data Science which is […].

Data Engineering 329

Data Engineering Data Engineering Data Engineering Data Engineer 329

Analytics Vidhya

JANUARY 8, 2023

Hey, are you the data science geek who spends hours coding, learning a new language, or just exploring new avenues of data science? If all of these describe you, then this Blogathon announcement is for you! Analytics Vidhya is back with its 28th Edition of blogathon, a place where you can share your knowledge about […].

Data Science 328

Data Science Analytics Analytics Hadoop 328

Analytics Vidhya

NOVEMBER 7, 2022

Hello, fellow data science enthusiasts, did you miss imparting your knowledge in the previous blogathon due to a time crunch? Well, it’s okay because we are back with another blogathon where you can share your wisdom on numerous data science topics and connect with the community of fellow enthusiasts.

Data Science 306

Data Science Analytics Analytics Hadoop 306

Precisely

MARCH 9, 2023

Hadoop, Snowflake, Databricks and other products have rapidly gained adoption. We will also address some of the key distinctions between platforms like Hadoop and Snowflake, which have emerged as valuable tools in the quest to process and analyze ever larger volumes of structured, semi-structured, and unstructured data.

Data Warehouse 59

Data Warehouse Data Lakes Hadoop Big Data 59

Pickl AI

JANUARY 30, 2025

Hadoop MapReduce, Amazon EMR, and Spark integration offer flexible deployment and scalability. In this section, well focus on three prominent solutions: Hadoop MapReduce, Amazon EMR, and the integration of Apache Spark. Hadoop MapReduce Hadoop MapReduce is the cornerstone of the Hadoop ecosystem.

Big Data 52

Big Data Big Data Hadoop AWS 52

Pickl AI

DECEMBER 25, 2024

Big data platforms such as Apache Hadoop and Spark help handle massive datasets efficiently. They must also stay updated on tools such as TensorFlow, Hadoop, and cloud-based platforms like AWS or Azure. Programming languages like Python and R are commonly used for data manipulation, visualization, and statistical modeling.

Data Science 52

Data Science Analytics Analytics Data Scientist 52

Smart Data Collective

FEBRUARY 14, 2021

For example, if you want to sell on AWS marketplace , you will need to see what they expect from you. You need to use Hadoop tools to mine this data and find out more about your target customers and product requirements. Preparing your product will take lots of effort and planning, but it will also increase your chances of success.

Big Data 108

Big Data Big Data Deep Learning Deep Learning 108

Smart Data Collective

APRIL 24, 2023

Google’s Hadoop allowed for unlimited data storage on inexpensive servers, which we now call the Cloud. Searching for a topic on a search engine can provide us with a vast amount of information in seconds. Deighton studies how this evolution came to be. Innovations in the early 20th century changed how data could be used.

Big Data 119

Big Data Big Data Hadoop Azure 119

Data Science Dojo

JULY 3, 2024

Big Data Technologies : Handling and processing large datasets using tools like Hadoop, Spark, and cloud platforms such as AWS and Google Cloud. Cloud Computing : Utilizing cloud services for data storage and processing, often covering platforms such as AWS, Azure, and Google Cloud.

Data Science 150

Data Science Machine Learning Machine Learning Data Visualization 150

Smart Data Collective

MARCH 21, 2022

Most recently, JP Morgan built a ‘Mesh’ on AWS and locked its scalability fortune on a decentralized architecture. The Hadoop library enabled distributed processing across all points of data storage. More case studies are added every day and give a clear hint – data analytics are all set to change, again!

Data Lakes 75

Data Lakes Hadoop Data Silos Data Warehouse 75

phData

NOVEMBER 8, 2024

Evolution of Open Table Formats Here’s a timeline that outlines the key moments in the evolution of open table formats: 2008 - Apache Hive and Hive Table Format Facebook introduced Apache Hive as one of the first table formats as part of its data warehousing infrastructure, built on top of Hadoop.

Data Lakes 52

Data Lakes Data Warehouse Database Azure 52

Women in Big Data

OCTOBER 9, 2024

Data Ingestion: Data is collected and funneled into the pipeline using batch or real-time methods, leveraging tools like Apache Kafka, AWS Kinesis, or custom ETL scripts. This phase ensures quality and consistency using frameworks like Apache Spark or AWS Glue.

Big Data 59

Big Data Big Data Apache Kafka Data Pipeline 59

Precisely

JANUARY 19, 2023

Platforms like Hadoop and Spark prompted many companies to begin thinking about big data differently than they had in the past. With the emergence of cloud hyperscalers like AWS, Google, and Microsoft, the shift to the cloud has accelerated significantly. Mainframes have long been valued for those very same attributes.

AWS 52

AWS Cloud Computing Data Pipeline Big Data 52