### Example EMR Cluster Creation and Output Comparison (Bash) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This example demonstrates how to define two EMR cluster creation commands (emr-cli-1 and emr-cli-2) with different release labels for upgrade comparison. It also includes SQL statements to create external tables for cluster outputs and a final query to compare these outputs and calculate differences. ```bash #subscriber-email example@example.com #emr-cli-1 aws emr create-cluster \ --applications Name=Spark Name=Hadoop Name=Hive \ --release-label emr-5.0.0 \ ... #emr-cli-2 aws emr create-cluster \ --applications Name=Spark Name=Hadoop Name=Hive \ --release-label emr-5.24.1 \ ... #table1 CREATE EXTERNAL TABLE IF NOT EXISTS table1 ( label STRING, prediction STRING ) ... #table2 CREATE EXTERNAL TABLE IF NOT EXISTS table2 ( label STRING, prediction STRING ) ... #comparing-table CREATE TABLE IF NOT EXISTS result AS SELECT A.label, A.prediction as predictionA, B.prediction as predictionB FROM table1 A JOIN table2 B ON (A.label = B.label); #result SELECT count(*) as total_count, (SELECT count(*) FROM comp.result WHERE predictionA != predictionB) as diff_count, (SELECT count(*) FROM comp.result WHERE predictionA != predictionB)/CAST(count(*) as double) as ratio FROM result; ``` -------------------------------- ### SAM CLI Build, Package, and Deploy Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md These commands use the AWS Serverless Application Model (SAM) CLI to build, package, and deploy the EMR Basketball Tool application. This process requires SAM CLI, AWS CLI, Docker, and Python 3 to be installed. ```bash cd aws-emr-basketball-tool/ NotificationBucket='basketball-tool-bucket-20190821' sam build sam package \ --template-file .aws-sam/build/template.yaml \ --output-template-file packaged.yaml \ --s3-bucket baseket-ball-cfn-$Date sam deploy \ --template-file packaged.yaml \ --stack-name BasketBallTool \ --capabilities CAPABILITY_IAM \ --region ap-northeast-2 \ --parameter-overrides NotificationBucket=$NotificationBucket ``` -------------------------------- ### Clone Repository using Git Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This command clones the AWS EMR Basketball Tool repository from GitHub to your local machine. It requires Git to be installed. ```bash git clone git@github.com:aws-samples/aws-emr-basketball-tool.git ``` -------------------------------- ### Main Lambda Function Code for Starting EMR Jobs (Python) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This Python script, 'shoot.py', is the main entry point for the Lambda function. It orchestrates the creation and initiation of the two EMR jobs (job1 and job2) that will be compared. It reads the request details and triggers the EMR cluster creation. ```python # src/shoot.py # Main Lambda function code for starting job1 and job2 of EMR ``` -------------------------------- ### Python Dependencies (Text) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This file lists the Python dependencies required for the Lambda functions. It is used by tools like pip and SAM CLI to install the necessary packages during the build process. ```text # src/requirements.txt # requirements.txt ``` -------------------------------- ### Build, Package, and Deploy EMR Basketball Tool with SAM CLI Source: https://context7.com/aws-samples/aws-emr-basketball-tool/llms.txt Complete bash workflow for cloning the repository, creating S3 bucket for CloudFormation artifacts, building and packaging the SAM application, deploying the stack with parameters, and updating Lambda function dependencies. Includes date-stamped bucket naming and region-specific deployment. ```bash # Clone repository git clone git@github.com:aws-samples/aws-emr-basketball-tool.git cd aws-emr-basketball-tool/ # Create S3 bucket for CloudFormation Date=$(date +'%d%m%Y-%H%M%S') aws s3 mb s3://baseket-ball-cfn-$Date # Build, package, and deploy with SAM NotificationBucket='basketball-tool-bucket-20190821' sam build sam package \ --template-file .aws-sam/build/template.yaml \ --output-template-file packaged.yaml \ --s3-bucket baseket-ball-cfn-$Date sam deploy \ --template-file packaged.yaml \ --stack-name BasketBallTool \ --capabilities CAPABILITY_IAM \ --region ap-northeast-2 \ --parameter-overrides NotificationBucket=$NotificationBucket # Update the Shoot Lambda function with dependencies cd src/ ./update-shoot-func.sh BasketBallTool-Shoot-XXXXXXXXXXXX ``` -------------------------------- ### Deploy Serverless EMR Comparison Stack with AWS SAM Source: https://context7.com/aws-samples/aws-emr-basketball-tool/llms.txt AWS SAM CloudFormation template defining a serverless application stack with Lambda function triggered by S3 events, IAM roles with EMR and S3 permissions, and S3 bucket configuration. The template automates EMR cluster creation and comparison workflow initiation when request files are uploaded. ```yaml AWSTemplateFormatVersion: '2010-09-09' Transform: 'AWS::Serverless-2016-10-31' Description: checking differences of outputs when upgrading EMR. Parameters: NotificationBucket: Type: String Description: S3 bucket which is used for Lambda event notification Resources: Shoot: Type: 'AWS::Serverless::Function' Properties: Handler: shoot.lambda_handler Runtime: python3.7 CodeUri: ./src/ MemorySize: 256 Timeout: 300 Role: !GetAtt LambdaEmrCreationRole.Arn Events: BucketEvent1: Type: S3 Properties: Bucket: !Ref Bucket1 Events: - 's3:ObjectCreated:Put' - 's3:ObjectCreated:Post' Filter: S3Key: Rules: - Name: prefix Value: request - Name: suffix Value: request.txt Bucket1: Type: AWS::S3::Bucket Properties: BucketName: !Ref NotificationBucket LambdaEmrCreationRole: Type: 'AWS::IAM::Role' Properties: ManagedPolicyArns: - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole" - "arn:aws:iam::aws:policy/AmazonS3FullAccess" - "arn:aws:iam::aws:policy/AmazonElasticMapReduceFullAccess" AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Action: "sts:AssumeRole" Principal: Service: "lambda.amazonaws.com" ``` -------------------------------- ### Configure EMR Cluster Comparison Request File Source: https://context7.com/aws-samples/aws-emr-basketball-tool/llms.txt Request configuration file defining subscriber email, EMR cluster creation commands for two different EMR versions, and Athena SQL queries to compare prediction outputs. The file specifies cluster configurations with Spark, Hadoop, and Hive applications, instance groups, and automated job execution with output comparison logic. ```bash #subscriber-email user@example.com #emr-cli-1 aws emr create-cluster \ --applications Name=Spark Name=Hadoop Name=Hive \ --release-label emr-5.0.0 \ --ec2-attributes '{"KeyName":"mykey","InstanceProfile":"EMR_EC2_DefaultRole","SubnetId":"subnet-xxxxx"}' \ --instance-groups '[{"InstanceCount":1,"InstanceGroupType":"MASTER","InstanceType":"r3.xlarge","Name":"Master"},{"InstanceCount":3,"InstanceGroupType":"CORE","InstanceType":"m4.large","Name":"Core"}]' \ --steps '[{"Args":["s3://bucket/script.sh","app.jar","s3://bucket/output1"],"Type":"CUSTOM_JAR","Jar":"s3://region.elasticmapreduce/libs/script-runner/script-runner.jar","Name":"job"}]' \ --service-role EMR_DefaultRole \ --region us-east-1 \ --auto-terminate #emr-cli-2 aws emr create-cluster \ --applications Name=Spark Name=Hadoop Name=Hive \ --release-label emr-5.24.1 \ --ec2-attributes '{"KeyName":"mykey","InstanceProfile":"EMR_EC2_DefaultRole","SubnetId":"subnet-xxxxx"}' \ --instance-groups '[{"InstanceCount":1,"InstanceGroupType":"MASTER","InstanceType":"r3.xlarge","Name":"Master"},{"InstanceCount":3,"InstanceGroupType":"CORE","InstanceType":"m4.large","Name":"Core"}]' \ --steps '[{"Args":["s3://bucket/script.sh","app.jar","s3://bucket/output2"],"Type":"CUSTOM_JAR","Jar":"s3://region.elasticmapreduce/libs/script-runner/script-runner.jar","Name":"job"}]' \ --service-role EMR_DefaultRole \ --region us-east-1 \ --auto-terminate ``` -------------------------------- ### SAM Template for Serverless Application (YAML) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This YAML file defines the AWS resources for the serverless application using the AWS Serverless Application Model (SAM). It specifies the Lambda functions, API Gateway (if used), and other necessary infrastructure for the EMR Basketball Tool. ```yaml # template.yaml # SAM Template ``` -------------------------------- ### Create EMR Cluster with Spark, Hadoop, Hive (EMR 5.0.0) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/request.txt This AWS CLI command creates an EMR cluster with Spark, Hadoop, and Hive applications. It configures instance groups, logging, debugging, and defines custom steps for data statistics and clustering using Spark. ```bash aws emr create-cluster \ --applications Name=Spark Name=Hadoop Name=Hive \ --tags 'Name=TEST-EMR' \ --ec2-attributes '{"KeyName":"ec2key","InstanceProfile":"EMR_EC2_DefaultRole","SubnetId":"subnet-60716608","EmrManagedSlaveSecurityGroup":"sg-03a5b0c01e1ff3c87","EmrManagedMasterSecurityGroup":"sg-0e10dc7742dc738ef"}' \ --release-label emr-5.0.0 \ --log-uri 's3n://aws-logs-522781160594-ap-northeast-2/elasticmapreduce/' \ --steps '[{"Args":["s3://public-access-sample-code/scripts/spark-runner.sh","MovieDataStatistics-1.0.0.jar","-d","20190821","s3://samples-euijj/ml-latest/movies.csv s3://samples-euijj/ml-latest/ratings.csv s3://samples-euijj/ml-latest/tags.csv s3://basketball-tool-bucket-20190821/output1"],"Type":"CUSTOM_JAR","ActionOnFailure":"CONTINUE","Jar":"s3://ap-northeast-2.elasticmapreduce/libs/script-runner/script-runner.jar","Properties":"","Name":"statistics data"},{"Args":["s3://public-access-sample-code/scripts/spark-runner.sh","MovieDataClustering-1.0.0.jar","-d","20190821","s3://basketball-tool-bucket-20190821/output1 s3://basketball-tool-bucket-20190821/predict1"],"Type":"CUSTOM_JAR","ActionOnFailure":"CONTINUE","Jar":"s3://ap-northeast-2.elasticmapreduce/libs/script-runner/script-runner.jar","Properties":"","Name":"clustering data"}]' \ --instance-groups '[{"InstanceCount":1,"InstanceGroupType":"MASTER","InstanceType":"r3.xlarge","Name":"Master - 1"},{"InstanceCount":3,"BidPrice":"OnDemandPrice","EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":32,"VolumeType":"gp2"},"VolumesPerInstance":1}]},"InstanceGroupType":"CORE","InstanceType":"m4.large","Name":"Core - 2"}]' \ --auto-scaling-role EMR_AutoScaling_DefaultRole \ --ebs-root-volume-size 10 \ --service-role EMR_DefaultRole \ --enable-debugging \ --name 'test-emr-20190821-spark20-test' \ --scale-down-behavior TERMINATE_AT_TASK_COMPLETION \ --region ap-northeast-2 \ --auto-terminate ``` -------------------------------- ### Create EMR Cluster with Specific Instance Types (Bash) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md These AWS CLI commands create an EMR cluster with specified application and instance group configurations. The first command uses 'm4.large' for core instances, while the second uses 'r4.large', illustrating a right-sizing scenario. ```bash #subscriber-email example@example.com #emr-cli-1 aws emr create-cluster \ --applications Name=Spark Name=Hadoop Name=Hive \ --instance-groups '[ { "InstanceCount":1, "InstanceGroupType":"MASTER", "InstanceType":"r3.xlarge", "Name":"Master" }, { "InstanceCount":3, "InstanceGroupType":"CORE", "InstanceType":"m4.large", "Name":"Core" } ]' ... #emr-cli-2 aws emr create-cluster \ --applications Name=Spark Name=Hadoop Name=Hive \ --instance-groups '[ { "InstanceCount":1, "InstanceGroupType":"MASTER", "InstanceType":"r3.xlarge", "Name":"Master" }, { "InstanceCount":3, "InstanceGroupType":"CORE", "InstanceType":"r4.large", "Name":"Core" } ]' ... #table1 ... ``` -------------------------------- ### Create S3 Bucket for CloudFormation Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This command creates a new S3 bucket to store CloudFormation artifacts during the deployment process. It includes a timestamp to ensure a unique bucket name. AWS CLI must be configured. ```bash Date=$(date +'%d%m%Y-%H%M%S') aws s3 mb s3://baseket-ball-cfn-$Date ``` -------------------------------- ### Upload Request File to S3 Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This command uploads the 'request.txt' file to a specified S3 bucket and path. This action triggers the main Lambda function ('BasketBall-Shoot') via an S3 POST Event, initiating the EMR cluster comparison process. ```bash aws s3 cp ./request.txt s3://$NotificationBucket/request/request.txt ``` -------------------------------- ### Python Lambda: Initiate EMR Cluster Comparison Workflow Source: https://context7.com/aws-samples/aws-emr-basketball-tool/llms.txt The main Lambda function, 'Shoot Function', handles the initiation of the EMR cluster comparison. It parses a request file from S3, extracts EMR CLI commands and subscriber email, creates two EMR clusters using these commands, and stores their IDs back to S3. Dependencies include 'boto3' for S3 interaction and 'subprocess' for executing shell commands. It takes an S3 event as input and returns a status code and message. ```python import json import boto3 import re from subprocess import Popen, PIPE def lambda_handler(event, context): ret = 200 try: BucketName = event['Records'][0]['s3']['bucket']['name'] # Parse request.txt from S3 bucket s3 = boto3.resource('s3') obj = s3.Object(BucketName, 'request/request.txt') body = obj.get()['Body'].read().decode('utf-8').replace('\n', '') # Extract EMR CLI commands and configuration emrCli1 = re.search(r'#emr-cli-1\s*([^\#]+)', body).group(1).replace('\\','') emrCli2 = re.search(r'#emr-cli-2\s*([^\#]+)', body).group(1).replace('\\','') subscriberEmail = re.search(r'#subscriber-email\s*([^\#]+)', body).group(1) # Execute both EMR cluster creation commands process = Popen("/var/task/" + emrCli1, shell=True, stdout=PIPE, stderr=PIPE) stdout1, stderr1 = process.communicate() process = Popen("/var/task/" + emrCli2, shell=True, stdout=PIPE, stderr=PIPE) stdout2, stderr2 = process.communicate() # Store cluster IDs for later reference s3.Object(BucketName, 'request/clusterid1').put(Body=stdout1) s3.Object(BucketName, 'request/clusterid2').put(Body=stdout2) return { 'statusCode': 200, 'body': json.dumps('Clusters created successfully') } except Exception as e: print(f"Error: {e}") return { 'statusCode': 500, 'body': json.dumps('Error creating clusters') } ``` -------------------------------- ### Create EMR Cluster with Spark, Hadoop, Hive (EMR 5.24.1) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/request.txt This AWS CLI command creates an EMR cluster using a newer release version (5.24.1) with Spark, Hadoop, and Hive. It's similar to the previous command but targets a different EMR release and output location for clustering data. ```bash aws emr create-cluster \ --applications Name=Spark Name=Hadoop Name=Hive \ --tags 'Name=TEST-EMR' \ --ec2-attributes '{"KeyName":"ec2key","InstanceProfile":"EMR_EC2_DefaultRole","SubnetId":"subnet-60716608","EmrManagedSlaveSecurityGroup":"sg-03a5b0c01e1ff3c87","EmrManagedMasterSecurityGroup":"sg-0e10dc7742dc738ef"}' \ --release-label emr-5.24.1 \ --log-uri 's3n://aws-logs-522781160594-ap-northeast-2/elasticmapreduce/' \ --steps '[{"Args":["s3://public-access-sample-code/scripts/spark-runner.sh","MovieDataStatistics-1.0.0.jar","-d","20190821","s3://samples-euijj/ml-latest/movies.csv s3://samples-euijj/ml-latest/ratings.csv s3://samples-euijj/ml-latest/tags.csv s3://basketball-tool-bucket-20190821/output2"],"Type":"CUSTOM_JAR","ActionOnFailure":"CONTINUE","Jar":"s3://ap-northeast-2.elasticmapreduce/libs/script-runner/script-runner.jar","Properties":"","Name":"statistics data"},{"Args":["s3://public-access-sample-code/scripts/spark-runner.sh","MovieDataClustering-1.0.0.jar","-d","20190821","s3://basketball-tool-bucket-20190821/output2 s3://basketball-tool-bucket-20190821/predict2"],"Type":"CUSTOM_JAR","ActionOnFailure":"CONTINUE","Jar":"s3://ap-northeast-2.elasticmapreduce/libs/script-runner/script-runner.jar","Properties":"","Name":"clustering data"}]' \ --instance-groups '[{"InstanceCount":1,"InstanceGroupType":"MASTER","InstanceType":"r3.xlarge","Name":"Master - 1"},{"InstanceCount":3,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":32,"VolumeType":"gp2"},"VolumesPerInstance":1}]},"InstanceGroupType":"CORE","InstanceType":"m4.large","Name":"Core - 2"}]' \ --auto-scaling-role EMR_AutoScaling_DefaultRole \ --ebs-root-volume-size 10 \ --service-role EMR_DefaultRole \ --enable-debugging \ --name 'test-emr-20190821-spark24-test' \ --scale-down-behavior TERMINATE_AT_TASK_COMPLETION \ --region ap-northeast-2 \ --auto-terminate ``` -------------------------------- ### Request File for EMR Job Trigger (Text) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This text file contains the necessary parameters to trigger the main Lambda function ('BasketBall-Shoot'). Uploading this file to a specific S3 location initiates the EMR cluster comparison process. ```text # request.txt # request.txt will trigger main Lambda function ``` -------------------------------- ### Execute Athena Queries for Data Comparison with Python Lambda Source: https://context7.com/aws-samples/aws-emr-basketball-tool/llms.txt Lambda function that reads SQL query definitions from an S3 request file, parses multiple query sections using regex, and executes them sequentially against Athena. Creates external tables from EMR outputs and performs data comparison, storing the final query execution ID back to S3 for result retrieval. Requires proper Athena database and S3 output location configuration. ```python import json import boto3 import re database = 'comp' def lambda_handler(event, context): ret = 200 athena = boto3.client('athena') try: BucketName = event['Records'][0]['s3']['bucket']['name'] s3 = boto3.resource('s3') obj = s3.Object(BucketName, 'request/request.txt') body = obj.get()['Body'].read().decode('utf-8').replace('\n', ' ') # Parse SQL queries from request table1 = re.search(r'#table1\s*([^\#]+)', body).group(1) table2 = re.search(r'#table2\s*([^\#]+)', body).group(1) comparingTable = re.search(r'#comparing-table\s*([^\#]+)', body).group(1) result = re.search(r'#result\s*([^\#]+)', body).group(1) s3Output = f's3://{BucketName}/athena_output/' resultOutput = f's3://{BucketName}/result/' # Execute Athena queries sequentially responseTable1 = athena.start_query_execution( QueryString=table1, QueryExecutionContext={'Database': database}, ResultConfiguration={'OutputLocation': s3Output} ) responseTable2 = athena.start_query_execution( QueryString=table2, QueryExecutionContext={'Database': database}, ResultConfiguration={'OutputLocation': s3Output} ) responseComparingTable = athena.start_query_execution( QueryString=comparingTable, QueryExecutionContext={'Database': database}, ResultConfiguration={'OutputLocation': s3Output} ) responseResult = athena.start_query_execution( QueryString=result, QueryExecutionContext={'Database': database}, ResultConfiguration={'OutputLocation': resultOutput} ) # Store query ID for result retrieval QueryExecutionId = responseResult['QueryExecutionId'] s3.Object(BucketName, 'request/queryid').put(Body=QueryExecutionId) return { 'statusCode': ret, 'body': json.dumps(f'Query ID: {QueryExecutionId}') } except Exception as e: print(f"Error: {e}") return { 'statusCode': 500, 'body': json.dumps('Error running comparison queries') } ``` -------------------------------- ### Create Hive External Tables for EMR Output Comparison Source: https://context7.com/aws-samples/aws-emr-basketball-tool/llms.txt Hive SQL commands to create external tables reading prediction results from S3 locations and a comparison table that joins outputs from two EMR runs. Tables are tab-delimited with label and prediction columns, enabling cross-version output analysis. ```sql CREATE EXTERNAL TABLE IF NOT EXISTS table1 ( label STRING, prediction STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LOCATION 's3://bucket/predict1' CREATE EXTERNAL TABLE IF NOT EXISTS table2 ( label STRING, prediction STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LOCATION 's3://bucket/predict2' CREATE TABLE IF NOT EXISTS result AS SELECT A.label, A.prediction as predictionA, B.prediction as predictionB FROM table1 A JOIN table2 B ON (A.label = B.label) ``` -------------------------------- ### Compare Predictions from Two Tables Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/request.txt This SQL query creates a new table named 'result' by joining 'table1' and 'table2' on the 'label' column. It selects the label and the predictions from both tables, aliasing them as 'predictionA' and 'predictionB' respectively, to facilitate comparison. ```sql CREATE TABLE IF NOT EXISTS result AS SELECT A.label, A.prediction as predictionA, B.prediction as predictionB FROM table1 A JOIN table2 B ON (A.label = B.label); ``` -------------------------------- ### API Gateway Proxy Integration Event Payload (JSON) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This JSON file represents a sample event payload that API Gateway might send to a Lambda function when using proxy integration. It contains information about the request, context, and other relevant details. ```json # event.json # API Gateway Proxy Integration event payload ``` -------------------------------- ### Cleanup AWS Resources (Bash) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This Bash script demonstrates how to clean up AWS resources created by the basketball tool. It first empties and deletes an S3 bucket used as a working directory and then deletes the CloudFormation stack. ```bash NotificationBucket='basketball-tool-bucket-20190821' aws s3 rb $NotificationBucket aws cloudformation delete-stack --stack-name BasketBallTool ``` -------------------------------- ### Script to Update Lambda Function Code (Bash) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This bash script is used to update the 'shoot.py' Lambda function code and its deployment package. It takes the Lambda function name as an argument and automates the process of zipping and uploading the updated code. ```bash # src/update-shoot-func.sh # Updating shoot.py and Zip File to Lambda function code ``` -------------------------------- ### Update Lambda Function Zip File Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This script updates the 'shoot.py' Lambda function code and its corresponding zip file. You need to navigate to the 'src' directory and replace 'BasketBallTool-Shoot-1BLAHBLAH' with your actual Lambda function name. This is necessary after deployment. ```bash cd src/ ./update-shoot-func.sh BasketBallTool-Shoot-1BLAHBLAH ``` -------------------------------- ### Query Prediction Differences with Athena SQL Source: https://context7.com/aws-samples/aws-emr-basketball-tool/llms.txt Athena SQL query calculating total record count, count of differing predictions between EMR versions, and the ratio of differences. Results provide quantitative metrics for evaluating EMR upgrade impact on model predictions. ```sql SELECT count(*) as total_count, (SELECT count(*) FROM comp.result WHERE predictionA != predictionB) as diff_count, (SELECT count(*) FROM comp.result WHERE predictionA != predictionB)/CAST(count(*) as double) as ratio FROM result ``` -------------------------------- ### Python Lambda Handler for EMR Report Generation and Email Source: https://context7.com/aws-samples/aws-emr-basketball-tool/llms.txt The core Lambda function that orchestrates the EMR cluster comparison and reporting process. It relies on boto3 for AWS service interactions, reads configuration and data from S3, fetches EMR cluster descriptions, processes Athena query results, constructs an HTML report, and sends it via Amazon SES. It also handles cleanup of temporary files in S3. ```python import json import boto3 from botocore.exceptions import ClientError import re TARGET_REGION = 'ap-northeast-2' SES_REGION = 'us-east-1' SENDER = 'Example ' def lambda_handler(event, context): ret = 200 subscriberEmail = '' try: BucketName = event['Records'][0]['s3']['bucket']['name'] s3 = boto3.resource('s3') emr = boto3.client('emr') # Read all stored metadata reqBody = s3.Object(BucketName, 'request/request.txt').get()['Body'].read().decode('utf-8') clusterId1Body = s3.Object(BucketName, 'request/clusterid1').get()['Body'].read().decode('utf-8') clusterId2Body = s3.Object(BucketName, 'request/clusterid2').get()['Body'].read().decode('utf-8') queryIdBody = s3.Object(BucketName, 'request/queryid').get()['Body'].read().decode('utf-8') subscriberEmail = re.search(r'#subscriber-email\s*([^#]+)', reqBody).group(1) clusterId1 = json.loads(clusterId1Body)['ClusterId'] clusterId2 = json.loads(clusterId2Body)['ClusterId'] queryId = queryIdBody # Fetch EMR cluster details emrInfo1 = emr.describe_cluster(ClusterId=clusterId1) emrInfo2 = emr.describe_cluster(ClusterId=clusterId2) # Fetch Athena query results resultObj = s3.Object(BucketName, f'result/{queryId}.csv') result = resultObj.get()['Body'].read().decode('utf-8') resultHeader = result.split("\n")[0].split(",") resultRow = result.split("\n")[1].split(",") # Build HTML email html = f""" EMR Diff Tool

Comparison Results

MetricValue
{resultHeader[0]}{resultRow[0]}
{resultHeader[1]}{resultRow[1]}
{resultHeader[2]}{resultRow[2]}
""" # Send email via SES ses_client = boto3.client('ses', region_name=SES_REGION) response = ses_client.send_email( Destination={'ToAddresses': [subscriberEmail]}, Message={ 'Body': { 'Html': {'Charset': 'UTF-8', 'Data': html}, 'Text': {'Charset': 'UTF-8', 'Data': 'EMR Diff Tool Results'} }, 'Subject': {'Charset': 'UTF-8', 'Data': 'EMR Diff Tool'} }, Source=SENDER ) # Clean up S3 temporary files bucket = s3.Bucket(BucketName) bucket.objects.filter(Prefix="result").delete() bucket.objects.filter(Prefix="status").delete() bucket.objects.filter(Prefix="predict1").delete() bucket.objects.filter(Prefix="predict2").delete() bucket.objects.filter(Prefix="athena_output").delete() print(f"Email sent! Message ID: {response['MessageId']}") return { 'statusCode': ret, 'body': json.dumps(f"sent: {subscriberEmail}") } except Exception as e: print(f"Error: {e}") return { 'statusCode': 500, 'body': json.dumps('Error sending report') } ``` -------------------------------- ### Monitor S3 Output Files with Python Lambda Handler Source: https://context7.com/aws-samples/aws-emr-basketball-tool/llms.txt Lambda function that monitors S3 bucket for output files from the second EMR cluster. It counts files in the predict2 directory and, when both clusters have completed (size > 1), writes a success status file to trigger downstream comparison processes. Handles S3 events and returns HTTP status codes with JSON responses. ```python import json import boto3 def lambda_handler(event, context): ret = 200 retString = 'no file' s3 = boto3.resource('s3') try: BucketName = event['Records'][0]['s3']['bucket']['name'] bucket = s3.Bucket(BucketName) # Count files in predict2 directory size = sum(1 for _ in bucket.objects.filter(Prefix='predict2/')) print(f"predict2 file count: {size}") # If both clusters have completed, trigger comparison if size > 1: key = 'status/status.txt' object = s3.Object(BucketName, key) object.put(Body='success') retString = 'success' return { 'statusCode': ret, 'body': json.dumps(retString) } except Exception as e: print(f"Error: {e}") return { 'statusCode': 500, 'body': json.dumps('Error checking output') } ``` -------------------------------- ### Python Lambda: Monitor EMR Cluster Output Completion Source: https://context7.com/aws-samples/aws-emr-basketball-tool/llms.txt This Lambda function, 'S3 Output Checker 1', monitors an S3 bucket for output files generated by the first EMR cluster. Upon detecting more than one file in the 'predict1/' prefix, it signifies job completion and writes a 'success' status to 'status/status.txt' in S3. It relies on 'boto3' for S3 interactions. The input is an S3 event, and the output is a status code and a string indicating 'success' or 'no file'. ```python import json import boto3 def lambda_handler(event, context): ret = 200 retString = 'no file' s3 = boto3.resource('s3') try: BucketName = event['Records'][0]['s3']['bucket']['name'] bucket = s3.Bucket(BucketName) # Count files in predict1 directory size = sum(1 for _ in bucket.objects.filter(Prefix='predict1/')) print(f"predict1 file count: {size}") # If files exist, write success status if size > 1: key = 'status/status.txt' object = s3.Object(BucketName, key) object.put(Body='success') retString = 'success' return { 'statusCode': ret, 'body': json.dumps(retString) } except Exception as e: print(f"Error: {e}") return { 'statusCode': 500, 'body': json.dumps('Error checking output') } ``` -------------------------------- ### Analyze Prediction Differences Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/request.txt This SQL query calculates the total number of records, the count of records where predictions differ between 'predictionA' and 'predictionB', and the ratio of differing predictions to the total count. It assumes the 'result' table has already been created. ```sql SELECT count(*) as total_count, (SELECT count(*) FROM comp.result WHERE predictionA != predictionB) as diff_count, (SELECT count(*) FROM comp.result WHERE predictionA != predictionB)/CAST(count(*) as double) as ratio FROM result; ``` -------------------------------- ### Define Hive External Table for Prediction Results (predict1) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/request.txt This SQL statement creates an external Hive table named 'table1'. It's designed to read prediction results stored in tab-delimited format from the 's3://basketball-tool-bucket-20190821/predict1' S3 location. The table schema includes 'label' and 'prediction' columns. ```sql CREATE EXTERNAL TABLE IF NOT EXISTS table1 ( label STRING, prediction STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' location 's3://basketball-tool-bucket-20190821/predict1' ``` -------------------------------- ### Define Hive External Table for Prediction Results (predict2) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/request.txt This SQL statement creates an external Hive table named 'table2'. It's designed to read prediction results stored in tab-delimited format from the 's3://basketball-tool-bucket-20190821/predict2' S3 location. The table schema includes 'label' and 'prediction' columns. ```sql CREATE EXTERNAL TABLE IF NOT EXISTS table2 ( label STRING, prediction STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' location 's3://basketball-tool-bucket-20190821/predict2' ``` -------------------------------- ### Lambda Function Code for Sending Reports (Python) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This Python script implements the 'scorer' Lambda function, which is responsible for sending the final comparison report to the specified email address. It formats and dispatches the results of the EMR job comparison. ```python # src/scorer.py # Lambda function code for sending report ``` -------------------------------- ### Lambda Function Code for Job Termination Check (Python) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md Similar to s3_checker1.py, this Python script is a Lambda function designed to verify the termination status of a second EMR job (job2). It ensures the orderly execution of the EMR workflow. ```python # src/s3_checker2.py # Lambda function code for checking if job2 is terminated ``` -------------------------------- ### Lambda Function Code for Athena Querying (Python) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This Python script is part of the Lambda function responsible for querying data in Athena. It likely interacts with AWS services to execute SQL queries against data stored in S3 and returns the results. ```python # src/athena_query.py # Lambda function code for querying to Athena ``` -------------------------------- ### Lambda Function Code for Job Termination Check (Python) Source: https://github.com/aws-samples/aws-emr-basketball-tool/blob/master/README.md This Python script serves as a Lambda function to check if a specific EMR job (job1) has terminated. It is crucial for monitoring the status of EMR jobs in the workflow. ```python # src/s3_checker1.py # Lambda function code for checking if job1 is terminated ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.