Now that you have successfully completed this workshop you should be familiar with the concepts and operations of AWS Batch. This includes the basic steps required to build and register your own container images; create and run single and array batch jobs; create job dependencies; share data between jobs using Amazon S3; and decribe AWS Batch infrastructure and the status of jobs. These actions form the basic building blocks of more complex workflows that can be accomplished with AWS Batch. At this stage it is a good idea to reinforce your knowedge by applying these techniques to address your own problems and workflows.
Some related tasks that may be beneficial to try at this stage include:
Make sure to let us know what cool things you are building!
After you have completed all of the previous workshop steps and finished experimenting you can clean up your environment by following the instructions below. Note that you can complete each step in either the AWS Management Console or by running the specified commands in a terminal on your Cloud9 instance.
Alternatively, you can use the following script to terminate jobs in any queues with names that match “stress”.
cd ~/environment/
cat > kill-all-jobs.sh << EOF
#!/bin/bash
MATCH=stress
echo "Terminating all jobs in queues that match \"\$MATCH\""...
JQS=\$(aws batch describe-job-queues --query "jobQueues[].[jobQueueName]" --output text | grep \$MATCH)
for jq in \$JQS
do
echo "Terminating jobs in queue: \$jq"
for state in SUBMITTED PENDING RUNNABLE STARTING RUNNING
do
JOBS=\$(aws batch list-jobs --job-queue \$jq --job-status \$state --query "jobSummaryList[].[jobId]" --output text)
if [ ! -z "\$JOBS" ]
then
for job in \$JOBS
do
aws batch terminate-job --job-id \$job --reason 'Terminating job'
echo "Terminating \$job in \$jq with state \$state"
done
fi
done
done
EOF
bash kill-all-jobs.sh
Or, use the following script to deregister all job definitions with names that match “stress”.
cd ~/environment/
cat > remove-jd.sh << EOF
#!/bin/bash
MATCH=stress
echo "Deregistering all job definitions that match \"\$MATCH\""...
JD_LIST=\$(aws batch describe-job-definitions --query "jobDefinitions[*].[jobDefinitionArn]" | jq -r ".[]" | grep arn | cut -f2 -d\" | grep \$MATCH)
for jd in \$JD_LIST
do
echo "Deregistering: \$jd"
aws batch deregister-job-definition --job-definition \$jd
done
EOF
bash remove-jd.sh
Or, use the following script to delete all job queues with names that match “stress”.
cat > ~/remove-jq.sh << EOF
#!/bin/bash
MATCH=stress
echo "Removing all job queues that match \"\$MATCH\""
# Disable the queues.
JQ_LIST=\$(aws batch describe-job-queues --query "jobQueues[*].jobQueueArn" | jq -r ".[]" | grep \$MATCH)
for queue in \$JQ_LIST
do
# Disable the job queue.
aws batch update-job-queue --job-queue \$queue --state DISABLED
# Wait until queue is disabled.
while [ \`aws batch describe-job-queues --job-queue \$queue --query "jobQueues[*].state" | jq -r ".[]" | grep DISABLED | wc -l\` -ne 1 ]
do
echo "Waiting for job queue \$queue to be disabled..."
sleep 10
done
# Wait until queue status is valid.
while [ \`aws batch describe-job-queues --job-queue \$queue --query "jobQueues[*].status" | jq -r ".[]" | grep VALID | wc -l\` -ne 1 ]
do
echo "Waiting for job queue \$queue status to become valid..."
sleep 10
done
# Delete the job queue.
aws batch delete-job-queue --job-queue \$queue
done
EOF
bash ~/remove-jq.sh
Or, use the following script to delete all Compute Environments with names that match “stress”.
cat > ~/remove-ce.sh << EOF
#!/bin/bash
MATCH=stress
echo "Removing all compute environments that match \"\$MATCH\""
CE_LIST=\$(aws batch describe-compute-environments --query "computeEnvironments[*].computeEnvironmentArn" | jq -r ".[]" | grep \$MATCH)
for ce in \$CE_LIST
do
# Disable the compute environment.
aws batch update-compute-environment --compute-environment \$ce --state DISABLED
# Wait until they are disabled.
while [ \`aws batch describe-compute-environments --compute-environment \$ce --query "computeEnvironments[*].state" | jq -r ".[]" | grep DISABLED | wc -l\` -ne 1 ]
do
echo "Waiting for compute environments to be disabled..."
sleep 10
done
# Wait until the status bevomes VALID.
while [ \`aws batch describe-compute-environments --compute-environment \$ce --query "computeEnvironments[*].status" | jq -r ".[]" | grep VALID | wc -l\` -ne 1 ]
do
echo "Waiting for compute environment \$ce status to become valid..."
sleep 10
done
# Delete the compute environment.
aws batch delete-compute-environment --compute-environment \$ce
done
EOF
bash ~/remove-ce.sh
Or, use the following script to delete all images and repositories with names that match “stress”.
cd ~/environment/
cat > remove-repos.sh << EOF
#!/bin/bash
MATCH=stress
echo "Removing all repositories that match \"\$MATCH\""
REPO_LIST=\$(aws ecr describe-repositories --output text --query "repositories[].[repositoryName]" | grep \$MATCH)
for repo in \$REPO_LIST
do
echo "Deleting images in \$repo"
IMAGE_LIST=\$(aws ecr describe-images --repository-name \$repo --output text --query "imageDetails[].[imageTags]")
for tag in \$IMAGE_LIST
do
echo "Deleting image: \$repo:\$tag"
aws ecr batch-delete-image --repository-name \$repo --image-ids imageTag=\$tag
done
# Wait until all images are deleted.
while [ \`aws ecr describe-images --repository-name \$repo --output text --query "imageDetails[].[imageTags]" | wc -l\` -gt 0 ]
do
echo "Waiting for images in \$repo to be deleted..."
sleep 10
done
# Delete the repository.
echo "Deleting reposityory: \$repo"
aws ecr delete-repository --repository-name \$repo
done
EOF
bash remove-repos.sh
Or, use the following commands.
export STRESS_BUCKET="s3://$(aws cloudformation describe-stacks --stack-name $STACK_NAME --output text --query 'Stacks[0].Outputs[?OutputKey == `Bucket`].OutputValue')"
aws s3 rm ${STRESS_BUCKET} --recursive
Or, use the following commands.
aws cloudformation delete-stack --stack-name BatchWorkshop
Congratulations, you have now completed this AWS Batch Workshop.