Running Bash Scripts on AWS Lambda

I don't read blogs, just show me some example code

It would be rare to find an organisation that's been in existence for more than a few years, that didn't have a bunch of Bash scripts completing vital backend operational tasks. Inevitably these scripts are hosted on an unloved host with no resilience to failures. Actually the resilience plan in these cases involves an urgent call to a sleep deprived engineer to quickly restore a script from source control on to the next least unloved server.

Let's break that cycle of sleep deprivation. Or perhaps find a better excuse for not sleeping, like doom scrolling kpop drama, playing league of legends (and losing to kids) or winning an argument on the Internet.

I find that scripts written in Bash tend to be quite stable in most organisations. You have the period when a need is identified, there might be a small project to write the script and evaluate its utility and efficacy, after which the script just runs^TM. And it keeps just running^TM?, for years and years and eventually outliving the infrastructure it was deployed on. Sure you might revisit the script if there are changes in scale or scope related to the thing its gluing together. But generally these are minor tweaks.

But as sure as there is a vital Bash script[s] keeping your organisation running, there's a bunch of engineers nervously sacrificing actual paper technology books to the IT gods hoping to get one more month of stability out of that server because its replacement didn't make it into the IT budget.

I'll show you how you can move those scripts to serverless infrastructure. The learning curve won't be very steep and you won't need to make significant changes to your scripts. It will cost you a few cents per month.

If you're conversant with common unix tools cron and at then the capability to schedule various tasks using AWS services, will feel very familiar. With very little effort you can mimic your current schedule. Over time you may wish to improve failure handling and retries, but that capability is not required to migrate your Bash scripts and for many use cases, a simple schedule will be enough.

Why Serverless?

"But why?" I hear you ask as I write myself questions for which I already have answers. Well, there's the warm glow of telling your friends, social media contacts and random LinkedIn thinkfluencers that it's all about the serverless. And that's reason enough. Design by resume padding is a legitimate pattern IMHO.

More importantly consider that there's ~13,000 GB secs (3.5 hrs) free execution time a day, forever. There's quite a lot you can get done in that time with a Bash script or three.

Since we are talking about migrating working code, the effort to achieve this is limited to porting your host. Changes to your scripts will be minor. But the rewards will be significant in almost any organisation.

• Cap-ex: less money replacing servers.
• Op-ex: less effort and opportunity cost of fixing the servers you didn't replace
• Risk: reduced business impact of scripts not running on the servers you didn't replace and couldn't find time to administer

AWS Provided Lambda Runtime

For Bash scripts we only need the OS and some basic command line programs. AWS Lambda offers an 'OS only' runtime using Amazon Linux 2023. You'll see this listed sometimes as provided.al2023 or OS only.

But there's a catch. The bare OS only environment that AWS provides you is very lean. Lean is good I hear you say. Well there's lean and then there's missing some vital tools lean. There's a few pretty key omissions that most people wanting to write Bash scripts on Lambda will surely miss. First there's no jq installed. AWS delivers the event data that triggers a Lambda function as a JSON payload. Parsing any event data will represent a lot of effort without jq.

The second omission is perhaps even more baffling than the first. There's no AWS CLI pre installed. That means any interaction with an AWS service will have to be through hand written API calls via curl.

There's also a lack of some reasonably common tools that I suspect many Lambda scripts that interact with files will desire, such as gzip.

So if you don't need to parse the Lambda triggering event and you don't need to interact directly with AWS services, then the provided OS only runtime may be suitable. Even if your use-case fits that scenario the missing common tools might still make this option unworkable for you. Therefore for most people the OS only runtime will not be a practical option.

Build Your Own

While it isn't suitable as-is, we can modify the AWS al2023 container for our needs. We can customise the container to handle just about any kind of Lambda triggering event. We'll ensure that all the necessary tools for our script are installed and ready to use.

What will we build?

We'll build a container with the CLI tools we need and the script we want to execute. We'll store it in a container repository. Then we'll configure a Lambda function to use that container. Finally, I'll show you a couple of options for triggering the function.

In order to use a custom container for a Lambda function, the container image must be stored in an AWS Elastic Container Registry (ECR) repository. This is the one component that isn't free. But we can keep the cost to a few cents a month.

The size of the completed example container, once uploaded to AWS ECR, is ~270MB. AWS ECR storage does have a free tier offer for the first 12 months, after that you'll pay for storage. If you only keep the latest version of this container, you'll be paying ~$.03/month.

What will we need?

To build the container you'll use either Podman or Docker. All my examples use Podman. You should be able to swap podman with docker and leave all other parameters unchanged.

We'll use the AWS CLI to manage the AWS configuration.

Defining the container

I've mocked up a very simple container definition that should support most basic Bash tasks.

 1FROM public.ecr.aws/lambda/provided:latest
 2
 3WORKDIR /var/runtime/
 4COPY bootstrap bootstrap
 5RUN chmod 755 bootstrap
 6
 7WORKDIR /var/task/
 8COPY function.sh function.sh
 9RUN chmod 755 function.sh
10
11WORKDIR /tmp
12
13RUN dnf --nodocs install -y gzip unzip jq
14
15WORKDIR /tmp
16RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o awscliv2.zip && \
17unzip -u awscliv2.zip
18
19RUN ./aws/install --bin-dir /usr/local/bin --install-dir /usr/local/aws-cli
20
21RUN rm -rf /tmp/aws && rm -f /tmp/awscliv2.zip
22
23WORKDIR /var/task
24ENTRYPOINT ["/lambda-entrypoint.sh"]
25CMD ["function.handler"]

You can use the example function.sh as a primer to write your own code.

 1# function.sh
 2# A function which processes a file uploaded to an S3 bucket
 3function handler () {
 4  EVENT_DATA=$1
 5  NEWFILE=$(echo $EVENT_DATA | sed 's/\\//g' | jq ".Records[0].s3.object.key"| tr -d '"')
 6  aws s3 cp s3://$NEWFILE .
 7  FILENAME="$(basename $NEWFILE)"
 8  # Do something with the file
 9  # Return some status information if required
10  RESPONSE="Everything is OK"
11  echo $RESPONSE
12}

The repo for this post includes instructions that take you through customising the container file.

AWS provides an example bootstrap file. This is the file that will get called when the function first executes. It's purpose is to call the function.sh script and supply it with the event trigger information and then return any output from the function to the Lambda service. In the example script above $1 contains the event information supplied by the bootstrap file. I use the AWS example bootstrap without changes.

 1#!/bin/sh
 2set -euo pipefail
 3# Initialization - load function handler
 4source $LAMBDA_TASK_ROOT/"$(echo $_HANDLER | cut -d. -f1).sh"
 5# Processing
 6while true
 7do
 8  HEADERS="$(mktemp)"
 9  # Get an event. The HTTP request will block until one is received
10  EVENT_DATA=$(curl -sS -LD "$HEADERS" "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next")
11  # Extract request ID by scraping response headers received above
12  REQUEST_ID=$(grep -Fi Lambda-Runtime-Aws-Request-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)
13  # Run the handler function from the script
14  RESPONSE=$($(echo "$_HANDLER" | cut -d. -f2) "$EVENT_DATA")
15  # Send the response
16  curl -sS "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$REQUEST_ID/response"  -d "$RESPONSE"
17done

Building and storing the bash container.

Building the container is fairly straight forward. The only decision you need to make is what architecture you want to build it for. Because I'm a cheapskate and Graviton instances generally cost less, my example will build for the ARM architecture.

1FUNCNAME=funcwmyhart
2podman build --arch=arm64 -t ${FUNCNAME} -f /pathto/Containerfile

Create an ECR repository to store the container.

1aws ecr create-repository --repository-name ${FUNCNAME}

Login to ECR, tag the container with the AWS repo, then upload it.

 1MYACCOUNT=1234567890
 2MYREGION="nn-moonbase-1"
 3aws ecr get-login-password --region ${MYREGION} | \
 4  podman login --username AWS --password-stdin ${MYACCOUNT}.dkr.ecr.${MYREGION}.amazonaws.com
 5
 6podman tag ${FUNCNAME}:latest \
 7  ${MYACCOUNT}.dkr.ecr.${MYREGION}.amazonaws.com/${FUNCNAME}:latest
 8
 9podman push \
10  ${MYACCOUNT}.dkr.ecr.${MYREGION}.amazonaws.com/${FUNCNAME}:latest

Create the Lambda function

Now that the container is uploaded and sitting in an accessible ECR repo, we can create the Lambda function. We will need the URI of the container we uploaded. Here's two ways to get the URI:

1# Method 1. Fetch repo URI via aws cli then append 'latest' tag
2REPO=$(aws ecr describe-repositories \
3  --repository-name ${FUNCNAME} \
4  --query 'repositories[0].repositoryUri')
5URI="${REPO}:latest"
6# Method 2. Compose URI based on what we already know
7URI=${MYACCOUNT}.dkr.ecr.${REGION}.amazonaws.com/${FUNCNAME}:latest

Before we create the function, we need to create a role for it to assume. The most basic role for a Lambda function will contain a default trust policy for Lambda and a permission policy allowing it to create a log group in AWS CloudWatch and to write to that log group. If you plan on interacting with other AWS services, you'll need to add permissions policies allowing those actions to the role. However you do not need to add all permissions right now. You can always update this role later.

So let's create that most basic of roles now. We will need a trust policy and the logging permission policy. We'll create a file for both. The file lambda-Trust.json can be used for future Lambda function roles without change.

lambda-Trust.json

 1{
 2    "Version": "2012-10-17",
 3    "Statement": [
 4        {
 5            "Effect": "Allow",
 6            "Principal": {
 7                "Service": "lambda.amazonaws.com"
 8            },
 9            "Action": "sts:AssumeRole"
10        }
11    ]
12}

Since this is a json document we can't use variable substitution. In the repo for this post, I've included an example jinja2 template for use with Ansible, you should be able to convert this to your template of choice.

funcwmyhart-lambda-Policy.json

 1{
 2    "Version": "2012-10-17",
 3    "Statement": [
 4        {
 5            "Effect": "Allow",
 6            "Action": "logs:CreateLogGroup",
 7            "Resource": "arn:aws:logs:nn-moonbase-1:1234567890:*"
 8        },
 9        {
10            "Effect": "Allow",
11            "Action": [
12                "logs:CreateLogStream",
13                "logs:PutLogEvents"
14            ],
15            "Resource": [
16                "arn:aws:logs:nn-moonbase-1:1234567890:log-group:/aws/lambda/funcwmyhart:*"
17            ]
18        }
19    ]
20}

Let's create the permissions policy first.

1aws iam create-policy \
2  --policy-name ${FUNCNAME}-lambda-Policy \
3  --policy-document file://path_to/funcwmyhart-lambda-Policy.json

Note the ARN in the output, you'll need it in a moment. To make things easier, save it as an environment variable.

1POLICYARN="arn:aws:iam::01234567890:policy/funcwmyhart-lambda-Policy"

Create the role. We will need the role ARN when we create the Lambda function, so save it as another environment variable.

1aws iam create-role \
2  --role-name ${FUNCNAME}-lambda-Role \
3  --assume-role-policy-document file://path_to/lambda-Trust.json
4ROLEARN="arn:aws:iam::01234567890:role/funcwmyhart-lambda-Role"

Attach the policy we created earlier to the new role.

1aws iam attach-role-policy \
2  --policy-arn ${POLICYARN} \
3  --role-name ${FUNCNAME}-lambda-Role

With the role created and the container image stored in ECR, we can create the Lambda function.

1aws lambda create-function \
2  --role ${ROLEARN} \
3  --function-name ${FUNCNAME} \
4  --package-type Image \
5  --code ${URI}

Triggering the function

I started this post talking about scheduled scripts. However as I've illustrated in the example function.sh, you can use other triggers to start your Lambda as well. Below is a snippet from the example script showing how a trigger based on a put object (file upload) operation occurring to an AWS S3 Bucket might be parsed.

To better understand the content and structure of the JSON payload for any trigger type you can view the Templates in the Test tab on your Lambda function's page within the AWS console.

1# Responding to a file uploaded to S3
2NEWFILE=$(echo $EVENT_DATA | sed 's/\\//g' | jq ".Records[0].s3.object.key"| tr -d '"')

If you prefer to use a schedule to trigger the script, you can still provide additional context via a JSON payload if you wish. The payload is configured as part of the Amazon EventBridge schedule.

1{ "Things": [ "apple", "banana", "cricketbat" ], "MYNumber": 42, "SecondNumber": 100 }

What next?

The example code should be enough to get anyone started. This should allow you to migrate any of those vital, yet unloved scripts keeping your organisation functioning, to a serverless platform where host maintenance is no longer your problem. There'll also be opportunities to add conditional execution and additional context to these scripts via other AWS services like Event Bridge. That might help avoid adding unnecessary complexities to your scripts or allow you to react to events rather than predict their occurrence.

For some functions, it may eventually be worth migrating them to a faster and leaner runtime, such as Go. Especially if you find yourself having to use large amounts of memory. But for many Bash functions you may not need to touch them again for a very long time.

Updating the runtime

Regardless of the amount of interaction (or lack thereof) your Bash function has with other services, I strongly recommend updating the runtime on a regular basis. Doing this should be very simple and will take little time. Re-running the container build process will refresh the container base image used and if you're fetching all your packages via the build process, you have nothing else to do. Once you have your new image, upload to AWS ECR and make sure to point your Lambda function at the new image, it won't switch automatically.

Get the code

Code for everything I've discussed in this post can be found here