johnpfeiffer

Using AWS CDK to configure deploy a Golang Lambda with APIGateway

Infrastructure as Code helps guarantee the elusive determinism in infrastructure that we all seek in building applications and services. Often you can defer that complexity by using Heroku, Google App Engine, or other PaaS providers. But when you need to build something really complex (or with specific controls required including managing costs) then using IaaC to tame AWS Serverless reduces some of that pain.

AWS have created their own specific product (domain specific language) "CDK" which competes with the venerable Terraform and nicely focused Serverless.

All of these products use the json syntax of CloudFormation which is the foundational AWS syntax for describing resources

It is straightforward to iteratively setup and use AWS CDK (with the native Typescript syntax).

CDK vocabulary

It all begins with stacks.

From an object oriented perspective, a stack is a definition of components that can be instantiated. An "app" is a collection of related stacks.

Let's say you wanted your persistence (DynamoDB) in one stack, and your more execution oriented components (APIGateway and Lambda) in another stack, and your App might have environment specific parameters it needs to define and pass through to each stack.

This ability to use templates and customize how you structure things, reference other CDK files, etc. makes CDK very modular and re-usable (and yes Typescript is a programming language so you can have linters and tests).

Install the AWS CDK Tool

Have to use npm to install the cdk tool...

npm install -g aws-cdk

Setup the project directories

mkdir cdk-example

cd cdk-example

Create a subdirectory to compartmentalize all the infrastructure code (from the rest of the application, i.e. don't mix the business logic with the infrastructure plumbing)

mkdir infra cd infra

cdk init app --language typescript

creates the default files in this directory for an empty CDK app

cdk ls

   InfraStack

Names are hard but whatever you pick, like "Infra", will then show up in the AWS resources everywhere related to this project

Install the dependencies for AWS resources that CDK will manage

Option 1: manually install dependencies (it will automatically insert this into package.json)

npm install @aws-cdk/aws-s3 @aws-cdk/aws-lambda

Option 2: write the lines into packages.json and run at the command line in the infra directory: npm install

Writing files first (and ensuring they are in version control) is a more IaaC pattern

"dependencies": {
  "@aws-cdk/aws-s3": "*",
  "@aws-cdk/core": "*",
  "source-map-support": "^0.5.16"
}

Write the actual Infrastructure Code

The initial application template creates an empty "class" that represents the "stack", we customize and replace that code with the following...

lib/infra-stack.ts has only 2 changes to the default file, the import of S3 and the new Bucket resource "MyExampleBucket"

import * as cdk from '@aws-cdk/core';
import * as s3 from '@aws-cdk/aws-s3';

export class InfraStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new s3.Bucket(this, 'MyExampleBucket', {
      versioned: true
    });
  }
}

The import renames are important to be consistent in subsequent code, so if something is "cdk" then it is cdk.Construct

More complex applications will use a "main entry point" that can refer to specific files for various stacks. - https://docs.aws.amazon.com/cdk/latest/guide/serverless_example.html

Output CloudFormation from a CDK Typescript file

cdk synth

this command will output to the display the CloudFormation that will be sent/used by AWS

Resources:
  MyExampleBucket8D68EFCA:
    Type: AWS::S3::Bucket
    Properties:
      VersioningConfiguration:
        Status: Enabled
    UpdateReplacePolicy: Retain
    DeletionPolicy: Retain
    Metadata:
      aws:cdk:path: InfraStack/MyExampleBucket/Resource
  CDKMetadata:
    Type: AWS::CDK::Metadata
    Properties:

To preview what will occur between changes to the .ts file...

cdk diff

Deploy resources that CDK has defined

cdk deploy

warning, really making a change in AWS (based on your creds)

InfraStack

InfraStack: deploying...
InfraStack: creating CloudFormation changeset...
(3/3)

Stack ARN:
arn:aws:cloudformation:us-east-1:409670809604:stack/InfraStack/b5167030-00fb-11eb-9f36-12f8925a37c4

Verify the new bucket was created

aws s3 ls | grep MyExampleBucket

this assumes you have installed the AWS CLI like sudo apt install awscli

aws cloudformation list-stacks | grep InfraStack

this listing will also show deleted Stacks

Making updates with CDK

A tiny snippet change to allow bucket deletion...

new s3.Bucket(this, 'MyExampleBucket', {
  versioned: true,
  removalPolicy: cdk.RemovalPolicy.DESTROY
});

Preview changes with diff

cdk diff

Stack InfraStack
Resources
[~] AWS::S3::Bucket JohnPBucket JohnPBucket8D68EFCA
 ├─ [~] DeletionPolicy
 │   ├─ [-] Retain
 │   └─ [+] Delete
 └─ [~] UpdateReplacePolicy
     ├─ [-] Retain
     └─ [+] Delete

Apply the new changes

cdk deploy

InfraStack: deploying...
InfraStack: creating CloudFormation changeset...

cdk destroy

aws s3 ls | grep MyExampleBucket

aws cloudformation list-stacks | grep InfraStack

CDK enables "InfrastructureAsCode" and command line (or scripted) resource management, yet you must still understand the intricacies of the domain (i.e. that s3 buckets have policies and do not get destroyed by default)

s3 buckets are designed by default to not delete with a Stack, you must change the removal policy to do so

APIGateway plus Lambda plus Go

The trick to using Golang with AWS Lambdas is that it is a compiled language. Many tutorials for lambdas (even for CDK) use javascript or python since those dynamic languages can just be put in "one more file" without a build step.

A tiny Go Web Request Handler

Put a simple placeholder Golang Lambda in place using the Gin web framework for convenience

mkdir cdk-example/examplefunction/ ; cd cdk-example/examplefunction/

vim main.go

package main

import (
        "context"
        "log"

        "github.com/aws/aws-lambda-go/events"
        "github.com/aws/aws-lambda-go/lambda"
        "github.com/awslabs/aws-lambda-go-api-proxy/gin"
        "github.com/gin-gonic/gin"
)

var ginLambda *ginadapter.GinLambda

// for convenience leverage the Go init startup concept to define a global web server object
func init() {
        // stdout and stderr are sent to AWS CloudWatch Logs
        log.Printf("Gin starting")
        r := gin.Default()
        r.GET("/ping", func(c *gin.Context) {
                c.JSON(200, gin.H{
                        "message": "pong",
                })
        })
        ginLambda = ginadapter.New(r)
}


// Handler is the function that executes for every Request passed into the Lambda
func Handler(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
        return ginLambda.ProxyWithContext(ctx, req)
}

func main() {
        lambda.Start(Handler)
}

go mod init

Ensure dependencies (like the AWS SDK) are recognized by the Go package manager

go test

This is the simplest way to trigger downloading the dependencies (imported packages), you may have to create a tiny main_test.go in order to force this to work

https://blog.golang.org/using-go-modules

Manually build your go binary for AWS Lambda

This will create an output file named "main"

GOOS=linux GOARCH=amd64 go build -o main main.go

compile for the target arch , AWS Lambda ("firecracker") is Linux =]

zip examplefunction.zip main

AWS requires these precompiled binaries to in .zip format

CDK with a Golang Lambda

Focusing on just the "infra" subdirectory in our project:

  • cdk-example/infra/package.json
  • cdk-example/infra/lib/infra-stack.ts

cd cdk-example/infra/

First update package.json

"dependencies": {
  "@aws-cdk/core": "*",
  "@aws-cdk/aws-s3": "*",
  "@aws-cdk/aws-s3-assets": "*",
  "@aws-cdk/aws-lambda": "*",
  "source-map-support": "^0.5.16"
}

Do not forget to npm install

Next update the lib/infra-stack.ts

Layout the resources from the deepest dependency first, so in this case a place for the golang function to be zipped

import * as cdk from '@aws-cdk/core';
import * as s3 from '@aws-cdk/aws-s3';
import * as lambda from '@aws-cdk/aws-lambda';
import assets = require("@aws-cdk/aws-s3-assets")
import path = require("path")

export class InfraStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Golang binaries must have a place where they are uploaded to s3 as a .zip
    const asset = new assets.Asset(this, 'ExampleFunctionZip', {
      path: path.join(__dirname, '../../examplefunction.zip'),
    });

    const handler = new lambda.Function(this, "ExampleFunction", {
      runtime: lambda.Runtime.GO_1_X,
      handler: "main",
      code: lambda.Code.fromBucket(
        asset.bucket,
        asset.s3ObjectKey
      ),
    });
  }
}

This crucial glue code indicates that the Lambda will be named "ExampleFunction", that it will get the binary (zipped) from S3, and that the handler expects to have a binary "main"

Note that CDK will handle actually uploading the .zip to an s3 bucket

cdk synth

outputting the CloudFormation is a quick way to valied the syntax and see any warnings

cdk diff

outputting and previewing the changes that will appear in AWS

cdk deploy

InfraStack: deploying...
[0%] start: Publishing 05e95f6b38c932a779e68a7a685e9950eca688e775c77f84787f6fa3e2ade474:current
[100%] success: Published 05e95f6b38c932a779e68a7a685e9950eca688e775c77f84787f6fa3e2ade474:current
InfraStack: creating CloudFormation changeset...
 (4/4)
 InfraStack

Stack ARN:
arn:aws:cloudformation:us-east-1:409670809604:stack/InfraStack/248d2960-0104-11eb-8cc5-0ac853a0932f

Use the AWS Console UI and visually look at the cloud formation stacks

e.g. https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/

When you filter and click on your new stack (e.g. InfraStack) you will be able to see the Resources associated with that stack

The default policy "AWSLambdaBasicExecutionRole" will actually allow the Lambda to output logs to Cloudwatch

CDK with an APIGateway integrated with a Go Lambda

Create the APIG and attach it to the Lambda:

First update infra/package.json

  "dependencies": {
    "@aws-cdk/aws-apigateway": "*",
    "@aws-cdk/aws-lambda": "*",
    "@aws-cdk/aws-s3": "*",
    "@aws-cdk/aws-s3-assets": "*",
    "@aws-cdk/core": "*",
    "source-map-support": "^0.5.16"
  }

npm install

Define the API Gateway in CDK (only adding a few more lines)

https://docs.aws.amazon.com/cdk/api/latest/docs/aws-apigateway-readme.html#aws-lambda-backed-apis

import * as cdk from '@aws-cdk/core';
import * as s3 from '@aws-cdk/aws-s3';
import * as lambda from '@aws-cdk/aws-lambda';
import assets = require("@aws-cdk/aws-s3-assets")
import apigw = require("@aws-cdk/aws-apigateway")
import path = require("path")


export class InfraStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Golang binaries must have a place where they are uploaded to s3 as a .zip
    const asset = new assets.Asset(this, 'ExampleFunctionZip', {
      path: path.join(__dirname, '../../examplefunction'),
    });

    const myhandler = new lambda.Function(this, "ExampleFunction", {
      runtime: lambda.Runtime.GO_1_X,
      handler: "main",
      code: lambda.Code.fromBucket(
        asset.bucket,
        asset.s3ObjectKey
      ),
    });

    // all routes (and REST verbs) will pass through to the lambda
    const api = new apigw.LambdaRestApi(this, 'examplefunction', {handler: myhandler});
  }
}

one gotcha is the ordering of imports, have the "path" one last

cdk synth

now the CloudFormation is very verbose =|

upon inspection you will see the type of integration Lambda is as desired, AWS_PROXY

cdk deploy

There is a warning about security because AssumeRole

InfraStack: deploying...
[0%] start: Publishing 05e95f6b38c932a779e68a7a685e9950eca688e775c77f84787f6fa3e2ade474:current
[100%] success: Published 05e95f6b38c932a779e68a7a685e9950eca688e775c77f84787f6fa3e2ade474:current
InfraStack: creating CloudFormation changeset...
  (14/14)
InfraStack

Outputs:
InfraStack.examplefunctionEndpoint65D9943D = https://abc123.execute-api.us-east-1.amazonaws.com/prod/

Stack ARN:
arn:aws:cloudformation:us-east-1:409670809604:stack/InfraStack/248d2960-0104-11eb-8cc5-0ac853a0932f

Verify your new API Gateway and Golang Lambda with CURL

The very helpful output means you can use CURL directly (as you get to ignore the whole "this apigateway needs to be deployed to a stage")

curl https://abc123.execute-api.us-east-1.amazonaws.com/prod/ping

you should get back either 404 or "pong" =)

To see how much more work it is to do (and understand) the API Gateway concepts...

Published

Category

build-CI-CD-devops

~1539 words

Tags