Blog Content Deploy with AWS Code Commit and Code Build
Table of Contents
In a previous post, I discussed a new static site generation process being used for this blog. More recently, I discussed moving and hosting in AWS Now, I want to briefly discuss how it's now, finally, being auto deployed via Git and AWS Code Build.
Overview
The basic idea is fairly straight forward and is typical of most continuous deployment pipelines found elsewhere. Upon pushing to a particular branch, submit a job to build and deploy to some environment. Since this blog has low risks we push straight to "production", where production is simply an S3 bucket as described before.
Examining this deployment flow from the AWS perspective, a branch is updated in AWS CodeCommit, this submits a message to an AWS SNS topic. From here, a Lambda function receives the event and submits a build request to AWS CodeBuild. This certainly feels as complex as it sounds. Unfortunately, this complexity is necessary as AWS doesn't currently provide a batteries included solution that is appropriately sized for the current problem.
The motivation for this choice in "architecture" is as such, CodeCommit can only send events ("triggers") to either SNS or Lambda; furthermore, sending the event to SNS allows for more flexibility in later subscriptions if necessary (as is for cases that are not this blog).
Another available option explored earlier was using CloudWatch Events to trigger the Lambda job and in doing so, being able to access little more information about the commit submitted. However, this has other filtering issues when considering its usage with many CodeCommit repositories.
CloudFormation
Let's consider the specifics of creating the necessary components in AWS CloudFormation.
Notice, the values will likely be very specific to this blog. If attempting to replicate for your own usage (which you should feel free to do so!), you will likely need to update a few values to your needs.
First, we need an SNS topic:
"CodeCommitEventsSnsTopic": {
"Type": "AWS::SNS::Topic",
"Properties": {
"DisplayName": "CodeCommit Events",
"TopicName": "codecommit-events"
}
}
Next, we need we will need a few IAM roles and policies for CodeBuild and Lambda.
Here are the two IAM resources for CodeBuild:
"CodeBuildIamManagedPolicy": {
"Type": "AWS::IAM::ManagedPolicy",
"Properties": {
"Description": "CodeBuild Service Policy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
{"Fn::Join": [":", [
"arn:aws:logs",
{"Ref": "AWS::Region"},
{"Ref": "AWS::AccountId"},
"log-group:/aws/codebuild/CodeBuild*"]]},
{"Fn::Join": [":", [
"arn:aws:logs",
{"Ref": "AWS::Region"},
{"Ref": "AWS::AccountId"},
"log-group:/aws/codebuild/CodeBuild*",
"log-stream:*"]]}
]
}, {
"Effect": "Allow",
"Action": [
"codecommit:GitPull"
],
"Resource": [
{"Fn::Join": [":", [
"arn:aws:codecommit",
{"Ref": "AWS::Region"},
{"Ref": "AWS::AccountId"},
"*"]]}
]
}, {
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:Get*",
"s3:List"
],
"Resource": [
{"Fn::GetAtt": ["BlogContentBucket", "Arn"]},
{"Fn::Join": ["", [{"Fn::GetAtt": ["BlogContentBucket", "Arn"]}, "/*"]]}
]
}
]
}
}
},
"CodeBuildIamServiceRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "codebuild.amazonaws.com"
},
"Effect": "Allow"
}
]
},
"ManagedPolicyArns": [
{"Ref": "CodeBuildIamManagedPolicy"}
]
}
}
Next are the two for Lambda.
"LambdaCodeCommitBuildIamManagedPolicy": {
"Type": "AWS::IAM::ManagedPolicy",
"Properties": {
"Description": "Lambda CodeCommit-Build Execution Policy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
{"Fn::Join": [":", [
"arn:aws:logs",
{"Ref": "AWS::Region"},
{"Ref": "AWS::AccountId"},
"log-group:/aws/lambda/codecommit-build-bae089e8-3871-4067-9a3d-bac114f08438:*"
]]}
]
}, {
"Effect": "Allow",
"Action": [
"codebuild:StartBuild"
],
"Resource": [
{"Fn::Join": [":", [
"arn:aws:codebuild",
{"Ref": "AWS::Region"},
{"Ref": "AWS::AccountId"},
"project/*"]]}
]
}
]
}
}
},
"LambdaCodeCommitBuildIamServiceRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow"
}
]
},
"ManagedPolicyArns": [
{"Ref": "LambdaCodeCommitBuildIamManagedPolicy"}
]
}
}
Finally, the Lambda function needs to subscribe to the SNS topic.
"CodeCommitBuildSnsSubscription": {
"Type": "AWS::SNS::Subscription",
"Properties": {
"Protocol": "lambda",
"Endpoint": {"Fn::GetAtt": [
"CodeCommitBuildLambdaFunction", "Arn"]},
"TopicArn": {"Ref": "CodeCommitEventsSnsTopic"}
}
}
Lest we forget, an all to often forgotton resource necessary for creating Lambda functions via CloudFormation, we need a Lambda Permission resource:
"CodeCommitBuildLambdaPermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": {"Fn::GetAtt": [
"CodeCommitBuildLambdaFunction", "Arn"]},
"Action": "lambda:InvokeFunction",
"Principal": "sns.amazonaws.com",
"SourceArn": {"Ref": "CodeCommitEventsSnsTopic"}
}
}
Finally, we need to add the CodeBuild resources:
"BlogCodeBuildLogGroup": {
"Type": "AWS::Logs::LogGroup",
"Properties": {
"LogGroupName": {"Fn::Join": ["-", [
"/aws/codebuild/CodeBuild",
{"Ref": "BlogBucketName"}]]},
"RetentionInDays": 14
}
},
"BlogCodeBuild": {
"Type": "AWS::CodeBuild::Project",
"Properties": {
"Name": "BlogCI",
"Description": "Blog Build Project",
"Artifacts": {
"Type": "NO_ARTIFACTS"
},
"Environment": {
"ComputeType": "BUILD_GENERAL1_SMALL",
"Image": "kennyballou/debian-pandoc:latest",
"Type": "LINUX_CONTAINER"
},
"LogsConfig": {
"CloudWatchLogs": {
"GroupName": {"Fn::Join": ["-", [
"/aws/codebuild/CodeBuild",
{"Ref": "BlogBucketName"}
]]},
"Status": "ENABLED"
}
},
"ServiceRole": {"Ref": "CodeBuildIamServiceRole"},
"Source": {
"Type": "CODECOMMIT",
"Location": {"Fn::GetAtt": ["BlogContentRepository",
"CloneUrlHttp"]}
}
}
}
With these resources added, we can now move onto some of the other details necessary.
buildspec.yml
Depending on how complicated the blog content is, the buildspec.yml file can
be trivial to very complex. If most of the build instructions are already
captured in a script or Makefile, the build specificiation will
likely be fairly straightforward.
For this blog, the buildspec.yml file is as follows:
version: 0.2
phases:
build:
commands:
- make
- make deploy
Realistically, a line could be removed but is left for clarity.
Docker
Since this blog is built using Pandoc and some Bash scripts, a custom build image was created.
It's referenced in the CodeBuild resource defined above.
However, if using different tools to generate content, using the provided images from AWS may be possible.
Git Remote
A git repository may have any number of remote repositories associated with it. Consider forked projects or repositories on GitHub for a moment: before opening a pull request against the parent project, it's good practice to make sure the changes are based on the latest changes in the parent branch. To trivially achieve this, the local clone of the repository (the fork) can be configured to have both the remotes associated, e.g.:
% git clone ssh://github.com/yours/${forked_project}
% cd ${forked_project}
% git remote add upstream ssh://github.com/parent/${forked_project}
Now, ensuring the changes to be submitted are based on the latest changes in the parent only requires a few commands (and possible some merge conflict resolution):
% git remote update -p % git rebase upstream/master % git push --force-with-lease origin pr-branch
I am making some assumptions of workflow and that the PR branch is yours and you're, therefore, allowed to do whatever you want to its history.
Similarly, for auto deploying blog content, we need to add the new repository from CodeCommit to the blog's remotes.
git remote add aws ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/blog.kennyballou.com
I recommend using SSH Config files to ease using Git, SSH, and CodeCommit. Especially so if multiple AWS accounts are involved, each with their own set of repositories.
Afterwhich, when content is ready to be published, it is as simple as pushing
the branch to the other remote. Assuming that we're already on the master
branch, push to the different remote:
% git push aws master
Parting Thoughts
Honestly, there may be easier and cheaper ways to host some simple infrastructure for running and building projects. GitHub now has Actions. GitLab has CI/CD pipelines as part of their offering. A new forge, Source Hut, has builds. There likely are many more variations I fail to mention as I'm not aware of them. That said, AWS does provide a 100 minutes of CodeBuild free each month and CodeCommit has some pretty high thresholds before AWS begins incuring charges.
However, for me, when already hosting the content via S3 and CloudFront, having the ability to implicitly authorize write access to the CodeBuild job, it is more convincing to run everything within AWS, even if AWS doesn't always bring the batteries.
Finally, setting up these resources via the AWS web console may be easier than setting them up via CloudFormation, it is the hope that the pain suffered in configuring and connecting the various resources together is helpful to someone else in a similar position.