Blue/Green Deployments with AWS CodeDeploy and Terraform

The hooks are executed as a Lambda function. For each hook you want to perform some steps in, you define the name of the Lambda function to be invoked. At various stages of the deployment lifecycle, CodeDeploy will invoke the Lambda associated with that hook, passing through the ID of the deployment and a unique token. The Lambda can then perform any steps required and then report a success or a failure status using the token it was passed when invoked.

This can potentially be a problem since no useful data about the deployment is passed to the hook Lambda in the event, so any other metadata about the deployment must be provided through other means. Lambda is able to fetch some general CodeDeploy deployment metadata using the CodeDeploy APIs from the SDK . But there isn’t any way of passing any specific configuration in there. So the simplest way to get data about what operations to perform on what resources is to get Terraform to deploy the hook Lambdas when it runs its apply operation as part of your application and embed the tasks to perform and any other data they require into the Lambda artefact (code, environment variables, IAM permissions etc). The side effect of this though is that you can end up with lots of extra lambdas deployed alongside each service in support of its blue/green deployment pipeline. This can however be solved quite easily if there are common conventions around your services within your team. You could look into using common deployment hook Lambdas which are used by all of your services along with some config store which can be used by these Lambdas during a deployment to share metadata about the deployment. DynamoDB would be an excellent choice for this. It’s also worth mentioning that CodeDeploy deployments have a hard limit of 60 minutes, this means that all of your hooks should have completed within this window.

CodeDeploy’s blue/green features are actually implemented in CloudFormation, so it’s much more challenging to decouple this pairing when using Terraform since there isn’t a way to invoke a CodeDeploy deployment directly using Terraform. Getting CodeDeploy to work with other IaC tools, in particular a “desired state” type of tool like Terraform (more about this can be learnt in the Terraform docs and also “CloudFormation or Terraform?” AWS Bites podcast), is somewhat more involved so let’s see how we can achieve that.

Firstly, we still can use Terraform to create, update and destroy almost all of our resources. However, there are some significant changes required to how you declare resources. Usually, the first public facing entry point into a service is typically an API handler. This is the only component that CodeDeploy is going to manage the process of running 2 instances side by side for the duration of the deployment. All other components of the service like databases and data ingestion resources will need to be managed through other means. That means, you will have 2 instances of the resources like databases, ingest handlers, queues etc. This is straightforward to implement in Terraform using for_each functionality on resources. Below is an example of what this looks like for an SQS queue:

locals {
 stack_refs = ["a", "b"] 
} 

resource "aws_sqs_queue" "ingest_queue" { 
 for_each = toset(local.stack_refs) 
 name = "bg-service-queue-${each.value}-${var.environment}" 
 policy = data.aws_iam_policy_document.sqs_queue_policy.json 
 visibility_timeout_seconds = 31
 redrive_policy = jsonencode({ 
  deadLetterTargetArn: aws_sqs_queue.ingest_dlq[each.value].arn, 
  maxReceiveCount: 3
 }) 
}

The result of the above definition is 2 SQS queues, named bg-service-queue-a-dev and bg-service-queue-b-dev. You can also see how they refer to other associated resources which are also duplicated in the the reference to the DLQ’s ARN above: aws_sqs_queue.ingest_dlq[each.value].arn where each.value is referencing either a or b.

In addition to creating the required duplicated resources, we also need to address the following:

1. Stopping Terraform from modifying any live and active resources. For example, if the deployment involves changing the name of a database, and the a set of resources are currently in live and in use, we only want to modify the b database since that will be associated with the next version of the deployment.
2. Stopping Terraform from undoing anything that is being managed by CodeDeploy. For example, if we’re using CodeDeploy to modify the live alias for a Lambda, we wouldn’t want Terraform to update that when the apply operation is run, and for an ECS service, we wouldn’t want Terraform to update the container definitions for that service.

To solve problem 1, we can use Terraform’s ignore_changes lifecycle meta-argument. In the snippet below, there is an example to demonstrate this with a DynamoDB table set to ignore changes:

resource "aws_dynamodb_table" "service_table" { 
 for_each = toset(local.stack_refs) 
 name = "bg-service-table-${each.value}-${var.environment}"
 billing_mode = "PAY_PER_REQUEST" 
 hash_key = “uuid"
 attribute { 
  name = "uuid" type = "S" 
 } 
 lifecycle {
  ignore_changes = all 
 } 
}

The effect of setting ignore_changes to the special value of all is that Terraform will only create or destroy the resource, but it won’t modify the table in place. In the after hooks of our CodeDeploy deployments, we can delete the set of resources associated with the old version of the service (a or b), so when Terraform apply next runs, it will recreate them with the latest parameters, but more critically , it will not modify the existing version of those resources even if any of the parameters have changed. Additionally, we can use a SSM parameter that can be set in the after hook of the deployment to determine whether the a or b set of resources is now live and active. Terraform can read that value in using a data source and use it to decide if the property on a resource should be updated or not. If it shouldn’t be updated, we can read the current value from the remote state or a data source and use that instead of the new value.

To solve problem 2, again we can make use of ignore_changes but this time on the specific parameter, that will be managed by CodeDeploy hooks instead, for example ignoring the container definition changes on an ECS task:

resource "aws_ecs_task_definition" "api" { 
 family = “service"
 requires_compatibilities =["FARGATE"]
 cpu = var.cpu 
 memory = var.memory 
 container_definitions = data.template_file.container_definitions.rendered 
 execution_role_arn = aws_iam_role.ecs_task_execution.arn 
 task_role_arn = aws_iam_role.ecs_task.arn 
 network_mode = var.network_mode 
 lifecycle { 
  ignore_changes = [container_definitions] 
 } 
}

Now that we can have Terraform and CodeDeploy set up in a way to not conflict with each other, we now need to orchestrate running one after the other. Initiating a CodeDeploy deployment involves creating an appSpec config that defines the resource to update along with what hooks to run, then calling the createDeployment API with the appSpec passed in. This action can be performed using a Lambda, invoked through a CloudFormation stack with a custom Lambda resource. The CloudFormation stack can reside in the Terraform code itself as a resource and should depend on every other Terraform resource to ensure it’s the very last thing that should be applied. Any metadata required for setting up the CodeDeploy deployment can be passed to this Lambda via Terraform and the CloudFormation stack’s custom resource properties. The Lambda can then create the deployment by having it invoked as part of the CloudFormation stack update. The effect of this is that as the apply operation is completing on the Terraform front, a new deployment is created in CodeDeploy which immediately starts running.

There is another gotcha to watch out for here however. While the Terraform apply operation is running, Terraform will lock its remote state to stop other deployments occurring at the same time as a safety mechanism to avoid infrastructure corruption. However, since we are doing blue/green deployments, the deployment hasn’t completed when Terraform completes, there is now a CodeDeploy deployment running along with the hooks to perform all the setup and teardown hooks. This means that another CI deployment can be initiated behind the current one, which would start the Terraform plan and apply operations while CodeDeploy may still be running and the hooks are still modifying or deleting resources. This can result in your service ending up in a completely broken state, which is obviously not the intention. To prevent this, there are several strategies:

• Concurrency control in CI – ensure that your workflows only ever allow for one deployment pipeline to run at any point in time. This can be achieved quite easily using GitHub Actions’s concurrency control as an example. This control should be used with a waiting script as suggested in the next strategy
• Script based manual terraform locks – have a simple script that is executed as a step in your CI pipeline after both Terraform has finished the apply operation and the CodeDeploy deployment has completed to lock and unlock the state lock respectively
• Script based CodeDeploy wait – a simple script that runs after the apply operation and will poll the status of the CodeDeploy deployment until it either succeeds or fails. The CodeDeploy deployment ID can quite easily be retrieved from the CloudFormation stack’s output value which can be set by the deployment Lambda. If the CodeDeploy deployment fails, the script can exit with a non-0 exit code causing the pipeline to fail
• null_resource – use this resource within Terraform so that waiting for CodeDeploy is also done as part of the apply operation

Using these strategies together in the deployment pipeline can allow for safe and non-overlapping deployments. It also stops the pipeline from showing as complete when the CodeDeploy job is still running. Since the pipeline can now account for both Terraform and CodeDeploy status checks, the overall deployment success/failure can be correctly displayed in CI.

Everything we have talked about so far has been quite focused on how to get these tools to work well together. But to really give you an idea of what is actually going to happen during a deployment, it helps to look at some production-like examples. We’ll describe the flow for 2 services, one an ECS Fargate based API backed by an RDS cluster and ingestion Lambdas, the other a Lambda based API backed by DynamoDB and ingestion Lambdas.

This service has an API running as an ECS Fargate service fronted by an ALB, reading data from an RDS cluster which is seeded by an ingestor Lambda. A single running instance of the service looks like this:

Since we need to run side by side during a blue/green deployment, the resources created by Terraform after an apply will look more like this:

During the deployment process, CodeDeploy will start up another ECS API TaskSet which reads from the Catalogue B database, attach it to another listener on the load balancer and move the traffic over from the older version to the new version. To complete the deployment, numerous other steps must be carried out by the CodeDeploy lifecycle hooks.

Prepare new version

1. The replacement database cluster is created by Terraform during the Terraform apply operation
2. Autoscaling minimum capacity on the new database cluster is set to match the current capacity of the live cluster by the BeforeInstall hook, this ensures that when production traffic hits the new stack, the database cluster is warmed up and there is enough capacity to handle the present load
3. Replacement database is initialised by running schema migrations in the BeforeInstall hook
4. Data sources such as EventBridge rules or SNS subscriptions which feed the SQS data messages queue are enabled for the replacement queue in the BeforeInstall hook
5. SQS event source mapping to the data investor Lambda is enabled for the replacement queue in the BeforeInstall hook
6. A new task set is created in the ECS service, and is bound to replacement stack’s database, CodeDeploy creates this task set by being provided a task definition containing environment variables for the replacement set’s version resources
7. An AfterInstall hook sends a message to an upstream system to trigger the new version’s data ingestor Lambda to seed the complete dataset required for the service
8. The new task set is placed in a testing target group which is attached to a testing port on the load balancer by CodeDeploy

Test and shift traffic to new version

1. Validation and automated test steps are performed against the replacement version of the service using the testing port by the AfterAllowTestTraffic hook. These steps can include sending traffic to ensure the service is responding. For really advanced checks, you can scrape a sample of the live requests logs, and replay them against the live and replacement versions of your service to ensure the responses match fully or to a specific threshold. This process can be retried multiple times whilst the ingestion process takes place and the database is seeded since it’s likely the first few runs will always fail. To go even further, if you have semantic versioning of your service in place, you can even conditionally bake in scenarios to ignore breaking changes of responses as an example. The possibilities are endless since you are able to perform any customised validation inside the Lambda. All CodeDeploy cares about is that the hook either succeeds or fails.
2. The replacement stack becomes the live and active receiving traffic on the primary port. What was the live stack is now the old stack and CodeDeploy manages this by swapping the live ALB listener on the primary port 80

Teardown and deactivate old version

1. Autoscaling the minimum capacity on the active set is reverted to the normal value in the AfterAllowTraffic hook, and the database will now scale in when traffic lower
2. Data sources subscriptions like EventBridge rules and SNS subscriptions are disabled for the old stack in the AfterAllowTraffic hook so that the old data messages SQS queue is no longer being used
3. The SQS event source mapping on the data ingestor Lambda is disabled by the AfterAllowTraffic hook. This will be enabled by Terraform on the next deployment to achieve the desired state.
4. The old database cluster is deleted in the AfterAllowTraffic hook
5. The old SQS queue is purged of any messages in the AfterAllowTraffic hook ready for use in the next deployment
6. The old ECS task set is destroyed by CodeDeploy

The majority of the operations are carried out by the hooks, with CodeDeploy mostly acting as an orchestrator to shift traffic and trigger the hooks at various points in the deployment lifecycle to manage all these operations. Since this is an ECS deployment, we have a lot of choice of hooks to use, and use 4 of them in this example.

There is also a sample demo service we have put together to showcase how a simple ECS based service backed by DynamoDB and data ingestors looks like as code –  https://github.com/fourTheorem/bg-tf-ecs-demo – complete with Terraform resource pairs, custom CloudFormation stacks to invoke a CodeDeploy deployment, CI and hooks.

This service has an API running as a Lambda fronted by v2 HTTP API Gateway and backed by DynamoDB tables for holding service and config data which is ingested by Lambdas using SQS and Kinesis:

The resources created by Terraform after an apply will look like this:

As this is a Lambda target for a CodeDeploy deployment, CodeDeploy is going to be manipulating Lambda aliases instead of ECS task sets and target groups. Sending any validation test traffic to the new version service is going to involve pointing a new API Gateway stage at the new version, whilst the live stage will always point to the live alias. The following steps will occur during the deployment:

Prepare new version

1. DynamoDB tables are created for the new version of the service along with deployment and new tag versions of the ingestor Lambdas, SNS/SQS subscriptions and event source mappings by Terraform
2. The BeforeAllowTraffic hook sends a message to an upstream system to trigger the new version’s data ingestor Lambda to seed the complete dataset required for the service.
3. The similar can be done for Kinesis config source or, alternatively no actions are required for the config ingestor since the stream will have data retained by ensuring it has the full dataset being added at least once before the retention period of the stream expires. When the event source mapping is created on each deployment, you can simply use TRIM_HORIZON as the starting position and ingest data in an idempotent manner. A word of caution, this only works if the service is able to ingest that data at a relatively quick speed since there may be huge amounts of data retained in the stream.

Test and shift traffic to new version

1. The BeforeAllowTraffic hook starts to perform validation steps on the new version of the service. Again, similar to ECS, you can get quite advanced here by scraping a sample of request logs and replaying them against both versions of the service and ensuring responses match. With API Gateway specifically, you can associate the live and next stages each with an API Gateway endpoint which can then be used by the hook as endpoints to distinguish the 2 versions of the service when sending test traffic.
2. CodeDeploy modifies the live alias of the API Lambda to point to the new Lambda version of the API handler, which routes all prod traffic to the new version. Since this service uses DynamoDB on demand billing mode, capacity shouldn’t be a major concern, especially if the service is ultimately frontend by a CloudFront distribution. However, if provisioned throughput was being used, there would have to be some mechanism in place in the hook to match the present throughput if the table is autoscaled.

Teardown and deactivate old version

1. The AfterAllowTraffic hook deletes the old dynamo tables
2. The  AfterAllowTraffic disables the old SNS/SQS subscriptions and deletes the associated event source mapping on the data ingestor Lambda
3. The AfterAllowTraffic deletes Kinesis event source mapping on the old config ingestor Lambda
4. The old SQS queue is purged of any messages in the AfterAllowTraffic hook ready for use in the next deployment
5. The AfterAllowTraffic updates the active version stored in the SSM parameter to a or b dependent on what version just became active

There are only 2 hooks available for Lambda based CodeDeploy deployments, and as a result, the setup hook is going to be doing a lot of tasks. The hooks have retry mechanisms baked in to ensure the Lambda can reinvoke itself manually if timeout limits are approaching and other sensible defaults.

Hopefully, this post has inspired you to start thinking about blue/green deployments with your own serverless based AWS services. It’s worth emphasising that this is just one approach to deployments using this strategy and certainly not the standard by any means. We expect more refined tooling and services offered by AWS in the future to help developers to build their suite of deployment tools and promote these types of deployment strategies. An example of this is the recent announcement of fully managed blue/green for Amazon RDS which we believe is a step in the right direction but still doesn’t solve the problem for all other resources belonging to an application’s stack.

There is a good deal of gaps and assumptions in this post that may not apply to your own services, but it’s certainly a great starting point to performing safe deployments and reducing the risk of things going wrong. We are confident in this approach as it has been tried and tested in production environments over many years on mission critical services. If you too want to start implementing blue/green as a deployment strategy, do take a look at the sample demo repository at https://github.com/fourTheorem/bg-tf-ecs-demo. This is an example of an ECS based service with blue/green deployment and includes all of the techniques related to CodeDeploy with ECS services that have been stated in this post.

Finally, if you need help with your AWS projects, don’t hesitate to reach out to us, we’d love to help!