Part 2 – Managing accounts using IaC with OrgFormation

In our previous article, we learned how to set up IAM Identity Center and how to use Granted to get credentials for one of our accounts when working with the CLI.

That article served as an introduction to the various concepts that we need to get familiar with when dealing with AWS Organizations and IAM Identity Center. But we did lots of things manually and the problem with manual things is that they are not repeatable and don’t scale.

In this article, we will learn how to manage multiple AWS accounts in a much more reliable and scalable way using infrastructure as code (IaC). In particular, we will deep-dive into the beautiful world of OrgFormation, an open-source tool that brings the full power of infrastructure as code to AWS Organisations.

At this point, you might be wondering why do we need a new tool to do Infrastructure as Code and why can’t we rely on CloudFormation alone.

And that’s a great question! It would be awesome if we could just use CloudFormation to fully manage all our accounts and our landing zone. But don’t worry, we are not going to give up on our CloudFormation knowledge. CloudFormation is still going to be an integral part of our toolchain when it comes to managing organisations, but there are quite a few things that CloudFormation alone doesn’t make too easy. This includes:

• Creating accounts and organising them into OUs
• Creating service control policies for various accounts
• Creating shared resources or resources that need to exist in every single account or a subset of your accounts

The main obstacle with CloudFormation is that, when you apply a CloudFormation template, that template is executed in the context of one account and one region, and the syntax of the template does not allow us to reference resources in other regions or accounts.

OrgFormation wraps CloudFormation and extends its capabilities providing us with a more seamless experience. We can keep all our infrastructure as code related to our organisation in one place and perform all the necessary changes with just a few CLI commands or through an automation pipeline, regardless of which accounts and regions we intend to update.

The syntax of OrgFormation is a superset of the CloudFormation syntax, so we can still do everything that we can do with CloudFormation using the syntax that we are familiar with. And we can do even more using the additional syntax provided by OrgFormation. The terminology often used in the official OrgFormation docs is “OrgFormation annotated CloudFormation templates”.

But before deep diving into OrgFormation, let’s see how it compares to another relevant AWS tool: Control Tower.

AWS Control Tower is an AWS service designed to help you set up and manage a secure, multi-account AWS environment based on AWS best practices. Through the AWS web console, you can use Control Tower to automate the creation of a landing zone, including account structure, governance controls, and automated account provisioning. It is very simple and intuitive and it doesn’t require an extensive knowledge of the subject to be used effectively. For this reason, Control Tower can be a great choice for quickly establishing a compliant and secure AWS environment with minimal setup effort.

But the main focus of Control Tower is creating new accounts, and therefore, it is sometimes called an “account vending machine”. This kind of tools are great for getting started, but they offer very limited functionality when it comes to evolving the account structure and the related resources as your business and compliance needs evolve.

For this reason, if you are planning to create an organisation that you will need to maintain long-term for enterprise use cases and you want to use Infrastructure as Code to do that, OrgFormation is a more appropriate tool that can cover the full lifecycle of your organisation, from getting started to maintaining things long-term.

But if you like Control Tower, you can still use it to bootstrap your organisation and then move to OrgFormation for longer-term maintenance. The two tools are not mutually exclusive and OrgFormation makes it easy to work with existing organisations.

OrgFormation is written in JavaScript and runs on Node.js, so before you can get started, make sure to have a recent version of Node.js installed on your system.

Once you have done that, you can run:

npm install -g aws-organization-formation

This command will download and install the org-formation CLI helper in your system.

To make sure that everything was done correctly, you can run the following command:

org-formation --version

This command will print the current version of the tool (at the time of writing, the latest version is 1.0.16).

You can also run:

org-formation --help

To get some help on how to use the CLI tool and what commands are available.

Now that we have OrgFormation installed in our system, we can create a new project by running the following command (make sure to do that with admin credentials for the management account):

org-formation init organization.yml --region eu-west-1

This command will generate a YAML file (organization.yml) containing a definition of the current accounts structure. This should contain all of the accounts and OUs that we created manually in the first article of this series. It should look like this:

# organization.yml

AWSTemplateFormatVersion: '2010-09-09-OC'
Description: default template generated for organization with master account 730335603479

Organization:
  ManagementAccount:
    Type: OC::ORG::MasterAccount
    Properties:
      AccountName: loigetemp
      AccountId: '730335603479'
      RootEmail: '[REDACTED]'

  OrganizationRoot:
    Type: OC::ORG::OrganizationRoot
    Properties:
      DefaultOrganizationAccessRoleName: OrganizationAccountAccessRole

  MarketplaceOU:
    Type: OC::ORG::OrganizationalUnit
    Properties:
      OrganizationalUnitName: Marketplace
      Accounts: !Ref MarketplaceAccount

  SandboxOU:
    Type: OC::ORG::OrganizationalUnit
    Properties:
      OrganizationalUnitName: Sandbox
      Accounts:
        - !Ref ExperimentsAccount
        - !Ref WorkshopsAccount

  SharedOU:
    Type: OC::ORG::OrganizationalUnit
    Properties:
      OrganizationalUnitName: Shared
      Accounts: !Ref SecurityAccount

  WorkloadsOU:
    Type: OC::ORG::OrganizationalUnit
    Properties:
      OrganizationalUnitName: Workloads
      Accounts:
        - !Ref ProductionAccount
        - !Ref QaAccount
        - !Ref DevelopmentAccount

  DevelopmentAccount:
    Type: OC::ORG::Account
    Properties:
      AccountName: Development
      AccountId: '992382505208'
      RootEmail: '[REDACTED]'

  ExperimentsAccount:
    Type: OC::ORG::Account
    Properties:
      AccountName: Experiments
      AccountId: '992382488722'
      RootEmail: '[REDACTED]'
      
# ... truncated for brevity

Now, the first cool thing that we can do with OrgFormation is to add a new account in this file, for example:

# organization.yml

Organization:
  # ...
  
  IntegrationAccount:
    Type: OC::ORG::Account
    Properties:
      AccountName: Integration
      RootEmail: youremail+aws-integration@gmail.com

There are a couple of things worth pointing out here:

• As we have to do when creating accounts manually, we need to provide a unique email for the account
• We don’t need to specify an AccountId property. This makes sense, because AccountId is something that AWS generates for us when the creation of the new account is completed. This file follows the same principles of Infrastructure as Code, we are describing a desired state, one that needs to be explicitly applied by running a specific command.

But before applying our changes, it’s interesting to see that we can also add this new account to an OU, for example the WorkloadsOU:

# organization.yml

Organization:
  # ...
  
  WorkloadsOU:
    Type: OC::ORG::OrganizationalUnit
    Properties:
      OrganizationalUnitName: Workloads
      Accounts:
        - !Ref ProductionAccount
        - !Ref QaAccount
        - !Ref DevelopmentAccount
        - !Ref IntegrationAccount  # <- new line!

Of course, we could also create new OUs if we want to.

Now, to make sure that our organization.yml file is valid we can run the following command:

org-formation validate-stacks organization.yml

And to apply the changes we can run:

org-formation update organization.yml

If all went as expected, we should see something in the output that looks like this:

WARN: Master account Id not set, assuming org-formation is running in the master account
OC::ORG::Account              | IntegrationAccount            | Create (381491877544)
OC::ORG::Account              | IntegrationAccount            | CommitHash
OC::ORG::OrganizationalUnit   | WorkloadsOU                   | Attach Account (IntegrationAccount)
OC::ORG::OrganizationalUnit   | WorkloadsOU                   | CommitHash
INFO: done

If we login to our AWS dashboard in our management account and go to AWS Organizations we should see the new integration account created, but there are some small issues we still need to address:

1. The organization.yml is not automatically updated to include the newly generated accountId. In reality, this is not a big deal, in fact if you try to run the update command again (without any change), it will recognise that everything is up to date and no action is needed. If you like to keep explicit account IDs in your organization.yml then you can simply copy it (e.g. from the output of the previous update command) and paste it into your organization file. The small advantage in doing this is that if you look at your organization file 3 months down the line, you can be sure that all the accounts with an AccountId property have been created in AWS.
2. If we go to the start page of our landing zone, we won’t see this new integration account. This makes sense because we haven’t defined permissions sets that give us access to it!
3. Similarly, our assume CLI command, is not going to be aware of the newly created account.

We will fix these last 2 points in a moment, but let’s introduce some important OrgFormation concepts first!

OrgFormation can be a bit intimidating at first because it brings to the table a few new concepts on top of what’s already supported by CloudFormation. So, let’s start by clarifying the most important features.

As part of your organisation’s IaC, you will want to deploy resources to multiple accounts and regions. Organization bindings are a way to dynamically define subsets of accounts to which you can deploy certain resources. In a way, we can think of it as a mini query language to select the accounts that are relevant for a given resource.

Let’s try to understand this concept with an example.

If you want to define an arbitrary resource in plain CloudFormation, let’s say a Lambda Function, you might be writing something like this in a plain CloudFormation template:

Resources:
  PrintLambdaEvent:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.lambda_handler
      Runtime: python3.12
      Code:
        ZipFile: |
          def lambda_handler(event, context):
              print(event)

This is a dead simple Lambda function written in Python that prints the incoming event.

Now let’s pretend for a second that this can be a useful Lambda for all our development accounts. After all, our devs might use it as a rudimentary debugging tool to inspect the shape of new event types. With this idea in mind, it might be useful to have this Lambda function available by default in all our development accounts.

We can do this with OrgFormation by using an organization binding. Let’s update our example to learn the syntax of this feature (this file can be saved in templates/test-lambda.yml):

# templates/test-lambda.yml

AWSTemplateFormatVersion: '2010-09-09-OC'

Organization: !Include ../organization.yml

OrganizationBindings:
  DevAccounts:
    OrganizationalUnit: !Ref DevelopmentOU
    Region: "eu-west-1"

Resources:
  PrintEventLambda:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.lambda_handler
      Runtime: python3.12
      Code:
        ZipFile: |
          def lambda_handler(event, context):
              print(event)
    OrganizationBinding: !Ref DevAccounts

Note that this snippet is now OrgFormation code, since we are extending the CloudFormation syntax with properties that are provided by OrgFormation. In other words, you won’t be able to use this file with just CloudFormation.

If we compare this snippet with the previous one, we can see that we added a few things:

• AWSTemplateFormatVersion: '2010-09-09-OC' : This indicates that we want to use the extended OrgFormation syntax
• Organization: !Include ../organization.yml: here we include the organisation file. We will see how to generate this file in the next section. For now, you only need to know that this file will contain the list of all the accounts and organisation units in our organisation. Let’s imagine that we will have defined an OU called DevelopmentOU that contains all our development accounts.
• OrganizationBindings : This is where the magic happens. In this section, we can define the various set of accounts that we are interested in referencing throughout the template. Every definition is dynamic. Again, you can think of it as a query that you can construct by using different properties available. In this particular example, we are using two properties: OrganizationalUnit to reference the Development OU and Region to limit the scope to one specific region (in this case eu-west-1).
• OrganizationBinding (in the resource): This is where we reference the set of accounts in which we want to deploy the Lambda function. Note how we are referencing the DevAccounts organization binding that we defined in the OrganizationBindings section of the template.

Other properties you can use to define organisation bindings other than Region and OrganizationalUnit are:

• Account: the ID of an account, a list of IDs of account or "*" to indicate all accounts (except the master account).
• IncludeMasterAccount: a boolean value that, if set to true , indicates that the master account should be included in the list of accounts where we want to deploy the resource.
• ExcludeAccount: It supports the same values as accounts, but it will remove the matching account from the list of accounts in which the resource is going to be deployed.
• AccountsWithTag: Allows us to select accounts with a specific tag.

If you want to know learn more about the properties that you can use when defining organisation bindings, you can check out the official documentation.

Org-formation allows you to define task files. These files are another kind of YAML file where you can define different actions that you might want to perform through Org-formation templates.

A sample task file (e.g. organization-tasks.yml) could look like this:

# organization-tasks.yml

OrganizationUpdate:
  Type: update-organization
  Template: ./organization.yml

TestLambdaResources:
  Type: update-stacks
  Template: ./templates/test-lambda.yml
  StackName: testLambda
  StackDescription: A simple test Lambda function that prints the received event

In this example we can see 2 different tasks. The first one, with type update-organization allows us to update our organization structure (e.g. create new accounts or OUs). This is effectively an alias for what we did before when we executed: org-formation update organization.yml.

The second task if of type update-stacks and it references our test Lambda template that we defined before. What this task does is effectively applying the template based on the organization bindings defined in it. It also provides a stack name and a stack description that is applied to all the matching accounts and regions where the stack is deployed.

We can now execute all the tasks in this file by running:

org-formation perform-tasks organization-tasks.yml

This will update our organization structure (creating any new account or OU that we might have added in the organization.yml file) and deploy our test lambda in all the development accounts.

As we said, OrgFormation extends the CloudFormation language with additional features. If you are curious to find out more about what’s supported, make sure to check out the official documentation.

One thing you might have noticed is that we were able to import all our accounts easily by generating our organization.yml file, but currently we have no reference to our users, groups and permission sets.

Unfortunately there is no automated way (that I am aware of) to import all of this stuff into our OrgFormation code, so we have 3 options:

1. 1. Start from scratch: Delete all your AWS SSO configuration and recreate everything as IaC in your OrgFormation template. Of course, this means that you might have some downtime where your users might not be able to access specific accounts or use specific profiles.
   2. Manage forward: Leave your existing AWS SSO configuration as is and start managing new resources with OrgFormation. This is generally not recommended as it will lead to a split-brain configuration where some resources are managed by OrgFormation and some are managed manually.
   3. Manual import: Manually import all your AWS SSO configuration into your OrgFormation template. This is probably the best approach because it allows to keep your existing resources and start managing them with OrgFormation. The problem is that this is a very manual and error-prone process. After all, importing resources in an existing CloudFormation stack is like performing brain surgery with a spoon! 🧠😨 So, unless you are a CloudFormation ninja (or a brain surgeon), you should probably avoid this approach.

We are working on an experimental script that should allow you to import groups, permission sets and assignments automatically. If you feel brave enough, we you can give that a shot, but for the rest of this article we will proceed with a more manual approach.

But before doing that, let’s recap all the entities we have to think about. The following diagram tries to represent them in a Entity-Relationship diagram:

Let’s walk through it.

At the center, we have AWS::SSO::PermissionSet. As we learned in the previous article, this is an entity that allows us to define a set of permissions and some other properties related to the user session that is going to assume these permissions like session duration and starting URL (RelayStateType).

These permission sets need to be assigned to combinations of users (or groups) and accounts. This can be done by creating AWS::SSO::Assignment resources. Every one of these resources gives a specific user (or group) the ability to access a specific account with a set of permissions.

Let’s see a quick example (templates/sso-assignments.yml):

# templates/sso-assignments.yml

AWSTemplateFormatVersion: '2010-09-09-OC'
Description: SSO Assignments for Users and Groups

Organization: !Include ../organization.yml

OrganizationBindings:
  ManagementAccountBinding:
    # Only the management account
    IncludeMasterAccount: true

DefaultOrganizationBinding: !Ref ManagementAccountBinding
DefaultOrganizationBindingRegion: eu-west-1

Parameters:
  IdentityStoreId:
    Type: String
    Default: "REPLACE THIS WITH YOUR IDENTITY STORE ID"
  ManagingInstanceArn:
    Type: String
    Default: "REPLACE THIS WITH THE ARN OF YOUR IDENTITY CENTER"

Resources:
  # Groups
  AdminGroup:
    OrganizationBinding: !Ref ManagementAccountBinding
    Type: AWS::IdentityStore::Group
    Properties:
      Description: Administrator Access
      DisplayName: Admin
      IdentityStoreId: !Ref IdentityStoreId
      
  # PermissionSets
  AdministratorAccessPermissionSet:
    OrganizationBinding: !Ref ManagementAccountBinding
    Type: AWS::SSO::PermissionSet
    Properties:
      Name: AdministratorAccess
      Description: "Administrator access"
      InstanceArn: !Ref ManagingInstanceArn
      ManagedPolicies:
        - arn:aws:iam::aws:policy/AdministratorAccess
      SessionDuration: PT4H
      
  ReadOnlyAccessPermissionSet:
    OrganizationBinding: !Ref ManagementAccountBinding
    Type: AWS::SSO::PermissionSet
    Properties:
      Name: ReadOnly
      Description: ReadOnlyAccess
      InstanceArn: !Ref ManagingInstanceArn
      ManagedPolicies:
        - arn:aws:iam::aws:policy/ReadOnlyAccess
      SessionDuration: PT4H
      
	# Assignments
	AdminGroupToAdministratorAccessPermissionSetToManagementAccount:
    OrganizationBinding: !Ref ManagementAccountBinding
    Type: AWS::SSO::Assignment
    Properties:
      InstanceArn: !Ref ManagingInstanceArn
      PermissionSetArn: !GetAtt AdministratorAccessPermissionSet.PermissionSetArn
      PrincipalId: !GetAtt AdminGroup.GroupId
      PrincipalType: GROUP
      TargetId: !GetAtt ManagementAccount.AccountId
      TargetType: AWS_ACCOUNT
	
	AdminGroupToReadOnlyAccessPermissionSetToManagementAccount:
    OrganizationBinding: !Ref ManagementAccountBinding
    Type: AWS::SSO::Assignment
    Properties:
      InstanceArn: !Ref ManagingInstanceArn
      PermissionSetArn: !GetAtt ReadOnlyAccessPermissionSet.PermissionSetArn
      PrincipalId: !GetAtt AdminGroup.GroupId
      PrincipalType: GROUP
      TargetId: !GetAtt ManagementAccount.AccountId
      TargetType: AWS_ACCOUNT
      
  # ... truncated for brevity

Now that we have this file, we can create a task that allows us to propagate any change we have done to our AWS accounts ( organization-tasks.yaml ):

# organization-tasks.yml

OrganizationUpdate:
  Type: update-organization
  Template: ./organization.yml

SsoAssignments:
  Type: update-stacks
  Template: ./templates/sso-assignments.yml
  StackName: SsoAssignments
  StackDescription: Manage SSO Assignments

And, finally we just need to run:

org-formation perform-tasks organization-tasks.yml

to get the changes deployed! 🎉

It’s important to realise that the number of assignment resources can easily grow and become very tricky to manage effectively. In our example from the previous article we have defined 8 different accounts. Another was was just added during this tutorial, which brings us to a total of 9 different accounts. Let’s imagine we have an Admin group which should get Admin and ReadOnly access to all of these accounts. If you do the maths, you can see that this means creating 18 different assignment resources! This might become tricky to maintain, even when using infrastructure as code, unless we find some abstraction mechanism.

OrgFormation allows us to use a powerful templating language called Nunjucks that can be used to render repeated parts of our yaml code with an iteration syntax (for loop). While this could be a valid approach, it can create complexity in our templates, so use this tool with moderation!

The aws-sso-util macro

Thankfully there’s an alternative approach, we can use an open-source CloudFormation macro called aws-sso-util which allows us to use a higher-level virtual resource (SSOUtil::SSO::AssignmentGroup) that can easily map multiple users (or groups) to multiple permission sets and accounts. Behind the scenes, when we deploy this resource, it will manage all the necessary AWS::SSO:PermissionSet resources.

To install this macro in the management account, you need to have SAM installed and then run the following commands:

git clone https://github.com/benkehoe/aws-sso-util
cd aws-sso-util/macro
sam build --use-container sam deploy --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM CAPABILITY_NAMED_IAM --stack-name aws-sso-util-cf-macro --guided

The --guided option will prompt you to specify all the various configuration options. In my experience, the default values should work in most cases. Also, note that the --use-container option will require you to have docker installed and running.

If all goes well, you should see the following output in your terminal:

Successfully created/updated stack - aws-sso-util-cf-macro in eu-west-1

Once you have done all of this, you can now use the AWS-SSO-Util-2020-11-08 CloudFormation macro which allows for a more concise syntax to define assignments. Let’s see an example:

# templates/sso-assignments.yml

AWSTemplateFormatVersion: '2010-09-09-OC'
Description: SSO Assignments for Users and Groups
Transform: AWS-SSO-Util-2020-11-08 # Enables the macro!

# ...

OrganizationBindings:
  AllAccountsBinding:
    Account: '*'
    IncludeMasterAccount: true

  ManagementAccountBinding:
    IncludeMasterAccount: true
    
# ...

Resources:

  # ...
  
  AdminGroupAssignments:
    OrganizationBinding: !Ref ManagementAccountBinding
    Type: SSOUtil::SSO::AssignmentGroup # Special type introduced by the macro
    Properties:
      Name: AdminGroupAssignments
      InstanceArn: !Ref ManagingInstanceArn
      Principal:
        - Type: GROUP
          Id:
            - !Ref AdminGroup
      PermissionSet:
        - !GetAtt AdministratorAccessPermissionSet.PermissionSetArn
        - !GetAtt ReadOnlyAccessPermissionSet.PermissionSetArn
      Target:
        - Type: AWS_ACCOUNT
          Id:
            - Fn::EnumTargetAccounts AllAccountsBinding ${account}

With just one (virtual) resource we are able to manage all the assignments for the Admin group. In our example this will effectively expand to all the 18 assignment resources for us!

Note that in the last line we are using a special OrgFormation function: Fn::EnumTargetAccounts. This function allows us to list all the accounts in a given account binding. In this case we are using the AllAccountsBinding which will fetch all the existing accounts in our organization.

If you want to find out more about Fn::EnumTargetAccounts and other special OrgFormation functions, check out the official documentation.