Well, That Escalated Quickly

Posted by Dorien Koelemeijer on November 14, 2021

Just like the cliché says, security is only as strong as the weakest link. We know that a single Identity and Access Management (IAM) misconfiguration in our AWS environment can lead to compromise of our entire cloud environment. It’s the task of the Security Team at Afterpay to manage this risk, while at the same time making sure that engineers are not slowed down in their everyday tasks. To tackle this (not-so-easy) problem, we have developed an automated and scalable solution to consistently and flexibly manage IAM in our AWS accounts.

Motivation

AWS IAM is at the heart of AWS Security, and as our organisation scales, we need to ensure a robust and consistent IAM model across our AWS accounts. One of the most common tactics attackers use to escalate privileges in cloud environments is to abuse overly permissive identity and access policies. Cloud Cover was born to reduce the chance of IAM misconfigurations in our AWS environments.

Goals

When we designed Cloud Cover, a few things were important to us:

Integrate well-known security best practices, such as the principle of least privilege, into our IAM model
Ensure engineers get their work done securely, but without unnecessarily compromising their efficiency. Part of this requirement for us is to allow engineers to request additional IAM permissions in a self-service, but safe and governed, way
Leverage Okta (our single sign-on solution) as the source of truth for IAM role federation (i.e. which engineers can assume which roles)
Have a central place where we can keep track of, and control permissions. This includes keeping an audit trail to see who requested certain permissions and when these were deployed
Introduce the ability to provide automatic, day one access (birthright) to the AWS accounts, and make it easy for IT support staff to process access requests quickly

What is Cloud Cover?

Cloud Cover is a tool that provides employees access to our AWS accounts through the use of temporary IAM role credentials. We use AWS roles that can be assumed, rather than users, to prevent the use of long-lived static credentials. IAM roles and policies are defined by Cloud Cover as infrastructure as code (IAC), and are deployed into our AWS accounts using a CI/CD pipeline.

IAM roles deployed by Cloud Cover are mapped to Okta groups, and engineers are assigned to these groups based on their team and responsibilities. We developed a set of AWS IAM base roles that users can federate into using Okta, which ensure least-privilege permissions. The goal was to capture 90% of the use cases for engineers who work with AWS with these base roles. As this base set of roles will not fit all use cases, individual teams can attach missing permissions to roles in their AWS accounts. Additional permissions can be requested by creating a pull request in Cloud Cover”s Github repo.

Endorsed and Custom Policies

To allow engineers to request additional IAM permissions and attach these to their base roles, Cloud Cover makes a distinction between “endorsed” and “custom” policies in its GitHub repo.

“Endorsed” policies are policies that are endorsed by the Security Team and are deemed secure enough to be used freely. It is our intent that these policies can be attached to any role definition without requiring approval through pull requests.

“Custom” policies can be created by individual teams when no endorsed policies fit their use case. These policies are defined in the Cloud Cover Github repo in the AWS account folder of the team owning that particular AWS account. The creation and attachment of “custom” policies requires approval, via a pull request in the Cloud Cover GitHub repo, before additional permissions can be granted.

Service-Based Permissions

To further enforce the principle of least privilege, some role policies are constrained to prefixes of services that are deployed in an AWS account. An example where this may be useful is to have more granular control over what secrets in AWS Secrets Manager can be accessed by different IAM roles within an AWS account, based on the services deployed there (i.e. only secrets for certain services can be accessed by engineers in a given AWS account).

Permissions Boundaries

The roles and policies managed by Cloud Cover are to a great extent only applicable to roles humans federate into. However, the principle of least privilege should not only apply to “human” IAM roles, but should also be applied to service roles created by these human roles. Therefore, the current Cloud Cover base roles can only create machine roles under certain conditions. An example of a condition is that roles can only be created if the new role has an AWS Permissions Boundaries attached to it, to restrict what actions that new role can perform. Another condition is that roles can only be created and passed to roles and policies that are prefixed with certain “prefixes” (as explained above).

It is important to note that there is no impact on the way machine roles are created using CI/CD pipelines, since these are checked for security issues prior to deployment using our code scanning tool Intersect.

Service Control Policy

To ensure that permissions attached to the Cloud Cover IAM roles are only defined as code in Cloud Cover”s GitHub repo, we have developed an AWS Service Control Policy (SCP) that prevents unauthorised modifications to the base roles in our AWS ecosystem.

Technical Details

Since all Cloud Cover IAM roles and permissions are defined as code, we had to come up with a way to convert the code into IAM roles in our AWS environment. To achieve this, a set of Python scripts was developed to create CloudFormation templates based on the role definitions in Cloud Cover, with the help of a tool called Troposphere.

Role definitions are specified in a yaml file for each AWS account in our organization. This role definition yaml file includes a specification of policies attached to the role (including assume role policies, permissions boundaries and regular role policies). These policies can comprise “endorsed”, “custom” or AWS managed policies.

engineer-role.yml

AssumeRolePolicy: endorsed:saml
Boundaries:
- custom:permissions-boundary
Policies:
- aws:PowerUserAccess
- endorsed:engineer-policy
- endorsed:iam-create-role
- custom:s3

Example of a role definition

For each account, the role definitions are fetched from that specific account’s folder, and validated. Subsequently, a CloudFormation template is created, containing all roles and policies as defined in the role definitions.

Whenever changes are made to role or policy definitions, and these changes are pushed to GitHub on feature branches, the Cloud Cover CI/CD pipeline validates the modifications to the IAM roles. The deployment of these changes occurs in this CI/CD pipeline as well, but only after the feature branch has been merged into the main branch.

An example of how permissions are managed and requested in Cloud Cover

Stay Tuned!

Cloud Cover was created to improve cloud hygiene and security, by providing an automated, self-service way of managing permissions and access to our cloud environment.

Over the course of the coming months, we’ll be rolling Cloud Cover out across the organisation. Stay tuned for our lessons learned!

We should also mention that we are growing! If you are the type of person that likes to solve tricky security problems at scale, check out our open roles or let us know what you’re looking for out of your next security role.

← →