Introduction
TeamCity offers seamless integration with AWS, allowing you to deploy on AWS quickly and efficiently. Until now, the main problem with integrating TeamCity and AWS has been that all AWS-related features required dedicated credentials. In most cases, this meant that you had to manage static IAM keys for each feature instance. That increased the chances of keys being leaked and made managing them a burden.
We’ve introduced a new feature called AWS Connection, which makes it possible to work with AWS credentials more flexibly and safely.
Our main goal with AWS Connection is to make it possible to use one set of credentials everywhere they are required. Additionally, while working on AWS Connection, we have introduced multiple features that simplify the experience of working with AWS and improve security.
Let’s take a closer look at what’s new. This feature is available starting from TeamCity 2022.10.
Session credentials
TeamCity now provides temporary AWS credentials by default when you need them instead of exposing saved static ones.This drastically reduces the impact if credentials are leaked via the build log or during your builds. Temporary credentials will be rotated automatically and frequently and will have the same permission set as the permanent ones. You can find more information in the AWS documentation.
Assuming IAM roles
It is a good security practice to authorize only actions that are necessary, following the principle of least privilege. In other words, you shouldn’t use administrator privileges in cases where you could use those of a user.
That’s why we have updated how TeamCity works with identity and access management (IAM) roles. In addition to common session credentials, you can assume an IAM role to get a specific permission set. This set also provides session credentials but with specific role permissions.
For example, imagine you have two roles. One role is for storing data in an AWS S3 bucket. The second role is for sending AWS SNS messages. You can now create an IAM user with no permissions and allow this user to assume both of these roles.
To do this in TeamCity, you need to create three AWS connections: one with static user credentials, one to assume the first role using the initial connection, and one for the second role, again using the first connection.
You can now use different permission sets for different cases. This approach allows you to use only one static key for a variety of situations.
Default provider chain
“Default provider chain” is the name AWS uses for their SDK feature that looks up credentials in specific places, making it possible to completely avoid using static keys. This means TeamCity administrators can provide system-wide credentials externally, and the AWS SDK will use those credentials automatically if it finds them.
This feature opens the door for a new use case where TeamCity does not store any static credentials at all. In this case, project administrators can provide all specific permissions with IAM roles.
Expose credentials to builds
The new build feature allows you to specify an AWS connection and expose credentials automatically. The AWS SDK-based code will detect them automatically. This is useful for anyone interacting with the AWS API from their builds.
Key rotation
There is a security concern associated with using static credentials. They can be leaked. For this reason, it’s good practice to rotate static credentials as often as possible. Sadly, this best practice is not always adhered to.
That’s why TeamCity now provides the ability to rotate AWS keys on request, right from the UI. This ensures that even in the scariest scenario of a TeamCity backup with encryption keys being leaked, malefactors won’t get working AWS keys because the project administrator will have already rotated them.
The confused deputy problem
Despite these advantages, there is a problem. If all AWS accounts authorize the TeamCity instance itself to do something in them, how can we prevent it from being misused with malicious intentions?
At first glance, there is nothing to prevent one project from using the AWS resources of another. This problem is known as the “confused deputy” problem.
TeamCity solves this problem by providing an external ID with each API call to AWS, with the external ID being a unique string for each AWS connection. Unique external ID strings allow AWS administrators to restrict operations only to principals who provide the proper unique mark.
HashiCorp Vault and AWS Connection
TeamCity can retrieve secrets from the HashiCorp Vault. The Vault itself can provide static or session AWS keys.
At JetBrains, we use the Vault a lot, as it has been the recommended solution for getting AWS credentials for years. The introduction of AWS Connection thus raises the question of which solution to use.
The answer is simple. You don’t need the Vault if your builds only require AWS credentials. Moreover, the integration with the Vault is limited and cannot be used everywhere you need AWS credentials, specifically with other TeamCity features.
TeamCity now provides the same level of security and feature parity as the Vault when working with AWS credentials, and it also covers the cases where the Vault cannot help.
Conclusion
We haven’t yet finished switching all existing AWS-related features to AWS Connection. The next step is to support AWS Connection within all the plugins available in TeamCity. This will take the burden of manually configuring access to EC2, ECR, S3, and other resources off your team’s shoulders.
While we’re working on this feature, you can already start benefiting from it. Don’t hesitate to provide your feedback as a comment to this post, in the forum, or in the tracker.