Investigate Service Account Key Origins and Usage with Best Practices

gcp
IAM
access-control

Required permissions:

You’ll need the following IAM predefined roles to follow the steps discussed in this blog post:

iam.serviceAccountViewer

monitoring.viewer

logging.privateLogViewer

Cloud monitoring authentication events

Google’s Cloud Monitoring API allows you to view and query service account and service account key authentication events that occurred within the past six weeks. An “authentication event” is defined as any time a service account or service account key was used to call any Google API.

Google Cloud Console

If you know which service account you are interested in, you can navigate to the “service accounts” page for your desired project here. You can then click on a specific service account and view its metrics.

You’ll be shown authentication traffic for the account, separated by impersonation events and key authentication events, as well as data on the API calls made from the service account.

You can also view aggregated metrics for all service accounts and keys via the Cloud Monitoring Metrics explorer, located here.

Use the drop-down and search to select either the “Service account authentication events” metric or the “Service account key authentication” metric.

“Service account authentication events” will include requests directly from an attached service account or requests made via service account impersonation, while “Service account key authentication events” will include requests authenticated with a specific key.

Google also provides various options for filtering, grouping, and aggregating the data.

You can group by unique_id to view the data grouped by Service Account ID, or group by key_id to view the data grouped by Service Account Key ID.

Google REST APIs

To query service account authentication events directly from the Cloud Monitoring API you can use the projects.timeSeries.list API route (docs).

You’ll need to specify metric.type = "iam.googleapis.com/service_account/authn_events_count" or metric.type = "iam.googleapis.com/service_account/key/authn_events_count" depending on whether you are looking for key events or direct authentication events. By default, this call will return raw time series data grouped by service account ID and service account key ID, but you can also request aggregated data via the aggregation parameter.

Identity access monitoring

The Cloud Monitoring data provides us with a lot of insights into service account usage, but some details are missing. For example, who accessed these service accounts? Which of these accesses are human users and which are machines? To answer these questions, we can look to Google’s Data Access audit logs.

To view the data in this section, you’ll need to have had data access audit logs enabled in your project for the time period you wish to analyze. These logs are not enabled by default, but can be very helpful for analyzing access patterns in your cloud environment. See Google’s documentation for steps on how to enable data access logs.

Uncover service account key origins

One difficulty of understanding service account key usage is that there’s no way to know who has access to a key. Users may have downloaded it to their computer and shared it with others, which is something that you cannot track. However, you can make progress towards understanding which users have access to a key by determining who created the key, since Google only lets you download the key at creation time.

To do this, we can query the Logs Explorer to search for the CreateServiceAccountKey event with the desired key ID.


protoPayload.methodName="google.iam.admin.v1.CreateServiceAccountKey"
protoPayload.response.name="projects/[PROJECT]/serviceAccounts/[SA_EMAIL]/keys/[KEY_ID]"

You can narrow your search to the day the service account key was created by using the “Creation Time” field in the Cloud Console or returned by the IAM Service Account Keys API.

The resulting log will have an authenticationInfo field that will tell you who created the key.

Investigate service account impersonation

In the previous section, we covered obtaining timestamps for service account impersonation events. With logs, you can get more details on each event, including the principal that performed the impersonation and the specific request that was made.

Each audit log contains an authenticationInfo field with information on how the request was authenticated. If the method was service account impersonation, there will be a serviceAccountDelegationInfo field inside authenticationInfo, which contains an array of all the principals in the impersonation chain. You can use this to query for all impersonation events.

If you want to query for all impersonation events involving human users, you can use the following:


protoPayload.authenticationInfo.serviceAccountDelegationInfo.firstPartyPrincipal.principalEmail =~ "@[YOUR DOMAIN]"

You can also query for all impersonation events involving service accounts via:


protoPayload.authenticationInfo.serviceAccountDelegationInfo.firstPartyPrincipal.principalEmail =~ "iam.gserviceaccount.com"

Filter for a specific service account by adding the following to your query:


protoPayload.authenticationInfo.principalEmail="[SA_EMAIL]"

Examine service account key usage

Similarly, you can use the audit logs to get a more detailed log of service account key usage. Because there is no identity tied to the key usage, this unfortunately will not provide information on who used the key.

However, the audit log includes the IP address from which the request was made, which can allow you to further investigate the source of the key usage.

To search for all events on a specific key, you can use:


protoPayload.authenticationInfo.serviceAccountKeyName="//iam.googleapis.com/projects/[PROJECT_ID]/serviceAccounts/[SA_EMAIL]/keys/[KEY_ID]"

To search for events on any key, you can use the following:


protoPayload.authenticationInfo.serviceAccountKeyName =~ "//iam.googleapis.com/"

The resulting log will have a requestMetadata field that contains a callerIp field and a callerSuppliedUserAgent field. You can use these to try to determine whether the key is being accessed programmatically or by a human user: for example, by cross-referencing the caller IP with your workload IP addresses.

Take action and reduce risk with IAM best practices

It’s important to secure your service account permissions, because they may be allowing your users to have unintended access in your system and a common privilege escalation attack vector in Google Cloud is managing to authenticate as a privileged service account. Having excessively privileged service accounts in your infrastructure can leave you more vulnerable to both internal and external threats.

Here are some best practices to manage privileged access:

Delete unused service accounts.

Right-size service account roles. You should not have service accounts with “Editor” on your project. Instead, give them access to only the APIs that they’re being used to call. If you are using the same service account for a large range of functionality, instead consider replacing it with several different service accounts that each have more limited access.

Delete unused service account keys.

Avoid using service account keys unless absolutely necessary. Oftentimes, you can instead use service account impersonation or direct user credentials, both of which are more secure. See this flowchart from Google to understand alternative authentication methods you may be able to use.

Apply the Disable service account key creation and Disable service account key upload organization policy constraints.

Secure service accounts and keys for free with P0

Tired of having to check multiple Google APIs and querying logs in search for the information you need in order to secure your cloud configuration? Sign up for P0 for free and conduct your first IAM assessment in fifteen minutes or less. P0 will show you all your cloud principals (human and service accounts) along with all resources they have access to and information about their authentication history. P0 allows you to view service accounts and service account keys by usage easily, and will even alert you to changes in your security posture, such as new unused service account keys.