How Afresh automated access escalations, improving security and developer experience

About Afresh

Afresh is a San Francisco-based technology company that builds an AI-powered platform to help grocery stores reduce the amount of wasted produce. They serve the largest grocery chains in the United States, and their platform trains state-of-the-art machine learning models on terabytes of customer data daily.
‍

Meet the team

The challenge

To maintain compliance with SOC2 and other certifications, Afresh wanted to implement strict access controls on sensitive cloud systems and customer data. Before engaging with P0, Afresh faced several security and operational challenges with their IAM set up.

Over provisioned birthright access: Many systems (and users) were provisioned with greater access than necessary. In many instances, engineers, who needed occasional break-glass access, failed to relinquish the escalated privileges even after an incident was over. They attempted to solve this via Azure PIM, but the migration was not straightforward.

Overhead with configuring escalated access: Afresh wanted to move to a framework of just-in-time escalated access. However, setting up this framework in Azure PIM (Privileged Identity Management) put a lot of overhead on the platform team, due to a number of reasons:

• Afresh’s sensitive assets resided across different systems, such as Snowflake, Kubernetes, Azure and GitHub. All of these had very different RBAC models, and configuring Azure AD groups with the different categories of permissions required a lot of time, knowledge and effort.

• Like any typical cloud environment, Afresh had thousands of sensitive resources. Over time, attempts to implement granular access controls resulted in a sprawl of groups and roles across systems. Managing this complexity was very cumbersome.

• Some microservices, especially ETL jobs on production databases, caused changes to the permissions of individual objects in the datastores. This ‘permissions drift’ could be potentially dangerous, and the platform team had to write (and maintain) numerous scripts to bring the system RBAC back to its desired state.

“P0 helps us control entitlements to sensitive data and systems much more easily than before. Previously, to provide engineers safe access to critical resources in Snowflake and Kubernetes, we created a patchwork of static groups and roles, used Azure PIM to provide escalated access, and spent a lot of time managing group membership.”

Eugene Yedvabny, Senior Staff Software Engineer

Poor developer experience: Existing processes were a drag on developer productivity.

• PIM’s UX was not intuitive, and was not a core part of an engineer’s day-to-day workflows. Most of the time, engineers did not know the right group/role to request. Navigating to the right permission or group could take time.

• Approval workflows could take hours. This was especially challenging for on-call engineers. The last thing someone needs, when they are paged in the middle of the night, is to find out that they not have the right level of access.

‍

The solution

The P0 team helped onboard Afresh onto the platform during an hour-long Zoom call. Subsequently, the platform team was able to use P0 docs to configure P0 to suit their needs.

‍

Key Features:

• Automated privileged access for Azure resources: Afresh’s engineering teams use P0 for just-in-time and short-lived permissions to Azure, Kubernetes and GitHub resources, via group membership in Azure AD groups.

• Automated privileged access for Snowflake: Engineers can use P0 for just-in-time access to sensitive datastores in Snowflake. They love that they can even request access to execute specific SQL queries. Further, P0 automatically remediates drift in production Snowflake permissions, which may be caused by different microservices.

• Slackbot: P0’s Slackbot is used by the engineering team for access requests and approvals.

• On-call automation via PagerDuty: Using P0’s integration with PagerDuty, on-call engineers can get automated access without requiring a human approval.

The impact

Afresh’s security posture has improved measurably, since P0 is helping them move away from loosely scoped roles and coarse grained groups. P0 has automated many of the security team’s repetitive tasks. These include processing requests for escalated access, and building and maintaining ad-hoc scripts for remediating ‘permissions drift’ in birthright access.
‍

“P0 is a game-changer. Prior to P0 we had to choose between access granularity and ease of use. P0 gives us the best of both worlds by scoping permissions exactly to what our users need, when they need it. It helps me sleep well at night knowing that my team always has the right-sized access to production, and long-standing escalated access is not lurking in any group.”

Eugene Yedvabny, Senior Staff Software Engineer

Mean-time-to-resolution of access requests has dropped from hours to minutes. For on-call engineers, resolution time is instantaneous, thanks to P0’s PagerDuty integration. Engineers regularly report how much faster they can accomplish their tasks when using P0, since the complexity of groups and roles has been abstracted away.
‍

“P0 worked great. Would use again. A++”

Anonymous Afresh engineer