December 1, 2022: So .. what exactly happened at Uber? What can we learn from this episode?
The recent breach at Uber provided a timely reminder, if one was needed, of the different cybersecurity risks that enterprises face in a cloud-native world. The exploding attack surface of the cloud has made even the most security conscious organizations susceptible to unforeseen attacks. In this blog post, elaborating on Uber’s example, I write about how cloud-native infrastructure poses a unique problem, particularly for access management. I’ll also share what we are doing to help developers deploy applications more securely, without sacrificing velocity.
Let’s take a look at the sequence of events in the Uber breach.
- Malware Attack: An Uber contractor’s device was infected with malware, which likely exposed their corporate password
- Stolen Credentials: A hacker associated with the Lapsus$ extortion group purchased those login credentials, likely off the dark web,
- MFA Fatigue Attack: The hacker flooded the contractor with multiple MFA requests, and eventually tricked them into approving one of those requests
- Access to Slack: The hacker began posting messages on internal channels
- Insecure Admin Credentials: The hacker also found some Powershell scripts on the Uber Intranet. One of these contained username/password for an admin user in Thycotic (a PAM solution), in hardcoded plaintext (!!)
- Access to AWS/GCP: In Thycotic, the hacker found credentials to Uber’s AWS/GCP accounts (amongst other things), from where they could, well, do whatever they pleased.
Each of these steps aptly illustrates security risks due to a growing attack surface (increasing number of users, devices, apps, credentials, cloud-providers etc.). In principle, there are known best practices to counter each one of these threats, and Uber has only itself to blame for a weak security posture. In practice however, even with the best of intentions, many organizations are unable to implement adequate security controls.
Why is this the case, even at organizations such as Uber who have no shortage of motivation, resources, budget, talent? There is no easy answer to this question, but in our opinion, it is part technological, part philosophical, and part organizational.
For the sake of simplicity, let’s restrict our analysis to events 5 and 6 above, i.e., why was the hacker able to get access to sensitive systems such as Thycotic and AWS. Put another way, why did ANYONE have access to an admin role for such systems? Best practice, particularly in a cloud-native environment, can conceivably be to restrict admin access to only identities that have a demonstrable need (with a very high bar), and even in such instances, to only grant them access for a limited period of time. Why was this not done? Naturally, we can’t speak for Uber, but we can put forward educated guesses, based on our observations at other companies.
Technologically, in-and-of-itself, granting time-bound and just-in-time access to an individual sensitive system (such as Thycotic, AWS, GCP etc.) is not a challenging problem. Every system usually comes with an RBAC (role-based access control) API, that can be used to give access to individuals according to organizational policies. However, with the explosion of cloud-native development, the complexity of such APIs, and the number of services in the stack of any organization has grown exponentially. As an example, AWS IAM alone has almost 10,000 different permissions. In many cases, developers know what they want to do, but they do not know what permissions they need to perform their task, because they do not know the language of AWS IAM. In most organizations, this creates a problem of insurmountable complexity.
Philosophically, limiting access to sensitive systems usually comes at the expense of developer velocity. When faced with a situation where developers need some unknown access to perform a particular task, they are prone to asking for/hardcoding admin credentials “one last time”. Or they may use a very adult-sounding justification “we are taking on technical debt, but it is for the sake of speed”. Except that the time to pay back that debt never comes.
Organizationally, security and development teams at most companies are quite siloed. Security teams want to implement best practices, but the tools they use are not developer-friendly, and developer teams are not incentivized to break their workflows to adopt security best practices. As a result, even with the best intentions, many security best practices never get adopted.
Today, I’m proud to announce my new endeavor: P0 Security. Together with my colleagues, I’m working on helping you improve the security of your cloud-native stack. Our tool:
- Works within your developers’ existing workflow (on the command line and in the IDE), preventing friction between development and security.
- Automatically grants time-bounded, just-in-time, least-privilege access with the correct permissions the first time.
- Gives you detailed contextual visibility into access, not only telling you what is happening in your systems, but why.
Stay tuned for more details! And of course, if our mission resonates with you, please reach out to us. We are looking for design partners to help us bring our vision to the market.