Security continues to be the No. 1 concern for organizations using Kubernetes, and what’s interesting is that it has little to do with the inherent unsafety of Kubernetes itself. Instead, it has a lot to do with how complex Kubernetes actually is, and the fact that even skilled cloud-native developers often have a hard time finding their way around. According to a recent report by StackRox, in addition to a steep learning curve and shortage of skilled labor, exposures due to misconfigurations is the biggest cause for Kubernetes security incidents. After receiving input from over 540 respondents, the conclusion was that over 94 percent had experienced security incidents in the last year!
They were hardly alone, as this short list of major Kubernetes security incidents shows.
1. Capital One
This Kubernetes security incident was a big one, no pun intended, and the cause for a lot of people to wake up and take notice. Occurring exactly one year ago, this breach saw 30GB of credit application data affecting about 106 million people being exfiltrated. What actually caused the breach is something we see quite a lot of in the world of Kubernetes, a misconfiguration. In particular, a misconfigured firewall that allowed an attacker to query internal metadata and gain credentials of an Amazon Web Services’ IAM role that had no business being that “broad” in the first place.
One of the important lessons we learned from this incident is to be more careful while assigning IAM roles. Most people are just in a hurry to get Kubernetes to work and often sidestep important tasks like managing secrets and services, and assigning IAM roles on a per-pod basis as opposed to per-application. Another important task is to “roll” credentials regularly, preferably by an automated service that puts a cap on how long before credentials need to be renewed. This also puts an upper limit on how long a breach can continue.
2. Docker Hub
With Kubernetes, containers, and distributed environments, the attack surface is only getting bigger exponentially, and you never know where an attack is going to come from. One example is how attackers managed to plant malicious images in the Docker hub last year, causing anyone who uses those images to be “cryptojacked.” What this means is users unknowingly deployed cryptocurrency miners in the form of Docker containers that then diverted compute resources toward mining cryptocurrency for the attacker. This is just one in a number of similar attacks we see of late.
Similar to the Capital One breach, changing passwords and rolling credentials is a must to avoid this sort of situation. Additionally, for Kubernetes environments, rotating your secrets and auditing images to ensure only verified images are being used are key steps to ensuring security. Malicious images can be pretty hard to detect, especially since a lot of the time, the containers work as expected. This is why additional checks that highlight any deviations in application behavior are necessary to ensure that no stowaway processes are running in the background. This kind of attack is quite lucrative.
3. Microsoft Azure
Microsoft is another organization that’s been seeing a lot of cryptojacking woes of its own. After disclosing that there was a large-scale cryptomining attack against Kubernetes cluster in Azure in April this year, a similar campaign was uncovered in June that targets misconfigured Kubeflow containers to turn them into cryptominers. Similar to the compromised image situation with Docker hub, Kubeflow uses a number of services, which in turn allow users to use custom images like Katib and Jupyter notebook server. In the case of Jupyter, the chosen image doesn’t have to be a legitimate notebook image, and that is where the attackers found an entry point.
If we look into what caused this misconfiguration, it’s pretty much what causes every misconfiguration — impatience, laziness, and lack of knowledge. Kubflow’s UI dashboard is, by default, only accessible internally through an Istio ingress gateway. Some users, however, took a short cut to access the dashboard directly without going through the Kubernetes API server and didn’t realize that while what they were doing was saving time, it was also opening a number of backdoors in the process. In this case, they were exposing the Istio ingress gateway to the internet, allowing anyone to access the dashboard. The moral of the story here is that there are security implications to every setting or configuration change that takes place.
With the value of cryptocurrencies skyrocketing, and limitless compute resources located in the cloud, hijacking resources has become a lot more lucrative than stealing info. Automaker Tesla was one of the earlier victims of cryptojacking when a Kubernetes cluster was compromised due to an administrative console not being password protected. The discovery was made by RedLock Cloud Security Intelligence and made public in a report stating the misconfiguration had helped attackers get hold of Tesla’s AWS S3 bucket credentials. Those credentials were then used to run a cryptomining script on a pod.
What was interesting about this attack was the number of “ingenuine” precautionary measures taken to avoid detection. Not only did the attackers refrain from using a known mining pool, and used an unlisted one instead, they also used popular CDN service Cloudflare to hide their IP. The attackers also made sure that the mining script didn’t use enough CPU resources to cause an alarm or get detected, and listened on a nonstandard port, making detection based on port traffic virtually impossible. The only way to detect such a breach is to actively monitor configurations to ensure all policies are being followed.
To put a dollar value on what this kind of a hijack of resources is worth, around the same time as the Tesla incident, hackers managed to exploit a vulnerability in Jenkins to cryptomine to the tune of about $3.5 million, or 10,800 Monero in 18 months. Monero is the same cryptocurrency involved with the malicious Docker images we mentioned earlier. In Docker’s case, it was discovered that six malicious images had been collectively pulled over 2 million times, that’s 2 million users potentially mining Monero for the enemy, quite a feat.
The Jenkins Kubernetes security incident is by far one of the most audacious breaches discovered yet, in addition to the fact that it uses vulnerable Windows machines and personal computers running Jenkins, it also targets Jenkins CI servers. This is a recent update as the malware goes through a number of lifecycles where it keeps updating itself and changing mining pools to avoid detection. The fact that it can target servers now is a step-up by the attackers who are of Chinese origin. If they could pull over $3 million from beat-up desktops, powerful servers are going to add a few zeros to that number, at least.
Kubernetes security: The attack surface keeps growing
Kubernetes security incidents are growing because the attack surface is now an infinite army of hybrid clouds, on-premises datacenters, IoT devices, personal computers, edge devices, and more. Closed-minded security is dead — gone are the days where you could just focus on your application and leave the rest to a firewall. A threat could potentially come from a service being used by a service, that’s then being used by a service that you’re using. Additionally, security is now everyone’s responsibility, and your cloud provider or Kubernetes manager can only do their part if you also do yours. Cryptojacking is only going to get more common as more hackers catch on that it’s pretty hard to detect except for some noticeably high cloud service bills.
Featured image: Pixabay