Lessons to be learned from the Elasticsearch data breach

Open source software being associated with the National Football League for any reason whatsoever would normally sound like a good idea, but not this time. Especially not if that open source software is responsible for private information on about 1,100 NFL players and their agents being leaked. Kromtech Researchers estimated that due to a lack of password security and authentication technology, over 4,500 Elasticsearch machines were infected by two types of malware, namely JackPOS and AlinaPOS. Additionally, in at least one case, data was accessed by a malicious attacker and a ransom note left in the files. The Elasticsearch data breach was said to have exposed sensitive information about the players and agents.

POS malware

The POS in their names means these are point-of-sale malware that attempts to scrape sensitive information like credit card details with a bunch of different techniques. One example of how this is so effective is the fact that JackPOS can encrypt stolen data to be sent back undetected. It uses the MAC address as a bot ID and gets in by tricking the system that it is a Java utility. It can then copy itself into the %APPDATA% directory or any Java-based sub-directory. Once the malware is in place, attackers can remotely access the server’s resources and basically do whatever they want like wiping out hard drives, remotely executing code with full admin rights, or stealing data.

The major problem here is that the malware can remain active on a number of systems even after being detected on individual servers. Additionally, every infected server became a part of a bigger POS botnet with Command and Control (C&C) functionality for other POS malware clients. Once infected, these clients are constantly collecting (stealing), encrypting, and transferring information from POS terminals, RAM memory, or infected Windows machines.

Lax password security and the Elasticsearch data breach

How did this Elasticsearch data breach occur? The vulnerability stemmed from a lack of basic password security with regards to authenticating user sessions. This lack of authentication, in turn, stems from something even more basic like the public configuration through which attackers can manage the whole system with full administrative privileges. An example of the public configuration issue is that before ES 6.0, an X-Pack installation would automatically create three users: elastic, kibana, and logstash_system with the password of “changeme.” The intent was obviously so you would change those passwords right away, but basic human laziness and the tendency to skip through installations make this a real problem. This certainly could have been a factor in the Elasticsearch data breach.

There is even a setting called xpack.security.authc.accept_default_password to disallow this password once your cluster is running, but people who actually do that are probably few and far between. Though the problem of how to bootstrap a password is being solved in version 6.0, default passwords are less than ideal in today’s environment. A bootstrap password is a temporary password that is used only until a real password has been set for the elastic user. A bootstrap password can be auto-generated or placed in the Elasticsearch keystore before the node is started so you don’t have to manually set passwords for each node.

Another issue before is the time gap between the time when the node is started and the default password changed. Even if the password is changed quickly, there is still a window for an outsider to authenticate and get in. To prevent this situation, the setup-passwords command line tool is used to set or automatically generate strong bootstrap passwords for the internal elastic, kibana, and logstash-system users.

Elasticsearch held for ransom

A series ransom attacks last year that have already hit over 33,000 MongoDB databases. expanded their targets to Elasticsearch instances. Affected Elasticsearch administrators are greeted in one actor’s attacks with a message reading:

Send 0.2 bitcoins to this wallet: 1DAsGY4Kt1a4LCTPMH5vm5PqX32eZmot4r if you want recover (sic) your database! Send to this email your service IP after sending the bitcoins [email protected] (sic).

Elastic published a blog post to signal the risk of leaving servers exposed to the Internet and to provide instructions on how to secure them. No. 1 on the list of good practices was “upgrade to the latest version.”

TLS and Zero Trust Networking

Security has actually become such a major issue for ES that version 6.0 is incorporating something called Zero Trust Networking, which when explained sounds like something from a deep espionage movie or an episode of “Narcos.” The basic concept is that the world has changed drastically in the last few years and we’ve reached a point where your own network can no longer be trusted because your ranks have been compromised. That’s right, any of the good guys could be working for the other side, and that’s the assumption you need to start with.

Since data traveling even across your own wires are now suspect, TLS communication between nodes in an Elasticsearch cluster will be mandatory in ES 6.0. Transport Layer Security (TLS) and its predecessor, Secure Sockets Layer (SSL), both frequently referred to as “SSL,” are cryptographic protocols that provide communications security over a computer network. TLS relies on using certificates, signatures, and encryption to ensure that the communication between two endpoints will be secured, even if the network between those endpoints is hostile. There are also “bootstrap checks” that run when Elasticsearch enters production mode, verifying that TLS is enabled is now part of the bootstrap checking process.

Securing your Elastic

There are nearly 4,000 infected Elasticsearch servers and about 99 percent of them are hosted on Amazon. The Amazon hosting platform gives users the possibility to configure the Elasticsearch cluster just in few clicks, but usually, people skip all security configurations during the quick installation process. This is where a simple mistake can have big repercussions and before you can say Elasticsearch data breach, you’re in the news for exposing massive amounts of sensitive data. That’s without a doubt the worst publicity a firm handling customer data could hope for. But there are a few things that can be done to avoid the situation.

Kromtech Security Center highly recommends you take following actions required for an effective incident response. Install the latest Elastic patch or completely reinstall it. Close all unused ports from external access, whitelist only trusted IPs, check your log files, connections, and traffic on all servers and back up all running systems. Additionally, you can also reinstall all compromised systems if you don’t want to check each suspicious one separately and monitor them closely for the next three months. You can also extract samples of malware and provide it to Kromtech ([email protected]) for further analysis.

Staying one step ahead

Keeping records, securing all the way down to the field level, password protection, and user management are all things that ES recommends to keep your ES cluster secure. Like everything else in the enterprise, if you stop, you are a sitting duck for attackers, and you die, period. The solution is to keep moving (forward) and actively thinking one step ahead of your attackers to make sure you’ve thought of and planned for every possible attack scenario. It’s also a good practice to look at every failure like the Elasticsearch data breach as a learning experience because you just got a great example of your attacker thinking one step ahead of you.

Photo credit: Shutterstock