Hacks and hackers are everywhere. All of you should be familiar with what happened with Equifax in 2017. If you don’t know about that breach, where have you been living? When I started writing this article, I came across a breach at British Airways, and even at Cisco.
Seems no company is immune. Crowdstrike co-founder Dmitri Alperovitch, is known for saying this:
There are only two types of organizations: those that know that they’ve been hacked and those that don’t yet know.
Unlike what seems to have happened with Equifax, many companies do make an effort to avoid being hacked, especially internally. According to a study done by IBM, many breaches are an inside job.
So a few years ago, I was working on a project to perform a network security analysis. With a network of well over 10,000 devices, it’s a challenge to make sure that all of these devices are configured correctly, and following corporate security standards.
A large percentage of breaches occur due to misconfigured devices. So this company wanted to make sure that its devices are configured properly and are in compliance with various standards, to prevent a breach of any kind.
Don’t be two steps behind
Hackers always seem like they are one step ahead. Security is one area you cannot afford to fall too many steps behind. To have a chance at preventing a breach, a company needs to be proactive. Unfortunately, too often in IT, we are not. We often don’t seem to anticipate that we are all human, and mistakes can and will be made. For this reason, this company developed their own internal tool to scan its network devices for configuration issues that could potentially lead to security breaches.
How many times have you typed a “9” instead of a “0”? Or, how about “a” instead of a “9”? Gotta love that shift button!
Human error — it happens. Sometimes, making mistakes is what we do. But other times, these “errors” can be by design. Someone could purposely add a configuration to your network device that goes against your standards for their own dark reasons. Whatever the reason, what you need is an ability to automatically identify these configuration errors when they happen. And that’s what this company did. They developed a tool to detect such errors. Depending on who you are and where you’ve worked, you can probably see a few problems with that method. One I thought of is technical debt.
Now, having created something internally, this company has to maintain this internal tool from now until eternity. Well, maybe not eternity, but they now have to update it as technology changes. New device interface commands? Change control! New device technology capabilities? Change control! Not keeping up with these changes can leave your network open to unfortunate circumstances. See Equifax. And oh, by the way, your lead network engineer? The person to lead the team to build this tool? Yeah, they just took a new job. So they’re leaving.
If your organization is like many other IT organizations I’ve been at, documentation isn’t where you want it to be. So it’s possible this engineer just left with much of the knowledge about your in-house tool. I’ve seen that happen with vendor software too. The product oftentimes becomes shelfware. So what do you do?
I’m all for doing things in-house. But if you’re in an industry where developing software tools is not where you make money, it may not make sense. This company came to this same conclusion. So they set out to replace their internal configuration scanning tool for security analysis with a vendor product. And my job was to help them implement this and set them up with a solution to automate the security analysis of their whole network.
Fail to plan or plan to fail
So how do we go about doing that? How do we replace an existing security tool with a vendor product?
Well, you develop an implementation plan, and then love it when the plan comes together. You need a plan to be proactive. So we had a plan to gather as much detail from the client about their network and this in-house tool they were using. This included the number of devices on the network that was currently being scanned for configuration issues. The client had several thousand devices along with several hundred rules in their internal tool to perform their scans. My job was to map each of their rules to a rule in the vendor product. If no rule existed, we would create a new rule. In the end, mapped the clients several hundred custom rules to less than 50 existing vendor product rules. From a performance standpoint, that’s a win!
I believe that one of the best ways to analyze your network, whether for security or performance, is to collect data from your network infrastructure and analyze them “offline.” You want to minimize any impact on your production network. Whether it’s configuration data from your routers and switches, logs from your servers, or packet captures from your network devices, you want to minimize directly connecting to them to pull analysis data. With that in mind, we collected production data from our network on a scheduled basis, ideally during lower network usage, to reduce any potential impact. So we were able to accomplish what the client set out to do - replace its existing configuration tool with a vendor product that performs security analyses.
Once this project was completed, there were a number of challenges that were overcome, but also important takeaways. Now let’s focus on the three things I learned that can adversely impact the security of your network if you’re not careful.
1. Are you there?
When I first engaged with this client, the number of devices on their network was pegged at over 10,000 devices of routers, switches, firewalls, etc. The first step in being able to check your network for security is knowing all of your devices that run on your network. So the first thing I do is to perform a network discovery.
There are multiple ways this can be done. One way is to use good ole SNMP, which has had its own security challenges historically, but often times does a good enough job. Another method is to log into each device and get the configs from those devices. A third is to utilize existing tools that a client already has and pull the device information from there. The most accurate method at the time was being able to log into the devices and doing a sh run. But many times, I was either not able to log into a device or the device didn’t even respond.
So what I found out was that only about 60 percent of the network devices the client believed to be out there were actually reachable. The other 40 percent were not accessible for one reason or another. This meant that either the login credentials for those devices had changed, the device’s role had changed, or the device was simply removed from the network.
Not having a handle on the network devices you have on your network is one surefire way to suffer a security issue. A device that you think is on your network but is not pingable can suggest that it’s there, but using a different IP that you don’t have in your database or it’s blocking your login and/or ICMP requests. This leaves you in the dark with what’s on your network, and potentially opens you up to vulnerabilities that you won’t know about until your network experiences an adverse security event.
2. ACL importance
Access-control lists are a basic method of starting to secure a network. You want to ensure that only certain devices, ports, and protocols have access to certain parts of your network. And you definitely want to make sure that the stuff that is denied does not find a way in. On this project, the client had many ACL rules and in some cases, very specific ACLs in portions of the network, so they wanted to make sure that their newly minted network security scan tool could identify ACLs that were being added or removed unexpectedly, whether deliberate or not.
Access lists can be added or changed whenever configuration changes are made. The wrong person with the right access to your network devices can make a deliberate change. But what can also happen is the right person with the right access could have fat-fingered a different port number and allowed or denied the wrong IP. So you want to have some sort of tool that will parse your changing ACLs and alert you when your rules are broken.
3. Accurate routing
Your access lists will permit or deny traffic as required. But when a network is under attack, those may not be the only areas that can be targeted. Your routing protocols can be a target as well.
In a large network such as the client I was working with, you can have large segments of your network running BGP to route traffic between each other. But also, you can have EIGRP or OSPF running within those network segments to route traffic.
Certain flaws in these protocols can be targets, so you want to be able to check that the expected routing is regularly occurring. Or, you want to ensure that if a route fails, as it will inevitably do, your network reroutes appropriately and is not allowing unexpected traffic through certain device interfaces.
This client had two large network segments with very different needs. There was little reason for much traffic to traverse the few devices that connected these two networks. Only certain traffic was permitted.
But when I analyzed certain ACLs between these two networks, I found that some unexpected traffic was actually being routed, surprisingly to the client. Unlike the above, where we were checking for ACLs that existed or not, this helped us identify not only whether the correct ACLs existed, but we were able to test whether traffic was actually being allowed through or not.
The client was able to close this loophole before it led to a bigger issue. You want to be able to do the same on your network before the wrong person exploits something like that and access a part of your network improperly.
Don’t be vulnerable
Keeping your network secure is obviously required. Clients like this one I worked with understand that, and had the right tools in place to help with that. However, their internal tool led to more technical debt than they preferred, and chose instead to implement a vendor product that could do what it’s in-house to do, and more.
Helping this client identify some things they could do better at the time, and going forward, is something we all need to look for. Whether we use tools or not, we can’t afford to leave our networks open to changes without knowing when these changes can leave our networks, and therefore, our companies, vulnerable to attack.
Take a cue from these three lessons to make sure you audit your network for potential security issues. Have you done this type of security analysis? What did you discover or take away?
Featured image: Pixabay