CSI: Crime Scene Investigation was a popular TV series that started in 2000 and led to several spinoff series. The program focused on careful use of forensic science and procedures to help solve difficult crimes, and being of a scientific bent myself I occasionally found it interesting. Though like most mystery/thriller stories I could usually see how it was going to develop. Because as actor Armin Mueller-Stahl said in a scene from one of my favorite action crime movies “The International” starring Clive Owen and Naomi Watts, “This is the difference between truth and fiction: fiction has to make sense.” Enterprise software has to make sense too. It has to be manageable and supportable. It has to keep running so your business can keep running. It has to have value and be understandable and meet all the basic requirements, both technical and business, that are necessary to meet the needs of your company.
And yet things often break. Sometimes it’s your fault as an administrator. Or as a bean counter. Or a CEO more interested in protecting his stock options than building a successful business. Or maybe it’s the vendor’s fault for not keeping their software up to date. Or someone inside or outside the organization perpetrates a criminal action against your software infrastructure through a hacking attempt of some form. Or your software environment just seems to have gotten out of hand somehow and has become too complex to handle properly, and bad stuff starts to happen.
How do you deal with the situation when something goes wrong like this and you suspect a crime has been committed? You work it like you’re working a crime scene. You call in your Incident Response Team to investigate and try to resolve so your business can get back on track again as smoothly and quickly as possible. Soumik Sarkar knows something about doing this. Soumik is chief technology officer and co-founder of Crosscode, a company whose solution called Panoptics provides enterprise code-level dependency mapping and discovery that can help untangle the baffling complexity of an enterprise software crime scene. He recently shared with me a three-step approach he recommends for investigating what’s gone wrong when your enterprise software infrastructure begins to fall apart.
Working the crime scene
“The modern enterprise is only as good as its software,” Soumik says. “Today, everything from team communications and collaboration to product and service delivery depends on the code that underpins these processes. But the enterprise software universe is complex — comprised of thousands of software modules spanning millions of lines of code. Some programs are developed in-house, others are purchased from vendors, still others are from the open-source communities. Unfortunately, a problem in one software module within an enterprise architecture can cripple others interacting with it. As a result, understanding how many software modules run within an enterprise, how they were developed, and how they depend on and interact with one another becomes critical. Tracking the breadcrumbs of a company’s software environment is not very different from a detective looking for clues to solve a crime. While their goals vary, the two processes have much in common.”
Step 1: Evaluating the scene of the crime
“A good detective always begins solving a complex crime by getting a good understanding of the scene and gathering information on the different people and events connected. Discovering the software makeup of an organization is not very different. It begins with a thorough and in-depth understanding of the tools and programs in use. This not only involves taking a full inventory of all applications but also understanding how the different lines of codes within each work together and how data are connected.”
Step 2: Interrogation
“Once a detective determines the circumstances surrounding the crime, the next step involves a methodical and rigorous examination of all possible suspects. Going by this analogy, organizations must understand their application infrastructure and track changes to the underlying source code in order to identify system changes, detect when and how often they occur, and assess their impact on other applications.
“Unfortunately, even those with robust methods of tracking application changes are likely to overlook critical modifications. Despite software peer reviews or audits (the former is a process conducted by the development team, the latter involves an independent party to assess compliance with specifications, standards, and contracts) changes such as alterations to database schemas, key/value stores, and message queues can fly under the radar.
“In such cases, it can help to have an audit trail that automatically logs the time and date of each change in the run time environment to applications, packages, classes, methods, tables, columns, and message queues. Having such a feature can save organizations a lot of time and money and help build better, more effective disaster recovery plans. The process also helps uncover hidden changes to the database, code, and configurations.”
Step 3: Identification of the culprit
“Once a detective uncovers the circumstances of the crime, the evidence pointing to all potential suspects and the possible motive, identifying the criminal isn’t too difficult. On the enterprise software side, the ability to uncover dependencies, risks, and changes that lead to outages or production issues is firmly connected to an understanding of the software development life cycle (SLDC). This always begins with a set of requirements describing what’s needed to solve a problem and ends with the deployment of the software in different runtime environments, such as development, QA, staging, and production.
“One of the best ways to identify what’s changed in a complex heterogeneous site is to extract information from the runtime environment, which is the real source of truth when it comes to enterprise applications. After that, all that organizations need to do is backtrack to identify the real problem.
“In today’s fast-paced application environments, organizations need accurate insights into their infrastructure, across the entire software development lifecycle. Detectives rely on their sources for valuable information and sophisticated technologies for evaluating evidence and examining the scene of the crime. Similarly, to accelerate their innovation and development projects, organizations must work with tools that enable automatic discovery of all the applications and databases running in the enterprise, analyze them, and generate a detailed map of their code-level dependencies. Only through a comprehensive understanding of what lies within an enterprise architecture, organizations can ensure effective and efficient software changes and upgrades.”
Featured image: Shutterstock