By: Michael Robinson
For approximately four years, I was responsible for the operation of a Local Area Network (LAN). Over that time the user population and data stored on the LAN more than the tripled. However, the most explosive growth on that network was not its usage; it was the number of malware attacks seen on a daily basis. The number of malware incidents increased tenfold. In 2010 the number of incidents was on the rise, the amount of user downtime doubled, the amount of time to correct the issue was increasing from hours to days, and the grumbling of users and Service Desk personnel alike was getting louder. Without getting into specific details, the problem was getting out of control. In order to address the issue from a holistic approach, detailed metrics were gathered for approximately five months while the Service Desk personnel modified its approach to addressing malware. Here’s what was learned.
1. The role of “First Responder” moved away from security personnel…and it made matters worse.
End users and Help Desk technicians became the de facto first responders to malware incidents. The role of first responder no longer resided in the hands of security engineers or acquisition personnel. It was discovered that end users often attempted to clean up malware issues on their own. Unfortunately, initial cleanup efforts were rarely effective and they served only to muddy the waters. Many users did not want downtime, nor did they want to contact the Help Desk. When the Help Desk technicians were finally called to fix problems that were often described as “significant slowness” and “sticky systems” a number of reboots already occurred, some files were manually deleted, and changes were made. The Help Desk personnel frequently treated symptoms through a cookbook approach without diagnosing systemic problems. While the intention was to maximize up-time, it usually resulted in clearing caches, patching systems, and, if necessary, wiping/reimaging the troubled system(s). These approaches made matters significantly more difficult for security and forensic personnel. It was determined that training was required for both users and Help Desk Technicians in how to treat potential malware situations.
2. The vast majority (90%+) of malware came from users surfing the web.
While phishing attacks and spam were real concerns, the majority of malware attacks (90%+) came from users surfing the web. The majority of those occurrences arose from users visiting reputable news sites and infections being delivered through advertisements. It was very similar to playing the slot machines in Las Vegas: play often enough and eventually a player wins (or, in the case of malware, loses). LAN users would frequent the same news websites and advertisement banners would rotate. Eventually an infected banner would appear in a browser and compromise a machine. When the source of this malware was identified, the firewall engineers created rules to block traffic from 20 specific advertisers. By blocking only these sites, the number of malware infections on the LAN dropped by over 80%.
3. Most garden variety malware appeared to be infecting similar locations.
In the latter half of 2010, details regarding malware infections from a five month span were collected. The goals were to identify common behavior in malware infections and to put this data in the hands of Help Desk technicians, who were responding to malware issues. While this didn’t solve every malware issue, this ultimately drove down the amount of time it took to diagnose garden-variety malware and the amount of down-time experienced by users. Response time improved which meant the likelihood of having compromised systems on the LAN and of having data exfiltrated from the LAN was reduced. Here is the list of common locations where malware resided:
C:\Documents and Settings\All Users\Documents\
C:\Documents and Settings\[username]\
C:\Documents and Settings\[username]\Application Data\
C:\Documents and Settings\[username]\Application Data\Microsoft\
C:\Documents and Settings\[username]\Application Data\Microsoft\Windows\
C:\Documents and Settings\[username]\Desktop\
C:\Documents and Settings\[username]\Local Settings\
C:\Documents and Settings\[username]\Local Settings\Application Data\
C:\Documents and Settings\[username]\Local Settings\Application Data\Microsoft\
C:\Documents and Settings\[username]\Local Settings\Application Data\Microsoft\Windows\
C:\Documents and Settings\[username]\Local Settings\Temp\
In the Registry:
These three items were meant to help triage the rapidly growing malware problem and reduce the risk of computers and data on the LAN being compromised. It also allowed security engineers and forensic analysts to focus on more significant issues.