APT Detection & Network Analysis

Until recently, the majority of organizations believed that they do not have to worry about targeted attacks, because they consider themselves to be “flying under the radar”. The common belief has been: “We are too small, only big organizations like financial service providers, military industry, energy suppliers and government institutions are affected”.

However, this assumption has been proven wrong since at least the detection of Operation ShadyRAT[0], DarkHotel[1] or the recent “RUAG Cyber Espionage Case”[2]. The analysis of the Command & Control (C&C) servers of ShadyRAT revealed that a large-scale operation was run from 2006 to 2011. During this operation 71 organizations (private and public) were targeted and spied on. It is assumed that these so-called Advanced Persistent Threats (APT) will even increase in the near future.

We at Compass Security are often asked to help finding malicious actions or traffic inside corporate networks.

The infection, in most cases, is a mix of social engineering methods (for example spear phishing) and the exploitation of vulnerabilities. This actually varies from case to case. Often we observe in proxy logs, that employees were lured into visiting some phishing sites which are designed to look exactly like the corporation’s Outlook Web Access (OWA) or similar applications/services as being used by the targeted company.

Typically, this is not something you can prevent exclusively with technical measures – user awareness is the key here! Nevertheless, we are often called to investigate when there still is some malware activity in the network. APT traffic detection can then be achieved with the correlation of DNS, mail, proxy, and firewall logs.

Network Analysis & APT Detection

To analyze a network, Compass Analysts first have to know the network’s topology to get an idea of how malware (or a human attacker) might communicate with external servers. Almost every attacker is going to exfiltrate data at some point in time. This is the nature of corporate/industrial espionage. Further, it is important to find out whether the attacker gained access to other clients or servers in the network.

For the analysis, log files are crucial. Many companies are already collecting logs on central servers [3], which speeds up the investigation process, since administrators don’t have to collect the logs from many different sources (which sometimes takes weeks), and off-site logs are more difficult to clear by attackers.

To analyze logs and sometimes traffic dumps, we use different tools like:

ELK offers many advantages when it comes to clustering and configuration, but it doesn’t offer many pre-configured log parser rules. If you are lucky, you can find some for your infrastructure on GrokBase[4]. Otherwise there are plenty of tools helping you to build them on your own, such as e.g. Grok Debugger[5].

However, when analysis has to be kick-started fast, and you do not have time to configure large rulesets – Splunk comes with a wide range of pre-configured parsers.

After we gathered all logs (and in some cases traffic dumps), we feed them into Splunk/ELK/Moloch for indexing.

In a first step we try to clean the data set by removing noise. To achieve this, we identify known good traffic patterns and exclude them. Of course it is not always straight forward to distinguish between normal and suspicious traffic as some malware uses for example Google Docs for exfiltration. It takes some time to understand what the usual traffic in a network looks like. To clean the data set even more, we then look for connections to known malware domains.

There are plenty of lists available for this:

If we are lucky, the attacker used infrastructure provided by known malware service providers (individuals and organizations are selling special services just for the purpose of hosting malware infrastructure). But more sophisticated attacks will most likely use their own infrastructure.

After cleaning out the data sets, we look for anomalies in the logs (e.g. large amounts of requests, single requests, big DNS queries, etc.). Some malware is really noisy and as a consequence, easy to find. Some samples are connecting to their C&C servers in a high frequency. Other samples are requesting commands form C&C servers at regular time intervals (Friday 20:00 for example). Others connect just once.

Sometimes we also detect anomalies in the network infrastructure which are caused by employees, for example heavy usage of cloud services such as Google Drive or Dropbox. Often these constitute to so-called false positives.

To share our experiences and knowledge in this field, we created the Security Training: Network Analysis & APT.

This training will cover:

  • Configuration of evidence (What logs are needed?)
  • Static and Dynamic Log Analysis with Splunk
    • Splunk Basics and Advanced Usage
    • Detecting anomalies
    • Detecting malicious traffic
  • Attack & Detection Challenges

If you are interested please visit our “Security Trainings” section to get more information: https://www.compass-security.com/services/security-trainings/kursinhalte-network-anlysis-apt/ or get in touch if you have questions.

The next upcoming training is on 22. and 23. September 2016 in Jona, click here to register.

Sources & References:
[0] Operation Shady RAT, McAfee, http://www.mcafee.com/us/resources/white-papers/wp-operation-shady-rat.pdf
[1] The Darkhotel APT, Kaspersky Lab Research, https://securelist.com/blog/research/66779/the-darkhotel-apt/
[2] Technical Report about the Malware used in the Cyberespionage against RUAG, MELANI/GovCERT, https://www.melani.admin.ch/melani/en/home/dokumentation/reports/technical-reports/technical-report_apt_case_ruag.html
[3] “Challenges in Log-Management”, Compass Security Blog, http://blog.csnc.ch/2014/10/challenges-in-log-management/
[4] GrokBase, http://grokbase.com/
[5] Grok Debugger, https://grokdebug.herokuapp.com/
[x.0] APT Network Analysis with Splunk, Compass Security, Lukas Reschke, https://www.compass-security.com/fileadmin/Datein/Research/White_Papers/apt_network_analysis_w_splunk_whitepaper.pdf
[x.1] Whitepaper: Using Splunk To Detect DNS Tunneling, Steve Jaworski, https://www.sans.org/reading-room/whitepapers/malicious/splunk-detect-dns-tunneling-37022

Netzwerktraffic und APT Analyse

Compass Security wird vermehrt von Kunden bzgl. Verdacht auf Advanced Persistent Threat (APT) kontaktiert. Unter die Bezeichnung “APT” fallen komplexe, zielgerichtete und äusserst effektive Angriffe auf kritische und zuweilen gar unternehmenswichtige Computersysteme bzw. deren gespeicherte Informationen.
Die Analyse von potentiell infiltrierten Netzen und Systemen gestaltet sich jedoch als enorm aufwändig, da Unmengen von Datensätzen und Logs ausgewertet werden müssen. Compass hat deshalb immer wieder verschiedene Aspekte im Bereich APT, Forensik und Incident Response beleuchtet. Einerseits betreiben wir Research mit internen “Hack Labs” und “Research Weeks”, wo unsere Spezialisten sich mit den neusten Erkenntnissen der Scene auseinandersetzen bzw. diese weiter treiben und andererseits bearbeitet Compass in Zusammenarbeit mit den Security Fachabteilungen einer Vielzahl von Hochschulen, entsprechende Themen.

Eine entsprechend gewürdigte Maturaarbeit aus dem letzten Sommer möchten wir der Öffentlichkeit nicht länger vorenthalten und publizieren darum die Resultate im Rahmen dieses Posts. Im Mittelpunkt des Whitepapers steht die Analyse von APT mittels Splunk, einer spezialisierten Software zur Analyse von grossen Mengen maschinengenerierter Logdaten. Es werden darin auch alternative Wege zur Auswertung eruiert und ein Standardvorgehen für APT Fälle vorgeschlagen. Das Paper greift auch das bei Compass übliche Vorgehen bei forensischen Analysen auf und gibt dem technischen Leser in gewohnter Compass manier, viele technische Details mit auf den Weg. Natürlich auch einige Ideen, wie man das Logging von bestimmten Diensten optimieren könnte.

Möchten Sie gerne mehr zum Thema wissen? Möchten Sie auch erfahren, was dies in der Praxis bedeutet? Dann können wir Ihnen unser nächstes “Hands-on Seminar” mit dem Titel: Network Analysis & Advanced Persistent Threats vom 25. und 26. August 2015 in Bern empfehlen.

In der Zwischenzeit wünschen wir Ihnen viel Spass beim Schmökern. Behalten Sie einen kühlen Kopf compass_security_schweiz_whitepaper_apt_network_analysis_w_splunk_v1.1.pdf.

Compass Security Crew,
Cyrill Brunschwiler

Challenges in Log Management

Recently, SANS Institute has published the 9th log management survey (2014). The paper identifies strengths and weaknesses in log management systems and practices. It further provides advice to improve visibility across systems with proper log collection, normalization and analysis. Log management is very important to Compass as it heavily influences forensic investigations. Evidently, accurate information needs to be available to track down incidents. This post provides a short summary of the paper and reflects Compass research and experiences in these fields.

TL:DR; Positive is, that most of the companies have some sort of log management, at least most collect logs in some form – many do log to a central log server. In summary, log management is a well-established control within companies, but there are challenges (e.g. cloud services, differences in logging by different vendors) which companies cannot solve on their own and depend on the vendors and hosting providers. To differentiate between “good” and “bad” traffic is one of the biggest challenges.

The respondents of the survey rated the following activities as the biggest challenges in log management:

  • Distinguish between normal and suspicious traffic
  • Analysis of “big data” (large amount of volumes and types of log and events)
  • Normalization and categorization of logs and security information
  • Correlation of logs from various sources
  • Cloud causes log management headaches
  • Vendors log similar events differently

The first point “distinguish between normal and suspicious traffic” is clearly a problem – especially, if the infrastructure includes different technologies and vendors and exceeds a small environment. The bad thing is, malware and therefore the malicious traffic uses also “good” traffic to communicate with C&C servers. Here, baselining your logs could help. You might also want to understand applications in-depth and get some meaning from the user’s behavior – network analysis of the given parts could help you to understand the ‘average’ traffic.

The challenges with the cloud log management are rather new – but behind the scenes the same challenges exust. Look at cloud systems as they would be systems managed “by others” and not simply “by the cloud”. Challenge yourself with the same questions as you would challenge a hosted Unified Communications (UC) or a storage provider etc. What is logged? Where is it logged? How long are the logs being kept? Are the logs collected by the central log server? Will they be processed by the security information and event management (SIEM) systems?

The respondents in the survey clearly stated that collecting logs from the cloud is still difficult. Around half of the respondents say that they feel no need to monitor apps in the cloud. Many respondents say they rely on their cloud operator’s ISMS and security services, management and controls. Compass Security has some concern with this view – ONE SHOULD log and monitor all the required information as one would with in-house services. Graham Cluley said in a blog post: “Don’t call it ‘the cloud’. Call it ‘someone else’s computer’.”. Moreover, with the shift to the “cloud”, forensic analysis is getting a big challenge which companies are facing. If a cloud provider is not willing or simply unable to provide logs, you might want to evaluate another one. Some cloud providers actually allow to export logs. See Amazon and Cloudstack.

The top three reasons to collect logs are

  • detect and/or track suspicious behavior (e.g. unauthorized access, insider abuse)
  • support IT/Network routine maintenance and operations
  • support forensic analysis

Unfortunately, the respondents have issues to make meaningful use of the logs for:

  • detection/tracking of suspicious behavior
  • detection of APT-style malware
  • prevention incidents

In a recent presentation in Jona, Compass Security highlighted the difficulties to detect suspicious behavior and thus to detect APTs. It was presented how monitoring and APT traffic detection can be achieved with the correlation of logs of DNS, Mail, Proxy and Firewall. For this purpose, the logs have been enriched with external data like IP Reputation Lists, ZeuS Tracker, DNS Blacklists, Mail Black- and Greylists to identify potential malicious traffic. There are lots of other tricks which help to identify malicious traffic.

Besides the challenges and difficulties, the survey pointed out that SIEM infrastructures have become widely used to claim some form of automated processing and/or alerting of suspicious events. Automation is the key to managing and analyzing the large amounts of data. In recent years, normalization improved but fully “normalized” log information is still not available. Log engines will help to normalize and categorize events and log information systems for many different formats.

Interesting to see is how long the different respondents spend their time on analysis their logs. Most of them spend around 4h-8h a week on log analysis (of course, this depends on the company size). Not surprising was the fact, that regulatory compliance has been one of the main drivers for determining log data retention policies.

Regarding the current SSL (padding vulnerability) discussions, here are two examples of SSLv3 logging shown to identify downgrade attacks or to just see which clients still uses SSLv3. For apache this could be used:

CustomLog logs/ssl_request_log "%t %h \"{User-agent}i\" %{SSL_PROTOCOL}x %{SSL_CIPHER}x "

For nginx the following line could be used within your nginx configuration:

log_format ssl ''$remote_addr "$http_user_agent" $ssl_cipher $ssl_protocol

Offtopic: there is a good overview of different products and how to disable SSLv3.

How Compass can help you

If you like to have some hands-on practice and get a deeper inside how to detect APTs and how they work, Compass has the following upcoming courses regarding this hot topic:

These trainings use our Hacking-Lab in order to practice with log engines to analyze real-world examples. Furthermore, our classic “Beer Talk” series in September was about APT.

Compass Security can help you in the regards of testing your log environment with simulating directed attacks or simulating APT-style malware or by analyzing your log management concept.


While companies implemented log management with some basic log search functionality, detecting malware in real environments or collecting logs from the cloud is still difficult. Environments grow overtime and understanding the traffic within the infrastructure is key but a somewhat tedious and time consuming task. Log engines (e.g. IDH Framework, Splunk , Log Correlation Engine (LCE from TENABLE), ELK) help to collect and analyze log information. SIEM systems help to match and correlate different events. Script languages are needed to normalize data where the log engines reach their limits. Cloud providers must support the companies to log the relevant information or provide connectors for log engines. Furthermore, there is also a “Splunk in the cloud” solution.

Please comment, if I missed challenges or difficulties. I would also be interested in your experiences regarding log management.

Keywords: SIEM, log management, logging, normalization, APT, cloud, SANS