Introduction

Every year Bitkom and others publish the statistics of detected attacks.
These numbers are increasing (30% in 2014, 51% in 2015, 69% in 2016 each regarding the last two years).

According to annual Mandiant reports, the duration until an attack is detected is 148 days in median. 148 days is a long time to gain persistence, hide, and gather assets.

Have you yet been targeted too? Would you notice? How long before you would actually notice? Would you be prepared? What would you do if it happened right now?
You would probably call in the forensic specialists. They most likely don’t know your network yet.
Which systems do they have to analyze and what information can you provide? Would you find the anomalies that should not be there if systems seem to work but are just a little slow?
Have you recorded all information and traces that the attacker left?
Analyzing a system takes days. The same can be said for log aggregation etc. You will want to know what happened and how to kick adversaries out of your perimeter now and not start with an inventory of your assets and assigning responsibilities.
The importance of forensic readiness increases. Attackers are moving forward and refine their skills and methodologies and so should you.
A skilled attacker might quickly know the network better than you. By the time your forensic analyst starts the investigation the analyst might already be 148 days behind the attacker, in their knowledge of your networks and topologies. And he is covered with work and status meetings. How can you compete when the bar is forever being raised by ever more competent adversaries?
Remember you need as much capacity for business continuation processes as possible. If this is the time you start firefighting the battle might get lost.

What about insurance and Forensic Teams?

Insurance won’t prevent an incident from occurring. You make a bet and invest money that you can recuperate once an incident occurs.
Most damage is however not only monetary. It does also not just help to detect an incident. It does not help against loss of intellectual property or negative publicity.
A DFIR (Digital Forensic and Incident Response) Specialist can only work on the basis of the information that you have gathered and can provide. Evidence that is lost can’t be acquired retroactively. Extending the downtime also costs money apart from the costs for the investigation. Analyzing a corporate network as a blackbox that is not prepared for an investigation and search for anomalies is inefficient, error-prone and in the worst case not effective. In practice the internal incident response team and external forensic analysts will have to work hand in hand.

Where to start?

Let’s structure this a bit:

you, enemy, allies, plan, equip, train, test
 

Know yourself

How well are you prepared and what are your strength and your weaknesses?

Is your staff ready for an incident? Forensic Analysts will dig into log files, images of hard disks and memory dumps.
The analysts will sniff for traffic asking you whether that specific stream is legit or Ask you if your system should have those patches not installed.
Do you know where the offline backups of X and Y are? Did you log the important stuff? Do you know the resources you can offer for investigation?
Answer yourself questions like What assets do you have? What is critical? What information can I monitor? Do I know the state of my network and systems?
Where is it documented? How far can I see in the past? What can I see? If one asset is lost what happens to the other?

How good are the defenses? On whom and what do you rely on? What happens to your defenses if a resource fails? How quickly can I build up new defenses? What is your reach?

In short, what is your forensic readiness?

Know your Enemy

Your imminent question will be who the threat actor is and what they have done. The truth is you will probably never know for sure. The origin is the last thing you will ever get to know, and attribution is a hard problem.
IP addresses can be abused. Tools usage  obfuscated and false flag attacks are common. (e.g. vault 7)

But you can look at the “Lessons Learned” sections from APT reports. State of the Art of Attacks. Gather information from CERTs. Stay informed about security news.

Turn the table – if you were the attacker what would you do? Where is the profit? The damage?
What would hurt? Consult specialists to improve that point of view.

What can an attacker achieve with a specific level of access. How can the attacker elevate privileges? Where can the attacker pivot in the network? What traceable actions left information in our logs? Do you collect them?

Categories of Attackers

There have been lots attempts to categories of adversaries depending on skill levels, focus and motivation. The reality is blurry. The complexity of an attack does not solely depend on the individual skill level as tools and services and information are shared and can be acquired for an operation. Toolkits and hacker gadgets are available on low costs. The attacker does not have to break a vulnerability himself or to develop the tools himself. Parts can be hired as a service. Motivation and skill level do not have to be aligned.
You have to establish clear barriers between the attack, the persistence and the actual malicious actions:

Attacks

Depending on the objective the attacker might use different approaches. Physical access, Social Engineering, Phishing, Drive By, Hacking, Fraud are only a few to mention.
Social engineering targets the human being as a weakness. It is easy to find at least someone but the action can later be identified. Phishing leaves digital traces but is successful if suffices to affect only one in a group. Drive by Exploits performed by exploit kits like angler depend on the security level of the target machine and the Patchlevel. Good Zero Day exploits are rare and expensive and probably burned after detection. The availability and costs against older systems or known vulnerabilities is good. Hacking in this context means an attacker from the internet analyses an external system and pivots step by step into the internal perimeters. This is more advanced but can be achieved even against aware targets .
Depending on the objective combinations can be used.

Persistence

After the initial infection of a system, the attacker tries to escalate privileges, gain persistence and establishes command and control channels (C2 channel) to a C2 Server where further instructions to the bots are sent.

Rootkits

The aim of a rootkit is to gain persistence on a system and protect against detection of the intrusion. Depending on the location of the rootkit there are several categories:

• Ring 3 – Userland (additional programs or libraries, patched files, configuration etc.) Hooking and injection
• Ring 0 – Kernelmode. Hard to detect since all calls from Userspace are controlled by the Operating system kernel
• Ring -1 – Hypervisor (bluepill )
• Ring -2 – SMM (BIOS, UEFI etc.)
• Ring -3 – Hardware

If the target is part of a Command and Control structure it will create a covered C2 Channel to a C2 Server and call home for further instructions. Systems located behind a perimeter firewall will have to initiate the connection.

Backdoor vs. Bug-Door

Another hard to detect backdoor is the so-called “bug-door”. A system is either downgraded or configured in an insecure way. This is hard to detect if the analyst searches for malware or unknown files on the system. The system consists of legitimate files (that might even be signed) besides the fact that it has known vulnerabilities. Patch your systems and keep logs as proof of your due diligence.

“Golden Ticket”

Another persistence attack is the “Golden Ticket”, a Kerberos ticket with unlimited lifetime. With such an undetected ticket an attacker might perform actions on any machine on any later time. To renew the keys and invalidate the golden ticket the krbtgt password has to be reset twice.

Fileless and in Memory Attacks

There are several techniques to run code on a machine without writing files to disk. This limits the success of filebased analysis like disk forensics or file based AV scanners. One common technique is reflective DLL loading. Here a DLL is loaded from memory and e.g. injected into the address space of another process. To make the DLL active in a foreign process DLL hooking techniques are used. This can be detected by memory analysis.

Multi Stage Attacks

Multi stage attacks start with a dropper and inject the actual more valuable in malware later.
Fileless attacks e.g. like reflective DLL loading and advanced obfuscation techniques can be used to cover detection.
Note up to this point it is not possible to know what the compromised machine is actually up to.
Even with memory analysis only a snapshot of the machine can be analyzed.
Now we know why we can’t answer the imminent questions at once.
But attackers also make mistakes.
It is hard to cover all tracks especially if there are so many devices listening and logging out there.
The attacker has to tamper with or scrub all possible evidence. The analyst just has to find one footprint or indicator of compromise.
One footprint is not enough to describe the path but enough to proof existence.

Unknown attacks

Expect the unexpected. Attacker are the first movers here. We watch and learn how their techniques improve. Just looking for known patterns might not be enough.
If you just focus on what you know and what you see you might spend your time solely on smoke bombs placed to consume the time of the investigator.

Acquire Allies

Depending on the size of your organisation, you will need:

• Incident Manager
• Information Security
• IT Staff
• IT Auditor
• Security
• Attorney
• Human Resources
• Public Relations
• Financial Auditor
• Forensic Analysts
• Law Enforcement

You will also want to have contacts to CERTs and other CSIRTs to exchange information and know how.

Make a plan

You have organizational and technical preparation tasks.

Organizational

Build a CSIRT (Computer Security Incident Response Team) in advance. Note down contacts lists and replacement candidates. Be clear who has which role and which capabilities and responsibilities.

Also make Business Continuity and Disaster Recovery plans in advance.

Communication

Define the first point of contact for the detection of an incident.
Define rules who is informed as well as when and how.

These rules should cover insider attacks. Avoid FUD – fear, uncertainty and doubt – as this will deplete resources for managing the situation unnecessarily.

Verify Incident

Define rules for a quick check whether the Indicator of Compromise (IoC) is real. For details see Live Forensics.

Define:

• Who verifies?
• What is verified?
• How is it verified?
• How is it documented?
• What is the escalation plan?
• Communication?
• Skills

DefCon

Define the Defense Condition. According to the threat level the DefCon defines the resources that are required for the handling of the incident. According to the DefCon everybody should know how much resources are required for the handling of an incident.

War Room

You will need a central location where your information is centralized and your course of action pre-planned. All participants in the investigation should leave information and status here and get new directions. The war room should be equipped with out of band communication and infrastructure that is air-gapped from the compromised networks.

Chain of Custody

When dealing with IoC (Indicators of Compromise) that can be used as evidence in a court it is crucial to obey the chain of custody. There are Incident Response Sheets that have to be filled out for evidence. And procedures for acquiring IoCs and data.

Equip

Hardware

Acquire equipment for forensic analysis of your exotic hardware. Even well equipped forensic jump bags have limitations.
(harddisks and adapters, raid controllers, hardware for memory snapshots, etc.)
If you use specialized hardware it could cost valuable time to get these things in place.

Network and Systems

• Separate network in zones
• Silence our network
• Learn the regular network traffic and logs
• Limit the accessible services
• Use the “least privilege” principle
• Do not reuse accounts or passwords
• Know all internet breakouts and limit them
• Control also outbound traffic
• Have non-administrator accounts that are not part of the domain especially for crucial systems.
• Use jump hosts
• Use Netflow and network monitoring
• Inventory
• Network plans
• Track of configuration
• List active accounts and permissions
• Identify suitable places for network monitoring
• Be aware of what is and what is not logged
• Be able to do full packet capture if needed
• IDS Systems
• Identify technical and organizational obstacles for reaction

Logging

• Use centralized log server (collect also Windows Event Logs centrally)
• Synchronize the time and timezone of each device, document drifts
• Adjust Log-Level and Log the right events
• Successful and unsuccessful Windows Log On and Log Off Events (AD and local)
• Log DHCP,  DNS, Firewall and proxy
• Log Post Data on Proxy Server (fine tuning necessary)
• Store also successful connections in your firewall logs
• Track your logging configuration with the logs
• Use offline Backups with integrity checks
• Use log aggregation Software and be able to monitor and query in realtime
• Use Firewall logging and adjust firewalls to log successful connections as well
• Keep offline backups of log files with integrity checksums and protect them
• Keep your logs long enough
• Be aware of the amount of logs and do not flood yourself
• If systems are run by third parties make sure you have access to logs and configuration

You may want to read more about logging and SIEM here.

Exchange has a lot of logging features that are interesting for forensic analysis.

Memory Forensics

Essential for fileless attacks and those where tracks on harddisk are covered.

• Identify possibility of getting memory images per system
• Identify how a memory image can be taken
• How long does it take to get a memory Image
• Is additional hardware necessary to gather memory images (DMA, Firewire, Thunderbolt, Forensic RAM Extraction Device)
• Are there preconditions that have to be fulfilled to gather memory images (BIOS/UEFI Settings)

You might be interested in volatility lime, winpmen, inception and commercial solutions.

BIOS / UEFI Analysis

This is the dark arts chapter .
Persistence below Ring 0 are extremely difficult to analyse.
To be sure that you can trust your data you need to be one layer lower than your foe. A rootkit running at a lower layer might fool the analysis Tools.
When doing Live Response the analysed data is taken from a running system.

Some layer -1 attacks can be detected by side channel analysis. Forensic analysis of layer 0 rootkits is advanced all layers below are objective to research and bleeding edge solutions.

First Malware scanners for filebased UEFI Bootkits are available (and here. Runtime and in memory analysis is not possible by the time writing.

But here is what you can do:

• Dump all UEFI / BIOS versions
• Document firmware updates and downgrades

Train

Your staff will have to fulfill some tasks during a forensic investigation. At least be able to provide all information that is needed for the investigation and provide insight to the systems.

Live Forensics

Interaction with a potentially compromised system should only be performed by trained experts.
First target is to identify if the Indicator of Compromise is actually an Incident and not a regular anomaly.

Dos and Don’ts

Dos

• Be aware that the tools on the system might already be compromised and provide an incorrect picture. Better use external write protected, statically built tools that do not rely on system libraries.
• Rebooting a system removes all in memory traces. Fileless and in memory attacks can’t be found anymore – if you aren’t lucky to find them in swap spaces. If taking memory snapshots and live analysis is not possible it might be better to either suspend the system or turn it physically of to avoid shutdown hooks. Memory analysis of hibernate files is possible but data on disk might get overwritten
• Think before making backups. Access timestamps are overwritten by the time you make your archives. If you want to find out if data leakage happened you cover the tracks.

Don’ts

• Don’t start an upgrade or install or uninstall software. You destroy evidence and the state of the system and probably overwrite deleted data related to the attack.
• An AV Scan run on a compromised system might give false results and also overwrites access timestamps and probably traces in memory.
• Starting an AV Scan from a bootable media needs a reset destroying in memory information and if the disk is mounted read write the AV can also destroy evidence. Better run the AV scan after the forensic image is taken and only on a read only copy.
• Never log in as a domain administrator on a potentially compromised system for analysis. An attacker with local administrator privileges might use memory carving tools like mimikatz and steal the credentials and perform Pass-the-Hash (PtH) Attacks later on. The information that the investigation is taking place is also leaked (Fog of War). You will probably need privileges Access for most investigative actions but better use local administrative accounts.
• Do not resolve DNS Names or contact suspicious IP Addresses. Never. You will inform the attacker. If you want to check DNS Names and IP addresses use ipvoid or urlvoid instead (www.ipvoid.com)
• Do not upload malware from targeted attacks directly to sites like virustotal (www.virustotal.com). You will generate a permanent, searchable database entry with timestamp source and filename informing your attacker. Search the hash instead and be alert if your binary is not found in the database.

OODA Loop

The model for reactions to a specific situation is called OODA Loop. This is short for Observe Orient Decide Act. You gather information, put the information in context, decide how to react and perform the action. Afterwards you observe the effect of your reaction.

OODA Loop Flooding

An attacker can try to break your OODA Loop by flooding you with actions you have to react on. Ideally the attacker interrupts your before you are actually able to perform a proper reaction holding you in the loop without leaving you the possibility to act. Identify such situations and avoid being flooded.

Fog of War

Both attacker and responders have a Fog of War. The attacker learns the premises starting from the initial attack point pivoting through the assets.
The responders have a fog of war starting with the indicators of compromise and the level of information that is available.
Both parties try to move without revealing the proceedings to the opposing party.
If you warn the attacker the will cover their tracks and stop the attack. It won’t be possible to trace or get them caught in the act anymore.

Test

“We don’t rise to the level of our expectations, we fall to the level of our training.” ― Archilochos

You will have to test whether your forensic readiness actions have been successful.
What has changed? What have you achieved? What is your new forensic readiness level and what are the lessons learned?
Better train the response in a simulated attack.
Incident Response can be a challenging adventure. Whether fortune is on your side might be a matter of your forensic readiness. What if it happened while you were reading…