Sean look at all processes, tools, commands, and











8 – Advanced Pen Testing Paper









Hunting takes cybersecurity
to the next level by making it an active process in which security analysts
sniff out traces of cyber attackers and go in pursuit, relentlessly tracking and
hunting down their prey (Ashford, 2015). In anticipating opposition to
cyberattacks, organizations can build stronger defenses, because they can find
and fix vulnerabilities in their networks and systems before they are attacked
maliciously. Proactive defense is key to mitigating operational risk, because
cleaning up the aftermath of an attack is much more costly than proactive
defense strategies.

Hunters typically look at all
processes, tools, commands, and network file shares that are running in an
environment to find potential vulnerabilities that typical security systems,
like firewalls, antiviruses, etc., would miss because they are not malicious in
and of themselves, but a trained eye can recognize if something is
inappropriate, unlikely, or unusual, which can signal that something is wrong.

According to an interview by Computer Weekly, Ben Johnson of Bit9 + CarbonBlack
says that this innovation in cybersecurity arose because large, well-resourced
companies are getting hacked on a daily basis (Ashford, 2015). Because
attackers are always innovating and evolving their capabilities, there must
also be innovation and evolution of defense capabilities. Hunting typically
involves the most enthusiastic, passionate, and security driven security
analysts, because it is these individuals that enjoy proactively investigating
and not waiting for alerts or emergency calls to come in. They know how to
think like an attacker, act like an attacker, attack like an attacker, how to
communicate with the attackers, and the good ones can even infiltrate cyber
criminals’ minds and organizations to learn their techniques and find out what
their plans and deeds are. For example, according to another article by
Ashford, many hunters that work for security companies, such as RSA
FraudAction, do this, and are long-standing members of hacker forums, talking
directly to hackers (2016). This kind of proactive security is a bit extreme,
and as such, these actions are carried out by only the most dedicated hunters.

At the most basic level, hunters are looking for abnormal, unusual or
suspicious behavior, especially in relation to high-value data assets, wherever
there is risk and attackers may be active (Ashford, 2015), which could be
anywhere on a network, at any time, with or without real login information or
administrator privileges.

One of the reasons hunters must
exist and are in high demand is because attackers can mask their attacks to
look like normal network and/or system usage, which doesn’t get flagged by
automated security systems. For example, when an attacker steals valid user
credentials and uses them to log on to a network or network device, it is
difficult to detect them because there is no malware or malicious code; it
simply looks like a user has logged in to their account. A hunter would look
for multiple logins at the same time. A hunter could look for the terminal or
command line command to pull password hashes into a file, like the bkhive command, which dumps the syskey
bootkey from a Windows system hive, and the samdump2
command, which dumps Windows (up to Vista) passwords and hashes. This command
is not a command that a typical user would know, and so a hunter could collect
all processes and commands running on all endpoints of a network, making it
possible to identify compromised computers by tracking commands, like the
aforementioned Windows command, that most people don’t know about.

Advanced persistent threats
present significant challenges to the security community and changes how
organization need to view, implement and manage security operations, according
to Rackspace (2017). Advanced persistent threats occur when  attackers capable of breaching data
infrastructure through continuous targeting, and then remaining within that
infrastructure, undetected, to locate and access valuable information, and as
Daniel Clayton, a former British intelligence officer who now serves as a
director of security operations at Rackspace, describes, advanced persistent
threats are typically “groups of individuals that have the resources and
manpower to persistently target a company or organization 24 hours a day for as
long as it takes to get the job done” (Rackspace, 2017). While prevention
measures, like web application firewalls, intrusion detection and preventions
systems, and anti-virus software, can be effective against some attacks, like
DDoS, viruses, Trojans, and other attacks that remain consistent across all
platforms, the reality of advanced persistent threats has made many of these
measures obsolete in the modern world of cyber security. Effective security now
requires firms to assume penetration and continually and actively scan their
environments for malicious activity.

Modern security providers
deploy sophisticated technology and highly skilled analysts to actively patrol
environments and locate anomalies. Cyber hunting is a focused and iterative
approach to searching out, identifying, and understanding adversaries internal
to the defender’s networks, according to Lee (2016). The formal process of
threat hunting should not be confused with an attempt to prevent adversaries
from breaching the environment or for defenders to eliminate vulnerabilities in
the network (Lee, 2016).

There are three factors to
consider when judging an organization’s hunting ability: the quality of the
data they collect for hunting, the tools they provide to access and analyze the
data, and the skills of the analysts who use the data and the tools to find
security incidents. Bianco describes a hunting maturity model based on
primarily the skills of the analysts, because they are the ones who turn data
into detections (2015). The quality of the data that an organization routinely
collects from its IT environment is also a strong factor in determining the HMM
level. The more data (and the more different types of data) you provide to an
expert hunter, the more results they will find. The toolset for collecting and
analyzing the data is a factor as well, but a less important one. Given a high
amount of analyst skill and a large amount of good quality data, it’s possible
to compensate for toolset deficiencies, at least to a degree. The hunting
maturity model ranges from HMM0, the initial stage of maturity, in which, an
organization relies primarily on automated alerting tools, such as IDS, SIEM,
or antivirus, may incorporate feeds of signature updates or threat intelligence
indicators, and routinely collects little or no data, to HMM4, the leading
stage of maturity, in which an organization automates the majority of
successful data analysis procedures and routinely collects high levels of data.

In HMM4, organizations will turn any successful hunting process into operational,
automated detection, which frees analysts from the burden of running the same
processes over and over, and also allows them instead to concentrate on
improving existing processes or creating new ones. This makes HMM4
organizations extremely effective at resisting adversary actions, by allowing
them to focus their efforts on creating a stream of new hunting processes,
resulting in constant improvement to the detection program as a whole (Bianco,
2015). Both HMM0 and HMM4 organizations carry out automation, but the
automation that they carry out is different. HMM4 organizations always have
automation in the front of their minds as they create new hunting techniques,
whereas HMM0 organizations rely entirely on their automated detection, whether
it’s provided by a vendor or created in-house. They may spend time improving
their detection by creating new signatures or looking for new threat intel
feeds to consume, but they are not fundamentally changing the way they find
adversaries in their network. Even if they employ the most sophisticated security
analytics tools available, if they are sitting back and waiting for alerts,
they are not hunting. HMM4 organizations, on the other hand, are actively
trying new methods to find the threat actors in their systems. They try new
ideas all the time, knowing that some won’t pan out but others will. They are
inventive, curious, and agile, qualities you can’t get from a purely automated
detection product. Although a good hunting platform can certainly give your
team a boost, you can’t buy your way to HMM4. Bianco recommends HMM2 for CISOs
looking to start hunting operations (2015). HMM2 describes organizations that
are able to learn and apply procedures developed by others, and may make minor
changes, but are not yet capable of creating wholly new procedures themselves.

They routinely apply these procedures, if not on a strict schedule, then at
least on a somewhat regular basis.

A couple recommendations I
would make for organizations looking to implement hunting operations would be
to monitor endpoint process creation, as well as searching for indicators of
compromise. Many organizations look for logs to analyze but as Carvey describes
in his Dell SecureWorks presentation, a malicious attacker can repurpose syslog
so that logs aren’t giving proper information, and this would have to be
detected by monitoring for these processes (Carvey, 2015). Organizations should
look for endpoint processes that show artifacts or indicators that malicious
activity is occurring in the network. Indicators, like endpoint process
artifacts, can show lateral movements in internal networks. Web shells can be
used to gain access to an infrastructure, by compromising a web server, and
then moving to internal systems. Examples include is a Windows server running
Apache and WordPress or by manipulating an IIS server. An attacker can also
gain access with a web shell to an SQL server from a web server. The attacker
can gain access to a web server, put a web shell on it, and with RDP access on
both servers, the attacker can access the web shell in Internet Explorer by
connecting it to localhost. Then they
use the web shell to issue SQL injection commands, using xp_cmdshell and then create a user account on the SQL server. This
can be found by looking through the browsing history, to see where the attacker
was accessing localhost. In this
case, the organization wouldn’t have event logs, because the attacker deleted
the web shell after they were done, but there would be logs in the web server.

Another file system indicator can be found on IIS servers with ASPX web shells,
because the first time it is accessed, the .NET framework creates a page called

In other words, the framework actually compiles it. These are file system artifacts
that a hunter should look for when looking for advanced persistent threats,
because attackers can come in, install a web shell, delete it after use, and
repeat this process as much as they want, all the while going undetected in the
network, because they actually created a legitimate login to the SQL or IIS
server. Only if someone was actively looking for those indicators would they
find out that web shells had been installed by malicious users. If an attacker
crashes a web browser, it will create a session restore file, which, if the
attacker doesn’t reinitiate the browser to delete that file, will remain on the
system. Parsing through a compromised system after it has been taken offline
will allow the forensics team to find these files and see what commands were
issued through the web shell, as well as the username and password that the
attacker used to access the SQL server, because the username and password would
get stored in a config file. Carvey
states that clusters of indicators, not individual artifacts, should be looked
for, “because there are a lot of things that go on within an infrastructure
that, if you look at them in isolation from everything else, could look like
threat actor activity, because a lot of the stuff that we see threat actors
doing is stuff that a normal admin might do” (2015).

Process creation monitoring
is useful in live detection of attacks being carried out. This enables security
professionals to see commands used by attackers, as they are being used, like
checking the time of the remote system, checking to see if the task is
completed, reissuing the task. Hunters should look to see if a process was
created, when it was created and compare that to the hours of operation of that
organization or the working hours of the person that normally uses that endpoint
and other clusters of indicators like registry keys, passwords that were used,
event logs, file systems, etc. Take, for example, the sticky keys attack. In
the Windows registry, there is a key called image file execution options with
spaces between all the words, that Microsoft left in place so that users can
add debugging capabilities to binaries. An attacker can modify this registry
key, via RDP access to the system, with the reg.exe
command line utility. The attacker creates a subkey for one of the two
accessibility tools, hc.exe or utilman.exe, and points the debugger
value to cmd.exe. Even if all the
passwords in the organization’s infrastructure are changed, and the attacker
can still access the infrastructure, all they must do is RDP to that system,
and when the login screen shows up, instead of inputting credentials, they just
hit the shift key five times, and get a system level command prompt. Attackers
use command line tools to do anything on a system. Once in, they can create
users, change passwords, dump passwords, and anything else. The only way to
detect this is to monitor for process creation and see that cmd.exe is being launched in places that
it shouldn’t be, perhaps at times or on systems that should show no use, or on
systems on which users should not be launching cmd.exe. Another suggestion I would make is to make use of shimcache and amcache. This allows systems administrators to see what has been run
on a system and when and for how long. This can be started by running it
through Python directly or by making a Windows EXE from the Python script,
provided on its GitHub Page,

ShimCache data should be collected and analyzed from all Windows endpoints in
an organization, both clients and servers. Servers are particularly important,
because they are “usually the number one initial entry point for breaches,
especially internet-facing servers, or other servers and DMZs,” says David
Sharpe, in his DerbyCon 2015 talk (Sharpe, 2015). Amcache replaced Shimcache,
starting with Windows Server 2012 and Windows 8, and provides the same function
as Shimcache, but has more useful fields for hunting, such as an SHA1 hash of
the file, as well as more useful timestamp fields. Data from these caches
should be stacked and analyzed for sequences of recon activity, net commands,
pings, archivers, like RAR, being ran, and EXEs running out of abnormal
locations on the disk. An example would be if an Amcache timeline were created,
and EXEs were found, being run from the C:users location, this would be an
abnormal location for EXEs to be run, as this does not normally occur.

I would also recommend mining
server antivirus logs, because they are a consistent, high yield data source to
hunt for intrusions, which is especially true for internet-facing assets.

According to Sharpe, about 20% of all targeted intrusions have AV fire
somewhere along the timeline (2015). If an intrusion attempt has progressed far
enough along to where an AV product triggers, then that is helpful. At best, there
will be a blocked intrusion, but there will still be an exploitable hole that
needs to be addressed. The worst case scenario is that the intrusion is far
along and AV picked up one tool in a long series of events that need to be
addressed. Things to look for include web shells, AV detections while the file
is under webroot or C:windows, any kind of backdoors and malware street names
identified by intelligence sources and experience. This should be supplemented
by custom host intrusion prevention systems detection, with HIPS rules
targeting how malware tools work. Look for credential dumpers, like WCE,
pwdump, gsecdump, fgdump, or Mimikatz.

Netstat data should be mined
to find rogue listeners across all endpoints, especially servers on the network
edges. The command netstat -nabo to
pull the data and mine it. An example of an indicator of compromise could be if
one TCP port has bound to it multiple process names and paths on a single
system. This would be impossible on a normally-running system. In this case,
intruder activity could be interleaved with legitimate SQL server activity.

Netstat data output should be stacked for all internet-accessible servers by
listening port, and see how many ports show up just once. This data should also
be stacked by the full path to the process’ binary, and see how many paths show
up just once. Additionally, all output should be preserved as a baseline, and
all new listeners that appear over time, especially those across
internet-facing systems, should be tracked (Sharpe, 2015).

There are many companies that
offer proactive hunting services for fees, but I would recommend that an
organization also have in-house hunters that are proactively seeking out
cyberattacks. Outside consultation should be utilized in order to improve
in-house hunting. In striving to be an organization with competent
cybersecurity measures in place, the organization should collect very large
amounts of data from across the enterprise and at all endpoints. All the
suggestions I have made in this paper have involved compiling large amounts of
data sets to find abnormalities in the operations of the organization. It is
only with these data sets that we can analyze the data and find indicators of












Ashford, W. (2015, October 13). Cyber security
innovation is crucial, says security evangelist. Retrieved December 15, 2017,

Ashford, W. (2016, March). Hunters: a
rare but essential breed of enterprise cyber defenders. Retrieved December 15,
2017, from

Bianco, D. (2015, October 15). A Simple
Hunting Maturity Model. Retrieved December 15, 2017, from

Carvey, H. (2015, July 25). BsidesCincy
2015 01 Lateral Movement Harlan Carvey. Retrieved December 15, 2017, from

Lee, R. M. (2016, February). The Who,
What, Where, When, Why and How of Effective Threat Hunting. Retrieved December
15, 2017, from

Rackspace. (2017, September 29). AGE OF
PARADIGM. Retrieved December 15, 2017, from

Rackspace. (2017). ENTERPRISE SECURITY
TODAY – WHY SPEED MATTERS. Retrieved December 15, 2017, from

Sharpe, D. (2015, September 28). Fix Me19
Intrusion Hunting for the Masses A Practical Guide David Sharpe. Retrieved
December 15, 2017, from