Introduction To Splunk & SPL

What Is Splunk?

Splunk is a highly scalable, versatile, and robust data analytics software solution known for its ability to ingest, index, analyze, and visualize massive amounts of machine data. Splunk has the capability to drive a wide range of initiatives, encompassing cybersecurity, compliance, data pipelines, IT monitoring, observability, as well as overall IT and business management.

Splunk's (Splunk Enterprise) architecture consists of several layers that work together to collect, index, search, analyze, and visualize data. The architecture can be divided into the following main components:

1. Forwarders:

• Universal Forwarder (UF)

• Heavy Forwarder (HF)

2. Indexers

3. Search Heads

4. Deployment Server

5. Cluster Master

6. License Master

Splunk's key components include:

• Splunk Web Interface

• Search Processing Language (SPL)

• Apps and Add-ons

• Knowledge Objects

Splunk As A SIEM Solution

When it comes to cybersecurity, Splunk can play a crucial role as a log management solution, but its true value lies in its analytics-driven Security Information and Event Management (SIEM) capabilities. Splunk as a SIEM solution can aid in real-time and historical data analysis, cybersecurity monitoring, incident response, and threat hunting. Moreover, it empowers organizations to enhance their detection capabilities by leveraging User Behavior Analytics.

Let's assume that main is an index containing Windows Security and Sysmon logs, among others.

Basic Searching

The search command is typically implicit at the start of each SPL query and is not usually written out. However, here's an example using explicit search syntax:

search index="main" "UNKNOWN"

By specifying the index as main, the query narrows down the search to only the events stored in the main index.

Note: Wildcards (*) can replace any number of characters in searches and field values. Example (implicit search syntax):

index="main" "*UNKNOWN*"

Fields and Comparison Operators

Splunk automatically identifies certain data as fields (like source, sourcetype, host, EventCode, etc.), and users can manually define additional fields.

index="main" EventCode!=1

The fields command

The fields command specifies which fields should be included or excluded in the search results. Example:

index="main" sourcetype="WinEventLog:Sysmon" EventCode=1 | fields - User

After retrieving all process creation events from the main index, the fields command excludes the User field from the search results. Thus, the results will contain all fields normally found in the Sysmon Event ID 1 logs, except for the user that initiated the process.

The table command

The table command presents search results in a tabular format. Example:

index="main" sourcetype="WinEventLog:Sysmon" EventCode=1 | table _time, host, Image

The rename command

The rename command renames a field in the search results. Example:

index="main" sourcetype="WinEventLog:Sysmon" EventCode=1 | rename Image as Process

This command renames the Image field to Process in the search results. Image field in Sysmon logs represents the name of the executable file for the process.

The dedup command

The 'dedup' command removes duplicate events. Example:

index="main" sourcetype="WinEventLog:Sysmon" EventCode=1 | dedup Image

The sort command

The sort command sorts the search results. Example:

index="main" sourcetype="WinEventLog:Sysmon" EventCode=1 | sort - _time

This command sorts all process creation events in the main index in descending order of their timestamps (_time), i.e., the most recent events are shown first.

The stats command

The stats command performs statistical operations. Example:

index="main" sourcetype="WinEventLog:Sysmon" EventCode=3 | stats count by _time, Image

This query will return a table where each row represents a unique combination of a timestamp (_time) and a process (Image). The count column indicates the number of network connection events that occurred for that specific process at that specific time.

The chart command

The chart command creates a data visualization based on statistical operations. Example:

index="main" sourcetype="WinEventLog:Sysmon" EventCode=3 | chart count by _time, Image

This query will return a table where each row represents a unique timestamp (_time) and each column represents a unique process (Image).

The eval command

The eval command creates or redefines fields. Example:

index="main" sourcetype="WinEventLog:Sysmon" EventCode=1 | eval Process_Path=lower(Image)

This command creates a new field Process_Path which contains the lowercase version of the Image field. It doesn't change the actual Image field, but creates a new field that can be used in subsequent operations or for display purposes.

The rex command

The rex command extracts new fields from existing ones using regular expressions. Example:

index="main" EventCode=4662 | rex max_match=0 "[^%](?<guid>{.*})" | table guid

• rex max_match=0 "[^%](?<guid>{.*})" uses the rex command to extract values matching the pattern from the events' fields. The regex pattern {.*} looks for substrings that begin with { and end with }. The [^%] part ensures that the match does not begin with a % character. The captured value within the curly braces is assigned to the named capture group guid.

• The max_match=0 option ensures that all occurrences of the pattern are extracted from each event. By default, the rex command only extracts the first occurrence.

The lookup command

The lookup command enriches the data with external sources. Example:

Suppose the following CSV file called malware_lookup.csv.

filename, is_malware
notepad.exe, false
cmd.exe, false
powershell.exe, false
sharphound.exe, true
randomfile.exe, true

This CSV file should be added as a new Lookup table as follows.

index="main" sourcetype="WinEventLog:Sysmon" EventCode=1 
| rex field=Image "(?P<filename>[^\\\]+)$" 
| eval filename=lower(filename) 
| lookup malware_lookup.csv filename OUTPUTNEW is_malware 
| table filename, is_malware

• | rex field=Image "(?P<filename>[^\\\]+)$": This command is using the regular expression (regex) to extract a part of the Image field. The Image field in Sysmon EventCode=1 logs typically contains the full file path of the process. This regex is saying: Capture everything after the last backslash (which should be the filename itself) and save it as filename.

• | lookup malware_lookup.csv filename OUTPUTNEW is_malware: This command is performing a lookup operation using the filename as a key. The lookup table (malware_lookup.csv) is expected to contain a list of filenames of known malicious executables. If a match is found in the lookup table, the new field is_malware is added to the event, which indicates whether or not the process is considered malicious based on the lookup table.

In summary, this query is extracting the filenames of newly created processes, converting them to lowercase, comparing them against a list of known malicious filenames, and presenting the findings in a table.

An equivalent that also removes duplicates is the following.

index="main" sourcetype="WinEventLog:Sysmon" EventCode=1 
| eval filename=mvdedup(split(Image, "\\")) 
| eval filename=mvindex(filename, -1) 
| eval filename=lower(filename) 
| lookup malware_lookup.csv filename OUTPUTNEW is_malware 
| table filename, is_malware 
| dedup filename, is_malware

• | eval filename=mvdedup(split(Image, "\\")): This command is splitting the Image field, which contains the file path, into multiple elements at each backslash and making it a multivalue field. The mvdedup function is used to eliminate any duplicates in this multivalue field.

• | eval filename=mvindex(filename, -1): Here, the mvindex function is being used to select the last element of the multivalue field generated in the previous step. In the context of a file path, this would typically be the actual file name.

In summary, this SPL query searches the Sysmon logs for process creation events, extracts the file name from the Image field, converts it to lowercase, matches it against a list of known malware from the malware_lookup.csv file, and then displays the results in a table, removing any duplicates based on the filename and is_malware fields.

The inputlookup command

The inputlookup command retrieves data from a lookup file without joining it to the search results. Example:

| inputlookup malware_lookup.csv

This command retrieves all records from the malware_lookup.csv file. The result is not joined with any search results but can be used to verify the content of the lookup file or for subsequent operations like filtering or joining with other datasets.

Time Range

Every event in Splunk has a timestamp. Using the time range picker or the earliest and latest commands, you can limit searches to specific time periods. Example:

index="main" earliest=-7d EventCode!=1

The transaction command

The transaction command is used in Splunk to group events that share common characteristics into transactions, often used to track sessions or user activities that span across multiple events. Example:

index="main" sourcetype="WinEventLog:Sysmon" (EventCode=1 OR EventCode=3) 
| transaction Image startswith=eval(EventCode=1) endswith=eval(EventCode=3) maxspan=1m 
| table Image 
|  dedup Image 

• | transaction Image startswith=eval(EventCode=1) endswith=eval(EventCode=3) maxspan=1m: The transaction command is used here to group events based on the Image field, which represents the executable or script involved in the event. This grouping is subject to the conditions: the transaction starts with an event where EventCode is 1 and ends with an event where EventCode is 3. The maxspan=1m clause limits the transaction to events occurring within a 1-minute window. The transaction command can link together related events to provide a better understanding of the sequences of activities happening within a system.

In summary, this query aims to identify sequences of activities (process creation followed by a network connection) associated with the same executable or script within a 1-minute window.

Subsearches

A subsearch in Splunk is a search that is nested inside another search. It's used to compute a set of results that are then used in the outer search. Example:

index="main" sourcetype="WinEventLog:Sysmon" EventCode=1 NOT 
[ search index="main" sourcetype="WinEventLog:Sysmon" EventCode=1 | top limit=100 Image | fields Image ] 
| table _time, Image, CommandLine, User, ComputerName

• NOT []: The square brackets contain the subsearch. By placing NOT before it, the main search will exclude any results that are returned by the subsearch.

• search index="main" sourcetype="WinEventLog:Sysmon" EventCode=1 | top limit=100 Image | fields Image: The subsearch that fetches EventCode=1 (Process Creation) events, then uses the top command to return the 100 most common Image (process) names.

This query can help to highlight unusual or rare processes, which may be worth investigating for potential malicious activity. Be sure to adjust the limit in the subsearch as necessary to fit your environment.

As with any language, proficiency comes with practice and experience. Find below some excellent resources to start with:

• https://docs.splunk.com/Documentation/SCS/current/SearchReference/Introduction

• https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/

• https://docs.splunk.com/Documentation/SplunkCloud/latest/Search/

How To Identify The Available Data

Data and field identification approach 1: Leverage Splunk's Search & Reporting Application (SPL)

Splunk can ingest a wide variety of data sources. We classify these data sources into source types that dictate how Splunk formats the incoming data. To identify the available source types, we can run the following SPL command, after selecting the suitable time range in the time picker of the Search & Reporting application.

| eventcount summarize=false index=* | table index

This query uses eventcount to count events in all indexes, then summarize=false is used to display counts for each index separately, and finally, the table command is used to present the data in tabular form.

| metadata type=sourcetypes

This search uses the metadata command, which provides us with various statistics about specified indexed fields. Here, we're focusing on sourcetypes. The result is a list of all sourcetypes in our Splunk environment, along with additional metadata such as the first time a source type was seen (firstTime), the last time it was seen (lastTime), and the number of hosts (totalCount).

| metadata type=sourcetypes index=* | table sourcetype

Here, the metadata command retrieves metadata about the data in our indexes. The type=sourcetypes argument tells Splunk to return metadata about sourcetypes. The table command is used to present the data in tabular form.

| metadata type=sources index=* | table source

This command returns a list of all data sources in the Splunk environment.

Once we know our source types, we can investigate the kind of data they contain. Let's say we are interested in a sourcetype named WinEventLog:Security, we can use the table command to present the raw data as follows.

sourcetype="WinEventLog:Security" | table _raw

A better approach is to identify the fields you are interested in using the fields command as mentioned before, and then specifying those field names in the table command. Example:

sourcetype="WinEventLog:Security" | fields Account_Name, EventCode | table Account_Name, EventCode

If we want to see a list of field names only, without the data, we can use the fieldsummary command instead.

sourcetype="WinEventLog:Security" | fieldsummary

This search will return a table that includes every field found in the events returned by the search (across the sourcetype we've specified). The table includes several columns of information about each field:

• field: The name of the field.

• count: The number of events that contain the field.

• distinct_count: The number of distinct values in the field.

• is_exact: Whether the count is exact or estimated.

• max: The maximum value of the field.

• mean: The mean value of the field.

• min: The minimum value of the field.

• numeric_count: The number of numeric values in the field.

• stdev: The standard deviation of the field.

• values: Sample values of the field.

Please note that the values provided by the fieldsummary command are calculated based on the events returned by our search. So if we want to see all fields within a specific sourcetype, we need to make sure our time range is large enough to capture all possible fields.

index=* sourcetype=* | bucket _time span=1d 
| stats count by _time, index, sourcetype 
| sort - _time

This query retrieves all data (index=* sourcetype=*), then bucket command is used to group the events based on the _time field into 1-day buckets. The stats command then counts the number of events for each day (_time), index, and sourcetype. Lastly, the sort command sorts the result in descending order of _time.

index=* sourcetype=* | rare limit=10 index, sourcetype

The rare command can help us identify uncommon event types, which might be indicative of abnormal behavior. This query retrieves all data and finds the 10 rarest combinations of indexes and sourcetypes.

index="main" | rare limit=20 useother=f ParentImage

This command displays the 20 least common values of the ParentImage field.

index=* sourcetype=* 
| fieldsummary 
| where count < 100 
| table field, count, distinct_count

This search shows a summary of all fields (fieldsummary), filters out fields that appear in less than 100 events (where count < 100), and then displays a table (table) showing the field name, total count, and distinct count.

index=* | sistats count by index, sourcetype, source, host

We can also use the sistats command to explore event diversity. This command counts the number of events per index, sourcetype, source, and host, which can provide us a clear picture of the diversity and distribution of our data.

index=* sourcetype=* 
| rare limit=10 field1, field2, field3

The rare command can also be used to find uncommon combinations of field values. Replace field1, field2, field3 with the fields of interest. This command will display the 10 rarest combinations of these fields.

Practical Exercises

1) Navigate to http://[Target IP]:8000, open the "Search & Reporting" application, and find through an SPL search against all data the account name with the highest amount of Kerberos authentication ticket requests. Enter it as your answer.

index="main" sourcetype="WinEventLog:Security" EventCode=4769
| stats count by Account_Name
| sort - count

Answer: waldo

2) Navigate to http://[Target IP]:8000, open the "Search & Reporting" application, and find through an SPL search against all 4624 events the count of distinct computers accessed by the account name SYSTEM. Enter it as your answer.

index="main" sourcetype="WinEventLog:Security" EventCode=4624 
| stats dc(ComputerName) as Unique_Computers by Account_Name
| sort - Unique_Computers

3) Navigate to http://[Target IP]:8000, open the "Search & Reporting" application, and find through an SPL search against all 4624 events the account name that made the most login attempts within a span of 10 minutes. Enter it as your answer.

index="main" sourcetype="wineventlog:security" EventCode=4624 
| stats count as login_attempts, range(_time) as time_range by Account_Name 
| where time_range <= 600 
| sort - login_attempts 
| head 1

Answer: aparsa