Part One
Last updated
Last updated
A NIDS can be powerful if used properly, but ineffective if misused. Some just meet compliance with no real monitoring, like a client who only logged in once a day and did nothing else.
The client said they log into their NIDS daily because auditors require it. NIDS is a key part of NSM, so the answer is not "NIDS or NSM," but "NSM with NIDS included."
Execs often complain about paying staff to monitor screens. They prefer prevention over detection because it saves money through automation.
That would be fine if it worked, but history shows that weak detection has caused many major breaches.
Many large organizations learn about breaches from third-party reports. Both the Verizon DBIR and Mandiant M-Trends show that a large number of breaches are detected externally. Verizon's data shows internal detection is decreasing, while Mandiant's data shows a small improvement, but this might be due to fast-detecting attacks like ransomware.
In the past, worms were simple. If one system out of 1,000 was unpatched, a worm like Conficker could infect it but couldn't spread to the rest because they were protected.
That is now changing: NotPetya could hack one system, steal Windows credentials, and spread to others using PSExec or WMIC, eventually infecting all 1,000 systems, even if they were mostly patched.
NotPetya tries to get admin access on one machine, then spreads through the network by controlling other computers or stealing domain admin credentials from memory, taking over the entire Windows network.
Security Onion is a great open-source network security monitoring platform that offers many blue team tools in one user-friendly package. Doug Burks created it in 2008, and it has developed a lot since then.
The Security Onion platform has NIDS and SIEM features that are on par with commercial products. It supports various use cases:
NIDS: Gather network events from Zeek, Suricata, and more to fully monitor your network. This helps catch threats quickly and easily.
HIDS: Security Onion works with different host-based event collection tools like Wazuh, Beats, and osquery. Just connect them to your setup, and you're ready to go.
Static Analysis (PCAP Import): Use Security Onion to quickly analyze PCAP files. Set up a virtual machine fast and start in minutes.
SOC Workstation: SOC analysts can use a workstation with local Linux tools for analyzing network and host events. All necessary apps are included, so no extra installation is needed.
Key changes:
Replace the unsigned PF_RING kernel module with AF_PACKET.
Swap netsniff-ng for Google Stenographer for pcap collection.
Remove Sguil, Squert, and capME.
Rename Storage Nodes to Search Nodes.
Add new technologies: TheHive, Strelka, Grafana, Fleet, Playbook, and Security Onion Console (SOC).
Components:
The Elastic Stack (Elasticsearch, Logstash, Kibana, Filebeat, etc…)
Suricata, Zeek, osquery, strelka, Wazuh
Filebeat, Redis, Curator
And more!
Security Onion Solutions describes Import mode:
An Import node is a basic setup that can import a pcap file using the so-import-pcap command. This command uses Suricata and Zeek to analyze the pcap, and Filebeat sends the logs to Elasticsearch for parsing and indexing. You can view the logs in the Security Onion Console (SOC).
Import mode is good for testing important features. Unlike 16.04, all Security Onion modes need more RAM, but you can run Import Mode with about 8 GB. It can't capture live traffic but can import packets using the so-import-pcap command for processing with Kibana and Zeek.
Standalone is like Evaluation because everything runs on one machine. Instead of Filebeat sending logs straight to Elasticsearch, it sends them to Logstash, which puts them in Redis. Then, another Logstash pipeline gets the logs from Redis and sends them to Elasticsearch for processing. This setup is mainly used for testing, labs, proofs of concept, or low-traffic situations. It's less scalable than a distributed setup.
For full use, distributed is best. For testing or limited use, lighter modes need fewer resources. The easiest mode is Import, which lets you test key Security Onion features with minimal hardware.
Wireshark is a powerful tool for analyzing network protocols. It originated from Ethereal when its creator, Gerald Combs, joined CACE Technologies in 2006 and had to leave the Ethereal name behind. To keep the project going, he changed its name to Wireshark.
The screenshot above shows Wireshark's view of the connection. It's interesting because the first SYN packet appears only after the 10th packet. Packets 1-8 show the server trying to connect to the client, packet 9 shows the client resetting, and packets 10 and onward show a normal TCP connection.
The TLS client hello usually happens on the fourth frame (SYN, SYN/ACK, ACK, Client Hello), but here it occurs on the 186th frame. It should have happened by frame 13, considering the first nine packets.
We right-clicked a packet in Wireshark and selected Follow -> TCP Stream. It starts as an EXE transfer with a delayed TLS handshake, which is typical malware behavior we'll cover more in Security 511.
In the TCP stream, there's a lot of unencrypted data in the PCAP before the TLS connection starts. After scrolling past the middle, you can see this data. The cryptographic functions mentioned are part of the EXE. Once the transfer finishes, the EXE runs and establishes the TLS connection on the same socket pair.
Tshark searches better than other command-line tools like tcpdump or ngrep, especially when used with scripts.
Snort is the most popular IDS. It's an open-source system for detecting and preventing network intrusions, created by Sourcefire. It uses signature, protocol, and anomaly-based checks, making it the most widely used IDS/IPS globally, with millions of downloads and about 400,000 users.
Zeek goes beyond basic detection and offers in-depth analysis. Doug Burks explained that while traditional systems search for specific items in data, Zeek provides all your data and insights, letting you explore it freely. Zeek tracks network activity, logging connections, DNS requests, network services, SSL certificates, and various protocols like HTTP and FTP. It gives detailed visibility into your network events and includes analyzers for common protocols, checking file downloads against malware databases.
Zeek makes detailed logs that are useful for tools but hard for people to read. The "zeek-cut" command lets you easily select fields to see, like user_agent and URI.
SIEM is the main term used, but there are others like SIM (Security Information Management) and SEM (Security Event Management). We will stick with "SIEM."
Many technologies mentioned here have some ability to detect threats, even if they weren't specifically designed for that. Just because we can use them to spot attackers doesn't mean we actually do. Think about the times you've read about a company getting hacked; we often learn what happened and how long it went on.
Mandiant M-Trends and Verizon DBIR show that it can take months for organizations to realize they've been hacked, often because someone else informs them.
Think about this: How did Mandiant and Verizon find out how long a company was compromised? In the cases they looked at, there was enough evidence for investigators to piece together what happened. This suggests that the information needed for detection was usually there but either overlooked or missed.
This section emphasizes using a tool to combine and connect data from different sources. Remember, just combining data isn't helpful without a skilled analyst to understand, prioritize, and act on it.
When organizations start, they mainly want to gather all their data in one place. But this doesn’t help much except making it easier to overlook data. The threat hunting team can help identify important information from the overwhelming logs of today's enterprises.
Organizations today have many data sources, and their number and size keep growing. To deal with this, many try to collect everything, but this often leads to problems. Although organizations plan to narrow down what they collect later, few succeed in effectively deploying a SIEM system after starting with a "collect all" approach.
Gartner explains that SIEM helps detect threats and respond to security incidents by collecting and analyzing security events in real-time from many different sources. However, this is a challenging goal for any one solution to accomplish.
Highlighting some of the aspects Gartner posits for SIEM:
Threat detection
Incident response
Real-time collection/analysis
Historical analysis
Contextualizing sources of data
Compliance requirements
Intrusion/incident investigations
Most setups focus on collecting a lot of different data in one place but don't analyze it much. While you may have the data for investigations, tools often struggle to analyze it quickly without a lot of adjustments. Real-time analysis can be difficult, but analyzing after the fact and compliance reporting don’t need to be as timely. Think of this as a Compliance SIEM.
A Tactical SIEM focuses on quick, real-time analysis. It uses fewer data sources and less data overall, but it provides faster results and allows easy switching between datasets.
These goals might be met with just one vendor and one setup. However, viewing them as separate solutions helps customize the deployment and understand the different goals that each approach can achieve.
All SIEM solutions can enhance logs either when they are collected or after they are stored. Enrichment means adding extra details to a log, which is important for better analysis and detection. For example, a DNS log with the field "query" showing "www.google.com" can be expanded to include eight more fields.
The first enrichment fields split www.google.com into parts. WHOIS creation dates and Alexa's top million sites or Cisco Umbrella lookups are done for a domain like google.com. These values are saved in the creation_date and tags fields. Geographic info can be found using IP addresses from DNS entries, and the ASN shows who owns the IP address. The frequency_score field uses Mark Baggett's freq_server.py, and parent_domain_length counts the string length.
Instead of buying an existing company to enter the SIEM market, Microsoft chose to develop its own tools used internally for security. This strategy has worked well, and Microsoft Sentinel quickly became a major competitor in the field.
Sentinel stands out from other SIEMs due to its strong integration with Azure and Microsoft Defender tools. It is designed for the cloud, which means it lacks an on-premises option. While this may limit some use cases, it allows for faster updates since there’s no need to manage on-premises support.
The slide shows part of the MITRE CTID project, mapping Azure security controls to MITRE ATT&CK. It lists 14 key tactics and highlights the relevant ATT&CK techniques for Azure security.
The Elastic website describes the history of the Elastic Stack:
It began with Elasticsearch, a popular open-source search engine that is easy to use, flexible, and scalable. A company was created to support it.
Elasticsearch began as a search engine for logs. Users wanted to easily add and view their logs, so Logstash was created to help ingest them, and Kibana was made for visualization.
In 2015, Beats were added to ELK so users could easily tail files.
Elasticsearch is open-source software for searching large amounts of data quickly and cheaply. Unlike traditional databases, it can manage huge data sets easily.
Hadoop is another open-source big data tool created by Google:
In a few years, Hadoop, a free software named after a toy elephant, has become essential for major websites. It powers top search engines, controls ads, shapes Yahoo’s homepage, and helps find friends on Facebook.
By 2003, Google struggled to process and index the entire Internet regularly and needed an easier way to analyze its huge data for better search results and user behavior. To fix this, two Google engineers created MapReduce, which, when combined with Google’s file management system, solved the problem.
Many sites prefer Elasticsearch because it uses common web technologies like REST and JSON, making it easier to set up.
Elastic describes Logstash:
Logstash processes and prepares data of any type or complexity.
Use grok to create structure from unstructured data.
Extract location info from IP addresses.
Hide personal data and remove sensitive information.
Simplify data handling, no matter the source or format.
Logstash sends data to a storage location, usually Elasticsearch, but it can also use NoSQL databases like MongoDB and Riak. Unlike SQL databases, NoSQL databases are not structured with fixed rows and columns.
NoSQL databases use JSON documents to store data instead of the columns and rows of traditional databases. NoSQL means "not only SQL," so these databases can work without SQL or combine JSON's flexibility with SQL's power. They are designed to be flexible, scalable, and quick to meet modern business data needs.
Logstash is the main data tool in the Elastic Stack. It can send and pull data, reformat it, and more, but it uses more resources. Beats are lighter and only send data.
To set up a good centralized logging system, you need a tool that gathers data from different sources and makes it useful. Logstash does this by collecting data from various systems, organizing it into clear fields, and sending it to a storage place.
Logstash has a major issue with performance. It needs the JVM to run, and its Ruby-based design leads to high memory use, especially with many pipelines and complex filtering.
Elastic supports several tools besides Filebeat: Metricbeat (sends metrics), Packetbeat (sends network data), Winlogbeat (sends Windows event logs), Auditbeat (sends audit data), Heartbeat (sends uptime data), and Functionbeat (sends serverless cloud data).
Kibana provides a search and visualization front-end for Security Onion 2.
Elastic describes Kibana:
You can search and visualize data in Elasticsearch using bar charts, pie charts, tables, histograms, and maps. A dashboard combines these visuals and can be shared online for real-time analysis of large data sets. This supports various uses, including:
a. Log analysis b. Infrastructure and container monitoring c. Application performance tracking d. Geospatial analysis e. Security analysis f. Business analytics
Managing and securing an Elastic Stack instance through a web interface while centralizing access for its built-in observability, security, and search tools.
The above screenshot shows results from Zeek using the query event.module.keyword: zeek.
The image above shows the Elastic components we've talked about. Beats send data using a library called libbeats, either to Elasticsearch or to Logstash. Logstash can filter and transform data, like parsing Apache logs. Elasticsearch helps search through large data quickly, and Kibana is the user interface for searching and visualizing that data.
Analyze strings in a packet capture.
Carve Microsoft EXEs from a packet capture.
Scan carved EXEs with an antivirus program.
Gain experience using strings and Zeek.
Run the strings command with a minimum string length of 10 on /labs/merlin/capture.pcap. Save the output to /labs/merlin/ capture-strings.txt.
Use grep to search for strings indicating the following: EXE file transfer, cmd.exe banner, Windows registry changes, Use of PowerShell
Use Zeek to extract the EXEs from /labs/merlin/capture.pcap:Use this Zeek script to extract the files (/usr/local/zeek/share/zeek/policy/frameworks/files/extract-all-files.zeek)
Determine the sending and receiving IP addresses, filename and mime type used in transferring the EXE
Scan the extracted EXEs with the clamscan antivirus client.
First, let's start by running the strings command with a minimum string length of 10 on /labs/merlin/capture.pcap and save the output to /labs/ merlin/capture-strings.txt:
Next, let's use grep to search for these strings: EXE file transfer, cmd.exe banner, Windows registry changes, and PowerShell usage.
These strings are often linked to Windows malware:
"This program cannot be run in DOS mode" (exe transfers)
"Microsoft Corporation. All rights reserved" (cmd.exe banner)
"HKEY_LOCAL_MACHINE" (registry access or changes)
"powershell" (use of PowerShell, note the trailing space
Let's search for EXE transfers:
Let's search for cmd.exe banners:
Let's search for registry access or changes:
Let's search for use of PowerShell:
The trailing space helps to distinguish between the string "powershell," which can be used by harmless software, and the actual powershell command, which usually has a space followed by flags. Also, malware often uses the string "powershell.exe."
Now, we'll use Zeek to extract the EXEs from /labs/merlin/capture.pcap using this zeek script (/usr/local/zeek/share/zeek/policy/frameworks/files/extract-all-files.zeek)
The file "extract-all-files.zeek" is a Zeek script that extracts several types of files from a Pcap file.
Most files are labeled as "SSL" or "HTTP." To better understand what these files represent, we can check the mime type of each extracted file using the zeek-cut command, focusing on the "extracted" file name and its mime type.
Most of the SSL and HTTP files are OCSP (Online Certificate Status Protocol) traffic, which is safe and not related to the exploit we’re investigating. Let's run the following command to see the other types of files Zeek extracted.
The "egrep" command skips files with "SSL" or "HTTP" in their names. It found four files that mention SMB (Server Message Block).
The MIME type helps identify file types, but the file command is more accurate. Let's use this command to find the types of files Zeek extracted from SMB traffic.
Two of the SMB files are Windows executables, and the other two are labeled as "data."
Now, let's identify the sending and receiving IP addresses, along with the filename and MIME type used for transferring the EXE files. The filenames listed by Zeek for the extracted files are helpful but do not represent the actual filenames used during the transfer. The information we need, including the actual filenames, sending and receiving IP addresses, and MIME type, can be found in Zeek's files.log.
The Zeek output shows that four files were sent from 192.0.2.160 to 10.5.11.11.
The server set the mime type for two files transferred via SMB to application/x-dosexec (E32Ds9ROun.dll and HxuT0y1GjO.exe). We should keep in mind that Microsoft DLLs are executables and share the same mime type as EXE files.
Two other transferred files have empty mime_type fields (shown as "-") but have the same filename as the previous EXE: HxuT0y1GjO.exe. This is a common problem with different file carving methods and tools, which can either fail to extract files from saved packet captures or do so incorrectly.
Let's check if the EXEs are harmful by scanning them with the ClamScan antivirus program.
