Configure Fluent-Bit to send logs to ELK

Prerequisites:

Fluent-Bit: https://docs.fluentbit.io/manual/installation/windows#installing-from-exe-installer

Let's begin by installing Fluent-Bit on Windows.

We have a log file named network_sample.log that we need to be ingested into the ELK stack. To ensure accurate data extraction, we will begin by crafting an appropriate regular expression to parse the required information.

SRC=(?<src_ip>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\s+DST=(?<dst_ip>\d{1,3}.\d{1,3}.\d{1,3}.
\d{1,3})\s+PROTO=(?<protocol>\w+)\s+SPT=(?<src_port>\d+)?\s+DPT=(?<dst_port>\d+)?\s+
LEN=(?<lenght>\d+)?\s+ACTION=(?<action>\w+)

A line does not match the current regular expression. Let's create a new one to accommodate it.

SRC=(?<src_ip>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\s+DST=(?<dst_ip>\d{1,3}.\d{1,3}.\d{1,3}.
\d{1,3})\s+PROTO=(?<protocol>\w+)\s+TYPE=(?<type>\w+)\s+CODE=(?<code>\d+)\s+ID=(?<id>\d
+)\s+ACTION=(?<action>\w+)

Next, we need to modify the parsers.conf file located in C:\Program Files\fluent-bit\conf.

Next, we need to configure the fluent-bit.conf file, located at C:\Program Files\fluent-bit\conf, to forward logs to the ELK stack.

[INPUT]
    Name         tail
    Parser       firewall-logs-1
    Path         C:\Users\NV\Downloads\network_sample.log
    Tag          firewall.logs-1

[INPUT]
    Name         tail
    Parser       firewall-logs-2
    Path         C:\Users\NV\Downloads\network_sample.log
    Tag          firewall.logs-2
    
[OUTPUT]
    name    	  es
    match   	  *
    Host    	  192.168.204.146
    Port    	  9200
    Match   	  *
    HTTP_User     elastic
    HTTP_Passwd   =Op+25maKY3GqC=IrV7m
    tls           on
    tls.verify    off 
    Trace_Output  on 
    Suppress_Type_Name on

This configuration is for Fluent Bit to read logs from a file (C:/Users/NV/Downloads/network_sample.log) and forward them to an Elasticsearch instance.

• name tail: The tail input plugin reads log files line by line, similar to the tail -f command in Linux.

• parser firewall-logs-1: Defines the parser used for processing log entries. The firewall-logs parser is specified in the parsers.conf file to extract structured fields from the logs efficiently.

• path C:/Users/NV/Downloads/network_sample.log: The path to the log file to monitor. Fluent Bit will read new lines appended to this file.

For the OUTPUT:

• name es: The es output plugin sends logs to Elasticsearch.

• Host 192.168.204.146: The IP address or hostname of the Elasticsearch server.

• Port 9200: The port where Elasticsearch is listening (default is 9200).

• tls on: Enables TLS/SSL encryption for communication with Elasticsearch.

• tls.verify off: Disables certificate verification.

• Trace_Output on: Enables verbose logging for debugging purposes.

Now, let's run Fluent Bit:

 & 'C:\Program Files\fluent-bit\bin\fluent-bit.exe' -c 
 'C:\Program Files\fluent-bit\conf\fluent-bit.conf'

We need to duplicate specific lines within the network_sample.log file and save the changes.

Let's confirm whether the logs are successfully being forwarded to ELK.