Write Your Own Graylog Extractors For pfSense Using Regex

Описание

Graylog can be a powerful remote syslog server for pfSense. pfSense can easily write raw logs to Graylog, but the problem is the logs need to be interpreted, which can be done in Graylog by using extractors. However, there are no official pfSense syslog extractors, so people rely on the marketplace, but it may not always be a reliable source.

For example, I am using pfSense 2.5.2, but I tried several high rating extractors on the marketplace, unfortunately neither of them work. The root cause for my particular issue is the filterlog[xxxxx]:, so it may be easily fixed, but what if the next version of pfSense has more dramatic changes?

Even if the extractors from marketplace work, some of them are written in regex mixed with gorex. I prefer one language. There are also some very simple extractors on the web for pfSense, but they cannot correctly parse the fields, in my testing.

In this video, I will explain how to write your own pfSense syslog extractors or by yourself, from scratch. We will use Regex only. If you are not familiar with Regex, don’t worry I will explain every detail. And the only source of information is the official document from pfSense: the raw format of filter log.

About BNF: if you have ever written your own compilers, you may know BNF is a very formal representation of the syntax of a programming language or expression. In this pfSense filterlog case, we are just dealing with plain flat structure in CSV format, so the syntax is much simpler.even if you know nothing about BNF, you can easily understand it.

I tried to paste the exported extractors here, but it exceeded the allowed maximum text length. But the steps are so straightforward, hopefully you can complete the logics you need.