InTrust 11.6 - Setting Up Gathering of Syslog Data

Setting Up Gathering of Syslog Data

You can use the InTrust Deployment Manager console to collect and manage Syslog data that is received by InTrust Server. To enable Syslog data capture, you need to set up a Syslog collection, as follows:

• Specify the InTrust server that should listen for Syslog messages
• Specify the devices you want to audit
• Specify the repository where you want to store the collected Syslog data

You can add, delete and edit collections at any time.

The first time you run InTrust Deployment Manager, you are directed to the welcome page, where you are prompted to create a collection. Take the opportunity to create your Syslog collection.

You can create more collections at any time. For that, right-click Collections and select New Syslog Collection, and then follow the wizard steps.

Common Tasks for Syslog Collections

To add a Syslog collection

1. In the InTrust Deployment Manager console, go to the Collections view.
2. Right-click Collections and select New Syslog Collection.
3. In the New Syslog Collection wizard, specify a name and a description for the collection.
4. On the Set Up Collection step, specify the InTrust server from which you want to get Syslog audit data and repository. You can collect Syslog data from all devices that send Syslog messages to the InTrust server or specify certain devices by selecting one of the following options:
   1. All Syslog data received by InTrust server
   2. Syslog data only from devices you specify on the next step
5. If you select the Syslog data only from devices you specify on the next step option, add the devices you want on the next Specify Syslog Devices step. For that click the Add button and select Devices. In the Specify Syslog Devices dialog box, you can add devices from the list or specify the IP address (DNS name) of the certain device.
   
   Also you can upload a text file that contain a list of device IPs, for that click Add and select the Import from file option. A list file uses the plain text format. Each IP address must be a separate line in the file.

To add devices to a collection

Use any of the following methods:

• In the wizard that opens when you edit a Syslog collection, change the devices list on the Specify Syslog Devices step as described in the previous procedure.
• Select the devices you need in the Syslog devices not in a collection search folder in the navigation pane and click Add to collection, and then select the collection you need.

To delete Syslog devices from a collection

1. Right-click the Syslog collection and select Edit Collection.
2. In the wizard that opens, go to the Specify Syslog Devices step.
3. In the list of devices, select the devices you do not need, and click Remove.

To start a new repository

You can create a repository when you create a new Syslog collection or edit an existing collection, on the Set Up Collection step of the wizard. For finer-grained management of repositories, use the Storage view (for details, see Managing Repositories).

Passing Messages On

If both Syslog listening and forwarding are enabled for a repository at once, then incoming Syslog messages are forwarded unchanged. This happens independently of writing the messages to the repository.

Analyzing Syslog Collections

There are two predefined search folders for Syslog devices: Syslog devices by collection and Syslog devices not in a collection. Use them to locate the devices you need; for example, if you cannot find your device in any existing Syslog collection, it is probably available in the Syslog devices not in a collection search folder.

For all Syslog devices contained in the Syslog devices not in a collection search folder, the Received field shows the the time when the InTrust server last received an event from the device. For devices that are included in collections, the corresponding field is called Timestamp, and it contains the time when a Syslog message was last generated by the device (or, if this cannot be determined, the time when the InTrust server last received an event from the device).

A device can have the Collecting or Not Collecting status. If the InTrust server does not receive events from the device for a week, the device changes status to Not Collecting.

When a Syslog collection is selected, the right pane shows a table with information about the collection members. The table supports multi-level grouping of devices, so that you can organize them in tree-like views using any criteria. For example, to quickly find out which devices are currently not collecting any data, you can group computers by source status, then by device name.

To use multi-level grouping, drag table column names from the devices list to the area above the list. The device list changes accordingly.

Message Parsing Specifics

InTrust parses the Syslog messages it captures to store a useful representation of them in the repository. Only UDP v4 is used for receiving messages, and they can use either ASCII or UTF-8.

Messages are expected to conform to either RFC 3164 or RFC 5424. The fields of an event entry in the repository are filled in from the fields of a Syslog message.

A message is parsed until the end or until a mismatch occurs. The parser breaks down the message into as many insertion strings as it can. No matter how many fields InTrust is able to parse successfully—all of them, just the first three or none at all—the entire message text is saved in the Description field. This enables you to find the message in Repository Viewer (by using the Any field parameter) or IT Security Search even if the fields are not mapped properly.

RFC 3164 Specifics

The following pattern is defined in RFC 3164:

<PRI>TIMESTAMP HOSTNAME TAG: MSG

An example of a valid message is as follows:

<34>Oct 14 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8

The PRI field indicates the facility and severity. For details, see Treatment of Facility and Severity Information.

A message has the following parts:

Treatment of Timestamps (RFC 3164)

If the timestamp cannot be parsed, the Time event field stores a part of the time that the event was written to the repository (in the InTrust server's time zone). Note that this field is supposed to contain local times. The GMT timestamp is derived from the parsed value. The message contains no time zone information, so it is important that the Syslog device and the InTrust server should best be located in the same time zone; otherwise, the local and GMT timestamps will be wrong.

The RFC 3164 format does not specify the year in the timestamp. However, when capturing messages, InTrust must supply the year to form a valid event date. Normally, this only matters for a fraction of a second at midnight on January 1st, because the year can change while a message is in transit.

In rare cases, messages are accumulated and need to be submitted after a delay. For example, some applications that can send Syslog will hold events until Syslog forwarding is configured. When these stored messages cross a calendar year, they can end up with an incorrect timestamp that appears to be in the future when in reality they are from the past.

When InTrust receives a Syslog message without the year, it assumes the message has the current year value. However, it can decide to change the year. If this timestamp appears to be more than six months in the future (same year), InTrust changes it to the previous year. For example, in March 2018 you get messages from November. There's no same-year date six months before March, and November 2018 is more than six months ahead, so the resulting date is in November 2017.

RFC 5424 Specifics

The following pattern is defined in RFC 5424 (the header is bolded):

<PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID STRUCTURED-DATA MSG

A message has the following parts:

Examples of valid messages:

• <165>1 2015-05-11T22:14:15.003Z SUPERHOST1 myproc 8710 - - %% It's time to make the do-nuts.
• <165>1 2003-10-11T22:14:15.003Z mymachine.domain.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"]
  Message with structured data in the UTC time zone
• <140>1 2003-10-11T22:14:15.003+3:00 10.30.44.245 evntslog - ID47
  Message with a non-UTC time zone and IP address instead of host name

For an in-depth description of the format, see Section 6 of RFC 5424.

Treatment of Timestamps (RFC 5424)

The timestamp in a message can contain such details as the time zone and milliseconds. Millisecond information is lost when a message is converted to an event entry. It is also possible that the timestamp is omitted altogether, replaced by “-”.

The following are examples of valid timestamps:

2015-05-12T19:20:50.52-04:00
2015-05-11T22:14:15.003Z
2015-05-24T05:14:15.000003-07:00
-

The following timestamps are malformed:

2015-08-24T05:14:15.000000003-07:00
Too many decimal places (there should be no more than six).

08-24-2015T05:14:15-07:00
The order of units in the date is wrong.

2015/08/24T05:14:15-07:00
You cannot use separators other than “–“ and “:” in the time part.

If the timestamp cannot be parsed or it is omitted, InTrust substitutes the current time during event generation (in the InTrust server's time zone). The parsed (or substituted) timestamp goes to the Date and Time fields of the event. Note that messages are supposed to contain local times.

The GMT timestamp is derived from the resulting value, as follows:

• If the time zone is specified, it is used for offsetting the GMT timestamp.
• A message must have either time zone information or local offset information; if neither is available, the timestamp cannot be parsed.

Mapping of Event Fields

When InTrust generates an event entry based on a Syslog message, it uses the rules outlined in the table below. It shows what happens to the following example message:

• RFC 3164-compliant format
  <34>Oct 14 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8
• RFC 5424-compliant format
  <34>1 2014-10-14T22:14:15+03:00 mymachine su - ID47 - 'su root' failed for lonvick on /dev/pts/8

Treatment of Facility and Severity Information

In both RFC 3164 and RFC 5424, the PRI field indicates the facility and severity. The following table shows how PRI values are interpreted:

Treatment of Named Fields

Syslog messages can contain pieces of information that are semantically key–value pairs. There is no agreed-upon notation for such items, so the syntax may vary from provider to provider. InTrust attempts to analyze incoming messages to spot key–value information.

During message parsing, InTrust first tries to detect a pattern that indicates the presence of keys. The pattern for a key is like this: separator followed by single word followed by assignment marker. These three elements are distinguished as follows:

• The separator is white space, a comma or a semicolon.
• The word is alphanumeric without spaces in the middle and doesn't start with a number or underscore.
• The assignment marker is a colon or equals sign. It may or may not contain white space to the left or right of it. As long as the use of white space is consistent, the assignment marker will be spotted.

If a combination of these features occurs repeatedly in a message, then InTrust considers it a pattern and treats the matching words as keys and the substrings to the right of them as values.

InTrust puts the discovered key–value pairs in the named strings of event records.

Tags

Messages from some Syslog providers include a piece of information that is traditionally referred to as a tag. It isn't usually marked in any way, and its position is defined by a set of rules. For example, in a message like "<54>Dec 31 23:51:36 sm-w2k12-001 SymantecServer: sm-w2k12-001,Local: 10.30.38.109,Local: 514..." the tag is SymantecServer.

InTrust detects such tags and puts them in the Tag field. If they match the key–value pattern, it also forms named fields from them.

Example 1

<134> id=firewall sn=18B1690729A8 mgmtip=192.168.168.168 time=\"2016-08-19 00:21 : 40 UTC\" fw=10.205.123.15 m=96 n=24789 i=60 lic=0 pt=8080.8443 usestandbysa=0 dyn=n.e ai=1 fwlan=192.168.168.168 conns=18

Here is how the fields are extracted:

The parser decides that a white space means a separator, and an equals sign without white space on either side is the assignment marker. Therefore, the following fields are discovered:

Example 2

<134> tk_url=http://example.com:80/whatever.png,tk_malicious_entity=,tk_file_name=,tk_entity_name=,tk_action=,tk_scan_type=WebReputation,tk_blocked_by=policy,tk_rule_name=IJVIRUS,tk_opp_id=0,tk_group_name=None,tk_category=Web Reputation - Very Low,tk_uid=0000002836-626130ec1beb9bfdaab2,tk_filter_action=3

Here is how the fields are extracted:

A comma is recognized as the separator, and an equals sign without white space on either side is the assignment marker. Some fields (such as tk_action and tk_malicious_entity) are detected and receive empty values. The following non-empty fields are discovered:

Example 3

<54>Dec 31 23:51:36 sm-w2k12-001 SymantecServer: sm-w2k12-001,Local: 10.30.38.109,Local: 514,Local: 000C29B01D8B,Remote: 10.30.44.167,Remote: ,Remote: 58800,Remote: 0003BAF4AAD9,UDP,Inbound,Begin: 2017-12-31 23:57:38,End: 2017-12-31 23:57:39,Occurrences: 3,Application: ,Rule: Block all other IP traffic and log,Location: Default,User: Administrator,Domain: SM-W2K12-001,Action: Blocked

Here is how the fields are extracted:

Here a colon followed by white space is assumed to be the assignment marker and the separator is a comma. The Local and Remote fields occur multiple times, so both of them get concatenated values. The resulting named strings are as follows:

Example 4

<54>Oct 01 13:12:44 sm-w2k12-001 SymantecServer: sm-w2k12-001,2Action: Blocked,Local: 10.30.38.109,Local: 514,Local: 000C29B01D8B,Remote: 10.30.44.167,Remote: ,Remote: 58800,Remote: 0003BAF4AAD9,UDP,Inbound,Begin: 2017-12-31 23:57:38,End: 2017-12-31 23:57:39,Application: ,Rule: Block all other IP traffic and log,Location: Default,User: Administrator,Domain: SM-W2K12-001

In this example, see what happens to the substring 2Action that could be a key if it didn't start with a digit: