Foglight for Infrastructure 5.9.2

File Formats

The File Formats properties allow you to specify the format of the log files that you want to monitor.

•

File Formats: A list describing the structure of contents in the monitored log files. Any file that you monitor must have its format specified in this list. In most cases, all agent instances refer to one global list of file formats. The global list is pre-populated to describe some common log file formats.

Each entry in the list includes the following columns:

•

Name: The name of the file format.

•

Max Record Size: The maximum length of a record entry in bytes to use for pattern matching and submission to the Management Server. If a record is larger in size, it is truncated. The pattern is still applied to the entire record when searching for a match. This property is optional, and defaults to 1024 if not specified.

•

New Line Policy: The character or sequence of characters used to signify the end of a text line (EOL) in the file format. The following values are available:

CR: The carriage return character (\r), ASCII code 0x0d. This is common on Mac OS systems up to version 9.

LF: The line feed character (\n), ASCII code 0x0a. This is common on Unix and Unix-like systems (including Mac OS X systems).

CRLF: A carriage return character (\r, ASCII code 0x0d), followed by a line feed (\n, ASCII code 0x0a). This is common on Windows systems.

ANYCRLF: A carriage return character (\r, ASCII code 0x0d), a line feed (\n, ASCII code 0x0a), or both. This is common on mixed platform log files.

ANY: Any Unicode new line sequence, including CR, LF, CRLF, plus the single characters VT (vertical tab, U+000B), FF (form feed, U+000C), NEL (next line, U+0085), LS (line separator, U+2028), and PS (paragraph separator, U+2029).

•

Rollover Policy: Indicates to the agent the way the contents of the log files using this format are rotated when they reach their maximum size.

RECYCLE: The name of the active log file remains the same, while older logs are renamed by appending a ‘1’ to the name and increasing that number each time a new log file is created. For example, your system always writes log records to the same file, abc.log. When that file reaches its maximum size, the system renames it to abc1.log, and creates a new abc.log file for storing new logs. When that abc.log file becomes full, the system renames it to abc.log2 file, and creates a new abc.log file, and so on.

NEW: Each time a new log file is created, the number in the file name of the active log is increased by ‘1’. For example, your system starts writing log records to abc.log. When that file reaches its maximum size, the system creates a new log file, abc1.log, and continues to write log records to that file. When abc1.log file becomes full, the system creates an abc2.log file, and so on.

•

Record Separator RegEx: A regular expression indicating when a log record ends and a new one starts. This property is optional. The default is an empty string which indicates that each record is a single line.


	TIP: The agent supports PCRE (Perl Compatible Regular Expressions). For details about the PCRE syntax, visit http://perldoc.perl.org/perlre.html

•

Character Set: The character encoding used in the log file using this format. The character set must be installed on the remote host, and available through the iconv utility. The native character set is translated to UTF-8 when a log record is reported. This property is optional. If not specified, it defaults to UTF-8.

•

Maximum Match Count: The maximum number of records the agent can read during one collection interval. If the agent reaches the number of records before the end of the file, it continues to read the remainder of the file during the next collection interval, and so on.

Setting this value to a reasonable number of records (for example, 200) allows you to control the amount of time and resources the agent spends to read monitored logs during a single collection interval, and to prevent bottlenecks during data collection. If you do not want to specify a limit, type -1.

•

Max Processing Time(s): The amount time in seconds the agent can spend on reading all log files in one collection cycle while being connected to the remote host.

This value should be equal to or less than the Operation Timeout value in the Hosts list. For more information, see Monitored Hosts.

Setting this value to a reasonable number of seconds allows you to control the amount of time the agent spends to read monitored logs during a single collection interval, and to prevent bottlenecks during data collection. Once surpassed, the collector stops, and continues from that same point in the next collection cycle.

Record Transformations

The Record Transformations properties allow you to transform any log message before it is sent to the Management Server. This could be used to add extra information or to remove sensitive information from a log record.

•

Record Transformations: A list of record transformations that the agents must use in conjunction with the match patterns to convert any log messages. When no transformation is specified, the log record is transmitted to the Management Server without changes.

Each entry in the list includes the following columns:

•

RegEx Record Transformation Pattern: A regular expression that the agent uses to look for specific text in the collected log record.

•

Record Transformation: The replacement text that the agent uses in the log record to be transmitted to the Management Server.

Data Collection Scheduler

The Datacenter Collection Scheduler agent properties specify the data frequency settings the agent uses to read monitored log files.

•

Collector Config: A list containing the data collectors the agent uses. Each entry in the list includes the following columns:

•

Collector Name: The name of the collector the agent uses to gather data.

•

Default Collection Interval: The number of milliseconds, seconds, minutes, hours, or days during which the agent collects data.

•

Time Unit: The time unit associated with the Default Collection Interval.

•

Fast-Mode Collection Interval: The number of milliseconds, seconds, minutes, hours, or days during which the agent collects data when working in the fast collection mode.

•

Fast-Mode Time Unit: The time unit associated with the Fast-Mode Collection Interval.

•

Fast-Mode Max Count: The maximum number of the times the agent can stay in fast collection mode.

FileLogMonitor configuration example

Scan example

This example provides the configuration settings for monitoring the FglAM log files on a UNIX® system for WARN and ERROR records. The FglAM log files are located in the /home/user/FglAM/state/default/logs folder. FglAM log records have a date at the beginning of each record that look like this:

2014-11-17 13:37:22.315 ...

This format can be set as the regular expression for the record separator.

Table 1. Configuration settings for the Scan example
Group Name	Property Name	Property Details	Value set for this example
Monitored Hosts	Hosts	Host	host.domain.com
		Host name override	(optional)
		Host Type	UNIX
		SSH Port	22
		Operation Timeout	60000
		Collect System ID
		Remote Collector Executable	(optional)
		Secure Launcher	(optional)
Log Files	Log Files	Directory	/home/user/FglAM/state/default/logs
		Filename Pattern	FglAM_*\\.log$
		File Format Name
	Patterns	RegEx Match Patterns	WARN ERROR
		Match Severity
		Tags
File Formats	File Formats	Name
		Max Record Size	1024
		New Line Policy	ANYCRLF
		Rollover Policy	NEW
		Record Separator RegEx	^\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}
		Character Set	UTF-8
		Maximum Match Count	200
		Max Processing Time(s)	120
Record Transformations	Record Transformations	RegEx Record Transformation Pattern	(optional)
		Record Transformation	(optional)
Data Collection Scheduler	Collector Config	Collector Name	(default)
		Default Collection Interval	(default)
		Time Unit	(default)
		Fast-Mode Collection Interval	(default)
		Fast-Mode Time Unit	(default)
		Fast-Mode Max Count	(default)

This example only shows one scan, but the scan can be performed multiple times at regular intervals since more records can be added to the log files over time.

Please select your product:

To serve you better, please complete the Purpose of your Chat:

Recommended Solutions for Your Problem

Foglight for Infrastructure 5.9.2 - User Guide

File Formats

Record Transformations

Data Collection Scheduler

FileLogMonitor configuration example