Chat now with support

Get Live Help
Complete Registration

Sign In

Request Pricing

Contact Sales

syslog-ng Premium Edition 7.0.34 - Administration Guide

Table of Contents

Target audience and prerequisites Acknowledgments

Introduction to syslog-ng

What syslog-ng is What syslog-ng is not Why is syslog-ng needed? What is new in syslog-ng Premium Edition 7? Who uses syslog-ng?

Public references of syslog-ng Premium Edition

Supported platforms

Certified packages

The concepts of syslog-ng

The philosophy of syslog-ng Logging with syslog-ng

The route of a log message in syslog-ng

Modes of operation

Client mode Relay mode Server mode

Global objects Timezones and daylight saving

How syslog-ng PE assigns timezone to the message A note on timezones and timestamps

Versions and releases of syslog-ng Premium Edition Licensing

License benefits Licensing model and modes of operation

GPL and LGPL licenses High availability support The structure of a log message

BSD-syslog or legacy-syslog messages

The PRI message part The HEADER message part The MSG message part

IETF-syslog messages Enterprise-wide message model (EWMM)

Message representation in syslog-ng PE Structuring macros, metadata, and other value-pairs

Specifying data types in value-pairs

Things to consider when forwarding messages between syslog-ng PE hosts Using syslog-ng PE with NFS or CIFS (or SMB) file system for log files

Limitations of using syslog-ng PE with NFS or CIFS (or SMB) file system Risks of using syslog-ng PE with NFS or CIFS (or SMB) file system Recommendations for using syslog-ng PE with NFS or CIFS (or SMB) file system

Installing syslog-ng PE

Prerequisites to installing syslog-ng PE Security-enhanced Linux: grsecurity, SELinux Installing syslog-ng PE on RPM-based platforms (Red Hat, SUSE, AIX) Using syslog-ng PE on SELinux Installing syslog-ng PE on Debian-based platforms Installing syslog-ng in Docker Installing syslog-ng using the .run installer

Installing syslog-ng PE in client or relay mode Installing syslog-ng PE in server mode Installing syslog-ng PE without user-interaction

Upgrading syslog-ng PE

Upgrading from syslog-ng PE 7.0.x to version 7 Upgrading from syslog-ng PE 6.0.x to version 7 Upgrading syslog-ng PE to other package versions Upgrading from syslog-ng PE to syslog-ng OSE Upgrade from syslog-ng OSE to syslog-ng PE

Upgrading from syslog-ng OSE to syslog-ng PE

Upgrading from complete syslog-ng PE to client setup version of syslog-ng PE Upgrading the sql() source of syslog-ng PE

Differences in configuration

Uninstalling syslog-ng PE Configuring Microsoft SQL Server to accept logs from syslog-ng

The syslog-ng PE quick-start guide

Configuring syslog-ng on client hosts Configuring syslog-ng on server hosts Configuring syslog-ng relays

Configuring syslog-ng on relay hosts How relaying log messages works

Managing and checking syslog-ng PE service on Linux

The syslog-ng PE configuration file

Location of the syslog-ng configuration file The configuration syntax in detail Notes about the configuration syntax Defining configuration objects inline Using channels in configuration objects Global and environmental variables Logging configuration changes Modules in syslog-ng Premium Edition (syslog-ng PE)

Loading modules

Managing complex syslog-ng configurations

Including configuration files Reusing configuration blocks Generating configuration blocks from a script

Python code in external files Logging from your Python code

Collecting log messages — sources and source drivers

How sources work default-network-drivers: Receive and parse common syslog messages

default-network-drivers() source options

internal: Collecting internal messages

internal() source options

file: Collecting messages from text files

Notes on reading kernel messages file() source options

google-pubsub: collecting messages from the Google Pub/Sub messaging service

Prerequisites Limitations Supported platforms Declaration The Google Pub/Sub message format in syslog-ng PE

The contents of the Google Pub/Sub Message body on the syslog-ng Premium Edition (syslog-ng PE) side The contents of the Google Pub/Sub Message attributes on the syslog-ng PE (syslog-ng PE) side Processing incoming message contents in raw message format and in .JSON format

Processing incoming message contents in raw message format Processing incoming message contents in .JSON message format

google-pubsub() source options Preventing message duplication resulting from the At-Least-Once delivery behavior Error messages you may encounter while using the google-pubsub() source

wildcard-file: Collecting messages from multiple text files

wildcard-file() source options

linux-audit: Collecting messages from Linux audit logs

linux-audit() source options

mssql, oracle, sql: collecting messages from an SQL database

mssql(), oracle(), and sql() source options Customizing mssql() queries Configuring TLS encryption for MSSQL servers Possible connection errors between the MSSQL server (2019) and syslog-ng PE 7 LTS

network: Collecting messages using the RFC3164 protocol (network() driver)

network() source options Proxy Protocol support

The working mechanism behind the Proxy Protocol Proxy Protocol: configuration and output examples

office365: Fetching logs from Office 365

Configuring Office 365 to permit fetching logs office365() source options Troubleshooting audit logging in Office 365

osquery: Collect and parse osquery result logs

osquery() source options

pipe: Collecting messages from named pipes

pipe() source options

program: Receiving messages from external applications

program() source options

python: writing server-style Python sources

Python LogMessage API python() and python-fetcher() source options

python-fetcher: writing fetcher-style Python sources

Python LogMessage API python() and python-fetcher() source options

snmptrap: Read Net-SNMP traps

snmptrap() source options

syslog: Collecting messages using the IETF syslog protocol (syslog() driver)

syslog() source options

system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage

systemd-journal() source options

systemd-syslog: Collecting systemd messages using a socket tcp, tcp6,udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol

tcp(), tcp6(), udp() and udp6() source options — OBSOLETE

Change an old source driver to the network() driver

udp-balancer: Receiving UDP messages at very high rate

udp-balancer() source options

unix-stream, unix-dgram: Collecting messages from UNIX domain sockets

unix-stream() and unix-dgram() source options

windowsevent: Collecting Windows event logs

windowsevent() source options

Sending and storing log messages — destinations and destination drivers

elasticsearch2>: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED)

Prerequisites How syslog-ng PE interacts with Elasticsearch Client modes Elasticsearch2 destination options (DEPRECATED)

elasticsearch-http: Sending messages to Elasticsearch HTTP Event Collector

Batch mode and load balancing elasticsearch-http destination options

file: Storing messages in plain-text files

file() destination options

google_bigquery(): Sending logs to a Google BigQuery table

Limitations Configuring the google_bigquery() destination google_bigquery() destination options Error messages you may encounter while using the google_bigquery() destination

google_bigquery_managedaccount(): Sending logs to a Google BigQuery table authenticated by Google Cloud managed service account

Limitations

google_pubsub(): Sending logs to the Google Cloud Pub/Sub messaging service

Limitations Configuring the google_pubsub() destination google_pubsub() destination options Available endpoints for the google_pubsub() destination Error messages you may encounter while using the google_pubsub() destination

google_pubsub-managedaccount(): Sending logs to the Google Cloud Pub/Sub messaging service authenticated by Google Cloud managed service account

Limitations Configuring the google_pubsub_managedaccount() destination google_pubsub_managedaccount() destination options Available endpoints for the google_pubsub_managedaccount() destination Error messages you can encounter while using the google_pubsub_managedaccount() destination

hdfs: Storing messages on the Hadoop Distributed File System (HDFS)

Prerequisites How syslog-ng PE interacts with HDFS Storing messages with MapR-FS Kerberos authentication with syslog-ng hdfs() destination HDFS destination options

http: Posting messages over HTTP

Batch mode and load balancing HTTP destination options

kafka(): Publishing messages to Apache Kafka (Java implementation) (DEPRECATED)

Prerequisites How syslog-ng PE interacts with Apache Kafka Kafka destination options

kafka-c(): Publishing messages to Apache Kafka using the librdkafka client (C implementation)

kafka-c(): Prerequisites and limitations kafka-c(): Shifting from the Java implementation to the C implementation kafka-c(): Flow control in syslog-ng PE and the Kafka client Options of the kafka-c() destination

logstore: Storing messages in encrypted files

Displaying the contents of logstore files Journal files logstore() destination options

mongodb: Storing messages in a MongoDB database

How syslog-ng PE connects the MongoDB server mongodb() destination options

network: Sending messages to a remote log server using the RFC3164 protocol (network() driver)

network() destination options

pipe: Sending messages to named pipes

pipe() destination options

program: Sending messages to external applications

program() destination options

python: writing custom Python destinations

python() destination options

sentinel(): Sending logs to the Microsoft Azure Sentinel cloud

Configuring the sentinel() destination to send logs to the Microsoft Azure Sentinel cloud

Getting the required credentials to configure syslog-ng PE as a Data Connector for Microsoft Azure Sentinel Log types

sentinel() destination options

snmp: Sending SNMP traps

snmp() destination options

smtp: Generating SMTP messages (email) from logs

smtp() destination options

splunk-hec: Sending messages to Splunk HTTP Event Collector

Batch mode and load balancing splunk-hec destination options

sql(): Storing messages in an SQL database

Using the sql() driver with an Oracle database Using the sql() driver with a Microsoft SQL database The way syslog-ng PE interacts with the database

MySQL-specific interaction methods MsSQL-specific interaction methods

sql() destination options

stackdriver: Sending logs to the Google Stackdriver cloud

Configuring syslog-ng PE to send logs to Google Stackdriver stackdriver destination options

syslog: Sending messages to a remote logserver using the IETF-syslog protocol

syslog() destination options

syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers)

tcp(), tcp6(), udp(), and udp6() destination options

Change an old destination driver to the network() driver

unix-stream, unix-dgram: Sending messages to UNIX domain sockets

unix-stream() and unix-dgram() destination options

usertty: Sending messages to a user terminal — usertty() destination Client-side failover

Routing messages: log paths, flags, and filters

Embedded log statements

Using embedded log statements

if-else-elif: Conditional expressions Junctions and channels Log path flags

Managing incoming and outgoing messages with flow-control

Flow-control and multiple destinations Configuring flow-control

Using the disk-buffer option and memory buffering

Enabling the reliable disk-buffer option Enabling the normal disk-buffer option How to get information about disk-buffer files

Information about disk-buffer files Getting the list of disk-buffer files Getting the status information of disk-buffer files Printing the content of disk-buffer files Orphan disk-buffer files How to process messages from an orphan disk-buffer file using a separate syslog-ng PE instance

How to empty disk-buffer files Enabling memory buffering About disk queue files

Using filters Combining filters with boolean operators Comparing macro values in filters Using wildcards, special characters, and regular expressions in filters Tagging messages Filter functions

facility() filter() host() in-list() level() or priority() match() message() netmask() netmask6() program() source() tags()

Dropping messages

Global options of syslog-ng PE

Configuring global syslog-ng options Global options

TLS-encrypted message transfer

Secure logging using TLS Encrypting log messages with TLS

Configuring TLS on the syslog-ng clients Configuring TLS on the syslog-ng server

Mutual authentication using TLS

Configuring TLS on the syslog-ng clients Configuring TLS on the syslog-ng server

Password-protected keys TLS options

Advanced Log Transport Protocol

Logging using ALTP

How ALTP connections work Using ALTP in a client-relay-server scenario

ALTP options Examples for using ALTP

Reliability and minimizing the loss of log messages

Introduction Flow control, no disk-buffer option, no ALTP Flow control, normal disk-buffer option, no ALTP Flow control, reliable disk-buffer option, no ALTP Flow control, reliable disk-buffer option, ALTP Deciding which loss prevention mechanism to apply

Manipulating messages

Customizing message format using macros and templates

Formatting messages, filenames, directories, and tablenames Templates and macros Date-related macros Hard versus soft macros Macros of syslog-ng PE Using template functions Template functions of syslog-ng PE Modifying the on-the-wire message format

Modifying messages using rewrite rules

Replacing message parts Setting message fields to specific values Unsetting message fields Creating custom SDATA fields Setting multiple message fields to specific values Conditional rewrites

How conditional rewriting works

Anonymizing credit card numbers

Regular expressions

Types and options of regular expressions Optimizing regular expressions

parser: Parse and segment structured messages

Parsing syslog messages

Options of syslog-parser parsers

Parsing messages with comma-separated and similar values

Options of CSV parsers

Parsing key=value pairs

Options of key=value parsers

Options of JSON parsers

Limitations of the XML parser Options of the XML parsers

Parsing dates and timestamps

Options ofdate-parser() parsers

Cisco Parser Linux audit parser

Options of linux-audit-parser() parsers

Python parser Parsing enterprise-wide message model (EWMM) messages Sudo parser iptables parser

Processing message content with a pattern database

Classifying log messages

The structure of the pattern database How pattern matching works Artificial ignorance

Using pattern databases

Using parser results in filters and templates Downloading sample pattern databases

Correlating log messages using pattern databases

Referencing earlier messages of the context

Triggering actions for identified messages

Conditional actions External actions Actions and message correlation

Creating pattern databases

Using pattern parsers

Pattern parsers of syslog-ng PE

What's new in the syslog-ng pattern database format V5 The syslog-ng pattern database format

Element: patterndb Element: ruleset Element: patterns Element: rules Element: rule Element: patterns Element: urls Element: values Element: examples Element: example Element: actions Element: action Element: create-context Element: tags

Correlating log messages

Correlating messages using the grouping-by() parser

Referencing earlier messages of the context Options of grouping-by parsers

Enriching log messages with external data

Adding metadata from an external file

Using filters as selector Options add-contextual-data()

Looking up GeoIP2 data from IP addresses

Options of geoip2 parsers

Monitoring statistics and metrics of syslog-ng

Metrics and counters of syslog-ng PE Log statistics from the internal() source The monitoring() source

monitoring() source options The monitoring-welf() source

Multithreading and scaling in syslog-ng PE

Multithreading concepts of syslog-ng PE Configuring multithreading Optimizing multithreaded performance

Troubleshooting syslog-ng

Possible causes of losing log messages Creating syslog-ng core files Collecting debugging information with strace, truss, or tusc Running a failure script Stopping syslog-ng Reporting bugs and finding help Error messages

Best practices and examples

General recommendations Handling large message load Using name resolution in syslog-ng

Resolving hostnames locally

Collecting logs from chroot Configuring log rotation Load balancing logs between multiple destinations

Load balancing with a round robin load balancing method based on the R_MSEC macro of syslog-ng PE Configuration generator for the load balancing method based on MSEC hashing

The syslog-ng manual pages

The syslog-ng manual pages

dqtool.1 lgstool.1 loggen.1 pdbtool.1 persist-tool.1 syslog-debun.1 syslog-ng-ctl.1 syslog-ng.8 syslog-ng.conf.5

Third-party contributions

This appendix includes the open source licenses and attributions applicable to syslog-ng Premium Edition.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating

© ALL RIGHTS RESERVED. Terms of Use Privacy Cookie Preference Center