Advanced Text Log Data Source
This example shows how to create a data source for the Apache access log. The example assumes that the following settings are specified in the Apache configuration file httpd.conf:
- LogFormat "%h %l %u %t \"%r\" %>s %b" common
- CustomLog logs/access_log common
To create a data source for the Apache access log
- Run the New Data Source Wizard and start creating a custom text log data source in Advanced mode.
Specify a valid path to the log directory (for example, /etc/httpd/logs) and the file name of the log (in this case, access_log). - On the Regular Expressions step, specify the following expression:
^([^ ]+)[ ][^[]+\[([^ ]+)[ ][-+\d]+\][ ]"([^"]+)"[ ](\d+)[ ]([-\d]+)[\r\n]+ - Complete the wizard.
The expression you specified matches lines similar to the following:
- 192.168.10.1 – jane [03/Apr/2005:13:21:46 -0400] "GET /cgi-bin/eshop.pl?seite=;cat%20/etc/passwd| HTTP/1.0" 404 294
- 192.168.10.5 – – [03/Apr/2005:13:21:47 -0400] "HEAD / HTTP/1.0" 200 0
- somehost.somedomain.org – – [03/Apr/2005:15:17:51 -0400] "GET /somepage.html HTTP/1.1" 404 304
The expression matches five fragments in each line, and these fragments are mapped to fields as follows:
| Match | Field |
Description |
|---|---|---|
| 1 | Computer | Client IP address or hostname |
| 2 | Date/Time | The time when the server finished processing the request |
| 3 | Insertion String 1 |
Request line from the client |
| 4 | Insertion String 2 |
Status code that the server sends back to the client |
| 5 | Insertion String 3 |
The size of the object returned to the client |
To see an example of field mapping configuration
- Open the properties of a predefined text log data source, such as CheckPoint or Cisco PIX.
- On the Settings tab, click Edit to start the Edit Custom text Log Data Source Wizard
- On the Regular Expressions step, select a regular expression and click Edit.
Database Events Data Sources
If you use software that stores its audit trails in databases, you can set up InTrust to gather such logs and store them. This can be required by regulations, or you may want to create backup storage for those logs using a different platform.
To create a custom database events data source
- In InTrust Manager, expand the Configuration node.
- Right-click the Data Sources node and select New Data Source.
- Select the Database Events type and follow the steps of the wizard.
In the wizard, configure the following settings:
- Source of data
You can manually specify an ODBC connection string, or you can click Create, select the ODBC driver from a list and let the wizard generate the connection string for you. The connection string can be created automatically if the ODBC driver is installed on the same computer as InTrust Manager. If you decide to write the query, you can use the Keyword button to insert predefined keywords.- Include the %PASSWORD% keyword in connection strings that you compose. This keyword is there for security reasons and stands for the password to be used for connection. Supply the password in the text box on the same step of the wizard.
- You can also use the keyword %COMPUTER_NAME%. This keyword is resolved as the name of the database server from which data is gathered. Use it if you need a uniform query for several database servers and use the same credentials to access them. There are as many values to %COMPUTER_NAME% as there are computers in the site you are gathering from.
If you use agents for gathering, the keyword is resolved on the agent side, and gathering is performed from all of the site's computers simultaneously, which gives better performance. If you do not use agents, the keyword is resolved by the InTrust server, and each computer in the site is processed in turn.
- SQL query
Write the SQL query that retrieves necessary data from the database. - Log name
Specify the name that InTrust must give to the log with gathered events. - Database field mapping
Configure the matching between the original database fields and those that InTrust stores. This governs how the retrieved data is arranged for storage. - Cleanup query
Supply an SQL query to be executed after gathering. This query should clear gathered events from the database.
The query is not run by default. To make it run, enable the Clear log after gathering option for policy that uses the data source. - Name and optional description of the data source.
To edit an existing database events data source, right-click the data source and select Properties.
Marking Where to Start Gathering
The SQL query you specify must include the variable %LAST_GATHERED_EVENT%. This keyword defines where the gathering starts from. If you ignore this keyword, the entire contents of the database are gathered. Doing so multiple times severely impedes performance and, if you gather to a repository, results in duplicate data.
It is recommended that you use %LAST_GATHERED_EVENT% as part of a “where” statement, such as the following:
select Time, ID, TestString from TestDatabase where Time >= %LAST_GATHERED_EVENT% order by Time
It makes sense to associate %LAST_GATHERED_EVENT% with event time. This helps avoid duplication of data even if different selection parameters are used in the query for different sessions.
It is also a good idea to make the query order events by time. Otherwise, the value of %LAST_GATHERED_EVENT% may remain the same from session to session. If you order by time, which is unique for most events, the value of %LAST_GATHERED_EVENT% is updated after each gathering, so you do not have to specify it for subsequent gathering sessions.
When you first create the data source, specify a value for %LAST_GATHERED_EVENT%. If you use the keyword as recommended, supply the date and time from which you want gathering to start. If you want to gather everything, supply a time earlier than the earliest in the log.
Be careful to use a time formatting and conversion convention that is appropriate for your RDBMS. If you associate %LAST_GATHERED_EVENT% with time, it is best to use the 24-hour format rather than the 12-hour format to avoid confusion.
Mapping Database Fields
On the Database Fields Mapping step, establish a correspondence between the fields in the original database and the InTrust representation of that database. The following controls are used:
On the left, specify the InTrust event fields. On the right, supply the database fields that match them.
InTrust field names are predefined, and you cannot specify custom names. However, you can supply as many insertion strings as you like, such as “Insertion String #1” or “Insertion String #99”. These are provided for fields that are unrelated to any of the existing InTrust fields.
The GMT and LAST_GATHERED_EVENT fields are mandatory. GMT should be mapped to event date and time. LAST_GATHERED_EVENT should be mapped to the field whose content InTrust will look at to determine where to start gathering. It is best to map LAST_GATHERED_EVENT to event date and time as well.
To use the data source in a new gathering policy
- Start creating a new gathering policy.
- On the Data Sources step of the New Policy Wizard, click Add and select the data source you created earlier.
- Optionally, as you proceed with the wizard, you can enable database cleanup and apply a filter to include or ignore specific data.
- Finish the wizards.