InTrust 11.6 - Repository Indexing for Advanced Search Capabilities

About Repository Indexing

InTrust uses indexing on repositories for fast searching and data retrieval. Indexing is optional, but it gives you the following benefits:

• You get real-time feedback as you browse audit data.
• You do not need to configure reporting for data analysis, because interactive data filtering, organization and sorting functionality is available in Repository Viewer.

Centera-based repositories currently do not support indexing.

Note that repository indexing is a resource-intensive activity. To increase indexing performance, set up a powerful dedicated computer specifically for this purpose, as described in the Dedicated Indexing topic.

When you view the contents of a repository in Repository Viewer, the most recent events are found first.

Estimating the Resources Required for Indexing

Performance tests conducted by InTrust quality control on repositories with typical heterogeneous data show the following results.

Conditions

• There is only one InTrust server in the organization.
• Indexing and real-time event collection are the only activities that take place in the tests.
• Every agent processes data from 6 to 10 active event logs, and the rate of event generation is about 50 events per second per agent.
• The InTrust server is a virtual machine. Two virtual machine host configurations are used in tests:
  • VMware ESXi on Dell PowerEdge 1950 with Intel^® Xeon^® E5310 processors, 2 sockets, 4 cores per socket
  • Hyper-V Server on Windows 2012 R2 on Dell PowerEdge R620 with Intel^® Xeon^® E5-2670 processors, 2 sockets, 16 cores per socket, hyperthreading is on

Results

On the E5310 configuration

On the E5-2670 configuration

Configuration-independent repository growth statistics (including index)

Estimating in Your Environment

The typical hardware requirements for repository indexing are outlined above. If the conditions in your environment differ—for example, if the indexing computers also perform a lot of audit data gathering or monitor a lot of sites—then you need to adjust your estimates.

To estimate the required disk space, you need to know how much the actual audit data takes up. For that, the most readily available solution is to use the dir command with the /s switch.

The size of the fully indexed repository is approximately twice the data size.

Troubleshooting

When you configure indexing, it is useful to track indexing-related events in the InTrust Server event log. For details about these events, see Events from InTrust Repository Services. The following tips will only indicate the event IDs, which you can look up in that topic.

Event IDs 13875, 13877, 13878

These warnings mean the number of files that haven't been indexed has exceeded a threshold value. This warning normally indicates a temporary state. For example, it may keep recurring for only four hours a day.

Event ID 8321

This error indicates that the InTrust server's event queue for a particular agent has overflowed. Reduce the activity of that agent.

Mind Your Disk Queue

The average disk queue length on the indexing InTrust server's disk and on the disk that contains the repository should not exceed the value 2. A higher number means the disk is a bottleneck resource.

Mind Your Bandwidth

Make sure you have enough bandwidth to accommodate the traffic generated by all the agents that talk to the InTrust server.

Configuring Indexing

If a repository is created in InTrust Deployment Manager, indexing is enabled automatically for it, but if it is created in InTrust Manager, indexing is disabled by default. To configure indexing options for an existing repository, open its properties in InTrust Manager and go to the Indexing tab.

This tab lets you do the following:

• Enable indexing capabilities for the repository.
  This is an irreversible operation; indexing of the repository will continue until it is deleted from InTrust configuration.
• Select the InTrust server that will manage the index-processing operations.
• Specify the location of the index files.
  The most convenient method is to place the index in the repository folder (this is the default option).
• Configure whether the indexing work is offloaded elsewhere.
  You can make the index-managing server also build the index, or you can use dedicated computers for the purpose. For details about offloading indexing, see the Dedicated Indexing topic. If you select to set up dedicated indexing, you have the option to specify the account that will be used for it.

Make sure that Active Directory delegation is enabled for the following:

• The computer account of the InTrust server that manages the index
• The user accounts that perform indexing

Best Practice: Lean Indexed Short-Term Repoisitory

If you use InTrust Manager, the recommended setup is to have more than one repository:

1. An indexed repository with the most current data
2. Another repository (indexed or non-indexed) for archival of data that is no longer current

A freshly-deployed default repository is indexed. In an upgraded InTrust deployment, all repositories keep their indexing settings after the upgrade, so if indexing was disabled prior to the upgrade, it is not enabled automatically.

To keep the short-term repository current, set up regular repository cleanup jobs that clear all data older than specified. To move data from the short-term repository to the archive, use regular cleanup jobs that occur straight after the consolidation.

This configuration helps achieve the following:

• Maximize indexing performance
• Keep the current repository lean, fast and at a roughly constant volume

The size of your short-term repository depends on your auditing needs. To prevent indexing from slowing down the auditing workflow, you may want to consider the suggestions in the Dedicated Indexing topic.