About Repository Indexing
InTrust uses indexing on repositories for fast searching and data retrieval. Indexing is optional, but it gives you the following benefits:
- You get real-time feedback as you browse audit data.
- You do not need to configure reporting for data analysis, because interactive data filtering, organization and sorting functionality is available in Repository Viewer.
Centera-based repositories currently do not support indexing.
Note that repository indexing is a resource-intensive activity. To increase indexing performance, set up a powerful dedicated computer specifically for this purpose, as described in the Dedicated Indexing topic.
When you view the contents of a repository in Repository Viewer, the most recent events are found first.
|
|
Caution: In InTrust versions prior to 10.4, events were not prioritized by how recent they are. Events gathered to an indexed repository by InTrust 10.4 or earlier will not be prioritized in the current version of Repository Viewer—they will be found in arbitrary order. However, in a non-indexed repository events will be prioritized correctly, even though searches will be slower. |
Estimating the Resources Required for Indexing
Performance tests conducted by InTrust quality control on repositories with typical heterogeneous data show the following results.
Conditions
- There is only one InTrust server in the organization.
- Indexing and real-time event collection are the only activities that take place in the tests.
- Every agent processes data from 6 to 10 active event logs, and the rate of event generation is about 50 events per second per agent.
- The InTrust server is a virtual machine. Two virtual machine host configurations are used in tests:
- VMware ESXi on Dell PowerEdge 1950 with Intel® Xeon® E5310 processors, 2 sockets, 4 cores per socket
- Hyper-V Server on Windows 2012 R2 on Dell PowerEdge R620 with Intel® Xeon® E5-2670 processors, 2 sockets, 16 cores per socket, hyperthreading is on
Results
On the E5310 configuration
| Number of processors | Number of agents | Repository growth rate |
|---|---|---|
| 4 | 200 | 36MB/sec |
| 8 | 300 | 54MB/sec |
On the E5-2670 configuration
| Number of processors | Number of agents | Repository growth rate |
|---|---|---|
| 4 | 500 | 90MB/sec |
| 8 | 800 | 144MB/sec |
| 16 | 1100 | 198MB/sec |
Configuration-independent repository growth statistics (including index)
| Number of agents | Repository growth per minute, bytes |
Repository growth per hour, bytes |
Repository growth per day, bytes |
Repository growth per month, bytes |
|---|---|---|---|---|
| 100 | 9,750,000 | 585,000,000 | 14,040,000,000 | 421,200,000,000 |
| 500 | 48,750,000 | 2,925,000,000 | 70,200,000,000 | 2,106,000,000,000 |
| 900 | 87,750,000 | 5,265,000,000 | 126,360,000,000 | 3,790,800,000,000 |
Estimating in Your Environment
The typical hardware requirements for repository indexing are outlined above. If the conditions in your environment differ—for example, if the indexing computers also perform a lot of audit data gathering or monitor a lot of sites—then you need to adjust your estimates.
To estimate the required disk space, you need to know how much the actual audit data takes up. For that, the most readily available solution is to use the dir command with the /s switch.
The size of the fully indexed repository is approximately twice the data size.
Troubleshooting
When you configure indexing, it is useful to track indexing-related events in the InTrust Server event log. For details about these events, see Events from InTrust Repository Services. The following tips will only indicate the event IDs, which you can look up in that topic.
Event IDs 13875, 13877, 13878
These warnings mean the number of files that haven't been indexed has exceeded a threshold value. This warning normally indicates a temporary state. For example, it may keep recurring for only four hours a day.
Event ID 8321
This error indicates that the InTrust server's event queue for a particular agent has overflowed. Reduce the activity of that agent.
Mind Your Disk Queue
The average disk queue length on the indexing InTrust server's disk and on the disk that contains the repository should not exceed the value 2. A higher number means the disk is a bottleneck resource.
Mind Your Bandwidth
Make sure you have enough bandwidth to accommodate the traffic generated by all the agents that talk to the InTrust server.
Configuring Indexing
If a repository is created in InTrust Deployment Manager, indexing is enabled automatically for it, but if it is created in InTrust Manager, indexing is disabled by default. To configure indexing options for an existing repository, open its properties in InTrust Manager and go to the Indexing tab.
This tab lets you do the following:
- Enable indexing capabilities for the repository.
This is an irreversible operation; indexing of the repository will continue until it is deleted from InTrust configuration. - Select the InTrust server that will manage the index-processing operations.
- Specify the location of the index files.
The most convenient method is to place the index in the repository folder (this is the default option). - Configure whether the indexing work is offloaded elsewhere.
You can make the index-managing server also build the index, or you can use dedicated computers for the purpose. For details about offloading indexing, see the Dedicated Indexing topic. If you select to set up dedicated indexing, you have the option to specify the account that will be used for it.
Make sure that Active Directory delegation is enabled for the following:
- The computer account of the InTrust server that manages the index
- The user accounts that perform indexing
Best Practice: Lean Indexed Short-Term Repoisitory
If you use InTrust Manager, the recommended setup is to have more than one repository:
- An indexed repository with the most current data
- Another repository (indexed or non-indexed) for archival of data that is no longer current
A freshly-deployed default repository is indexed. In an upgraded InTrust deployment, all repositories keep their indexing settings after the upgrade, so if indexing was disabled prior to the upgrade, it is not enabled automatically.
To keep the short-term repository current, set up regular repository cleanup jobs that clear all data older than specified. To move data from the short-term repository to the archive, use regular cleanup jobs that occur straight after the consolidation.
This configuration helps achieve the following:
- Maximize indexing performance
- Keep the current repository lean, fast and at a roughly constant volume
The size of your short-term repository depends on your auditing needs. To prevent indexing from slowing down the auditing workflow, you may want to consider the suggestions in the Dedicated Indexing topic.