Agent-Side Audit Log Backup (Cache)
Agent-Side Audit Log Backup Concepts
Agent-side cache, referred to in the product GUI as "agent-side audit log backup", is an InTrust proprietary technology primarily aimed at protection of audit data from being lost even if an accidental or malicious log cleanup occurs on the target machine. Another benefit of this technology is that it allows InTrust to never access the same audit trail (most commonly, event log) more than once for one and the same event, even if multiple gathering jobs applied to the local agent by one or more InTrust Server(s) require the same events.
Agent-side log backup is used only with scheduled task-based event log gathering and doesn't affect real-time monitoring or real-time event collection in any way. It is also optional, so gathering can be performed without enabling it.
|
|
Note: Number of data sources processed by gathering jobs defines the number of accesses to native log. |
The cache is implemented as a set of folders and files on the agent machine. The structure and names of files and folders follows a specific pattern. Cached audit data is stored on disk in compressed form in order to provide for keeping a large amount of events for the period of time sufficient for InTrust to gather them all according to its gathering schedule, regardless of the original audit trail retention settings. That is, even if the events in their native log are overwritten (or the log is purged) by the time when InTrust comes for them, those events still can be gathered from the agent cache.
An agent with caching enabled receives events being logged to an audit trail specified for scheduled gathering in real time, as they are written to the native log.
|
|
Note: InTrust agent does not interfere with the native event logging workflow but listens to the events in parallel with the original event logging. The native process of writing events to the original audit trail is not interrupted when agent-side caching is stopped or something unexpected happens to it (except Syslog processing that can be restarted when caching is turned on/off). |
Since an agent may be engaged in gathering events from the same log with multiple gathering policies, it caches all the events in the log regardless of filters imposed by settings of a specific policy or InTrust data source the policy is based on. The elementary entity to be specified for caching is an audit trail (log) underlying the InTrust data source. It is a specific gathering job where any filtering is done, exactly as in the case of gathering non-cached events from a native log.
Cached events remain stored on disk and available for further gathering when the agent machine is rebooted or the agent is temporarily stopped. When the agent is started again, it proceeds with caching events generated since the agent start. For Windows NT data source type, however, the following takes place:
The data source of Windows NT type keeps the position of the last monitored event. If that event is not found in the log at the agent start, then the log data (for the specified period) will be re-read from the earliest records.
Events from a particular audit trail (log) remain stored in the agent cache until one of the following happens:
- The cache folder size limit is exceeded
- The retention period defined for an InTrust data source based on that log expires
- A gathering job with the runs for events from this specific log and clears events after gathering
The effect each of these conditions has on events stored in agent cache is different and will be described in more detail further in this document.
How It Works
Agent Data Location
Agent cache files are located in the proxy_manager subfolder of InTrust agent data folder. The location of the agent data folder is specified by the value of the adc_data_path variable defined in the agent.ini file.
The value of adc_data_path is defined in agent.ini through another variable, %ADC_INSTALLPATH%. The value of this variable is read from the local registry at HKEY_LOCAL_MACHINE\SOFTWARE\Aelita\ADC\ADCCoreInstallPath (for agents on Windows-based computers).
By default, the data folder is an immediate subfolder of the agent installation folder named data.
You can also use InTrust Manager to change the agent-side cache location for one or more agents selected (see the General tab of the agent's properties dialog box).
Agent Log Structure
An agent writes data to be cached into a file. When it reaches the ITRT_RTCacheFileLimitSize value (see Agent Log Controls and Settings below), the agent stops adding data to it, compresses the file and starts writing to a new one. Another thing the agent does at this point is create a new entry in the index file cfg.1 mapping the name (GUID) of the new archived file to the timestamps of the first and the last event stored in it. Use of this index file allows the agent to quickly retrieve events when a gathering time comes, with no necessity to search through all the compressed files in attempt to locate events matching the scope of the particular gathering job. This index also makes it easier for the agent to quickly clean up its cache since outdated event data is deleted on the per-file basis, and files to be removed from the cache are spotted based on the map from cfg.1.
As stated above, the most granular entity to be specified for caching is an audit trail (log) underlying the InTrust data source. That is, in terms of InTrust configuration objects, a data source. Therefore folders named after data sources look like the most obvious way to organize the cache files in, since each data source can be uniquely identified by its GUID organization-wide. But an agent may work for multiple InTrust Servers that not necessarily belong to one and the same InTrust organization, while a data source GUID is recognized within one organization only. For this reason, data source subfolders are located on the second hierarchy level, making way for folders named by InTrust organization GUIDs. An individual index files is maintained for each data source and is located in corresponding folder.
As a result, the tree of folders and files eventually follows the following pattern:
<Agent_data_folder>\proxy_manager (folder)
- Organization 1 folder GUID (folder)
- data source 1 GUID (folder)
- current cache file (uncompressed)
- cfg.1 (index file, uncompressed)
- archived cache file 1 (compressed)
- archived cache file 2 (compressed)
- archived cache file 3 (compressed)
- archived cache file 4 (compressed)
- <...>
- data source 2 GUID (folder)
- current cache file (uncompressed)
- cfg.1 (index file, uncompressed)
- archived cache file 1 (compressed)
- archived cache file 2 (compressed)
- archived cache file 3 (compressed)
- archived cache file 4 (compressed)
- <...>
- data source 3 GUID (folder)
- <...>
- <...>
- Organization 2 folder GUID (folder)
- data source 1 GUID (folder)
- <...>
- data source 2 GUID (folder)
- <...>
- <...>
- <...>
- <...>
- data source 1 GUID (folder)
Agent Log Retention and Cleanup
When the total size of cache files in all data source subfolders of an organization folder reaches the value defined by the ITRT_RTCacheFolderLimitSize organization parameter, the agent deletes the compressed file that stores the oldest events from the biggest data source folder to free up space for new compressed files.
If the new compressed file still does not fit into ITRT_RTCacheFolderLimitSize, the agent removes one more file using the same logic, and so on.
The condition imposed by the Agent-side audit log backup retention period (days) data source setting is checked every 24 hours (hardcoded) starting from the agent start time. Upon the condition verification, every file, compressed or uncompressed, with no event in it younger than the retention criterion is deleted from the data source folder.
When a gathering job runs against the agent computer and clears the log after gathering, all compressed cache files are deleted from the folder(s) matching the data source(s) included into the job.
Every time an agent creates or removes a cache file, it adds or removes an entry to or from the cfg.1 index file in the appropriate data source folder.
Every time a cache-enabled agent starts, it evaluates all the files in each data source folder against records in the cfg.1 file stored there. Every file found in a folder with no matching entry in the local index file is considered unrelated and deleted unconditionally.
When a last task with a gathering job in it that applies to an agent is unscheduled or deleted from an InTrust organization configuration, and caching of some (but not all, within an organization) data sources is switched off on this agent, folder with files for those data sources remains in cache for 10 more days (this value is hardcoded and does not depend on the data source retention period settings). The same is true if caching is stopped for all the data sources on the agent that remains engaged in any real-time monitoring for at least one InTrust Server in the same organization the cached data sources belong to.
If agent-side caching for all data sources belonging to an organization is stopped and the agent does not do any real-time monitoring for any InTrust Server in that organization, cached data under the organization GUID will remain on disk for 1 more day + 3 hours of uninterrupted agent uptime (defaults values, controlled by ITRT_ZombieTimeout and ITRT_ZombieLastChanceTimeout organization parameters respectively).
Agent Log Controls and Settings
The Agent-side audit log backup retention period (days) parameter is specified in properties of each particular InTrust data source. The default value is 2 days for data sources based on the Security event log and 8 days for other data sources.
The following settings related to cache behavior are defined as the organization parameters:
| Parameter Name | Default Value | Maximum Value | Description |
|---|---|---|---|
| ITRT_ZombieTimeout | 86400 | 2147483647 | Period of time, in seconds, after stopping all the real-time monitoring and event caching activity on an agent, during which it does not delete from disk any data related to its former activity (tasks and proxy_manager subfolders of the agent data folder). |
| ITRT_ZombieLastChanceTimeout | 10800 | 2147483647 | A period, in seconds, of uninterrupted uptime for an agent with the ITRT_ZombieTimeout period expired, during which the agent waits for appropriate InTrust Server to resume this agent’s real-time or event caching activity before it permanently deletes the tasks and proxy_manager subfolders of the agent data folder from disk. |
|
|
Note: Each of the organization parameters can be defined at either the organization (default) or InTrust Server or agent level. If an agent is caching events from the same data sources for scheduled gathering job on multiple InTrust servers, and effective parameter settings differ from of those Servers to another, the most strict (the least) of the conflicting parameter values within one organization becomes effective for the whole agent cache for that organization. |
In addition, the following settings are defined in agent.ini:
| Parameter Name | Default Value | Maximum Value | Description |
|---|---|---|---|
| ITRT_RTCacheFileLimitSize | 64 | 1796 |
Maximum size, in megabytes, of a single cached data file. When a file reaches this size, it is closed and saved in a compressed form, and a new file is started. |
| ITRT_RTCacheFolderLimitSize | 1024 |
2147483647(or -1 for 'unlimited') |
Maximum total size, in megabytes, of all cached data files. When a cache reaches this size, the oldest cached data file is irrevocably deleted and a new file is started. |
| ITRT_RTCacheLockFiles | 1 | — | When set to '1', the InTrust Agent service keeps each cached data file open while it works. This makes cache files locked by the agent process and protects them from being deleted by a user or some other application. Opening each file, however, takes time and may result in a long time required for an agent to start when its local cache contains a large number of cached data files. To disable cache file locking, set the value of this parameter to '0'. |
Agentless Gathering
In case of agent-less gathering, the GatheringEngine connects to the remote computer via RPC, SMB or ODBC (depending on data source type).
During gathering to a repository, the following takes place:
- GatheringEngine collects the data, applying filters as necessary, and writes the data into a “repository file” in a temporary folder on the InTrust server.
- GatheringEngine moves the “repository file” to the repository.
During gathering to an audit database, GatheringEngine collects the data and stores it directly in the audit database.