Domino Transaction Logging summary (4044544)

Date: June 2007
NV Version: NA
OS Version: NA
Application version: Domino Server 6.5, 7.0
Plugin version: NA


Description: Explanation of how Domino transaction logging works - INTERNAL ONLY.

Domino transaction logging Summary.

Domino transaction logging is required for Netvault to perform any incremental backups. Only Full backups can be taken of non transaction logged databases.

If Domino is enabled to use archive transaction logging, it writes all active transaction to the .txn files in the defined directory (default /LOGDIR).

When enabling transaction logging a maximum size of disk usage is set. This is the maximum amount of disk space that will be used by active transaction logs. If this is filled and Domino can write no further transactions, the Domino server will panic.

The transaction files are allocated in 64Kb chunks (this does not seem to be configurable). There is no obvious differentiation between a "live" transaction log and an "archived" transaction logs. Once a transaction log is full - but unarchived - the next transaction log is used. If there are no more useable transaction logs, a new transaction log is created (the number at the end of the filename is incremented). New transaction logs will be allocated up to the point where the maximum allowed space for transaction logs has been filled.

Transaction files are archived when a 3rd party backup program takes a safe copy of the transaction log, then sends an API call back to the Domino server to say that this transaction log has been "archived". Domino then allows this transaction log to be reused. Note that is the Domino server itself that cleans up and reuses archive log files and we have had reported cases of this process failing to remove redundant archived logs.

Once a transaction log has been archived, the next time Domino "switches" its logfile - the archived logfile will be renamed to the next sequential number and be made available for reuse.

Transaction logs have 3 modes:
Active - being currently written to
Archive - complete but as yet not written to a backup.
Inactive- has been backed up and is ready to be reused.


Example:

After some transactions - do an incremental backup:

In the backup log we see:
Backup Log File(s) for Hard Recovery:

C:\notes\data\logs\nlogctrl.lfh
C:\notes\data\logs\S0000000.TXN
C:\notes\data\logs\S0000001.TXN
C:\notes\data\logs\S0000002.TXN


Backup Archived Log File(s):

c:\notes\data\logs\S0000000.TXN


We can see these files in the directory:
C:\notes\data\logs\nlogctrl.lfh
C:\notes\data\logs\S0000000.TXN
C:\notes\data\logs\S0000001.TXN
C:\notes\data\logs\S0000002.TXN


So c:\notes\data\logs\S0000000.TXN should now be marked as archived and inactive - but currently remains which shows that it is not the backup process that does this.

After committing a large number of big changes we then see:

C:\notes\data\logs\nlogctrl.lfh
C:\notes\data\logs\S0000001.TXN
C:\notes\data\logs\S0000002.TXN
C:\notes\data\logs\S0000003.TXN

and the date-time stamp on 1 & 3 have been incremented. Note S0000000.TXN has been renamed to S0000003.TXN.


At this point if we re-run the incremental backup and we see:

Backup Log File(s) for Hard Recovery:

C:\notes\data\logs\nlogctrl.lfh
C:\notes\data\logs\S0000001.TXN
C:\notes\data\logs\S0000002.TXN
C:\notes\data\logs\S0000003.TXN


Backup Archived Log File(s):

c:\notes\data\logs\S0000001.TXN

So the actual backup operation also performs a "switch logfile" so that the currently active logfile is set into "archive" mode, backed up then set to "inactive".

After more BIG transactions. we have:

Directory of C:\notes\data\logs
07/03/2007 14:29 3,072 nlogctrl.lfh
07/03/2007 14:55 67,109,888 S0000002.TXN
07/03/2007 14:57 67,109,888 S0000003.TXN
07/03/2007 14:55 67,109,888 S0000004.TXN
07/03/2007 14:56 67,109,888 S0000005.TXN
07/03/2007 14:58 67,109,888 S0000006.TXN


So S0000001.TXN was archived and marked for rename/reuse but we then had insufficient logs for the outstanding transactions so extra files were allocated. It is not possible to reduce the number of logs once they have been allocated. The only way to remove these (that I know of) is:

Switch off transaction logging
reboot the domino server
delete the .txn files
switch ON transaction logging
reboot the domino server

Another incremental backup gives:
Backup Log File(s) for Hard Recovery:

C:\notes\data\logs\nlogctrl.lfh
C:\notes\data\logs\S0000002.TXN
C:\notes\data\logs\S0000003.TXN
C:\notes\data\logs\S0000004.TXN
C:\notes\data\logs\S0000005.TXN
C:\notes\data\logs\S0000006.TXN


Backup Archived Log File(s):

c:\notes\data\logs\S0000002.TXN
c:\notes\data\logs\S0000003.TXN



Caveats on transaction logging.
1) Enabling transaction logging in Domino creates a unique database instance ID (DBIID) for each database. These DBIIDs are used to link the transactions in the .txn files to the database with which they are associated. Any process which changes the DBIID of a database will render the incremental backups for that database useless. If an operation which changes the DBIID is used, a full backup of that database must be made to ensure backup integrity. Examples of operations that update DBIID are:
a) The first time transaction logging occurs
b) In some instances when the Compact task is executed, such as reducing
file size
c) When fixup is used to correct a corrupted database
d) When a database is moved to a server using transaction logging

2) Transaction logging works with databases in format ODS 41 (ODS=On Disk Format) or higher but not with databases that use formats from earlier releases (ODS 20 will not work). After you enable transaction logging, all databases are automatically logged. To check database formats, use the Files tab in Domino Administrator.

3) Transaction logging can be disabled for individual databases through the properties tab of that database.

4) the database dbdirman is NOT transactionally logged by default. DBDIRMAN holds information about all Notes databases on the Server. The memory cache version of dbdirman is required but the persistent .nsf file is not required so this will not be present for all databases. Transaction logging cannot be enabled for this database.

Further Notes:

1) When restoring a backup containing transaction logs - the replayed transaction logs are cleaned up after the restore process is finished.

2) When restoring a single database, all the transaction files are restored, the database then selectively rolls forward those transactions associated with that database, identified by DBIID. The side effect of this is that when restoring, the same number of .txn files are brought back from tape whether one or all database are being restored.