Can a shutdown of Shareplex cause data loss and consequent out of sync? (4330570)

You encounter an out of sync on target tables and wonder if this was caused by the shutdown of Shareplex you did earlier.

The data loss depends on a number of  factors, some of which may emanate from the type of shutdown done.

A normal shutdown of Shareplex should not usually result in data loss. An abnormal shutdown by "shutdown force" has rarely been known to cause data loss and consequent out of sync. A system crash is more likely to cause queue corruption and the ensuing data loss.

This is how it works. Whenever Shareplex is shutdown cleanly, its processes checkpoint, meaning that the queue data cached in memory is written to disk. The processes also checkpoint every certain # of queue messages processed by it so that in case of a forced shutdown, it does a checkpoint recovery whereby it reconciles what queue messages have already been sent to next process (during last checkpoint) and discards them and starts processing the ones that did not had to be discarded. Sometimes this recovery does not complete due to any snag and this may require deletion of queues for Shareplex to resume normal processing. Solution # 25047 covers this in context of Capture process and the principle applies to other processes as well. The symptoms of a process going into recovery are:

a. The messages keep accumulating in a queue
b. The event log has a message "Queue write recovery started" but no consequent message "Queue write recovery completed".

Sometimes the recovery is not proper even if it seemed to complete whereby Shareplex attempts to apply the messages that had been applied prior to the abnormal shutdown and this can either result in double posting (if no unique constraint is present on target) or the posting fails with a false alarm about out of sync (though no harm is done in this case since the duplicate data failed to post). Obviously the reconciliation as a part of checkpoint recovery was not done properly in this scenario.

At other times such as during a system crash, the queues get corrupted and it requires corruption to be fixed before Shareplex can move ahead. This may result in part or total data loss.  The reason for corruption is that Shareplex queues are partly cached in memory for performance reasons. In the event of power failure or system crash, some messages may not be fully written to disk, resulting in queue corruption.  Solution # 22995 talks about preventive aspects of queue corruption due to system crash.

It can be concluded that normal shutdown should not result in data loss and shutdown "force" can rarely result in data loss. If there were other factors responsible for shutdown, such as system crash, then it will most likely cause data loss.