svcfd_create error cause QoreStor to disconnect and/or crash (4378609)

Backup job fails because QoreStor container went offline. Further inspection of QoreStor indicates that application has crashed (sometimes periodically e.g. about the same weekday and time). If this happens multiple times, QoreStor will enter maintenance mode and filesystem check will be triggered. Logs include entries such as the following:

watcher.log

2024-12-24 07:14:54 watcher[1658]: [1733] NOTICE: QUEUED_EVENT: OCA_WATCHER_FSM_EV_OCAFSD_CRASHED

2024-12-31 07:14:30 watcher[1658]: [1733] NOTICE: QUEUED_EVENT: OCA_WATCHER_FSM_EV_OCAFSD_CRASHED

2025-01-07 07:15:43 watcher[1658]: [1733] NOTICE: QUEUED_EVENT: OCA_WATCHER_FSM_EV_OCAFSD_CRASHED

ocafsd.log

2024-12-24 07:14:39 ocafsd: svc_fd_create: could not retrieve remote addr
2024-12-24 07:14:39 ocafsd[3722390]: [3744499] WARN: NFS: closing socket fd(1306) due to svcfd_create error

2024-12-31 07:14:17 ocafsd: svc_fd_create: could not retrieve remote addr
2024-12-31 07:14:17 ocafsd[516877]: [535571] WARN: NFS: closing socket fd(1089) due to svcfd_create error

2025-01-07 07:15:31 ocafsd: svc_fd_create: could not retrieve remote addr
2025-01-07 07:15:31 ocafsd[1567734]: [1586375] WARN: NFS: closing socket fd(1072) due to svcfd_create error

A function call on the OS library libtirpc.so.3.0.0 is causing a double free error. Issue has been reported by Quest, and a version with the corrected function has been built. The updated version of the library has not made it upstream in RHEL 8.x and clones at the moment of the publication of this article.

Replace the libtirpc.so.3.0.0 library.

NOTE: The library is also used by PAM for authentication. Incorrectly removal of the library instead of overwriting it can leave system in a state where login to the system won't be possible. It is highly suggested to run the commands as root or, alternatively, having a different session open with root.

1. Download and extract the library file from archive
   new_libtirpc.so.3.0.0.zip
2. Upload the file to the /tmp directory on the QoreStor server
3. Log into the QoreStor server in single-user mode

4. Stop QoreStor service

   systemctl stop ocards

5. Backup original library

   cp /lib64/libtirpc.so.3.0.0 /lib64/libtirpc.so.3.0.0_orig

6. Put new library in place

   cp /tmp/new_libtirpc.so.3.0.0 /lib64/libtirpc.so.3.0.0 
   

7. remove old symbolic link

   rm -f /lib64/libtirpc.so.3

8. create new symbolic link

   ln -s /lib64/libtirpc.so.3.0.0 /lib64/libtirpc.so.3

9. Start Qorestor service

   systemctl start ocards

10. In a new session, test logging into the system using password (verify that PAM is working).