Embedded FMS repository error PANIC: could not open control file "global/pg_control": Permission denied (4374922)

Foglight embedded FMS repository running on PostgreSQL fails to start.

During an upgrade, the log file Foglight_[version]_Install_[date/time]_001.log shows messages like this:

2024-02-08 10:57:35.738 VERBOSE [ProcessReader-1] postgres/STDOUT - waiting for server to start....2024-02-08 10:57:35.751 PST: LOG:  listening on IPv6 address "::1", port 15432
2024-02-08 10:57:35.738 VERBOSE [ProcessReader-1] postgres/STDOUT - 2024-02-08 10:57:35.751 PST: LOG:  listening on IPv4 address "127.0.0.1", port 15432
2024-02-08 10:57:35.770 VERBOSE [ProcessReader-1] postgres/STDOUT - 2024-02-08 10:57:35.770 PST: PANIC:  could not open control file "global/pg_control": Permission denied
2024-02-08 10:57:37.058 VERBOSE [ProcessReader-1] postgres/STDOUT - . stopped waiting
2024-02-08 10:57:37.058 VERBOSE [ProcessReader-2] postgres/STDERR - pg_ctl: could not start server
2024-02-08 10:57:37.058 VERBOSE [ProcessReader-2] postgres/STDERR - Examine the log output.
2024-02-08 10:57:37.058 VERBOSE [Thread-31] com.quest.nitro.db.PostgreSQLEmbeddedDatabaseHandler - Process "pg_ctl" finished with exit code: 1.
2024-02-08 10:57:37.058 ERROR   [Thread-31] com.quest.nitro.db.EmbeddedDatabaseManager - Failed to exec embedded database process
java.lang.IllegalStateException: Exception while waiting for postgres to get ready for external connections: java.io.IOException: pg_ctl existed with error code  1
    at com.quest.nitro.db.PostgreSQLEmbeddedDatabaseHandler.startDatabaseServer(PostgreSQLEmbeddedDatabaseHandler.java:163)
    at com.quest.nitro.db.EmbeddedDatabaseManager.startup(EmbeddedDatabaseManager.java:184)
    at com.quest.nitro.installanywhere.database.EmbeddedDatabaseConfigurator.startupEmbeddedDB(EmbeddedDatabaseConfigurator.java:65)
    at com.quest.nitro.installanywhere.action.FoglightUpgradeRequirementCheckAction.install(FoglightUpgradeRequirementCheckAction.java:147)
    at com.zerog.ia.installer.actions.CustomAction.installSelf(Unknown Source)
    at com.zerog.ia.installer.util.GenericInstallPanel$2.run(Unknown Source)
Caused by: java.io.IOException: pg_ctl existed with error code  1
    at com.quest.nitro.db.PostgreSQLEmbeddedDatabaseHandler.startDatabaseServer(PostgreSQLEmbeddedDatabaseHandler.java:150)
    ... 5 more

This same issue can also happen when changing the service account being used for running the Foglight service.

When starting the FMS with global debug level 1 using [FMS_HOME]\bin\fms.exe -Dquest.debug=1, the log file ManagementServer_[date/time]_001.log shows messages like this:

2024-02-23 10:34:27.965 VERBOSE [forge-startup] com.quest.nitro.db.PostgreSQLEmbeddedDatabaseHandler - Process "pg_ctl" finished with exit code: 3.
2024-02-23 10:34:27.965 VERBOSE [forge-startup] com.quest.nitro.db.PostgreSQLEmbeddedDatabaseHandler - Executing: [E:\Quest\Foglight\postgresql\bin\pg_ctl.exe, start, -w, -D, E:\Quest\Foglight\state\postgresql-data, -o, --port=15432]
2024-02-23 10:34:28.028 VERBOSE [ProcessReader-1] postgres/STDERR - pg_ctl: another server might be running; trying to start server anyway
2024-02-23 10:34:28.168 VERBOSE [ProcessReader-2] postgres/STDOUT - waiting for server to start....2024-02-23 10:34:28.173 PST: LOG:  listening on IPv6 address "::1", port 15432
2024-02-23 10:34:28.168 VERBOSE [ProcessReader-2] postgres/STDOUT - 2024-02-23 10:34:28.173 PST: LOG:  listening on IPv4 address "127.0.0.1", port 15432
2024-02-23 10:34:28.184 VERBOSE [ProcessReader-2] postgres/STDOUT - 2024-02-23 10:34:28.189 PST: PANIC:  could not open control file "global/pg_control": Permission denied
2024-02-23 10:34:28.325 VERBOSE [ProcessReader-2] postgres/STDOUT -  stopped waiting
2024-02-23 10:34:28.325 VERBOSE [ProcessReader-1] postgres/STDERR - pg_ctl: could not start server
2024-02-23 10:34:28.325 VERBOSE [ProcessReader-1] postgres/STDERR - Examine the log output.
2024-02-23 10:34:28.325 VERBOSE [forge-startup] com.quest.nitro.db.PostgreSQLEmbeddedDatabaseHandler - Process "pg_ctl" finished with exit code: 1.
2024-02-23 10:34:28.325 ERROR   [forge-startup] com.quest.nitro.db.EmbeddedDatabaseManager - Failed to exec embedded database process
java.lang.IllegalStateException: Exception while waiting for postgres to get ready for external connections: java.io.IOException: pg_ctl existed with error code  1
    at com.quest.nitro.db.PostgreSQLEmbeddedDatabaseHandler.startDatabaseServer(PostgreSQLEmbeddedDatabaseHandler.java:163)
    at com.quest.nitro.db.EmbeddedDatabaseManager.startup(EmbeddedDatabaseManager.java:184)

Regardless of the scenario the solution is to grant full access to the service account being used for running the installer or service.

1. Using File Explorer in Windows, right click on the [FMS_HOME] folder (e.g. E:\Quest\Foglight) and select Properties
2. Select the Security tab
3. Click on Edit...
4. Click on Add.. and lookup the account to grant access to (e,g, foglight_svc)
5. Allow Full control
6. Click OK to close the Permissions dialog
7. Click OK or Cancel to close the [FMS_HOME] Properties dialog

Some environments generate an error for Failed to enumerate the objects in the container. Access is denied. while applying step 6.

If this happens, try running icacls [FMS_HOME] /setowner "EXAMPLEDOMAIN\exampleuser" /t /l to override the ownership of the directory and then retry the previous steps.

# Example
icacls E:\Quest\Foglight /setowner "TESTING\foglight_svc" /t /l