Not getting alerts for master-api, master-controllers & master-etcd pods after shutting down one of the master nodes in Kubernetes cluster (4280193)

Enhancement Request CON-717 has been submitted to Development to add a new Rule to detect abnormal status of PODs running on a node if a node is shutdown. It will be evaluated for a future version.

Please check the release notes of latest version of Container Management cartridge to find out which Enhancements have been included in this release.

Additional Information:

There are two fields that are used for checking the status. One is phase, and the other is conditions, all returned by Kubernetes API Server REST APIs.

Phase is a summary status in string format, like Running, Failed, Unknown.

Conditions is a list of detailed status related to pods lifecycle, Like Ready:True/False, Initialized:True/False, ...

So in a multi-master environment, all pods Phase status is Running and Condition is good.

If one of the master nodes is shutdown, some of the pods on that node except the master components will change to Terminating, however, the master component pods are always Running. however, the conditions Ready change to False.

In the Rule, we use the phase to report the alarm, so if the pods phase is always Running, we will not trigger the alarm. Instead there will be a Critical alarm for Nodes. "Kubernetes Node Health Check" something like NodeStatusUnknown: Kubelet stopped posting node status