Update monitoring scripts to track leftover OpenVSwitch 'veth' interfaces and clean them up pereodically to avoid performance degradation, split kickstart

1 files changed, 103 insertions, 0 deletions

diff --git a/docs/problems.txt b/docs/problems.txt
new file mode 100644
index 0000000..4be9dc7
--- /dev/null
+++ b/docs/problems.txt
@@ -0,0 +1,103 @@
+Actions Required
+================
+ * Long-term solution to 'rogue' interfaces is unclear. May require update to OpenShift 3.9 or later.
+ However, proposed work-around should do unless execution rate grows significantly.
+ * All other problems found in logs can be ignored.
+ 
+
+Rogue network interfaces on OpenVSwitch bridge
+==============================================
+ Sometimes OpenShift fails to clean-up after terminated pod properly. The actual reason is unclear.
+  * The issue is discussed here:
+        https://bugzilla.redhat.com/show_bug.cgi?id=1518684
+  * And can be determined by looking into:
+    ovs-vsctl show
+
+ Problems:
+  * As number of rogue interfaces grow, it start to have impact on performance. Operations with
+  ovs slows down and at some point the pods schedulled to the affected node fail to start due to
+  timeouts. This is indicated in 'oc describe' as: 'failed to create pod sandbox'
+
+ Cause:
+  * Unclear, but it seems periodic ADEI cron jobs causes the issue.
+  * Could be related to 'container kill failed' problem explained in the section bellow.
+     Cannot kill container ###: rpc error: code = 2 desc = no such process
+
+         
+ Solutions:
+  * According to RedHat the temporal solution is to reboot affected node (not tested yet). The problem
+  should go away, but may re-apper after a while. 
+  * The simplest work-around is to just remove rogue interface. They will be re-created, but performance
+  problems only starts after hundreds accumulate.
+    ovs-vsctl del-port br0 <iface>
+  
+ Status:
+   * Cron job is installed which cleans rogue interfaces as they number hits 25.
+
+
+Orphaning / pod termination problems in the logs
+================================================
+ There is several classes of problems reported with unknow reprecursions in the system log. Currently, I
+ don't see any negative side effects except some of these issues may trigger "rogue interfaces" problem.
+
+ ! container kill failed because of 'container not found' or 'no such process': Cannot kill container ###: rpc error: code = 2 desc = no such process"
+
+   Despite the errror, the containers are actually killed and pods destroyed. However, this error likely triggers
+   problem with rogue interfaces staying on the OpenVSwitch bridge.
+    
+  Scenario:
+    * happens with short-living containers 
+
+ - containerd: unable to save f7c3e6c02cdbb951670bc7ff925ddd7efd75a3bb5ed60669d4b182e5337dec23:d5b9394468235f7c9caca8ad4d97e7064cc49cd59cadd155eceae84545dc472a starttime: read /proc/81994/stat: no such process
+   containerd: f7c3e6c02cdbb951670bc7ff925ddd7efd75a3bb5ed60669d4b182e5337dec23:d5b9394468235f7c9caca8ad4d97e7064cc49cd59cadd155eceae84545dc472a (pid 81994) has become an orphan, killing it
+    
+  Scenario:
+    This happens every couple of minutes and attributed to perfectely alive and running pods. 
+    * For instance, ipekatrin1 was complaining some ADEI pod.
+    * After I removed this pod, it immidiately started complaining on 'glusterfs' replica.
+    * If 'glusterfs' pod re-created, the problem persist.
+    * It seems only a single pod is affected at each given moment (at least this was always true 
+    on ipekatrin1 & ipekatrin2 while I was researching the problem)
+    
+  Relations:
+    * This problem is not aligned with the next 'container not found' problem. One happens with short-living containers which
+    actually get destroyed. This one is triggered for persistent container which keep going. And in fact this problem is triggered
+    significantly more frequently.
+
+  Cause:
+    * Seems related to docker health checks due to a bug in docker 1.12* which is resolved in 1.13.0rc2
+        https://github.com/moby/moby/issues/28336
+        
+  Problems:
+    * It seems only extensive logging, according to the discussion in the issue
+
+  Solution: Ignore for now
+    * docker-1.13 had some problems with groups (I don't remember exactly) and it was decided to not run it with current version of KaaS.
+    * Only update docker after extensive testing on the development cluster or not at all.
+
+ - W0625 03:49:34.231471   36511 docker_sandbox.go:337] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "...": Unexpected command output nsenter: cannot open /proc/63586/ns/net: No such file or directory
+ - W0630 21:40:20.978177    5552 docker_sandbox.go:337] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "...": CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "..."
+  Scenario:
+    * It seems can be ignored, see RH bug.
+    * Happens with short-living containers (adei cron jobs)
+
+  Relations:
+    * This is also not aligned with 'container not found'. The time in logs differ significantly.
+    * It is also not aligned with 'orphan' problem.
+
+  Cause:
+    ? https://bugzilla.redhat.com/show_bug.cgi?id=1434950 
+
+ - E0630 14:05:40.304042    5552 glusterfs.go:148] glusterfs: failed to get endpoints adei-cfg[an empty namespace may not be set when a resource name is provided]
+   E0630 14:05:40.304062    5552 reconciler.go:367] Could not construct volume information: MountVolume.NewMounter failed for volume "kubernetes.io/glusterfs/4
+
+    I guess some configuration issue.... Probably can be ignored...
+
+  Scenario:
+    * Reported on long running pods with persistent volumes (katrin, adai-db)
+    * Also seems an unrelated set of the problems.
+
+
+
+
+