Second revision: includes hostpath mounts, gluster block storage, kaas apps, etc.

1 files changed, 49 insertions, 0 deletions

diff --git a/docs/troubleshooting.txt b/docs/troubleshooting.txt
index b4ac8e7..ef3c206 100644
--- a/docs/troubleshooting.txt
+++ b/docs/troubleshooting.txt
@@ -60,6 +60,8 @@ Debugging
         oc logs <pod name> --tail=100 [-p]                  - dc/name or ds/name as well
  - Verify initialization steps (check if all volumes are mounted)
         oc describe <pod name>
+ - Security (SCC) problems are visible if replica controller is queried
+        oc -n adei get  rc/mysql-1 -o yaml
  - It worth looking the pod environment
         oc env po <pod name> --list
  - It worth connecting running container with 'rsh' session and see running processes,
@@ -85,6 +87,7 @@ network
     * that nameserver is pointing to the host itself (but not localhost, this is important
     to allow running pods to use it)
     * that correct upstream nameservers are listed in '/etc/dnsmasq.d/origin-upstream-dns.conf'
+    * that correct upstream nameservers are listed in '/etc/origin/node/resolv.conf'
     * In some cases, it was necessary to restart dnsmasq (but it could be also for different reasons)
  If script misbehaves, it is possible to call it manually like that
     DEVICE_IFACE="eth1" ./99-origin-dns.sh eth1 up
@@ -96,6 +99,7 @@ etcd (and general operability)
  may be needed to restart them manually. I have noticed it with 
     * lvm2-lvmetad.socket       (pvscan will complain on problems)
     * node-origin
+    * glusterd in container     (just kill the misbehaving pod, it will be recreated)
     * etcd               but BEWARE of too entusiastic restarting:
  - However, restarting etcd many times is BAD as it may trigger a severe problem with 
  'kube-service-catalog/apiserver'. The bug description is here
@@ -181,6 +185,13 @@ pods (failed pods, rogue namespaces, etc...)
                 docker ps -aq --no-trunc | xargs docker rm
 
 
+Builds
+======
+ - After changing storage for integrated docker registry, it may refuse builds with HTTP error 500. It is necessary
+ to run:
+    oadm policy reconcile-cluster-roles
+
+
 Storage
 =======
  - Running a lot of pods may exhaust available storage. It worth checking if 
@@ -208,3 +219,41 @@ Storage
         gluster volume start <vol>
     * This may break services depending on provisioned 'pv' like 'openshift-ansible-service-broker/asb-etcd'
     
+ - If something gone wrong, heketi may end-up creating a bunch of new volumes, corrupt database, and crash
+ refusing to start. Here is the recovery procedure.
+    * Sometimes, it is still possible to start by setting 'HEKETI_IGNORE_STALE_OPERATIONS' environmental
+    variable on the container.
+        oc -n glusterfs env dc  heketi-storage -e HEKETI_IGNORE_STALE_OPERATIONS=true
+    * Even if it works, it does not solve the main issue with corruption. It is necessary to start a 
+    debugging pod for heketi (oc debug) export corrupted databased, fix it, and save back. Having
+    database backup could save a lot of hussle to find that is amiss.
+        heketi db export --dbfile heketi.db --jsonfile /tmp/q.json
+        oc cp glusterfs/heketi-storage-3-jqlwm-debug:/tmp/q.json q.json
+        cat q.json | python -m json.tool > q2.json
+        ...... Fixing .....
+        oc cp q2.json glusterfs/heketi-storage-3-jqlwm-debug:/tmp/q2.json 
+        heketi db import --dbfile heketi2.db --jsonfile /tmp/q2.json
+        cp heketi2.db /var/lib/heketi/heketi.db
+    * If bunch of disks is created, there are still various left-overs. First, the Gluster volumes
+    have to be cleaned. The idea is to compare 'vol_' prefixed volumes in Heketi and Gluster. And
+    remove ones not present in heketi. There is the script in 'ands/scripts'.
+    * There is LVM volumes left from Gluster (or even allocated, but not associated with Gluster for
+    various failurs. so this clean-up is worth making independently). On each node we can easily find
+    volumes created today
+        lvdisplay -o name,time -S 'time since "2018-03-16"'
+    or again we can compare lvm volumes which are used by Gluster bricks and which are not. The later
+    ones should be cleaned up. Again there is the script.
+     
+Performance
+===========
+ - To find if OpenShift restricts the usage of system resources, we can 'rsh' to container and check
+ cgroup limits in sysfs
+    /sys/fs/cgroup/cpuset/cpuset.cpus
+    /sys/fs/cgroup/memory/memory.limit_in_bytes
+
+
+Various
+=======
+ - IPMI may cause problems as well. Particularly, the mounted CDrom may start complaining. Easiest is
+ just to remove it from the running system with
+     echo 1 > /sys/block/sdd/device/delete