Various fixes before moving to hardware installation

1 files changed, 210 insertions, 0 deletions

diff --git a/docs/troubleshooting.txt b/docs/troubleshooting.txt
new file mode 100644
index 0000000..b4ac8e7
--- /dev/null
+++ b/docs/troubleshooting.txt
@@ -0,0 +1,210 @@
+The services has to be running
+------------------------------
+  Etcd:
+    - etcd 
+
+  Node:
+    - origin-node
+ 
+  Master nodes:
+    - origin-master-api
+    - origin-master-controllers
+    - origin-master is not running
+
+  Required Services:
+    - lvm2-lvmetad.socket 
+    - lvm2-lvmetad.service
+    - docker
+    - NetworkManager
+    - firewalld
+    - dnsmasq
+    - openvswitch
+ 
+  Extra Services:
+    - ssh
+    - ntp
+    - openvpn
+    - ganesha (on master nodes, optional)
+
+Pods has to be running
+----------------------
+  Kubernetes System
+    - kube-service-catalog/apiserver
+    - kube-service-catalog/controller-manager
+  
+  OpenShift Main Services
+    - default/docker-registry
+    - default/registry-console
+    - default/router (3 replicas)
+    - openshift-template-service-broker/api-server (daemonset, on all nodes)
+
+  OpenShift Secondary Services
+    - openshift-ansible-service-broker/asb
+    - openshift-ansible-service-broker/asb-etcd
+
+  GlusterFS
+     - glusterfs-storage (daemonset, on all storage nodes)
+     - glusterblock-storage-provisioner-dc
+     - heketi-storage
+
+  Metrics (openshift-infra):
+    - hawkular-cassandra
+    - hawkular-metrics
+    - heapster
+    
+
+Debugging
+=========
+ - Ensure system consistency as explained in 'consistency.txt' (incomplete)
+ - Check current pod logs and possibly logs for last failed instance
+        oc logs <pod name> --tail=100 [-p]                  - dc/name or ds/name as well
+ - Verify initialization steps (check if all volumes are mounted)
+        oc describe <pod name>
+ - It worth looking the pod environment
+        oc env po <pod name> --list
+ - It worth connecting running container with 'rsh' session and see running processes,
+ internal logs, etc. The 'debug' session will start a new instance of the pod.
+ - If try looking if corresponding pv/pvc are bound. Check logs for pv.
+    * Even if 'pvc' is bound. The 'pv' may have problems with its backend.
+    * Check logs here: /var/lib/origin/plugins/kubernetes.io/glusterfs/
+ - Another frequent problems is failing 'postStart' hook. Or 'livenessProbe'. As it
+ immediately crashes it is not possible to connect. Remedies are:
+    * Set larger initial delay to check the probe.
+    * Try to remove hook and execute it using 'rsh'/'debug'
+ - Determine node running the pod and check the host logs in '/var/log/messages'
+    * Particularly logs of 'origin-master-controllers' are of interest
+ - Check which docker images are actually downloaded on the node
+        docker images
+
+network
+=======
+ - There is a NetworkManager script which should adjust /etc/resolv.conf to use local dnsmasq server.
+ This is based on  '/etc/NetworkManager/dispatcher.d/99-origin-dns.sh' which does not play well 
+ if OpenShift is running on non-default network interface. I provided a patched version, but it
+ worth verifying 
+    * that nameserver is pointing to the host itself (but not localhost, this is important
+    to allow running pods to use it)
+    * that correct upstream nameservers are listed in '/etc/dnsmasq.d/origin-upstream-dns.conf'
+    * In some cases, it was necessary to restart dnsmasq (but it could be also for different reasons)
+ If script misbehaves, it is possible to call it manually like that
+    DEVICE_IFACE="eth1" ./99-origin-dns.sh eth1 up
+
+
+etcd (and general operability)
+====
+ - Few of this sevices may seem running accroding to 'systemctl', but actually misbehave. Then, it 
+ may be needed to restart them manually. I have noticed it with 
+    * lvm2-lvmetad.socket       (pvscan will complain on problems)
+    * node-origin
+    * etcd               but BEWARE of too entusiastic restarting:
+ - However, restarting etcd many times is BAD as it may trigger a severe problem with 
+ 'kube-service-catalog/apiserver'. The bug description is here
+        https://github.com/kubernetes/kubernetes/issues/47131
+ - Due to problem mentioned above, all 'oc' queries are very slow. There is not proper
+ solution suggested. But killing the 'kube-service-catalog/apiserver' helps for a while.
+ The pod is restarted and response times are back in order.
+    * Another way to see this problem is quering 'healthz' service which would tell that
+    there is too many clients and, please, retry later.
+        curl -k https://apiserver.kube-service-catalog.svc/healthz
+
+ - On node crash, the etcd database may get corrupted. 
+    * There is no easy fix. Backup/restore is not working.
+    * Easiest option is to remove the failed etcd from the cluster.
+        etcdctl3 --endpoints="192.168.213.1:2379" member list
+        etcdctl3 --endpoints="192.168.213.1:2379" member remove <hexid>
+    * Add it to [new_etcd] section in inventory and run openshift-etcd to scale-up etcd cluster.
+ 
+ - There is a helth check provided by the cluster
+    curl -k https://apiserver.kube-service-catalog.svc/healthz
+ it may complain about etcd problems. It seems triggered by OpenShift upgrade. The real cause and
+ remedy is unclear, but the installation is mostly working. Discussion is in docs/upgrade.txt
+ 
+ - There is also a different etcd which is integral part of the ansible service broker: 
+ 'openshift-ansible-service-broker/asb-etcd'. If investigated with 'oc logs' it complains 
+ on:
+        2018-03-07 20:54:48.791735 I | embed: rejected connection from "127.0.0.1:43066" (error "tls: failed to verify client's certificate: x509: certificate signed by unknown authority", ServerName "")
+        WARNING: 2018/03/07 20:54:48 Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
+ Nevertheless, it seems working without much trouble. The error message seems caused by
+ certificate verification code which introduced in etcd 3.2. There are multiple bug repports on
+ the issue.
+ 
+pods (failed pods, rogue namespaces, etc...)
+====
+ - After crashes / upgrades some pods may end up in 'Error' state. This is quite often happen to
+    * kube-service-catalog/controller-manager
+    * openshift-template-service-broker/api-server
+ Normally, they should be deleted. Then, OpenShift will auto-restart pods and they likely will run without problems.
+    for name in  $(oc get pods -n openshift-template-service-broker | grep Error | awk '{ print $1 }' ); do oc -n openshift-template-service-broker delete po $name; done
+    for name in  $(oc get pods -n kube-service-catalog | grep Error | awk '{ print $1 }' ); do oc -n kube-service-catalog delete po $name; done 
+ 
+ - Other pods will fail with 'ImagePullBackOff' after cluster crash. The problem is that ImageStreams populated by 'builds' will 
+ not be recreated automatically. By default OpenShift docker registry is stored on ephemeral disks and is lost on crash. The build should be 
+ re-executed manually.
+        oc -n adei start-build adei
+
+ - Furthermore, after long outtages the CronJobs will stop functioning. The reason can be found by analyzing '/var/log/messages' or specially
+        systemctl status origin-master-controllers
+  it will contain something like:
+        'Cannot determine if <namespace>/<cronjob> needs to be started: Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.'
+  * The reason is that after 100 missed (or failed) launch periods it will stop trying to avoid excive load. The remedy is set 'startingDeadlineSeconds'
+  which tells the system that if cronJob has failed to start in the allocated interval we stop trying until the next start period. Then, 100 is only 
+  counted the specified period. I.e. we should set period bellow the 'launch period / 100'.
+        https://github.com/kubernetes/kubernetes/issues/45825
+  * The running CronJobs can be easily patched with
+        oc -n adei patch cronjob/adei-autogen-update --patch '{ "spec": {"startingDeadlineSeconds": 120 }}'
+  
+ - Sometimes there is rogue namespaces in 'deleting' state. This is also hundreds of reasons, but mainly
+    * Crash of both masters during population / destruction of OpenShift resources
+    * Running of 'oc adm diagnostics'
+  It is unclear how to remove them manually, but it seems if we run
+    * OpenShift upgrade, the namespaces are gone (but there could be a bunch of new problems).
+    * ... i don't know if install, etc. May cause the trouble...
+
+ - There is also rogue pods (mainly due to some problems with unmounting lost storage), etc. If 'oc delete' does not
+ work for a long time. It worth
+    * Determining the host running failed pod with 'oc get pods -o wide'
+    * Going to the pod and killing processes and stopping the container using docker command
+    * Looking in the '/var/lib/origin/openshift.local.volumes/pods' for the remnants of the container
+        - This can be done with 'find . -name heketi*' or something like...
+        - There could be problematic mounts which can be freed with lazy umount
+        - The folders for removed pods may (and should) be removed.
+
+ - Looking into the '/var/log/messages', it is sometimes possible to spot various erros like
+    * Orphaned pod "212074ca-1d15-11e8-9de3-525400225b53" found, but volume paths are still present on disk.
+        The volumes can be removed in '/var/lib/origin/openshift.local.volumes/pods' on the corresponding node
+    * PodSandbox "aa28e9c7605cae088838bb4c9b92172083680880cd4c085d93cbc33b5b9e8910" from runtime service failed: ...
+        - We can find and remove the corresponding container (the short id is just first letters of the long id)
+                docker ps -a | grep aa28e9c76
+                docker rm <id>
+        - We further can just destroy all containers which are not running (it will actually try to remove all,
+        but just error message will be printed for running ones)
+                docker ps -aq --no-trunc | xargs docker rm
+
+
+Storage
+=======
+ - Running a lot of pods may exhaust available storage. It worth checking if 
+    * There is enough Docker storage for containers (lvm)
+    * There is enough Heketi storage for dynamic volumes (lvm)
+    * The root file system on nodes still has space for logs, etc. 
+  Particularly there is a big problem for ansible-ran virtual machines. The system disk is stored
+  under '/root/VirtualBox VMs' and is not cleaned/destroyed unlike second hard drive on 'vagrant
+  destroy'. So, it should be cleaned manually.
+  
+ - Problems with pvc's can be evaluated by running 
+        oc  -n openshift-ansible-service-broker describe pvc etcd
+   Furthermore it worth looking in the folder with volume logs. For each 'pv' it stores subdirectories
+   with pods executed on this host which are mount this pod and holds the log for this pods.
+        /var/lib/origin/plugins/kubernetes.io/glusterfs/
+
+ - Heketi is problematic.
+    * Worth checking if topology is fine and running.
+        heketi-cli -s http://heketi-storage-glusterfs.openshift.suren.me --user admin --secret "$(oc get secret heketi-storage-admin-secret -n glusterfs -o jsonpath='{.data.key}' | base64 -d)"
+ - Furthermore, the heketi gluster volumes may be started, but with multiple bricks offline. This can
+ be checked with 
+        gluster volume status <vol> detail
+    * If not all bricks online, likely it is just enought to restart the volume
+        gluster volume stop <vol>
+        gluster volume start <vol>
+    * This may break services depending on provisioned 'pv' like 'openshift-ansible-service-broker/asb-etcd'
+