1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
|
RBAC DENY: user # what it was?
Pod ended with: too old resource version
containerd: unable to save
an empty namespace may not be set when a resource name
timeout expired waiting for volumes to attach
Fast watcher, slow processing
# Temporary (current evaluation)
# ---------
apiserver.*http: Handler timeout # apiserver was not restarted. Why?
etcdserver: request timed out
Error syncing job: Timeout: request did not complete within allowed duration
Timeout: request did not complete within allowed duration
the server cannot complete the requested operation at this time, try again later
unable to create pods: the server cannot complete the requested operation at this time
Server.processUnaryRPC failed to write status # Whats what???? Why it was triggered?
route ip\+net: no such network interface # And what?
dockerd-current:.*Error getting v2 registry # OpenShift docker registry died by that moment
forbidden: no providers available to validate pod request # This when I tried to old supplementalGroup with coma. Unclear why this was wrong, but ok.
unable to validate against any security context constrain # This was supplementalGroup missing in namespace definition
ImagePullBackOff: "Back-off pulling image # This was due to registry diying, making it inaccessible
ImagePullBackOff: Back-off pulling image
PullImage.*getsockopt: connection refused
Error syncing pod.*getsockopt: connection refused
secret.*not registered # Unclear but was related to development of davmail
Couldn't get secret.*not registered
Chown failed on
Chmod failed on
# Too many parallel mounts cause malfunction (transient scope) https://github.com/kubernetes/kubernetes/issues/79194
# -----------------------
Mount failed: exit status 1
mount failed: exit status 1
Failed to start transient scope unit
Mounting command: systemd-run # potentially we can use different mounter?
Mounting arguments:
origin-node: Output: umount.*target is busy # this is probably a very different problem
origin-node: Unmounting arguments:
Error: UnmountVolume.TearDown failed for volume
# docker cgroup-driver related stuff (needs inverstigation) https://github.com/kubernetes/kubernetes/issues/76531
# --------------------
Error while processing event
/sys/fs/cgroup/memory
RecentStats: unable to find data for container
Failed to get RecentStats
Failed to read "/sys/fs/cgroup
Failed to update stats for container "/user.slice
Failed to create summary reader for
Failed to detect containers: readdirent: no such file or directory
unable to find data for container /system
origin-node: container /kubepods.slice
# Known errors which we tolerate
# ------------------------------
has become an orphan, killing it # left-over resources, etc.
is already stopped
but volume paths are still present on disk
Same superblock, different security settings
required revision has been compacted # etcd
Could not get instant cpu stats: different number of cpus
# performance (may be worth looking into)
# -----------
du and find on following dirs took
# Set of problems with sandboxes, cleaning resources (CronJobs). It was there before and probably not critical
# ------------------------------
Failed to stop sandbox
killPodWithSyncResult failed
Cannot find network namespace for the terminated container
rpc error: code = 4 desc = context deadline exceeded
rpc error: code = 2 desc = no such process
cannot delete builds.build.openshift.io
Unmount skipped because path does not exist
stderr: du: cannot access
CNI request failed with status 400 # https://github.com/kubernetes/kubernetes/issues/72044 (not a problem)
network: failed to Statfs
Error while adding to cni lo network
ns/net: No such file or directory # this is still the same
ns/net": no such file or directory
garbagecollector.go.*not found
# containers stopped/dying simmultaneously
# -------------------
not found in pod's containers # Unclear, containers died in the pod (before cgroups killed one container, but many more dissapperead for some reason shortly)
failed to exit within.*trying direct SIGKILL # Unclear, for some reason many containers (pods) were stopped simulatenously....
failed to exit within.*using the force
No ref for container # Related. Then, many of them seems exited before or what happened?
Can't make a ref to pod
Error: No such container
Error deleting network when building cni runtime
rpc error: code = 2 desc = container not running # Related. The same
rpc error: code = 2 desc = Error: No such container
rpc error: code = 2 desc = containerd: process not found for container
returned error: No such container
containerd: container not found
json returned error: No such container
devmapper: Unknown device # also emptyDir volumes seems missing
Both sandbox container and checkpoint for id.*could not be found
NetworkPlugin cni failed to teardown pod
Failed to retrieve checkpoint for sandbox
cannot exec a container that has run and stopped
cannot join network of a non running container
Container.*is not running
# Filtering some known and repeated stuff
# --------------------
adei # adei cron jobs mounting/unmounting adei-* volumes
# Normal errors, due to current development efforts
# -------------
restarting failed container
error processing PVC
Could not construct volume information: no volume plugin matched
Readiness probe for
Liveness probe for
Failing rollout for
is dead, but RestartPolicy says that we should restart it
the object has been modified; please apply your changes to the latest version and try again
container start failed: ErrImagePull
starting container process caused
returned error: No such image
Error setting up exec command in container
endpoints "gfs" not found # I forgot to create it while creating new ns
Unable to authenticate the request due to an error
volume-subpaths: no such file or directory
# Normal errors, like communication failures, etc.
# -------------
Error proxying data from client to backend
Error proxying data from backend to client
# Suppress multi-line output what we can't interpret after filtering
# --------------------------
In some cases useful info about processes that use # suppress multi-line output
the device is found by
with error: exit status
origin-node: - exit status
origin-node: '
the following error information was pulled from the glusterfs log to help diagnose this issue
# Non errors from openshift
# -------------------------
origin-master-controllers: I0923 # informational
origin-master-controllers: I0925
origin-master-controllers: I0926 # DS: May worth investigating, scheduling failures
origin-node: I0923
origin-node: I0924
origin-node: I0925
origin-node: I0926
tomic-openshift-master-api: I0926
atomic-openshift-master-api:.*Trace
origin-master-controllers: Trace
Starting watch for
updating pod condition for
Using node IP
Delete endpoint
is released and reclaim policy
found previous inflight deployment
already in flight, abandoning
entered phase "Released"
Checking for prebound volumes with node affinity
Discovered runtime cgroups name
is not a mountpoint, deleting
List /apis
LoadBalancerRR: Setting endpoints for
dockerd-current:.*cleaned up
dockerd-current:.*Attempting next endpoint for pull after error # This is a problem, but this is secondary messages...
dockerd-current:.*Failed to cancel upload: unauthorized: authentication required
# System informational Messages
# -----------------------------
kernel: XFS
systemd: Starting
systemd: Created
systemd: Stopping
systemd: Removed
systemd: Started
systemd: Stopped
systemd-logind: New session
systemd-logind: Removed session
etcd: store.index
etcd: segmented wal file
etcd: finished scheduled compaction
etcd: compacted rev
etcd: purged
etcd: start
etcd: saved
etcd: compacted
etcd: avoid queries with large range
device-mapper: ioctl: remove_all left.*device
dnsmasq.*setting upstream servers from DBus
dnsmasq.*using nameserver
kernel: nf_conntrack: falling back to vmalloc
IPv6: ADDRCONF
NetworkManager.*veth
ntpd.*veth
kernel: device veth # infamouse veth devices
veth.*link connected
Listen normally on
Creating netns
rotating log files
|
