Error while tearing down pod, "device or resource busy" on service account secret #28750
Description
I'm seeing errors in the kubelet now on 1.3.0+ that indicate a timeout on unmount of secrets into the pod, which looks like it causes some delay in shutdown (although not positive, since this was a long run).
I0711 00:13:30.469559 412 reconciler.go:138] UnmountVolume operation started for volume "kubernetes.io/secret/default-token-2uhpl" (spec.Name: "default-to
ken-2uhpl") from pod "32c22a0c-46fc-11e6-af1a-0e6a9f5112c1" (UID: "32c22a0c-46fc-11e6-af1a-0e6a9f5112c1").
I0711 00:13:30.469837 412 secret.go:249] Tearing down volume default-token-2uhpl for pod 32c22a0c-46fc-11e6-af1a-0e6a9f5112c1 at /tmp/openshift/test-end-t
o-end-docker/volumes/pods/32c22a0c-46fc-11e6-af1a-0e6a9f5112c1/volumes/kubernetes.io~secret/default-token-2uhpl
E0711 00:13:30.506681 412 goroutinemap.go:155] Operation for "kubernetes.io/secret/default-token-2uhpl" failed. No retries permitted until 2016-07-11 00:1
4:02.506671169 +0000 UTC (durationBeforeRetry 32s). error: UnmountVolume.TearDown failed for volume "kubernetes.io/secret/default-token-2uhpl" (volume.spec.Na
me: "default-token-2uhpl") pod "32c22a0c-46fc-11e6-af1a-0e6a9f5112c1" (UID: "32c22a0c-46fc-11e6-af1a-0e6a9f5112c1") with: rename /tmp/openshift/test-end-to-en
d-docker/volumes/pods/32c22a0c-46fc-11e6-af1a-0e6a9f5112c1/volumes/kubernetes.io~secret/default-token-2uhpl /tmp/openshift/test-end-to-end-docker/volumes/pods
/32c22a0c-46fc-11e6-af1a-0e6a9f5112c1/volumes/kubernetes.io~secret/wrapped_default-token-2uhpl.deleting~719439388: device or resource busy
E0711 00:13:46.461321 412 kubelet.go:1946] Unable to mount volumes for pod "cli-with-token_test(0ba7ffca-46fc-11e6-af1a-0e6a9f5112c1)": timeout expired wa
iting for volumes to attach/mount for pod "cli-with-token"/"test". list of unattached/unmounted volumes=[default-token-2uhpl]; skipping pod
E0711 00:13:46.461353 412 pod_workers.go:183] Error syncing pod 0ba7ffca-46fc-11e6-af1a-0e6a9f5112c1, skipping: timeout expired waiting for volumes to att
ach/mount for pod "cli-with-token"/"test". list of unattached/unmounted volumes=[default-token-2uhpl]
I0711 00:13:46.461412 412 server.go:655] Event(api.ObjectReference{Kind:"Pod", Namespace:"test", Name:"cli-with-token", UID:"0ba7ffca-46fc-11e6-af1a-0e6a9
f5112c1", APIVersion:"v1", ResourceVersion:"771", FieldPath:""}): type: 'Warning' reason: 'FailedSync' Error syncing pod, skipping: timeout expired waiting fo
r volumes to attach/mount for pod "cli-with-token"/"test". list of unattached/unmounted volumes=[default-token-2uhpl]
I0711 00:13:46.461483 412 server.go:655] Event(api.ObjectReference{Kind:"Pod", Namespace:"test", Name:"cli-with-token", UID:"0ba7ffca-46fc-11e6-af1a-0e6a9
f5112c1", APIVersion:"v1", ResourceVersion:"771", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Unable to mount volumes for pod "cli-with-token_test(0
ba7ffca-46fc-11e6-af1a-0e6a9f5112c1)": timeout expired waiting for volumes to attach/mount for pod "cli-with-token"/"test". list of unattached/unmounted volum
es=[default-token-2uhpl]
I0711 00:14:02.570569 412 reconciler.go:138] UnmountVolume operation started for volume "kubernetes.io/secret/default-token-2uhpl" (spec.Name: "default-to
ken-2uhpl") from pod "32c22a0c-46fc-11e6-af1a-0e6a9f5112c1" (UID: "32c22a0c-46fc-11e6-af1a-0e6a9f5112c1").
I0711 00:14:02.570848 412 secret.go:249] Tearing down volume default-token-2uhpl for pod 32c22a0c-46fc-11e6-af1a-0e6a9f5112c1 at /tmp/openshift/test-end-t
o-end-docker/volumes/pods/32c22a0c-46fc-11e6-af1a-0e6a9f5112c1/volumes/kubernetes.io~secret/default-token-2uhpl
E0711 00:14:02.590507 412 goroutinemap.go:155] Operation for "kubernetes.io/secret/default-token-2uhpl" failed. No retries permitted until 2016-07-11 00:1
5:06.590493939 +0000 UTC (durationBeforeRetry 1m4s). error: UnmountVolume.TearDown failed for volume "kubernetes.io/secret/default-token-2uhpl" (volume.spec.N
ame: "default-token-2uhpl") pod "32c22a0c-46fc-11e6-af1a-0e6a9f5112c1" (UID: "32c22a0c-46fc-11e6-af1a-0e6a9f5112c1") with: rename /tmp/openshift/test-end-to-e
nd-docker/volumes/pods/32c22a0c-46fc-11e6-af1a-0e6a9f5112c1/volumes/kubernetes.io~secret/default-token-2uhpl /tmp/openshift/test-end-to-end-docker/volumes/pod
s/32c22a0c-46fc-11e6-af1a-0e6a9f5112c1/volumes/kubernetes.io~secret/wrapped_default-token-2uhpl.deleting~619277515: device or resource busy
Occurred 4 times for roughly 30 pods. This is a rhel72 system but I don't have the kernel version handy. @kubernetes/rh-storage what sorts of frequency should we expect a device busy on unmount of a tmpfs?