Moby: Kill `docker exec` command will not terminate the spawned process

20

Whenever a process is launched via docker exec, it seems that killing docker exec will not terminate the process. For example:

> docker run -d --name test-exec busybox top
> docker exec -it test-exec sh
/ # # we have an exec shell now. assume pid of docker exec is 1234
> kill 1234
# docker exec process is terminated atm, but `nsenter-exec` process is still running with sh as its child

I would expect that killing docker exec -it process will also kill the spawned process, or there should be a way to stop the spawn process similar to how docker stop works.

My version of docker:

❯ docker version
Client version: 1.3.1-dev
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): c049949
OS/Arch (client): linux/amd64
Server version: 1.3.1-dev
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): c049949

❯ docker info
Containers: 1
Images: 681
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 693
Execution Driver: native-0.2
Kernel Version: 3.13.0-33-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 2
Total Memory: 1.955 GiB
Debug mode (server): true
Debug mode (client): false
Fds: 17
Goroutines: 16
EventsListeners: 0
Init Path: /home/action/bin/docker
Username: dqminh
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support
dqminh picture dqminh  ·  11 Nov 2014

Most helpful comment

17

Yo, 2017! Any love?

vincentwoo picture vincentwoo  ·  19 Jan 2017

All comments

0

mmm, I've just followed your example, and have a mildly different result?

[sven@t440s docker]$ docker run -d -name test-exec busybox top
Warning: '-name' is deprecated, it will be replaced by '--name' soon. See usage.
0daecd23a78f05990466c9f7d1094c737771a0cc15142588bb57ebd6b7f99c5f
[sven@t440s docker]$ docker exec -it test-exec sh
/ # ps
PID   USER     COMMAND
    1 root     top
    7 root     sh
   13 root     ps
/ # kill 7
/ # ps
PID   USER     COMMAND
    1 root     top
    7 root     sh
   14 root     ps
/ # kill -9 7
[sven@t440s docker]$ docker exec -it test-exec ps aux
PID   USER     COMMAND
    1 root     top
   15 root     ps aux
[sven@t440s docker]$ docker version
Client version: 1.3.1
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): 4e9bbfa
OS/Arch (client): linux/amd64
Server version: 1.3.1
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): 4e9bbfa
[sven@t440s docker]$ 

and the non-containerised version works the same:

[sven@t440s docker]$ sh
sh-4.2$ ps
  PID TTY          TIME CMD
11920 pts/3    00:00:00 bash
12090 pts/3    00:00:00 sh
12091 pts/3    00:00:00 ps
sh-4.2$ kill 12090
sh-4.2$ 
sh-4.2$ ps
  PID TTY          TIME CMD
11920 pts/3    00:00:00 bash
12090 pts/3    00:00:00 sh
12092 pts/3    00:00:00 ps
sh-4.2$ kill -HUP 12090
Hangup

so for me, its works as intended (

SvenDowideit picture SvenDowideit  ·  12 Nov 2014
1

@SvenDowideit ah, my use case is that the docker exec process is killed from outside of the container, not the process started by docker exec inside the container. For example, after running docker exec, the tree will look like ( pseudo pids here to illustrate the point ) :

1024 --- docker run -d -it --name test-exec busybox top
1025 --- docker exec -it --name test-exec sh
10 --- docker -d
  \ 10000 --- top
  \ 10001 --- nsenter-exec --nspid 23119 --console /dev/pts/19 -- sh
          \---- sh

Now if i do kill 1025, which kill the docker exec process, the process tree becomes:

1024 --- docker run -d -it --name test-exec busybox top
10 --- docker -d
  \ 10000 --- top
  \ 10001 --- nsenter-exec --nspid 23119 --console /dev/pts/19 -- sh
          \---- sh

I would expect nsenter-exec to be killed as well and/or maybe docker should expose a way to programatically stopped the exec-process from outside.

dqminh picture dqminh  ·  12 Nov 2014
0

ah, good to know more info :)

SvenDowideit picture SvenDowideit  ·  12 Nov 2014
0

ah, good to know more info :)

Yes, i should have included the process tree from the start as it's much easier to know what's going on. Should not submit an issue at 5am i guess :(

dqminh picture dqminh  ·  12 Nov 2014
2

mmm, ok, so I agree - I too would expect that docker exec would trap the kill signal and pass it on to the Docker daemon, which should then pass the signal on to the exec'd child

I don't see much in the way of support for this in the API, http://docs.docker.com/reference/api/docker_remote_api_v1.15/#exec-create, so bug?

SvenDowideit picture SvenDowideit  ·  12 Nov 2014
0

@vieux @proppy what do you think? (I'm going off the MAINTAINERs file :))

SvenDowideit picture SvenDowideit  ·  12 Nov 2014
0

Yes and I don't see where the pid for the child is (if at all) stored in the ExecConfig.

proppy picture proppy  ·  12 Nov 2014
0

/cc @vishh

proppy picture proppy  ·  12 Nov 2014
0

Terminating a running 'exec' session via an API has not been implemented
yet.
@proppy: Yes, the child pid is not stored as part of ExecConfig.

On Tue, Nov 11, 2014 at 10:41 PM, Johan Euphrosine <[email protected]

wrote:

/cc @vishh https://github.com/vishh


Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/9098#issuecomment-62676939.

vishh picture vishh  ·  12 Nov 2014
4

@vishh do you think adding support for POST /exec/:name/stop (and maybe POST /exec/:name/kill) make senses here ( similar to POST /containers/:name/stop and POST /containers/:name/kill ) ? That would actually solve majority of my usecase as I mainly consume the remote API ( which makes the exec process's unique id available with POST /exec/:name/create )

It's probably much harder to do it from the docker cli though as we don't really expose the exec's id anywhere.

dqminh picture dqminh  ·  12 Nov 2014
0

Yes. A stop/kill daemon api makes sense to me. For the CLI case, I need to
see if the daemon can automatically terminate an abandoned interactive
'exec' command.

On Tue, Nov 11, 2014 at 11:14 PM, Daniel, Dao Quang Minh <
[email protected]> wrote:

@vishh https://github.com/vishh do you think adding support for POST
/exec/:name/stop (and maybe POST /exec/:name/kill) make senses here (
similar to POST /containers/:name/stop and POST /containers/:name/kill )
? That would actually solve majority of my usecase as I mainly consume the
remote API ( which makes the exec process's unique id available with POST
/exec/:name/create )

It's probably much harder to do it from the docker cli though as we don't
really expose the exec's id anywhere.


Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/9098#issuecomment-62679132.

vishh picture vishh  ·  12 Nov 2014
0

@vishh I'm not sure how we can implement this auto-terminate. Maybe we can have some list api for exec? And make exec jobs dependent on container, so on container deleting - all abandoned jobs deletes too.

LK4D4 picture LK4D4  ·  12 Nov 2014
0

AFAIK exec jobs should get terminated on container deletion. Is that not
the case?

On Wed, Nov 12, 2014 at 9:23 AM, Alexandr Morozov [email protected]
wrote:

@vishh https://github.com/vishh I'm not sure how we can implement this
auto-terminate. Maybe we can have some list api for exec? And make exec
jobs dependent on container, so on container deleting - all abandoned jobs
deletes too.


Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/9098#issuecomment-62756154.

vishh picture vishh  ·  12 Nov 2014
0

Maybe we can have some list api for exec?

Perhaps add a way to see all processes related to a container? Eg

docker containers ps <containerid>

Which will include the exec process.

thaJeztah picture thaJeztah  ·  12 Nov 2014
1

Good point. We should expose exec jobs belonging to a container.

On Wed, Nov 12, 2014 at 9:59 AM, Sebastiaan van Stijn <
[email protected]> wrote:

Maybe we can have some list api for exec?

Perhaps add a way to see all processes related to a container? Eg

docker containers ps

Which will include the exec process.


Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/9098#issuecomment-62762031.

vishh picture vishh  ·  12 Nov 2014
0

@vishh eh, I meant internal execStore. Yeah, it is a little different, because I wanted to add method for getting exitCode of exec job and be sure that this job will be deleted from execStore. (All I can imagine is pretty ugly)

LK4D4 picture LK4D4  ·  12 Nov 2014
0

+1
this also causes a go routine leak. 3 go routines are leaked whenever this happens.

anandkumarpatel picture anandkumarpatel  ·  12 Nov 2014
0

I proposed additional extensions for the remote API to stop/kill exec command here #9167 . That should fix my particular use case ( programmatically managing exec commands )

The proposal doesn't include CLI changes as i'm not sure what is the appropriate interface for exposing exec sessions yet.

dqminh picture dqminh  ·  14 Nov 2014
0

An alternative to killing the spawned process would be to close stdin, stdout and stderr when docker exec is killed. In most cases, such as when a shell is being exected, the spawned process will quit when stdin is closed.

Currently, it seems that when docker exec is killed, the spawned process still has a stdin with nobody attached to.

I don't know if closing stdin would be a better alternative to killing the spawned process.

thomasthiriez picture thomasthiriez  ·  19 Nov 2014
0

I get this on latest:

/ # kill 3899
sh: can't kill pid 3899: No such process
/ # kill 3900
sh: can't kill pid 3900: No such process
/ # 

super weirrd

jessfraz picture jessfraz  ·  26 Feb 2015
0

Container becomes unresponsive after creating some random number of exec instances :(. Could be related to this .. +1 for ability to destroy them via the remote API.

kelonye picture kelonye  ·  2 Mar 2015
0

Happen to me as well, there is any ETA for this to be fixed?

camilonova picture camilonova  ·  9 Apr 2015
0

+1

sakurai-youhei picture sakurai-youhei  ·  15 May 2015
0

@dqminh can you share How are you guys workaround this issue right now? We need someway for docker to properly kill exec sessions..

thefallentree picture thefallentree  ·  17 Aug 2015
0

What if start any command in wrapper. Example for API usage (haven't checked so correct if I missed smth):
"/bin/bash" "-c" "trap '[ -z \"$(jobs -p)\" ] || kill $(jobs -p); EXIT" + yourcommand

So when you kill bash all child processes should be stopped

garagatyi picture garagatyi  ·  20 Aug 2015
0

@garagatyi That doesn't work, because when you kill your "docker exec" (external to container ,possibly remotely) process, the process in container doesn't get any signal, docker daemon should have closed its stdin pipe.

thefallentree picture thefallentree  ·  21 Aug 2015
0

+1

Or at least what should I be doing instead?

amcsi picture amcsi  ·  5 Nov 2015
5

The following bash snippet can be used as a workaround for this issue. I basically intercept the SIGTERM to docker exec and do a manual cleanup. It is based on this: http://veithen.github.io/2014/11/16/sigterm-propagation.html

function docker_cleanup {
    docker exec $IMAGE bash -c "if [ -f $PIDFILE ]; then kill -TERM -\$(cat $PIDFILE); rm $PIDFILE; fi"
}

function docker_exec {
    IMAGE=$1
    PIDFILE=/tmp/docker-exec-$$
    shift
    trap 'kill $PID; docker_cleanup $IMAGE $PIDFILE' TERM INT
    docker exec $IMAGE bash -c "echo \"\$\$\" > $PIDFILE; exec $*" &
    PID=$!
    wait $PID
    trap - TERM INT
    wait $PID
}

#use it like this:
docker_exec container command arg1 ...
stefanklug picture stefanklug  ·  27 Feb 2016
0

I'd be content to have an api to send signals to the exec'd processes, ie hup, wait a few seconds, kill, the way a shutdown would do it.

ala

POST /exec/(id or name)/kill
Signal: default hup

swynter-ladbrokes picture swynter-ladbrokes  ·  18 May 2016
3

Any updates? We still want this.

dreammaker picture dreammaker  ·  16 Dec 2016
17

Yo, 2017! Any love?

vincentwoo picture vincentwoo  ·  19 Jan 2017
2

+1

servercharlie picture servercharlie  ·  27 Mar 2017
0

/cc @mlaventure

thaJeztah picture thaJeztah  ·  27 Mar 2017
0

I guess the daemon could be updated to kill the associated process if the exec wasn't started in detached mode. But this behavior has been there for quite some time now, and I wonder if some people incorrectly relies on this side effect.

/cc @tonistiigi @crosbymichael @cpuguy83 for their opinions.

mlaventure picture mlaventure  ·  30 Mar 2017
1

docker run has a flag --sig-proxy, which is enabled by default when attached.

Not sure about changing defaults, but it would be nice to have docker exec be able to proxy signals to the process it's attached to.

cpuguy83 picture cpuguy83  ·  30 Mar 2017
1

+1

computersrmyfriends picture computersrmyfriends  ·  29 Jun 2017
0

@cpuguy83 there seems to be a part of the code that actually disables forwarding signals though; see https://github.com/moby/moby/issues/28872#issuecomment-269088870, and --sig-proxy is also ignored then

thaJeztah picture thaJeztah  ·  30 Jun 2017
2

We're hitting this as well, both on docker 1.12.6 and on 17.0.6.0-ce. We have some automated processes that attach to running containers via docker exec -it from an SSH session, interacting with the running processes and gathering output to logs. If the SSH connection is disrupted, the docker exec'd process remains running. We end up accumulating these stale processes over time.

bobziuchkovski picture bobziuchkovski  ·  11 Jul 2017
0

ping @mlaventure @crosbymichael any thoughts on https://github.com/moby/moby/issues/9098#issuecomment-312152980 and https://github.com/moby/moby/issues/28872#issuecomment-269088870 ?

thaJeztah picture thaJeztah  ·  12 Jul 2017
10

This issue was the root cause of a number of inconveniences I've experienced over the past several months, and only today did I finally land on this bug.

The workaround isn't too fun or easy either. I'm using Python to call docker exec in a subprocess, and what I settled on amounts to grepping docker exec ps ... to get the PID of the command I just ran, followed by docker exec kill ... to kill the process running inside the container. There were also some tricky aspects to what I had to do, but I won't describe them here.

I think this issue should be prioritized more highly because it's the kind of behavior one takes for granted, and in certain use cases (like in my case) it's easy not to notice this bug was happening all along.

cjerdonek picture cjerdonek  ·  2 Aug 2017
7

I had that issue with both docker run and docker exec not dispatching signals to the daemon/container. The root cause is when using --tty signal proxying is entirely disabled with no way to enable it (even with --sig-proxy). It affects at least docker run and docker exec that share the same code path.

Previously --sig-proxy was an option to force proxying signals when not using a tty, and hence when using --tty the proxy got forwarded. Passing both --sig-proxy and -tty was leading to an error. That looked like:

Options | Signal proxy?
--- | ---
_none_ | No
--tty | Yes
--sig-proxy | Yes
--tty --sig-proxy | error: TTY mode (-t) already imply signal proxying (-sig-proxy)

October 2013 patch e0b59ab52b87b8fc15dd5534c3231fdd74843f9f was made to "_Enable sig-proxy by default in run and attach _". It changed --sig-proxy to default to true and made --tty to always disable signal proxy.

Options | Signal proxy?
--- | ---
_none_ | Yes (was No)
--tty | No (was Yes)
--sig-proxy | Yes
--tty --sig-proxy | No (was an error)

So what happened with e0b59ab52b87b8fc15dd5534c3231fdd74843f9f is that signal proxying is now enabled by default for non-tty. BUT the patch has a fault: setting --tty always disable signal proxying.

I am pretty sure that signal-proxying should be enabled by default whether it is a tty or non-tty mode, it is still possible to disable it with --sig-proxy=false. A patch would thus have to implement the following changes:

Options | Signal proxy (current) | Expected
--- | --- | ---
_none_ | Yes | Yes
--tty | No | Yes
--sig-proxy | Yes | Yes
--sig-proxy=false | No | No
--tty --sig-proxy | No | Yes
--tty --sig-proxy=false | No | No

TLDR: --tty should not arbitrarily force sigProxy = false caused by e0b59ab52b87b8fc15dd5534c3231fdd74843f9f

Reference: https://phabricator.wikimedia.org/T176747#3749436

hashar picture hashar  ·  28 Nov 2017
0

It there any update? I have the same issue, I run container (as a sandbox for code), and then Jenkins run inside container some script many times, and in case if Jenkins build (which ran docker exec blah bash -c '<dangerous code>' was aborted bash process still run in a container.

pbelskiy picture pbelskiy  ·  30 Jan 2018
0

@th3mis if you can use docker run --init instead it shouldn't happen.
The init process in the container will ensure that signals are forwarded to the executed process in docker.

fho picture fho  ·  30 Jan 2018
0

@fho
No, it also doesn't work.

  1. In first terminal: docker run --init --rm -it --name ubuntu ubuntu:16.04
  2. In the second terminal: docker exec ubuntu bash -c 'sleep 77'

And when I press CTRL+C, I still see sleep 77 in container processes.

pbelskiy picture pbelskiy  ·  30 Jan 2018
0

No, it's not resolved. The docker CLI does not forward signals to the exec process. This is one of the areas where exec's usage is at odds with the original intent, which is debugging.
The functionality just needs to be rounded out

cpuguy83 picture cpuguy83  ·  30 Jan 2018
1

@th3mis ok, I meant running your command via
docker run --init --rm -it --name ubuntu ubuntu:16.04 bash -c 'sleep 77' instead.

If you still want to use docker-exec you have to run your command via the init-process manually:
docker exec ubuntu bash -c '/dev/init -s -- sleep 77'

Otherwise the signals are still not forward to your bash process

fho picture fho  ·  30 Jan 2018
3

@th3mis no one is saying this is an issue with docker run. It's an issue with docker exec and your example still doesn't work. Signals are not forwarded when you kill the docker exec process no matter what combination of commands you pass to it.

$ docker run --init -di ubuntu:16.04
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
21d2b053d10f        ubuntu:16.04        "/bin/bash"         3 minutes ago       Up 3 minutes                            reverent_perlman
$ ps -eaf --forest
root      1846     1  0 Mar06 ?        00:30:23 /usr/bin/dockerd -H fd://
root      8823  1846  0 Mar06 ?        00:16:40  \_ docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir 
root       563  8823  0 10:38 ?        00:00:00      \_ docker-containerd-shim 21d2b053d10f9d26e294ff055494b4136624fae1faa0952ba63e6d36cf21df74 /var/run/docker/libcontainerd/21d2b053d10
root       583   563  0 10:38 ?        00:00:00          \_ /dev/init -- /bin/bash
root       623   583  0 10:38 ?        00:00:00              \_ /bin/bash

$ docker exec reverent_perlman bash -c '/dev/init -s -- sleep 7777'
<open a new terminal>
$ ps -eaf --forest
root      1846     1  0 Mar06 ?        00:30:24 /usr/bin/dockerd -H fd://
root      8823  1846  0 Mar06 ?        00:16:41  \_ docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir 
root       563  8823  0 10:38 ?        00:00:00      \_ docker-containerd-shim 21d2b053d10f9d26e294ff055494b4136624fae1faa0952ba63e6d36cf21df74 /var/run/docker/libcontainerd/21d2b053d10
root       583   563  0 10:38 ?        00:00:00      |   \_ /dev/init -- /bin/bash
root       623   583  0 10:38 ?        00:00:00      |       \_ /bin/bash
root      1404  8823  0 10:43 ?        00:00:00      \_ docker-containerd-shim 21d2b053d10f9d26e294ff055494b4136624fae1faa0952ba63e6d36cf21df74 /var/run/docker/libcontainerd/21d2b053d10
root      1454  1404  0 10:43 ?        00:00:00          \_ /dev/init -s -- sleep 7777
root      1460  1454  0 10:43 ?        00:00:00              \_ sleep 7777

$ ps -eaf | grep 'docker exec'
root     1715 32646  0 10:45 pts/18   00:00:00 docker exec reverent_perlman bash -c /dev/init -s -- sleep 7777
$ kill 1715
<docker exec in original terminal has exited>
$ ps -eaf --forest
root      1846     1  0 Mar06 ?        00:30:25 /usr/bin/dockerd -H fd://
root      8823  1846  0 Mar06 ?        00:16:41  \_ docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir 
root       563  8823  0 10:38 ?        00:00:00      \_ docker-containerd-shim 21d2b053d10f9d26e294ff055494b4136624fae1faa0952ba63e6d36cf21df74 /var/run/docker/libcontainerd/21d2b053d10
root       583   563  0 10:38 ?        00:00:00      |   \_ /dev/init -- /bin/bash
root       623   583  0 10:38 ?        00:00:00      |       \_ /bin/bash
root      1944  8823  0 10:47 ?        00:00:00      \_ docker-containerd-shim 21d2b053d10f9d26e294ff055494b4136624fae1faa0952ba63e6d36cf21df74 /var/run/docker/libcontainerd/21d2b053d10
root      1961  1944  0 10:47 ?        00:00:00          \_ /dev/init -s -- sleep 7777
root      1967  1961  0 10:47 ?        00:00:00              \_ sleep 7777
matthewlenz picture matthewlenz  ·  15 Mar 2018
0

any process about this issue?
when "docker exec" process was killed, the docker daemon get no signal from docker cli.
the containerd-shim process and "docker exec cmd" process will not exit.
This happens even if we did not add the --detach flag in the docker exec command.
@thaJeztah @cpuguy83
any plan about this ?

Mark-50 picture Mark-50  ·  13 Nov 2018
0

i think the ”docker exec“ and "docker run" are different issues.

when "docker run" process was killed, the container started by "docker run" should exit.
when "docker exec" process was killed, the container started before should remain running, only the process started by "docker exec" inside the container should exit.

also, this is not only about signal proxying.
what if we run "docker exec" or "docker run -H " in a host and dockerd was started in a remote host?
signal proxy is not sufficient for this situation.

Mark-50 picture Mark-50  ·  14 Nov 2018
4

Is there any movement on this? We've just hit this issue as well - very unexpected behavior.

muellermichel picture muellermichel  ·  7 Nov 2019
0

I am working on https://github.com/kubernetes/kubernetes/pull/87281 where we need the ability to kill docker exec command.

When would the fix get to be reviewed ?

Thanks

tedyu picture tedyu  ·  16 Jan 2020
0

I do this nasty piece of work...
docker container exec -d my_container sleep 999

Then later when I want to kill the long running command i.e. the sleep:
docker container top my_container | grep "sleep 999" | awk '{print $2}' | xargs kill

In my case the container in question doesn't have a "ps" command so it's cleaner to kill from the host.

tonymcneil picture tonymcneil  ·  7 May 2020
4

This issue is still very much existing and causing unexpected and unwanted behavior.

nurmians picture nurmians  ·  9 Jun 2020
0

Spent several hours running around in circles yesterday, thanks to this "feature". (Well with kubectl, but I blame docker for starting this mess).

If it's not going to be fixed, can we at least have a clear warning in the documentation / --help output / man page?

ChanderG picture ChanderG  ·  13 Jun 2020
3

Hello. I've run into this issue too. I offer here a workaround done to the best of my limited understanding of how docker works and its internals (most learned while creating this workaround).

First, let me explain the use case:
I run a docker image which runs a basic piece of software (a driver, long-time running process, using a number of random ports).
Over time, I need to run further commands to interact with this driver, these commands are also long-time running processes. They interact with the driver and in doing so they need to get some resources and free them properly afterwards, which implies, they need to be killed cleanly, i.e. when getting SIGTERM or SIGINT they must do cleanup before exiting. All this software is automatically run by a tty-less environment (supervisord).
Summary: I do docker run -t FLAGS --name DRIVER_CONTAINER MYIMAGE MYDRIVER and later on every X time docker exec -t DRIVER_CONTAINER MYDRIVERCLIENT. I need the exec commands to exit cleanly and not remain as zombies.

Now, the fun part, my workaround: I made a wrapper, using the Docker API and some hacky bits, for docker exec to enable signal forwarding. It is contained in this gist which has the class DockerExecWithSignalForwarding. It contains a command line interface that imitates docker exec, supporting (almost) the same flags and nature of use. I install it as dockerexec to make it look similar to docker exec. It supports remote DOCKER_HOST via SSH.

I hope this may help by: giving an option to people falling into the same issue, and maybe providing an idea on how to do it officially (if there is no better way). And I'd also request feedback on how to improve it (in the comments of the gist probably).

I reproduce it here as it is right now for ease of reading:

#!/usr/bin/env python3
import sys
import argparse
import os
import subprocess
import signal
import time
import threading
import shlex
import docker

"""
This program is meant to substitute running 'docker exec <FLAGS> CONTAINER_NAME COMMAND'
to overcome the limitation of docker exec not forwarding signals to the
executed process.

This was reported here in Nov 2014: https://github.com/moby/moby/issues/9098#issuecomment-312152980)
Furthermore, here: https://github.com/docker/cli/pull/1841 they say

    moby/moby#9098 Kill docker exec command will not terminate the spawned process
        This patch does not fix the docker exec case; it looks like there's no API to kill an exec'd process, 
        so there's no signal-proxy for this yet

Note: -i is not supported, couldn't make it work but got as close as I could (maybe it can't work?).

Author: Sammy Pfeiffer <sam.pfeiffer at hullbot.com>
"""


class DockerExecWithSignalForwarding(object):
    def __init__(self, container_name, command,
                 # Offer the rest of the options from docker exec
                 detach=False,
                 # Note detach-keys is not implemented
                 environment=None,
                 # Not supported
                 interactive=False,
                 privileged=False,
                 tty=False,
                 # Leaving these as None makes the inner docker API deal with them correctly
                 user=None,
                 workdir=None,
                 # By default the timeout of Python's docker exec is 60s, change it to 1 year-ish
                 socket_timeout=60 * 60 * 24 * 365):
        """
        Provided a set of flags (same ones of docker exec), a container name and a command,
        do 'docker exec' but managing signals to be forwarded to the exec-ed process.
        """
        if interactive:
            raise RuntimeError("Interactive mode not supported, use docker exec.")
        # We inherit the rest of the configuration (including DOCKER_HOST) from the environment
        self.client = docker.from_env(timeout=socket_timeout)

        # Sanity check on the command, should be a string which we split with shlex or already a list/tuple
        if isinstance(command, str):
            command = shlex.split(command)
        if not (isinstance(command, list) or isinstance(command, tuple)):
            raise TypeError("Command is of type {} and it must be str/list/tuple. (command: {})".format(
                type(command),
                command))

        # Translate docker exec style arguments into exec_run arguments
        try:
            # Get a reference to the container
            self.container = self.client.containers.get(container_name)
            # Get the Id of the 'docker exec' instance (that is not yet being executed) so we can start it
            exec_create_response = self.client.api.exec_create(self.container.id,
                                                               command,
                                                               stdout=True,
                                                               stderr=True,
                                                               stdin=self.interactive,
                                                               tty=tty,
                                                               privileged=privileged,
                                                               user=user,
                                                               environment=environment,
                                                               workdir=workdir)
            self.exec_id = exec_create_response['Id']

            # The following block of code is to manage the situation of an interactive session
            # We would like to support it but the underlying API doesn't allow for it (writing into the socket
            # simply does not work as far as I could test) it was a lot of work to figure out the bits
            # to get this to this state, so I'm leaving it here
            # if interactive:
            #     # Because we want to support stdin we need to access the lower-level socket
            #     # instead of being able to use exec_start with stream=True
            #     self.exec_socket = self.client.api.exec_start(self.exec_id,
            #                                                   detach=detach,
            #                                                   tty=tty,
            #                                                   stream=False,
            #                                                   socket=True,
            #                                                   demux=True)

            #     # Recreate the function that offers the generator for output usually when using stream=True
            #     def _read_from_socket(socket, stream, tty=True, demux=False):
            #         """
            #         Adapted from docker/client.py in order to enable stdin... tricky.
            #         """
            #         gen = docker.api.client.frames_iter(socket, tty)

            #         if demux:
            #             # The generator will output tuples (stdout, stderr)
            #             gen = (docker.api.client.demux_adaptor(*frame) for frame in gen)
            #         else:
            #             # The generator will output strings
            #             gen = (data for (_, data) in gen)

            #         if stream:
            #             return gen
            #         else:
            #             # Wait for all the frames, concatenate them, and return the result
            #             return docker.api.client.consume_socket_output(gen, demux=demux)

            #     self.exec_output = _read_from_socket(self.exec_socket, True, tty, True)
            # else:
            self.exec_output = self.client.api.exec_start(self.exec_id,
                                                          detach=detach,
                                                          tty=tty,
                                                          stream=True,
                                                          socket=False,
                                                          demux=True)

            self.setup_signal_forwarding()
            self.program_running = True

        # Imitate the behaviour of the original docker exec up to a point
        except docker.errors.NotFound as e:
            print("Error: No such container: {}".format(container_name))
            os._exit(1)

        # Start a thread that monitors if the program died so we can end this when this happens
        self.monitor_thread = threading.Thread(target=self.monitor_exec)
        self.monitor_thread.start()

        self.output_manager_thread = None
        if self.interactive:
            # Deal with stdout and stderr in a thread and let the main thread deal with input
            self.output_manager_thread = threading.Thread(target=self.manage_stdout_and_stderr)
            self.output_manager_thread.start()
            self.manage_stdin()
        else:
            self.manage_stdout_and_stderr()

    def monitor_exec(self):
        """
        We loop (very slowly) to check if the underlaying command died, this is useful for
        commands executed in a remote docker daemon. It 'should' not happen locally, but it may.
        """
        try:
            # Check if the process is dead, the 'Running' key must become false
            exec_inspect_dict = self.client.api.exec_inspect(self.exec_id)
            while exec_inspect_dict.get('Running'):
                # Generous sleep, as this is to catch the program dieing by something else than this wrapper
                time.sleep(10.0)
                exec_inspect_dict = self.client.api.exec_inspect(self.exec_id)

            # If it's dead, we should exit with its exit code
            os._exit(exec_inspect_dict.get('ExitCode'))

        except docker.errors.APIError as e:
            # API error, we can't access anymore, exit
            raise RuntimeError("Docker API error when monitoring exec process ({})".format(e))

    def forward_signal(self, signal_number, frame):
        """
        Forward the signal signal_number to the container,
        we first need to find what's the in-container PID of the process we docker exec-ed
        then we docker exec a kill signal with it.
        """
        # print("Forwarding signal {}".format(signal_number))
        # Using a lock to attempt to deal with Control+C spam
        with self.signal_lock:
            pid_in_container = self.get_container_pid()
            kill_command = ["kill", "-{}".format(signal_number), str(pid_in_container)]
            try:
                exit_code, output = self.container.exec_run(kill_command,
                                                            # Do it always as root
                                                            user='root')
            except docker.errors.NotFound as e:
                raise RuntimeError("Container doesn't exist, can't forward signal {} (Exception: {})".format(
                    signal_number, e))

            if exit_code != 0:
                raise RuntimeError(
                    'When forwarding signal {}, kill command to PID in container {} failed with exit code {}, output was: {}'.format(
                        signal_number, pid_in_container, exit_code, output))

    def get_container_pid(self):
        """
        Return the in-container PID of the exec-ed process.
        """
        try:
             # I wish the stored PID of exec was the container PID (which is what I expected)
             # but it's actually the host PID so in the following lines we deal with it
            pid_in_host = self.client.api.exec_inspect(self.exec_id).get('Pid')
        except docker.errors.NotFound as e:
            raise RuntimeError("Container doesn't exist, can't get exec PID (Exception: {})".format(e))

        # We need to translate the host PID into the container PID, there is no general mapping for it in Docker
        # If we are running in the same host, this is easier, we can get the Docker PID by just doing:
        #       cat /proc/PID/status | grep NSpid | awk '{print $3}'
        # If the docker container is running in a different machine we need to execute that command in that machine
        # which implies using SSH to execute the command

        # Here we can only support DOCKER_HOST=ssh://user@host to use ssh to execute this command
        # as if we are using ssh:// to access the docker daemon it's fair to assume we have SSH keys setup
        # if docker host is tcp:// on another host or a socket file with SSH tunneling there isn't much we can do
        docker_host = os.environ.get('DOCKER_HOST', None)
        # If using SSH execute the command remotely
        if docker_host and 'ssh://' in docker_host:
            ssh_user_at_host = docker_host.replace('ssh://', '')
            get_pid_in_container_cmd = "ssh -q -o StrictHostKeyChecking=no {} ".format(ssh_user_at_host)
            get_pid_in_container_cmd += "cat /proc/{}/status | grep NSpid | awk '{{print $3}}'".format(pid_in_host)
        # Otherwise, execute the command locally
        else:
            get_pid_in_container_cmd = "cat /proc/{}/status | grep NSpid | awk '{{print $3}}'".format(pid_in_host)

        # Execute the command that gets the in-Docker PID
        try:
            pid_in_container = subprocess.check_output(get_pid_in_container_cmd, shell=True)
        except subprocess.CalledProcessError as e:
            raise RuntimeError(
                "CalledProcessError exception while trying to get the in-docker PID of the process ({})".format(e))

        return int(pid_in_container)

    def setup_signal_forwarding(self):
        """
        Forward all signals to the docker exec-ed process.
        If it dies, this process will die too as self.manage_stdout_and_stderr will finish
        and forward the exit code.
        """
        self.signal_lock = threading.Lock()
        # Forward all signals, even though we are most interested just in SIGTERM and SIGINT
        signal.signal(signal.SIGHUP, self.forward_signal)
        signal.signal(signal.SIGINT, self.forward_signal)
        signal.signal(signal.SIGQUIT, self.forward_signal)
        signal.signal(signal.SIGILL, self.forward_signal)
        signal.signal(signal.SIGTRAP, self.forward_signal)
        signal.signal(signal.SIGABRT, self.forward_signal)
        signal.signal(signal.SIGBUS, self.forward_signal)
        signal.signal(signal.SIGFPE, self.forward_signal)
        # Can't be captured, but for clarity leaving it here
        # signal.signal(signal.SIGKILL, self.forward_signal)
        signal.signal(signal.SIGUSR1, self.forward_signal)
        signal.signal(signal.SIGUSR2, self.forward_signal)
        signal.signal(signal.SIGSEGV, self.forward_signal)
        signal.signal(signal.SIGPIPE, self.forward_signal)
        signal.signal(signal.SIGALRM, self.forward_signal)
        signal.signal(signal.SIGTERM, self.forward_signal)

    def manage_stdout_and_stderr(self):
        """
        Print stdout and stderr as the generator provides it.
        When the generator finishes we exit the program forwarding the exit code.
        """
        # Note that if the application prints a lot, this will use some CPU
        # but there is no way around it as we are forced to read from the socket and decode to print
        for stdout, stderr in self.exec_output:
            # Note that if choosing tty=True output is always in stdout
            if stdout:
                print(stdout.decode("utf-8"), file=sys.stdout, end='')
            if stderr:
                print(stderr.decode("utf-8"), file=sys.stderr, end='')

        # When we come out of this loop, the program we exec-ed has terminated
        # so we can exit with its exit code just here
        exec_inspect_dict = self.client.api.exec_inspect(self.exec_id)
        exit_code = exec_inspect_dict.get('ExitCode')
        os._exit(exit_code)

    def manage_stdin(self):
        """
        Forward the input of this program to the docker exec-ed program.
        """
        raise NotImplemented("Managing stdin is not implemented.")
        # print(dir(self.exec_socket))
        # print(self.exec_socket.readable())
        # print(self.exec_socket.writable())
        # print(dir(self.exec_socket._sock))
        # self.exec_socket._writing = True
        # print(self.exec_socket.writable())
        # def write(sock, str):
        #     while len(str) > 0:
        #         written = sock.write(str)
        #         str = str[written:]
        # while True:
        #     # self.exec_socket._sock.sendall(input().encode('utf-8'))
        #     # self.exec_socket.flush()
        #     #print("sent")
        #     # Doesn't work either
        #     write(self.exec_socket, input().encode('utf-8'))
        #     print("--written--")
        #     #os.write(self.exec_socket._sock.fileno(), input().encode('utf-8'))
        #     #print("sent")
        #     #print("Received: {}".format(self.exec_socket._sock.recv(1)))
        #     # try:
        #     #     print(os.read(self.exec_socket._sock.fileno(), 4096))
        #     # except BlockingIOError as b:
        #     #     print("BlockingIOError: {} ".format(b))
        #     #     print(self.client.api.exec_inspect(self.exec_id))

    def __del__(self):
        """
        When the program ends this gets called so we can cleanup resources
        and exit with the exit code from the exec-ed command.
        Note it is unlikely this gets ever called.
        """
        # print("Calling __del__")
        # Wait for the output thread in case there are more prints to show
        if self.output_manager_thread:
            self.output_manager_thread.join()

        # Try to wait for the process to be dead in case it isn't yet
        try:
            exec_inspect_dict = self.client.api.exec_inspect(self.exec_id)
            while exec_inspect_dict.get('Running'):
                time.sleep(0.1)
                exec_inspect_dict = self.client.api.exec_inspect(self.exec_id)
        except docker.errors.APIError as e:
            # We may get an API error here, if so, return an exit code other than 0
            os._exit(127)
            pass

        # Forward the exit code of the exec-ed command if we got here
        exit_code = exec_inspect_dict.get('ExitCode')
        os._exit(exit_code)


if __name__ == '__main__':
    # Original docker exec --help
    """
Usage:  docker exec [OPTIONS] CONTAINER COMMAND [ARG...]

Run a command in a running container

Options:
  -d, --detach               Detached mode: run command in the background
      --detach-keys string   Override the key sequence for detaching a container
  -e, --env list             Set environment variables
  -i, --interactive          Keep STDIN open even if not attached
      --privileged           Give extended privileges to the command
  -t, --tty                  Allocate a pseudo-TTY
  -u, --user string          Username or UID (format: <name|uid>[:<group|gid>])
  -w, --workdir string       Working directory inside the container
"""
    parser = argparse.ArgumentParser(description="Run a command in a running container")
    parser.add_argument("container", help="Container name")
    parser.add_argument("command_and_args", help="Command and arguments", nargs=argparse.REMAINDER)

    parser.add_argument("-d", "--detach", action='store_true',
                        help="Detached mode: run command in the background")
    # We only support environment variables as a long string if there must be more than one
    # I.e. -e USER=user for one or -e "USER=user SOMETHING_ELSE=1"
    # Supporting multiple -e didn't work for me
    parser.add_argument("-e", "--env",
                        type=str, help="Set environment variables (like 'VAR1=1 VAR2=2')")
    # Interactive is not supported, but leaving it here just in case it is implemented in the future
    parser.add_argument("-i", "--interactive", action='store_true',
                        help="Keep STDIN open even if not attached (Note: not implemented, use 'docker exec')")
    parser.add_argument("--privileged", action='store_true',
                        help="Give extended privileges to the command")
    parser.add_argument("-t", "--tty", action='store_true',
                        help="Allocate a pseudo-TTY")
    parser.add_argument("-u", "--user",
                        type=str, help="Username or UID (format: <name|uid>[:<group|gid>])")
    parser.add_argument("-w", "--workdir",
                        type=str, help="Working directory inside the container")

    args = parser.parse_args()

    if len(args.command_and_args) < 1:
        print("dockerexec requires at least 2 arguments")
        parser.print_help()
        exit(1)

    if args.interactive:
        raise NotImplemented("Interactive mode not implemented, you should just use docker exec")

    dewsf = DockerExecWithSignalForwarding(args.container,
                                           args.command_and_args,
                                           detach=args.detach,
                                           # Note detach-keys is not implemented
                                           environment=args.env,
                                           interactive=args.interactive,
                                           privileged=args.privileged,
                                           tty=args.tty,
                                           user=args.user,
                                           workdir=args.workdir)

    # The following lines are tests done with a container running:
    # docker run --rm -t --name exec_signal_problem python:3 sleep 999
    # Proper testing should be implemented based on this
    # # Forward error test
    # de = DockerExec('exec_signal_problem',
    #                 'ls asdf',
    #                 tty=True,
    #                 interactive=False)
    # # simple working test
    # de = DockerExec('exec_signal_problem',
    #                 'ls',
    #                 tty=True,
    #                 interactive=False)
    # Test signal forwarding SIGINT Control C
    # de = DockerExec('exec_signal_problem',
    #                 'python -c "import sys;import signal;signal.signal(signal.SIGINT, print);print(\'hello\', file=sys.stderr);import time; time.sleep(600)"',
    #                 tty=True,
    #                 interactive=False)
    # Test signal forwarding SIGTERM
    # de = DockerExec('exec_signal_problem',
    #                 'python -c "import sys;import signal;signal.signal(signal.SIGTERM, print);print(\'hello\', file=sys.stderr);import time; time.sleep(600)"',
    #                 tty=True,
    #                 interactive=False)

    # Test output in stderr
    # de = DockerExec('exec_signal_problem',
    #                 'python -c "import sys; print(\'hello stderr\', file=sys.stderr);print(\'hello stdout\', file=sys.stdout)"',
    #                 tty=False,
    #                 interactive=False)

    # test input, doesn't work, not supported (not needed anyways)
    # de = DockerExec('exec_signal_problem',
    #                 'cat',
    #                 tty=True,
    #                 interactive=True)

awesomebytes picture awesomebytes  ·  2 Oct 2020