Openshift router pod (docker container) doesn't reap zombie processes

[ramr]

The openshift router pod running the openshift/origin-haproxy-router:v* image does not reap the haproxy zombie processes. Its probably not a big deal for now - but we will hit process limits when this list of unreaped children grows - dependent on how often the /var/lib/haproxy/reload-haproxy reload script is called.

[vagrant@openshiftdev ~]$ sudo nsenter -m -u -n -i -p -t $(sudo docker inspect --format "{{ .State.Pid }}" "$(sudo docker ps | grep origin-haproxy-router | awk '{print $NF}')")    
[root@router-1-paurv /]# ps -ef | cat    
UID        PID  PPID  C STIME TTY          TIME CMD    
root         1     0  0 01:09 ?        00:00:01 /usr/bin/openshift-router --template=/var/lib/haproxy/conf/haproxy-config.template --reload=/var/lib/haproxy/reload-haproxy    
haproxy     15     1  0 01:09 ?        00:00:00 [haproxy] <defunct>    
haproxy     19     1  0 01:09 ?        00:00:00 [haproxy] <defunct>    
haproxy     23     1  0 01:09 ?        00:00:00 [haproxy] <defunct>    
haproxy     27     1  0 01:09 ?        00:00:00 [haproxy] <defunct>    
haproxy     31     1  0 01:09 ?        00:00:00 [haproxy] <defunct>    
haproxy     35     1  0 01:09 ?        00:00:00 [haproxy] <defunct>    
haproxy     39     1  0 01:09 ?        00:00:00 [haproxy] <defunct>    
haproxy     43     1  0 01:10 ?        00:00:00 [haproxy] <defunct>    
haproxy     80     1  0 01:49 ?        00:00:00 /usr/sbin/haproxy -f /var/lib/haproxy/conf/haproxy.config -p /var/lib/haproxy/run/haproxy.pid -sf 43    
root        81     0  0 01:56 ?        00:00:00 -bash    
root        97    81  0 01:57 ?        00:00:00 ps -ef    
root        98    81  0 01:57 ?        00:00:00 cat    

[ramr]

@pweil- fyi.

[smarterclayton]

Sigh

[pweil-]

Hrm, well that's troublesome since we're checking for a pid file and sending finish signals in the restart if we found it:

if [ -n "$old_pid" ]; then
  /usr/sbin/haproxy -f $config_file -p $pid_file -sf $old_pid
else
  /usr/sbin/haproxy -f $config_file -p $pid_file
fi

[pweil-]

Some thoughts:

1. the plugin is the entry point for the image, it executes the reload commands
2. the plugin could be modified to reap child processes - this introduces os specific code with syscall
3. maybe the docker container can be set up differently so that haproxy is running under an init process that can reap?

cc @pmorie

[smarterclayton]

In general, the 3rd option is most sane. We have systemd container for this, also supervisor and a few others. None of them are ideal. The important thing is that haproxy must know it is being restarted.

There may be a golang library already for acting as pid 1 - you should investigate that.

[pweil-]

@smarterclayton

HAProxy knows it is being restarted, the old pid is stopped and is defunct. I think the issue is the parent process (the router plugin) doesn't care and isn't gathering the exit code so it lingers until the parent process stops (which running under init would do if it inherited the child).

But, I can do this without anything crazy or OS specific code if I have the restart script pass back the old pid and do something like this in a go routine:

proc, e := os.FindProcess(int(pid))
proc.Wait()

if that seems reasonable. Preliminary testing seems to work fine. Full test code: master...pweil-:router-reaper

I haven't run across any "do it for you" solutions. I did find a similar issue that led me to the SIGCHLD solution that was OS specific. I will look again since I was mainly searching for reaping with our process acting as pid 1.

[smarterclayton]

There's a golang process manager written by brad fitz. It's worth a look.

On Mar 6, 2015, at 12:29 PM, Paul [email protected] wrote:

@smarterclayton

HAProxy knows it is being restarted, the old pid is stopped and is defunct. I think the issue is the parent process (the router plugin) doesn't care and isn't gathering the exit code so it lingers until the parent process stops (which running under init would do if it inherited the child).

But, I can do this without anything crazy or OS specific code if I have the restart script pass back the old pid and do something like this in a go routine:

proc, e := os.FindProcess(int(pid))
proc.Wait()
if that seems reasonable. Preliminary testing seems to work fine.

I haven't run across any "do it for you" solutions. I did find a similar issue that led me to the SIGCHLD solution that was OS specific. I will look again since I was mainly searching for reaping with our process acting as pid 1.

—
Reply to this email directly or view it on GitHub.

[pweil-]

checking out dinit, and baseimage-docker. Will check that one out too.

[smarterclayton]

BaseImage docker is a terrible pattern

On Mar 6, 2015, at 2:35 PM, Paul [email protected] wrote:

checking out dinit, and baseimage-docker. Will check that one out too.

—
Reply to this email directly or view it on GitHub.

[pweil-]

rm -Rf pauls_brain/baseimage-docker

[smarterclayton]

My preferred solution here would be a simple addition to our Go binaries which detect when they are PID 1 and enable a simple minimal behavior for reaping (Wait on child processes in a goroutine) and proper signal handling. Docker may already have this from the old dockerinit code - I'd be surprised if someone hasn't handled it. @mrunalp you aware of anything like that?

[mrunalp]

@smarterclayton delved into that last week in lib container. Should be possible to put something together. I will take a closer look at the router tomorrow and see what we can do.

[pweil-]

@smarterclayton @mrunalp - that doesn't seem too difficult. I will wait to see if Mrunal knows of something already done. I looked at runsit, it seems to do the trick but makes setting up the image a little less obvious. Still solvable but if we'd prefer a simpler solution (I would) then I won't spend more time diving in.

@mrunalp - let me know if there is anything I can do to help

[mrunalp]

@pweil- I am going to prototype what @smarterclayton suggested; basically make the process act as a simple init that reaps zombies. If it works reliably, then we can use that else look for a real init.

[pweil-]

@mrunalp - this may not be the ideal implementation since it doesn't catch /any/ child process but it seemed to work fine for me as a general solution. master...pweil-:router-reaper . It was my prototype for option 2 in the comment above

[smarterclayton]

Fixt