Only restart containers from systemd

[pothos]

• Only restart containers from systemd
  Restarting etcd or the kubelet container from the Docker daemon
  introduces a race because the "docker logs" command will terminate and
  then the systemd unit will also restart the container in parallel to
  the Docker daemon. This race causes problems like terminating the
  container again or failing to start it, but it also introduces
  additional issues because the ExecStartPre and ExecStopPost commands
  run while the container is running, which means the state can get
  corrupted. Even if the systemd unit would not restart in parallel, it
  is not wanted to restart the container from the Docker daemon because
  the systemd unit should first run the ExecStartPre/Post commands, e.g.,
  for the "etcd-rejoin" script.

  Only restart the containers from the systemd unit, not from the Docker
  daemon.

• etcd: stop the service before modifying the etcd folder

  The etcd folder was modified while the etcd service was running which
  could lead to an earlier restart of the service because the service
  conditions for existing files are finally met and it may conflict
  with any commands in the ExecStopPost directive.

  First stop the etcd service unit before touching the files and only
  start the unit after the modification is done.

• aws|packet: align etcd unit file with the shared controller module

  The shared etcd CLC systemd unit has additional conditions for startup
  which were missing in the aws and packet etcd systemd units. It seems
  that these conditions help to make the bootstap more robust.

  Port the startup conditions over to aws and packet to align the
  platforms (which also makes debugging more clear).

How to use

Testing done

Looking at CI

[surajssd]

@pothos if this hypothesis works, could you also update for other platforms?

[pothos]

Yes, looks the kubelet was also affected, updated it there, too.

[invidian]

This should be mentioned in the upgrade guide.

[pothos]

Still got a failure and tried one more improvement in another commit

[invidian]

Looks good to me.

[surajssd]

👍