Docker Socket API broken after Python update

[felixlabrot]

Describe the bug

Since the latest update to some Python packages it is not possible anymore to access the Docker API via unix://var/run/docker.sock. All connection attempts fail with Not supported URL scheme http+docker universally on all Photon OS servers. This entirely breaks monitoring with all Python based monitoring solutions (like Check_MK) and likely much more.

The bug is in one of those packages:

python3-jinja2-3.1.2-5.ph5.noarch
python3-requests-2.28.1-6.ph5.noarch
python3-xml-3.11.9-7.ph5.x86_64
python3-libs-3.11.9-7.ph5.x86_64
python3-3.11.9-7.ph5.x86_64
expat-2.6.0-6.ph5.x86_64
expat-libs-2.6.0-6.ph5.x86_64

Reproduction steps

1. run Photon OS with all updates installed
2. enable and start Docker
3. install package python3-pip
4. run pip install docker==7.0.0 (with latest version 7.1.0 the library is broken as well)
5. run this script: https://gist.github.com/felixlabrot/c96a6d2018b7ee7dcb004a032f5c323a

Expected behavior

It should be possible to connect to the Docker API like it was before.

Additional context

No response

[dcasota]

Your script indicates a custom Photon OS compilation due to the fact that check_mk has specific libraries.
On a Workstation 17.x vm deployed by a recently compiled Photon OS 5.0 iso, an api test seems to be successful. Here the sample from that machine (with a custom docker plexserver).

curl --unix-socket /var/run/docker.sock http://api/containers/json | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2157    0  2157    0     0   262k      0 --:--:-- --:--:-- --:--:--  263k
[
  {
    "Id": "bec3e975976aeea0abb3e2b72420d78d4ccbba9b79b8661510ac7a0c812ca4dd",
    "Names": [
      "/plex"
    ],
    "Image": "lscr.io/linuxserver/plex:latest",
    "ImageID": "sha256:7c39c1281ab063d0e483b1a74f3e6be89e94e8f03b7162962933c474f734bb9f",
    "Command": "/init",
    "Created": 1735066503,
    "Ports": [],
    "Labels": {
      "build_version": "Linuxserver.io version:- 1.41.3.9314-a0bfb8370-ls250 Build-date:- 2024-12-23T09:23:54+00:00",
      "maintainer": "thelamer",
      "org.opencontainers.image.authors": "linuxserver.io",
...

Docker version

root@photon-f2966979acd9 [ ~ ]# docker version
Client: Docker Engine - Community
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.21.13
 Git commit:        3ab4256
 Built:             Tue Dec 24 04:06:55 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       3ab5c7d
  Built:            Tue Dec 24 04:08:04 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.14
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
root@photon-f2966979acd9 [ ~ ]#

From your script, did you use the RedHat check-mk-raw-2.3.0p23-el8-38.x86_64.rpm version of 2.3.0p23 ? I'm asking because it needs quite a few dependencies during installation.

rpm -i ./check-mk-raw-2.3.0p23-el8-38.x86_64.rpm
warning: ./check-mk-raw-2.3.0p23-el8-38.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID c4503261: NOKEY
error: Failed dependencies:
        bind-utils is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        binutils is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        cronie is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        dialog is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        freeradius-utils is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        glib2 is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        graphviz is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        graphviz-gd is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        httpd is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        libevent is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        libgsf is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        libpq is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        libtool-ltdl is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        logrotate is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        pango is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        perl-IO-Zlib is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        perl-Locale-Maketext-Simple is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        perl-Net-Ping is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        php is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        php-cli is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        php-gd is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        php-json is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        php-mbstring is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        php-pdo is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        php-xml is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        poppler-utils is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        rpcbind is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        rpm-build is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        rsync is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        time is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        traceroute is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        uuid is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        xinetd is needed by check-mk-raw-2.3.0p23-el8-38.x86_64

Or did you use the docker version [1] as described in the [2]?

Accordingly to the latest commits in the official Photon OS 5.0 repository, see [3], afaik most changes were not related to python. There was a cve related fix for python virtualenv [4]. Can you describe ~"broken"?

edited:
The Not supported URL scheme http+docker issue seems be docker-py related only, e.g. [5]. A few users downgraded/pinned the requests module pip install requests==2.31 but that was before docker-py 7.1.0.

[1] docker container run -dit -p 8080:5000 -p 8000:8000 --tmpfs /opt/omd/sites/cmk/tmp:uid=1000,gid=1000 -v monitoring:/omd/sites --name monitoring -v /etc/localtime:/etc/localtime:ro --restart always checkmk/check-mk-raw:2.3.0p23
[2] https://checkmk.com/download?method=docker&edition=cre&version=2.3.0p23
[3] https://github.com/vmware/photon/commits/5.0/
[4] 4730351
[5] docker/docker-py#3256

[felixlabrot]

You are entirely wrong with those packages. You thought about the Check_MK server, which is not compatible with Photon OS. I am talking about the monitoring agent, which is not even needed to replicate the issue. The monitoring agent just calls the mentioned Python script and processes the text output it creates. To replicate the problem, there isn't even a need to install the agent. Just follow exactly my replication steps and do nothing more than that. That means your [1] and [2] are entirely not needed and are about something completely different.

The PIP package "docker-py" must not be used as per Check_MK documentation.
But your hint with the "requests" package brought me onto something.
The general problem with PIP is, that there isn't any command to update all packages, but that is the key to fix the problem. There is apparently something entirely broken when updating python3-requests with tdnf but then not updating it with pip as well.

This command is the magic fix to the issue:
pip list --outdated | cut -d" " -f1 | tail -n +3 | xargs -L1 pip install --upgrade

[dcasota]

So your enhancement suggestion is that eligible 'python3-*'-packages are updating their 'pip'-version as well? Good catch.

@prashant1221 Could you discuss this enhancement suggestion internally?

[felixlabrot]

Yes, because my issue has proven that a version mismatch can lead to broken packages. When both things are automatically kept in sync that can prevent such issues in the future.
In my case this has caused an outage for all Docker monitoring in our datacenter. A very unpleasant thing to have.

[dcasota]

Please re-open the issue, so the Photon OS team can have a look at it. Be patient. Thank you for your help. Thankful.