Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] 3007.1 - salt-proxy - "there is no current event loop in thread" #66775

Open
1 of 9 tasks
ITJamie opened this issue Aug 1, 2024 · 5 comments · May be fixed by #67131
Open
1 of 9 tasks

[BUG] 3007.1 - salt-proxy - "there is no current event loop in thread" #66775

ITJamie opened this issue Aug 1, 2024 · 5 comments · May be fixed by #67131
Assignees
@dwoz
Labels
Bug broken, incorrect, or confusing behavior
Milestone
Chlorine v3007.2

Comments

@ITJamie
Copy link
Contributor

ITJamie commented Aug 1, 2024

Description
attempt to run a state.apply against a salt-sproxy device.

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior
setup a salt-sproxy host + master setup. both version 3007.1 (in my case i was using the napalm nxos sproxy config)
confirm that basic commands like test.ping work to the salt-proxy host.
when you attempt to apply the following error occurs:

salt "sproxyminionname" state.apply timeout=600 test=true
sproxyminionname:
    The minion function caused an exception: Traceback (most recent call last):
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/metaproxy/proxy.py", line 483, in thread_return
        return_data = minion_instance.executors[fname](
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
        ret = self.loader.run(run_func, *args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/executors/direct_call.py", line 10, in execute
        return func(*args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
        ret = self.loader.run(run_func, *args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 835, in apply_
        return highstate(**kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 1154, in highstate
        st_ = salt.state.HighState(
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 5007, in __init__
        self.client = salt.fileclient.get_file_client(self.opts)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 55, in get_file_client
        return {"remote": RemoteClient, "local": FSClient, "pillar": PillarClient}.get(
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1129, in __init__
        self.channel = salt.channel.client.ReqChannel.factory(self.opts)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 56, in factory
        return SyncWrapper(
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 76, in __init__
        self.obj = cls(*args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 134, in factory
        auth = salt.crypt.AsyncAuth(opts, io_loop=io_loop)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 695, in __new__
        auth.__singleton_init__(opts, io_loop=io_loop)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 742, in __singleton_init__
        self._authenticate_future = tornado.concurrent.Future()
      File "/opt/saltstack/salt/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
        raise RuntimeError('There is no current event loop in thread %r.'
    RuntimeError: There is no current event loop in thread 'ProcessPayload(jid=20240801160004816387)'.
ERROR: Minions returned with non-zero exit code

Expected behavior
state to apply and respond cleanly

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) 
Salt Version:
          Salt: 3007.1

Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: 1.7.1
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.13.1
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.16.0
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: debian 11.10 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-30-amd64
        system: Linux
       version: Debian GNU/Linux 11.10 bullseye

Additional context
downgrading the salt-minion (and thus salt-sproxy) to 3006.9 fixes and state.apply's from the salt 3007.1 master work correctly
@ITJamie ITJamie added Bug broken, incorrect, or confusing behavior needs-triage labels Aug 1, 2024
@dwoz dwoz self-assigned this Aug 1, 2024
@dwoz dwoz removed the needs-triage label Aug 1, 2024
@dwoz dwoz added this to the Chlorine v3007.2 milestone Aug 1, 2024
@r-mettner-skaylink
Copy link

any update here ?
i see a similar issue if i try to use salt.client.Caller() within a thread, looks like a general issue with threading and salt 3007.x.
This worked with 3006.x.

@frebib
Copy link
Contributor

frebib commented Jan 6, 2025

We've just hit more or less the same issue with __slot__s and the fileclient. Applying this patch seems to fix it

diff --git salt/utils/asynchronous.py salt/utils/asynchronous.py
index 316fc5c478d..2db45ec49bc 100644
--- salt/utils/asynchronous.py
+++ salt/utils/asynchronous.py
@@ -73,7 +73,8 @@ class SyncWrapper:
         self.cls = cls
         if loop_kwarg:
             kwargs[self.loop_kwarg] = self.io_loop
-        self.obj = cls(*args, **kwargs)
+        with current_ioloop(self.io_loop):
+            self.obj = cls(*args, **kwargs)
         self._async_methods = list(
             set(async_methods + getattr(self.obj, "async_methods", []))
         )

@r-mettner-skaylink
Copy link

^can confirm that diff also fixes my issue with salt.client.Caller() within a thread.

@ITJamie
Copy link
Contributor Author

ITJamie commented Jan 7, 2025

@frebib would you consider opening a PR for that or would you be ok with me opening one with that change?

@frebib
Copy link
Contributor

frebib commented Jan 7, 2025

I can open a PR 👍🏻

@frebib frebib linked a pull request Jan 7, 2025 that will close this issue
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dwoz dwoz

Labels
Bug broken, incorrect, or confusing behavior
Projects
None yet
Milestone
Chlorine v3007.2
Development

Successfully merging a pull request may close this issue.

Use Current ioloop in SyncWrapper frebib/salt
4 participants
@frebib @dwoz @ITJamie @r-mettner-skaylink