[kfp-persistence] config-change hook failed with ValueError

[rgildein]

Bug Description

After deploying kfp-persistence from 2.2/stable channel, the config-changed hook failed with ValueError.

To Reproduce

1. juju add-model kubeflow
2. juju deploy kubeflow --channel 1.9/stable --trust

Environment

Juju 3.5.3
MicroK8s v1.30.5 revision 7235

Relevant Log Output

unit-kfp-persistence-0: 12:55:12 INFO juju.worker.uniter awaiting error resolution for "config-changed" hook
unit-kfp-persistence-0: 12:56:28 INFO juju.worker.uniter awaiting error resolution for "config-changed" hook
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed Traceback (most recent call last):
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "./src/charm.py", line 31, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     from components.sa_token_component import SaTokenComponent
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/src/components/sa_token_component.py", line 11, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     import kubernetes
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/__init__.py", line 21, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     from . import dynamic
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/dynamic/__init__.py", line 15, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     from .client import *  # NOQA
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/dynamic/client.py", line 18, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     from kubernetes import watch
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/watch/__init__.py", line 15, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     from .watch import Watch
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/watch/watch.py", line 16, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     import pydoc
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "/usr/lib/python3.8/pydoc.py", line 73, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     import sysconfig
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap_external>", line 844, in exec_module
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap_external>", line 976, in get_code
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap_external>", line 640, in _compile_bytecode
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed ValueError: bad marshal data (invalid reference)
unit-kfp-persistence-0: 12:56:29 ERROR juju.worker.uniter.operation hook "config-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-kfp-persistence-0: 12:56:29 INFO juju.worker.uniter awaiting error resolution for "config-changed" hook
unit-kfp-persistence-0: 12:59:12 INFO juju.worker.uniter awaiting error resolution for "config-changed" hook
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed Traceback (most recent call last):
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "./src/charm.py", line 31, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     from components.sa_token_component import SaTokenComponent
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/src/components/sa_token_component.py", line 11, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     import kubernetes
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/__init__.py", line 21, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     from . import dynamic
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/dynamic/__init__.py", line 15, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     from .client import *  # NOQA
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/dynamic/client.py", line 18, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     from kubernetes import watch
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/watch/__init__.py", line 15, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     from .watch import Watch
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/watch/watch.py", line 16, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     import pydoc
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "/usr/lib/python3.8/pydoc.py", line 73, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     import sysconfig
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap_external>", line 844, in exec_module
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap_external>", line 976, in get_code
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap_external>", line 640, in _compile_bytecode
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed ValueError: bad marshal data (invalid reference)
unit-kfp-persistence-0: 12:59:13 ERROR juju.worker.uniter.operation hook "config-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-kfp-persistence-0: 12:59:13 INFO juju.worker.uniter awaiting error resolution for "config-changed" hook

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6384.

This message was autogenerated

[orfeas-k]

@rgildein Thank you for reporting this. We 'll need more information in order to assess the issue. Some questions that would help us with that:

• Did you make a specific configuration or does this refer to the config-changed hook run during deployment?
• Was this a one time thing or have you observed it consisently?
• Have you been able to reproduce this with Juju 3.4? We do not support 3.5.x right now, as we haven't tested our charms with it.

[rgildein]

Did you make a specific configuration or does this refer to the config-changed hook run during deployment?

No, I did not.

Was this a one time thing or have you observed it consisently?

It happened two times, but not I could not reproduce it.

Have you been able to reproduce this with Juju 3.4? We do not support 3.5.x right now, as we haven't tested our charms with it.

I did not try to use Juju 3.4, on the third try it worked fine with 3.5. I deployed it again now (doing some testing) and it worked fine again.