No such file or directory: 'jupyterhub-announcement-cookie-secret' #12

orboan · 2022-04-06T13:35:45Z

I am trying this service in my jupyterhub (deployed in a k8s cluster), where the jupyterhub container (in its pod) listens in port 8081. The config file used is:

c.JupyterHub.services.append({
    'name': 'announcement',
    'url': 'http://127.0.0.1:8081',
    'command': ["python3", "-m", "jupyterhub_announcement"]
})

But when accessing the announcement service from the services menu in the hub page, I get the 503 Service Unavailable. Cheking the pod logs I get:

[I 2022-04-06 13:28:57.073 JupyterHub service:132] Spawning python3 -m jupyterhub_announcement
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/jupyterhub_announcement/__main__.py", line 6, in <module>
    main()
  File "/usr/local/lib/python3.8/dist-packages/jupyterhub_announcement/announcement.py", line 349, in main
    app.initialize()
  File "/usr/local/lib/python3.8/dist-packages/jupyterhub_announcement/announcement.py", line 313, in initialize
    with open(self.cookie_secret_file) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'jupyterhub-announcement-cookie-secret'
[I 2022-04-06 13:28:58.483 JupyterHub log:189] 200 GET /hub/home ([email protected]) 74.78ms
[I 2022-04-06 13:29:00.376 JupyterHub log:189] 200 POST /hub/api/users/dvalencia19/activity ([email protected]) 79.35ms
[I 2022-04-06 13:29:01.417 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fservices%2Fannouncement%2F (@10.32.0.1) 3.66ms

Any ideas on how to fix this?

PS: Using the port 8888 I get exactly the same result (the same logs).

Thank you.

The text was updated successfully, but these errors were encountered:

rcthomas · 2022-04-06T13:50:46Z

You need to generate a cookie secret and put it at the expected location. Something like openssl rand -hex 32 might do the trick and get you started, but take a look in the tornado documentation for more details.

orboan · 2022-04-07T11:56:31Z

Hello rcthomas. Thank you for your hints.

The location of the tornado's cookie_secret for jupyterhub, in my case, is:

/usr/local/etc/jupyterhub/secret/hub.config.JupyterHub.cookie_secret

But in your file announcement.py you define the cookie_secret as:

cookie_secret_file = Unicode(
        "jupyterhub-announcement-cookie-secret",
        help="File in which we store the cookie secret."
    ).tag(config=True)

And the location of this file is:

/usr/local/lib/python3.8/dist-packages/jupyterhub_announcement/announcement.py

So which one could be the location of the jupyterhub-announcement-cookie-secret?

/usr/local/etc/jupyterhub/secret/
/usr/local/lib/python3.8/dist-packages/jupyterhub_announcement/
Another location. In that case, any clues on how to find it?

Thank you again for your help.

rcthomas · 2022-04-07T13:40:25Z

I think hub.config.JupyterHub.cookie_secret is the wrong secret. You need to create one for this service, and then set the service config option to point at the secret.

orboan · 2022-04-07T17:44:43Z

Thanks again rcthomas.

For the moment I am not successful in making this service to work.

After your latest message I did the following, without success. Maybe you could tell me what I am doing wrong:

I added a copy of announcement_config.py in the directory where all configurations files for jupyterhub are located, which is: /usr/local/etc/jupyterhub/
In this announcement_config.py file I modified the line you suggested with an absolute path for the cookie secret file, as this way I will know for sure where that file needs to be put:

c.AnnouncementService.cookie_secret_file = '/usr/local/etc/jupyterhub/secret/jupyterhub-announcement-cookie-secret'

I generated jupyterhub-announcement-cookie-secret file in that location using openssl rand -hex 32.

After this, I am still getting FileNotFoundError: [Errno 2] No such file or directory: 'jupyterhub-announcement-cookie-secret'

Options where I may be wrong:

The location where I put announcement_config.py is incorrect and it is not loaded.
The c.AnnouncementService.cookie_secret_file property does not accept absolute paths.
I do not know what else could be.

I appreciate all your help and patiente. Only If you could provide me with a little bit more of light in this issue...

PS: I also tried putting announcement_config.py in the same directory as announcement.py with the same result (same error).

Regards

I am wondering why the pip install command cannot fix this issue. Is there any reason I am not aware of?

rcthomas · 2022-04-07T18:16:26Z

The FileNotFoundError references the default name for the cookie secret file, indicating to me that the config file is maybe not being read by the service. By default it's read from the current working directory. When the service starts up, what path is it sitting in? If the announcement_config.py is found there, it will be executed. Otherwise you need to specify the path to it on the command line.

orboan · 2022-04-13T10:44:46Z

Thanks to rcthomas suggestions, I managed to fix the issue about the cookie secret and now the service is started (as a hub managed service).
But still cannot access to it. I am still getting:

503 : Service Unavailable

Your server appears to be down. Try restarting it from the hub

The logs for the jupyterhub pod are next:

[instructor@jupyter kube]$ k logs hub-55c48684d6-6bl55 
'/tmp/announcement_config.py' -> '/srv/jupyterhub/announcement_config.py'
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
Loading /usr/local/etc/jupyterhub/jupyterhub_config.d config: 01_jh_config_custom_spawner_longhorn.py
Loading /usr/local/etc/jupyterhub/jupyterhub_config.d config: 02_jh_config_custom_lti.py
Loading extra config: announcement
Loading extra config: cull
Loading extra config: lti-auth
Loading extra config: ngshare.py
Loading extra config: security
[I 2022-04-13 10:22:24.380 JupyterHub app:2769] Running JupyterHub version 2.2.0
[I 2022-04-13 10:22:24.381 JupyterHub app:2799] Using Authenticator: builtins.MyLTIAuthenticator
[I 2022-04-13 10:22:24.381 JupyterHub app:2799] Using Spawner: builtins.MyKubeSpawner
[I 2022-04-13 10:22:24.381 JupyterHub app:2799] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.2.0
[I 2022-04-13 10:22:24.739 JupyterHub app:1924] Not using allowed_users. Any authenticated user will be allowed.
[I 2022-04-13 10:22:24.882 JupyterHub roles:448] Added role token to token <APIToken('af5b...', service='cull-idle', client_id='jupyterhub')>
[I 2022-04-13 10:22:24.976 JupyterHub roles:448] Added role token to token <APIToken('421a...', service='announcement', client_id='jupyterhub')>
[I 2022-04-13 10:22:25.036 JupyterHub provider:609] Updating oauth client service-announcement
[I 2022-04-13 10:22:25.133 JupyterHub provider:609] Updating oauth client service-ngshare
[I 2022-04-13 10:22:25.486 JupyterHub reflector:275] watching for pods with label selector='component=singleuser-server' in namespace labs
[I 2022-04-13 10:22:25.516 JupyterHub reflector:275] watching for events with field selector='involvedObject.kind=Pod' in namespace labs
[W 2022-04-13 10:22:25.562 JupyterHub spawner:2116] Pod labs/jupyter-spena21 url changed! 10.32.0.23:50881 -> 10.32.0.23:8888
[I 2022-04-13 10:22:25.603 JupyterHub app:2498] spena21 still running
[W 2022-04-13 10:22:35.411 JupyterHub app:2854] init_spawners did not complete within 10 seconds. Allowing to complete in the background.
[I 2022-04-13 10:22:35.411 JupyterHub app:3051] Not starting proxy
[I 2022-04-13 10:22:35.425 JupyterHub app:3087] Hub API listening on http://:8081/hub/
[I 2022-04-13 10:22:35.425 JupyterHub app:3089] Private Hub API connect url http://hub:8081/hub/
[I 2022-04-13 10:22:35.426 JupyterHub app:3098] Starting managed service cull-idle
[I 2022-04-13 10:22:35.426 JupyterHub service:377] Starting service 'cull-idle': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2022-04-13 10:22:35.430 JupyterHub service:132] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[I 2022-04-13 10:22:35.444 JupyterHub app:3098] Starting managed service announcement at http://127.0.0.1:8888
[I 2022-04-13 10:22:35.446 JupyterHub service:377] Starting service 'announcement': ['/usr/bin/python3', '-m', 'jupyterhub_announcement', '--log-level', '10']
[I 2022-04-13 10:22:35.455 JupyterHub service:132] Spawning /usr/bin/python3 -m jupyterhub_announcement --log-level 10
[I 2022-04-13 10:22:36.166 JupyterHub log:189] 200 GET /hub/api/ ([email protected]) 82.74ms
[I 2022-04-13 10:22:36.258 JupyterHub log:189] 200 GET /hub/api/users?state=[secret] ([email protected]) 87.00ms
[AnnouncementService] Looking for announcement_config in /srv/jupyterhub
[AnnouncementService] Loaded config file: /srv/jupyterhub/announcement_config.py
[AnnouncementService] ephemeral queue, persist_path not set
[AnnouncementService] queue has 0 announcements
DEBUG:tornado.application:No user identified
DEBUG:tornado.application:Redirecting to login url: /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Fservices%2Fannouncement%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogImJkMjkzMzI3YTdjYjRmMzVhMTI0OGVkNDEzZDA4MDhmIiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIn0
INFO:tornado.access:302 GET /services/announcement/ (127.0.0.1) 20.23ms
[I 2022-04-13 10:22:36.650 JupyterHub app:3107] Adding external service ngshare at http://ngshare.labs.svc.cluster.local:8080
[I 2022-04-13 10:22:36.672 JupyterHub app:3156] JupyterHub is now running, internal Hub API at http://hub:8081/hub/
DEBUG:tornado.application:No user identified
DEBUG:tornado.application:Redirecting to login url: /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Fservices%2Fannouncement%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjlhYjc2NzhiNTA2NjRiNDA4NTA4MDg0YjUzMTkyYzVlIiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIn0
INFO:tornado.access:302 GET /services/announcement/ (127.0.0.1) 12.30ms
[I 2022-04-13 10:23:45.450 JupyterHub log:189] 200 POST /hub/api/users/spena21/activity ([email protected]) 94.84ms
[I 2022-04-13 10:24:26.545 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Forboan%2Fapi%2Fmetrics%2Fv1%3F1649845333293 (@10.32.0.1) 62.73ms
[I 2022-04-13 10:24:26.668 JupyterHub log:189] 200 GET /hub/api/user ([email protected]) 120.75ms
[I 2022-04-13 10:24:26.780 JupyterHub app:2498] orboan still running
[I 2022-04-13 10:24:26.786 JupyterHub app:2838] Initialized 2 spawners in 121.378 seconds
[I 2022-04-13 10:24:26.903 JupyterHub log:189] 200 POST /hub/api/users/orboan/activity ([email protected]) 117.98ms
DEBUG:tornado.application:No user identified
DEBUG:tornado.application:Redirecting to login url: /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Fservices%2Fannouncement%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjU0NDg0ZWFhMDUwZTQ3YjBiNTk1OTRjYjc1MmMwMDBlIiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIn0
INFO:tornado.access:302 GET /services/announcement/ (127.0.0.1) 11.66ms
DEBUG:tornado.application:No user identified
DEBUG:tornado.application:Redirecting to login url: /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Fservices%2Fannouncement%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjA0NDU1MTU0Njg0MTRiN2E5ZmNkYTBhNmU4N2NhNDQxIiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIn0
INFO:tornado.access:302 GET /services/announcement/ (127.0.0.1) 11.88ms
DEBUG:tornado.application:No user identified
DEBUG:tornado.application:Redirecting to login url: /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Fservices%2Fannouncement%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogImUzMTIzYWE3ZGFkNDQ0MWZiNDU2ZmMyNzNjZmE0ZjBjIiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIn0
INFO:tornado.access:302 GET /services/announcement/ (127.0.0.1) 11.98ms
DEBUG:tornado.application:No user identified
DEBUG:tornado.application:Redirecting to login url: /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Fservices%2Fannouncement%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjYzODJkM2YzNzk3NjRiMjFiNzgzYWNkNWY5MzcxODVhIiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIn0
INFO:tornado.access:302 GET /services/announcement/ (127.0.0.1) 12.15ms
[I 2022-04-13 10:28:21.479 JupyterHub log:189] 200 POST /hub/api/users/spena21/activity ([email protected]) 76.12ms
DEBUG:tornado.application:No user identified
DEBUG:tornado.application:Redirecting to login url: /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Fservices%2Fannouncement%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjJlZGIyYTQ5YzFkMDQwNmViMDhhODgzYTg0N2IyMGVjIiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIn0
INFO:tornado.access:302 GET /services/announcement/ (127.0.0.1) 11.90ms
[I 2022-04-13 10:29:27.289 JupyterHub log:189] 200 GET /hub/api/user ([email protected]) 60.29ms
[I 2022-04-13 10:29:29.665 JupyterHub log:189] 200 POST /hub/api/users/orboan/activity ([email protected]) 101.63ms
DEBUG:tornado.application:No user identified
DEBUG:tornado.application:Redirecting to login url: /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Fservices%2Fannouncement%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjdhZDA0ZjA3NTI3YTRkMGM5MDI1YTNmZDZiMGVmMDc4IiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIn0
INFO:tornado.access:302 GET /services/announcement/ (127.0.0.1) 12.29ms
DEBUG:tornado.application:No user identified
DEBUG:tornado.application:Redirecting to login url: /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Fservices%2Fannouncement%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjIxYzBkZjIzYTU1OTRkMzM5YjAyMzA5M2U2NDc2OTRjIiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIn0
INFO:tornado.access:302 GET /services/announcement/ (127.0.0.1) 12.29ms
[I 2022-04-13 10:30:45.740 JupyterHub log:189] 200 GET /hub/home ([email protected]) 49.54ms
[I 2022-04-13 10:30:48.480 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fservices%2Fannouncement%2F (@10.32.0.1) 3.72ms

Any ideas?

Thank you.

rcthomas · 2022-04-13T14:04:54Z

Can you post your hub's services configuration, and can you confirm if you have set this in load_roles:

{
    "name": "user",
    "scopes": ["access:services", "self"],
}

orboan · 2022-04-24T15:56:47Z

Hello rcthomas, thanks again.

In my jupyterhub I have these configured services in the config.yaml file used by the helm chart:

hub:
  image:
    name: {image_name}
    tag: {version}
  args:
    - "/tmp/start_hub.sh"
  extraConfig:
    lti-auth: |
      c.JupyterHub.authenticator_class = MyLTIAuthenticator
    ngshare.py: |
      c.JupyterHub.services.append({
        'name': 'ngshare',
        'admin': True,
        'url': 'http://ngshare.labs.svc.cluster.local:8080',
        'api_token': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'})
    announcement: |
      c.JupyterHub.services.append({
        'name': 'announcement',
        'url': 'http://127.0.0.1:8888',
        'command': [sys.executable, "-m", "jupyterhub_announcement","--log-level","10"]})
    roles: |
      c.JupyterHub.load_roles.append({
        "name": "user",
        "scopes": ["access:services", "self"],})
    security: |
      from kubernetes import client
      def modify_pod_hook(spawner, pod):
        pod.spec.containers[0].security_context = client.V1SecurityContext(
            privileged=True,
            capabilities=client.V1Capabilities(
                add=['SYS_ADMIN']
            )
        )
        return pod
      c.KubeSpawner.modify_pod_hook = modify_pod_hook
      c.JupyterHub.admin_access = True
    cull:
      timeout=3800

As you can see, I just now, as you suggested, added the load_roles data. Unfortunately I am still not able to acces the web UI of the service from the jupyterhub services menu. The logs are exactly the same as in my previous post.

Any other ideas?

Thanks again once more.

orboan · 2022-04-25T11:15:26Z

Following the example found here (for a managed service named service-notebook):

service-notebook

... I modified the config.yaml as next:

hub:
  ...
  extraConfig:
    announcement: |
      group_name = 'admins'
      service_name = 'announcement'
      service_port = 8888
      c.JupyterHub.load_groups = {group_name: ['orboan']}
      c.JupyterHub.services.append({
        'name': service_name,
        'url': f'http://127.0.0.1:{service_port}',
        'command': [sys.executable, "-m", "jupyterhub_announcement","--log-level","10"]})
      c.JupyterHub.load_roles = []
      c.JupyterHub.load_roles.append({
        "name": "user",
        "scopes": ["access:services", "self"],})
      c.JupyterHub.load_roles.append({
        "name": service_name,
        "groups": [group_name],
        "scopes": [f"access:services!service={service_name}"],})

Still I am getting:

[I 2022-04-25 11:07:59.823 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fservices%2Fannouncement%2F (@10.32.0.1) 13.04ms
DEBUG:tornado.application:No user identified
DEBUG:tornado.application:Redirecting to login url: /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Fservices%2Fannouncement%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogy9t679f6vfvfbvgMTQ0NDM4MmZhYjM4ODdlNjdkNDFlIiwgIm5leHRfdXJsIjogIi9zZXJ2aWt9t9t9togyg780W1lbnQvIn0
INFO:tornado.access:302 GET /services/announcement/ (127.0.0.1) 12.33ms

rcthomas · 2022-04-25T23:31:11Z

I'll need to recreate the same kind of setup and that'll take a little time here, hopefully I can come back to you later in the week.

rcthomas · 2022-04-26T04:00:04Z

I just started up JupyterHub in a Docker container, with the hub managing the service and it seemed to be just fine. I compared my log to yours and I don't understand your logs.

If I do something like omit the cookie secret (causing the announcement service to crash on startup) then I can replicate a 503 just like you get because the proxy can't find it. But I see messages in your log that indicate the service started up OK. In mine, the crash is obvious from the log.

I also don't see your hub saying it's adding routes to your proxy like [I 2022-04-26 03:57:39.007 JupyterHub proxy:267] Adding service announcement to proxy /services/announcement/ => http://127.0.0.1:8888 do your proxy logs show the routes being added? Here's my test setup:

Dockerfile:

FROM jupyterhub/jupyterhub:latest

WORKDIR /srv

RUN \
    apt-get update          &&  \
    apt-get upgrade --yes   &&  \
    apt-get install --yes       \
        git

RUN \
    pip install --no-cache-dir  \
        git+https://github.com/rcthomas/jupyterhub-announcement \
        html-sanitizer

RUN \
    adduser -q --gecos "" --disabled-password admin     && \
    echo admin:admin | chpasswd

RUN \
    adduser -q --gecos "" --disabled-password user1     && \
    echo user1:user1 | chpasswd

ADD jupyterhub_config.py jupyterhub_config.py
ADD jupyterhub-announcement-cookie-secret jupyterhub-announcement-cookie-secret

And the jupyterhub_config.py:

c.JupyterHub.default_url = "/hub/home"

c.JupyterHub.services = [{
    'name': 'announcement',
    'url': 'http://127.0.0.1:8888',
    'command': ["python3", "-m", "jupyterhub_announcement"]
}]

c.JupyterHub.load_roles = [{
    "name": "user",
    "scopes": ["access:services", "self"],
}]

c.Authenticator.admin_users = set(["admin"])

This ran just fine with docker run -it --rm -p 8000:8000 test:latest jupyterhub --debug

With the cookie secret initialized how I mentioned before. What's in your announcement_config.py ?

orboan · 2022-04-27T08:48:40Z

Hello rcthomas,

thank you again for taking your time to try to find a solution.

What I can see from your Dockerfile is that you are using the jupyterhub image that is specific for simple docker containers, whereas in my case I am running jupyterhub in a kubernetes cluster. For running jupyterhub in a kubernetes cluster, the image you need to use is:

jupyterhub/k8s-hub

instead of jupyterhub/jupyterhub.

Some of the paths are different between these 2 images. For example, the working directory in jupyterhub/k8s-hub is /srv/jupyterhub, while I see in your Dockerfile the working dir is /srv.

In a k8s cluster the proxy is running in a separate pod, not in the hub pod itself.

Overall seems that you service is only compatible with jupyterhub running in a simple docker container, but no in a kubernetes cluster.

I will continue searching for a solution, and in case I find it I will be happy to share here.

Thank you (again).

rcthomas · 2022-04-27T13:18:00Z

Hmmm, no, I don't buy that. My production hub is running in a k8s cluster (Rancher) and it's just fine. In that deployment the hub, proxy, announcement service, and everything are in their own separate pods.

Why don't you try running the announcement service in a separate pod? I was only trying to replicate your setup here where you had it running as hub-managed.

orboan · 2022-04-28T11:15:41Z

Ok, I may be wrong, obviously.
Have you tried to run the service as a hub-managed service in the kubernetes cluster? This is the situation where I am stuck.

I can try to run it as an external service. If that works in your case it should work also in mine. For that, could you add to this repo the Dockerfile of the container running the service? And if possible, the pod yaml or even better the deployment yaml corresponding to this service run in its own pod. This would be very helpful and give the repo completeness, to my unsderstanding.

Thanks once more ;)

orboan · 2022-04-30T16:16:24Z

As suggested, I am trying to deploy this service as external in its own pod.

For the moment I am getting now

400 : Bad Request
Missing client_id parameter.

The logs are:

[I 2022-04-30 15:59:39.868 JupyterHub app:3089] Private Hub API connect url http://hub:8081/hub/
[I 2022-04-30 15:59:39.868 JupyterHub app:3098] Starting managed service jupyterhub-idle-culler
[I 2022-04-30 15:59:39.868 JupyterHub service:377] Starting service 'jupyterhub-idle-culler': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2022-04-30 15:59:39.872 JupyterHub service:132] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[I 2022-04-30 15:59:39.894 JupyterHub app:3107] Adding external service announcement at http://announcement-announcementchart.labs.svc.cluster.local:8888
[I 2022-04-30 15:59:40.012 JupyterHub app:3107] Adding external service ngshare at http://ngshare.labs.svc.cluster.local:8080
[I 2022-04-30 15:59:40.032 JupyterHub app:3156] JupyterHub is now running, internal Hub API at http://hub:8081/hub/
[I 2022-04-30 15:59:40.566 JupyterHub log:189] 200 GET /hub/api/ ([email protected]) 57.29ms
[I 2022-04-30 15:59:40.636 JupyterHub log:189] 200 GET /hub/api/users?state=[secret] ([email protected]) 65.61ms
[I 2022-04-30 16:00:03.834 JupyterHub log:189] 200 GET /hub/home ([email protected]) 147.37ms
[W 2022-04-30 16:00:07.045 JupyterHub auth:70] /authorizations/cookie endpoint is deprecated in JupyterHub 2.0. Use /api/user with OAuth tokens.
[I 2022-04-30 16:00:07.054 JupyterHub log:189] 200 GET /hub/api/authorizations/cookie/jupyterhub-services/[secret] ([email protected]) 67.60ms
[W 2022-04-30 16:00:12.041 JupyterHub web:1787] 400 GET /hub/api/oauth2/authorize?client_id=&redirect_uri=%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjViM2U3MzVlMjc0YTQ3YjlhYThjZWFkZjk5MGU3MTFlIiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIiwgImNvb2tpZV9uYW1lIjogIi1vYXV0aC1zdGF0ZS1jSnJRa1BGaCJ9 (192.168.1.38): Missing client_id parameter.
[W 2022-04-30 16:00:12.089 JupyterHub log:189] 400 GET /hub/api/oauth2/authorize?client_id=&redirect_uri=%2Foauth_callback&response_type=code&state=[secret] ([email protected]) 57.46ms

Maybe it's the value of the token I use in:

      c.JupyterHub.services.append({
        'name': service_name,
        'url': f'http://announcement-announcementchart.labs.svc.cluster.local:{service_port}',
        'api_token': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'})
      c.JupyterHub.load_roles.append({
        "name": "user",
        "scopes": ["access:services", "self"],})

I got this token from the user interface of the jupyterhub, I do not know if that's the right way to obtain it.

By the way, I used the next Dockerfile to create the image used in the pod:

FROM jupyterhub/jupyterhub:latest

WORKDIR /srv

RUN \
    apt-get update          &&  \
    apt-get upgrade --yes   &&  \
    apt-get install --yes       \
        git

RUN \
    pip install --no-cache-dir  \
        git+https://github.com/rcthomas/jupyterhub-announcement \
        html-sanitizer

RUN \
    adduser -q --gecos "" --disabled-password admin     && \
    echo admin:admin | chpasswd

RUN \
    adduser -q --gecos "" --disabled-password user1     && \
    echo user1:user1 | chpasswd

#ADD jupyterhub_config.py jupyterhub_config.py
ADD jupyterhub-announcement-cookie-secret jupyterhub-announcement-cookie-secret
ENTRYPOINT ["python3", "-m", "jupyterhub_announcement"]

EXPOSE 8888

And I deployed it using a simple helm chart I created using helm create.

rcthomas · 2022-05-02T14:22:05Z

Maybe try setting the environment variable JUPYTERHUB_CLIENT_ID=service-announcement and maybe also JUPYTERHUB_SERVICE_NAME=announcement for the service? These were added with the switch to using oauth, I may need to document these.

orboan · 2022-05-02T15:45:26Z

Thanks rcthomas for your new suggestions, and for your patiente.

I added those 2 env variables and now I am getting next:

400 : Bad Request
Mismatching redirect URI.

The hub logs are next:

[I 2022-05-02 15:37:52.599 JupyterHub base:807] User logged in: orboan
[I 2022-05-02 15:37:52.602 JupyterHub log:189] 302 POST /hub/lti/launch -> /hub/ ([email protected]) 86.97ms
[I 2022-05-02 15:37:55.706 JupyterHub log:189] 302 GET /hub/ -> /hub/spawn ([email protected]) 9.99ms
[W 2022-05-02 15:37:59.283 JupyterHub provider:500] Redirect uri /oauth_callback != /services/announcement/oauth_callback
[W 2022-05-02 15:37:59.283 JupyterHub web:1787] 400 GET /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjYwOTFlOTE1Y2NlZTQwZjE4M2NkZjQxMDM2ZDJmZWI1IiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIn0 (192.168.1.38): Mismatching redirect URI.
[W 2022-05-02 15:37:59.286 JupyterHub log:189] 400 GET /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Foauth_callback&response_type=code&state=[secret] ([email protected]) 27.53ms
[W 2022-05-02 15:38:22.788 JupyterHub provider:500] Redirect uri /oauth_callback != /services/announcement/oauth_callback
[W 2022-05-02 15:38:22.788 JupyterHub web:1787] 400 GET /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjQ2OTEwMDM4MWRlZTRiYmVhODIyMDhkZjIxMzBlZmFiIiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIiwgImNvb2tpZV9uYW1lIjogInNlcnZpY2UtYW5ub3VuY2VtZW50LW9hdXRoLXN0YXRlLWl6d1dVR0lvIn0 (192.168.1.38): Mismatching redirect URI.
[W 2022-05-02 15:38:22.792 JupyterHub log:189] 400 GET /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Foauth_callback&response_type=code&state=[secret] ([email protected]) 53.08ms

And the logs of the annoucement pod are:

[instructor@jupyter announcement]$ k logs announcement-announcementchart-554f88d944-l888k 
WARNING:tornado.application:Detected unused OAuth state cookies

Everything is behing an ingress ngnix reverse proxy, and the other external service I have (ngshare) is working fine.

rcthomas · 2022-05-02T16:02:19Z

Do you have JUPYTERHUB_SERVICE_PREFIX=/services/announcement set?

orboan · 2022-05-02T16:31:09Z

I did not have it set. I added it to the announcement service Dockerfile, which now is next:

FROM jupyterhub/jupyterhub:latest

WORKDIR /srv

RUN \
    apt-get update          &&  \
    apt-get upgrade --yes   &&  \
    apt-get install --yes       \
        git

RUN \
    pip install --no-cache-dir  \
        git+https://github.com/rcthomas/jupyterhub-announcement \
        html-sanitizer

RUN \
    adduser -q --gecos "" --disabled-password admin     && \
    echo admin:admin | chpasswd

RUN \
    adduser -q --gecos "" --disabled-password user1     && \
    echo user1:user1 | chpasswd

#ADD jupyterhub_config.py jupyterhub_config.py
ADD jupyterhub-announcement-cookie-secret jupyterhub-announcement-cookie-secret
ENTRYPOINT ["python3", "-m", "jupyterhub_announcement"]

EXPOSE 8888
ENV JUPYTERHUB_CLIENT_ID=service-announcement
ENV JUPYTERHUB_SERVICE_NAME=announcement
ENV JUPYTERHUB_SERVICE_PREFIX=/services/announcement

Is this Dockerfile different than the one you use in your cluster for this service?

The problem is that now I am getting a 404 not found.

Relevant hub logs are now:

[W 2022-05-02 16:10:34.449 JupyterHub app:2412] Cannot connect to external service announcement at http://announcement-announcementchart.labs.svc.cluster.local:8888

And announcement pod's logs are:

WARNING:tornado.access:404 GET /services/announcement/ (10.34.0.0) 1.71ms

rcthomas · 2022-05-02T16:41:16Z

Ah, I think I forgot to include a trailing / on the prefix. Try instead

ENV JUPYTERHUB_SERVICE_PREFIX=/services/announcement/

orboan · 2022-05-02T18:29:52Z

I added now the trailing slash: the 404 Not found is not showing anymore ;)

But... now I amb getting 500: Internal Server Error

Relevant hub logs are:

[W 2022-05-02 16:28:30.678 JupyterHub provider:500] Redirect uri /oauth_callback != /services/announcement/oauth_callback

It pretty clear the problem, but I do not know how to fix it.

Annoucement service pod's logs are:

[instructor@jupyter announcement]$ k logs announcement-announcementchart-9b4c5fc95-nzv5s 
ERROR:tornado.application:Error connecting to http://127.0.0.1:8081/hub/api: HTTPConnectionPool(host='127.0.0.1', port=8081): Max retries exceeded with url: /hub/api/oauth2/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd386dd84f0>: Failed to establish a new connection: [Errno 111] Connection refused'))
WARNING:tornado.general:500 GET /services/announcement/oauth_callback?code=jlT1XZKF4vxzF7YUJasypWMEIkOKhP&state=eyJ1dWlkIjogIjcyZDA5ZjM5YTg5MTRlMzk5Njg4MWU2NmJiNDQ3MDAxIiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIn0 (10.35.128.0): Failed to connect to Hub API at 'http://127.0.0.1:8081/hub/api'.  
Is the Hub accessible at this URL (from host: announcement-announcementchart-9b4c5fc95-nzv5s)?  
Make sure to set c.JupyterHub.hub_ip to an IP accessible to single-user servers if the servers are not on the same host as the Hub.
ERROR:tornado.access:500 GET /services/announcement/oauth_callback?code=jlT1XZKF4vxzF7YUJasypWMEIkOKhP&state=eyJ1dWlkIjogIjcyZDA5ZjM5YTg5MTRlMzk5Njg4MWU2NmJiNDQ3MDAxIiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIn0 (10.35.128.0) 8.77ms

rcthomas · 2022-05-02T18:56:57Z

Is your hub running at 127.0.0.1 (that would be the same container as the service)? If this is a separate pod it's most likely not, and you need to set that accordingly, eg via JUPYTERHUB_API_URL=http://whatever:8081/hub/api

orboan · 2022-05-03T09:53:18Z

Thank you rcthomas. I added this env variable JUPYTERHUB_API_URL=http://whatever:8081/hub/ap, and now I am getting this:

Hub logs:

[I 2022-05-03 09:48:10.795 JupyterHub log:189] 200 GET /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Fservices%2Fannouncement%2Foauth_callback&response_type=code&state=[secret] ([email protected]) 55.04ms
[I 2022-05-03 09:48:12.494 JupyterHub log:189] 302 POST /hub/api/oauth2/authorize?client_id=service-announcement&redirect_uri=%2Fservices%2Fannouncement%2Foauth_callback&response_type=code&state=[secret] -> /services/announcement/oauth_callback?code=[secret]&state=[secret] ([email protected]) 62.26ms
[W 2022-05-03 09:48:12.631 JupyterHub log:189] 401 POST /hub/api/oauth2/token (@10.34.0.0) 17.57ms

Service logs:

WARNING:tornado.application:Failed to check authorization: [401] Unauthorized
WARNING:tornado.application:{"error": "invalid_client"}
WARNING:tornado.general:500 GET /services/announcement/oauth_callback?code=26tGjp9MowJKaTe6oh3D0NUT0rdF8n&state=eyJ1dWlkIjogImU2YzI5ODNlODkyMDQ5Y2M5NDU2ZDcxN2QxNGZkZGE5IiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIn0 (10.35.128.0): Failed to check authorization: invalid_client
ERROR:tornado.access:500 GET /services/announcement/oauth_callback?code=26tGjp9MowJKaTe6oh3D0NUT0rdF8n&state=eyJ1dWlkIjogImU2YzI5ODNlODkyMDQ5Y2M5NDU2ZDcxN2QxNGZkZGE5IiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9hbm5vdW5jZW1lbnQvIn0 (10.35.128.0) 122.05ms

Is this related with the token used in the jupyterhub to access the external service? I generated the token in the jupyterhub itself (client side from the service perspective) using the hub web user interface (menu token). Maybe I should generate that token from the service side, but I do not know how to.

rcthomas · 2022-05-03T13:12:36Z

Hmmm. You didn't use whatever:8081/hub/ap literally did you? It should be the URL of the hub container, and it should be api not ap --- just checking if that was a typo!

orboan · 2022-05-03T16:28:03Z

Oh sorry, yes there is a typo but in my message here ;), not in the Dockerfile- I double cheked my Dockerfile, and I'm pasting it here again:

FROM jupyterhub/jupyterhub:latest

WORKDIR /srv

RUN \
    apt-get update          &&  \
    apt-get upgrade --yes   &&  \
    apt-get install --yes       \
        git

RUN \
    pip install --no-cache-dir  \
        git+https://github.com/rcthomas/jupyterhub-announcement \
        html-sanitizer

RUN \
    adduser -q --gecos "" --disabled-password admin     && \
    echo admin:admin | chpasswd

RUN \
    adduser -q --gecos "" --disabled-password orboan     && \
    echo orboan:orboan | chpasswd

#ADD jupyterhub_config.py jupyterhub_config.py
ADD jupyterhub-announcement-cookie-secret jupyterhub-announcement-cookie-secret

ENTRYPOINT ["python3", "-m", "jupyterhub_announcement"]

EXPOSE 8888
ENV JUPYTERHUB_CLIENT_ID=service-announcement
ENV JUPYTERHUB_SERVICE_NAME=announcement
ENV JUPYTERHUB_SERVICE_PREFIX=/services/announcement/
ENV JUPYTERHUB_API_URL=http://hub.labs.svc.cluster.local:8081/hub/api

stubclan · 2022-05-03T17:24:53Z

@orboan Loooking at your dockerfile, where have you set the api token?
'api_token': os.environ["ANNOUNCEMENT_JUPYTERHUB_API_TOKEN"]

The token can be anything and you can generate and add it yourself or if you have admin can prob use the UI to generate token and set the value

@rcthomas I do have the same issue above but I'm running annoucement as hub managed service, on annoucement.py line 313 you open the cookie-secret file, what is this really used for? API Token is diff than cookie-secret-file right ?

rcthomas · 2022-05-03T17:41:01Z

@stubclan it's for oauth, it's different than API token yes. @orboan had api token set before though? That is still needed.

stubclan · 2022-05-03T19:57:26Z

@rcthomas in that case, we just create a random token and add it to the library location ? Does it have to be tied to SSO service in any way?

rcthomas · 2022-05-03T20:00:31Z

Not sure what you mean by library location. But see #12 (comment) above, you should be able to put the cookie secret at the default location specified by the config, it just has to be in whatever directory is being used when the application starts. For the above Dockerfile @orboan has that would be /srv

stubclan · 2022-05-03T20:53:53Z

@rcthomas I added the token as you mentioned, now I'm getting this error:
[W 2022-05-03 20:51:00.993 JupyterHub user:242] Failed to retrieve encrypted auth_state for sujan_aryal because
[I 2022-05-03 20:51:03.865 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fservices%2Fannouncement%2F (@172.25.10.87) 2.40ms

I'm debugging but if you have any insights please lmk. Also going to try the separate pod install as well.

rcthomas · 2022-05-03T21:03:32Z

I think the auth_state line isn't related directly, it seems to be coming from

https://github.com/jupyterhub/jupyterhub/blob/main/jupyterhub/user.py#L303

which in turn is caused by an exception here

https://github.com/jupyterhub/jupyterhub/blob/main/jupyterhub/crypto.py#L167

See also https://jupyterhub.readthedocs.io/en/stable/reference/authenticators.html#authentication-state

As for the 503 I can't tell much from that one line by itself, sorry

orboan · 2022-05-04T10:35:02Z

@stubclan I did not add 'api_token': os.environ["ANNOUNCEMENT_JUPYTERHUB_API_TOKEN"] in the hub pod , but instead the token value itself:

c.JupyterHub.services.append({
        'name': service_name,
        'url': f'http://announcement-announcementchart.labs.svc.cluster.local:{service_port}',
        'api_token': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'})

Is is mandatory to use the env variable ANNOUNCEMENT_JUPYTERHUB_API_TOKEN ?

rcthomas · 2022-05-04T13:04:14Z

No, you should be able to set the token the way you have, though it is really not advisable to put a secret like that into your config.

https://jupyterhub.readthedocs.io/en/stable/reference/services.html#properties-of-a-service

orboan · 2022-05-04T15:42:01Z

Thanks @rcthomas

In

https://jupyterhub.readthedocs.io/en/stable/reference/services.html#properties-of-a-service

we can read:

Each Externally-Managed Service will need a unique API token, because the Hub authenticates each API request and the API token is used to identify the originating Service or user.

My question is how the token can be assigned to the announcement service running as external in its own pod? For the moment, I only set the token to the jupyterhub config file, but I haven't set it in the service.

In the announcement_config.py I see no option for token. Am I wrong?

Thank you

stubclan · 2022-05-04T15:54:32Z

@orboan Here's my understanding of this and I had to do thing for other API services.

The token has to be generated from hub, its also encrypted using jupyterhub crypt key. Once the token is saved in hub database, then you can use that token in these services.

Since you are using external service you have to give it a token, in the readme they are using env variable ANNOUNCEMENT_JUPYTERHUB_API_TOKEN which is the token generated for this service. You can inject this however you want but adding to config directly is risky.

You could just add that directly as env variable for the pod/ deployment or read from github secret/ vault/ or other if you're running via pipeline.

rcthomas · 2022-05-04T19:30:58Z

Maybe this is the problem then? The API token here just needs to be a shared secret. On the announcement service side you have JUPYTERHUB_API_TOKEN set and get the same secret to the hub service configuration. The same value is used in the hub config. You don't have to mint that API token from the hub, you can just set it on both services or share it. like @stubclan suggests.

orboan · 2022-05-05T09:17:48Z

Maybe this is the problem then? The API token here just needs to be a shared secret. On the announcement service side you have JUPYTERHUB_API_TOKEN set and get the same secret to the hub service configuration. The same value is used in the hub config. You don't have to mint that API token from the hub, you can just set it on both services or share it. like @stubclan suggests.

This latest information has turned out what I had missing to make the service work. Now it is working. Thank you again @rcthomas for your help and patiente.

rcthomas · 2022-05-05T15:51:00Z

Great, sorry for the difficulty. I'm going to review this thread here and make updates to the documentation to clarify things about configuring the service. The feedback is really useful.

stubclan · 2022-05-05T15:55:54Z

@orboan could you share what worked like your final dockerfile. I couldn't make it work with jupyterhub service itself so gonna have to run a separate container too. Also glad it worked for you :)

@rcthomas thanks for that, maybe adding dockerfile template would be helpful too

orboan · 2022-05-06T10:01:40Z

@rcthomas @stubclan I am going to publish all steps I needed to do to make this service work, including files needed. Just give some time cause right now I amb exhausted.

stubclan · 2022-05-09T16:21:32Z

@orboan Thank you and take your time

stubclan · 2022-05-16T15:55:14Z

@orboan sorry to bother you, just wondering if you had time, Thanks

orboan · 2022-05-19T18:08:54Z

Hello,

I added some documentation and files, according to the steps I needed to follow in order to make this service to work. You can find it in this fork:

annoucement-service - fix to some problems

I hope this could help someone.

Thanks for all your help.

rcthomas · 2022-05-20T13:18:41Z

@orboan you don't want to contribute these as a PR?

orboan · 2022-06-10T09:59:09Z

@orboan you don't want to contribute these as a PR?

Yes, I will submit a PR. Thank you

stubclan · 2022-06-20T17:09:47Z

Hello @orboan I was wondering if you could help with a small thing. I used your fork to deploy external annoucement service but getting the following error:

I am using SSO for jupyterhub and its trying to authenticate services-annoucement as a user. Was wondering if you had same setup and you did run into similar issue below?:

Does this client id have to match SSO service's client ID ? If so, then probably need to add secret as well right? I'll test this out but wondering if you had seen something like this already

rcthomas · 2022-06-21T16:54:38Z

What's the services config look like on the hub side?

nikhilbansalintel · 2024-02-23T00:41:35Z

Hi @orboan I am facing the same issue as you. I am trying to solve it for a while but seems like this link is not working anymore. Can you please send a copy of this if possible? annoucement-service - fix to some problems

orboan closed this as completed May 20, 2022

rcthomas reopened this May 20, 2022

No such file or directory: 'jupyterhub-announcement-cookie-secret' #12

No such file or directory: 'jupyterhub-announcement-cookie-secret' #12

Comments

orboan commented Apr 6, 2022 • edited Loading

rcthomas commented Apr 6, 2022

orboan commented Apr 7, 2022

rcthomas commented Apr 7, 2022

orboan commented Apr 7, 2022 • edited Loading

rcthomas commented Apr 7, 2022

orboan commented Apr 13, 2022 • edited Loading

rcthomas commented Apr 13, 2022

orboan commented Apr 24, 2022

orboan commented Apr 25, 2022

rcthomas commented Apr 25, 2022

rcthomas commented Apr 26, 2022

orboan commented Apr 27, 2022

rcthomas commented Apr 27, 2022

orboan commented Apr 28, 2022

orboan commented Apr 30, 2022 • edited Loading

rcthomas commented May 2, 2022

orboan commented May 2, 2022

rcthomas commented May 2, 2022

orboan commented May 2, 2022 • edited Loading

rcthomas commented May 2, 2022

orboan commented May 2, 2022

rcthomas commented May 2, 2022

orboan commented May 3, 2022 • edited Loading

rcthomas commented May 3, 2022 • edited Loading

orboan commented May 3, 2022

stubclan commented May 3, 2022 • edited Loading

rcthomas commented May 3, 2022

stubclan commented May 3, 2022

rcthomas commented May 3, 2022

stubclan commented May 3, 2022

rcthomas commented May 3, 2022

orboan commented May 4, 2022

rcthomas commented May 4, 2022

orboan commented May 4, 2022 • edited Loading

stubclan commented May 4, 2022

rcthomas commented May 4, 2022

orboan commented May 5, 2022

rcthomas commented May 5, 2022

stubclan commented May 5, 2022 • edited Loading

orboan commented May 6, 2022

stubclan commented May 9, 2022

stubclan commented May 16, 2022

orboan commented May 19, 2022

rcthomas commented May 20, 2022

orboan commented Jun 10, 2022

stubclan commented Jun 20, 2022

rcthomas commented Jun 21, 2022

nikhilbansalintel commented Feb 23, 2024

orboan commented Apr 6, 2022 •

edited

Loading

orboan commented Apr 7, 2022 •

edited

Loading

orboan commented Apr 13, 2022 •

edited

Loading

orboan commented Apr 30, 2022 •

edited

Loading

orboan commented May 2, 2022 •

edited

Loading

orboan commented May 3, 2022 •

edited

Loading

rcthomas commented May 3, 2022 •

edited

Loading

stubclan commented May 3, 2022 •

edited

Loading

orboan commented May 4, 2022 •

edited

Loading

stubclan commented May 5, 2022 •

edited

Loading