Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pull Through Cache is not functioning. 500 internal server error #1801

Open
PotentialIngenuity opened this issue Oct 29, 2024 · 5 comments
Open

Pull Through Cache is not functioning. 500 internal server error #1801

PotentialIngenuity opened this issue Oct 29, 2024 · 5 comments
Labels
Issue Triage-Needed

Comments

@PotentialIngenuity
Copy link

Version
core: 3.66.0
container: 2.21.1

Describe the bug
The pull through caches cannot be listed. Some of them can be pulled from but some of them cannot be.

http --verify false GET https://localhost/pulp/api/v3/distributions/container/pull-through/
HTTP/1.1 500 Internal Server Error
Access-Control-Expose-Headers: Correlation-ID
Connection: keep-alive
Content-Length: 145
Content-Type: text/html; charset=utf-8
Correlation-ID: 2eb919f124c24a94a8673a9f1fbb56c4
Cross-Origin-Opener-Policy: same-origin
Date: Tue, 29 Oct 2024 21:41:20 GMT
Referrer-Policy: same-origin
Server: nginx/1.16.1
X-Content-Type-Options: nosniff
X-Frame-Options: DENY


<!doctype html>
<html lang="en">
<head>
  <title>Server Error (500)</title>
</head>
<body>
  <h1>Server Error (500)</h1><p></p>
</body>
</html>

http --verify false GET https://localhost/pulp/api/v3/remotes/container/pull-through/
HTTP/1.1 500 Internal Server Error
Access-Control-Expose-Headers: Correlation-ID
Connection: keep-alive
Content-Length: 145
Content-Type: text/html; charset=utf-8
Correlation-ID: ea372ef5b7414c65af625adfe8d6c9f8
Cross-Origin-Opener-Policy: same-origin
Date: Tue, 29 Oct 2024 21:44:28 GMT
Referrer-Policy: same-origin
Server: nginx/1.16.1
X-Content-Type-Options: nosniff
X-Frame-Options: DENY


<!doctype html>
<html lang="en">
<head>
  <title>Server Error (500)</title>
</head>
<body>
  <h1>Server Error (500)</h1><p></p>
</body>
</html>

I can see that the pull through still exists but trying to create it again.

http --verify false POST https://localhost/pulp/api/v3/remotes/container/pull-through/ name=docker-cache url=https://registry-1.docker.io
HTTP/1.1 400 Bad Request
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, POST, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 39
Content-Type: application/json
Correlation-ID: 3014e1050aab43a39116aa704c8446e7
Cross-Origin-Opener-Policy: same-origin
Date: Tue, 29 Oct 2024 21:44:02 GMT
Referrer-Policy: same-origin
Server: nginx/1.16.1
Vary: Accept
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "name": [
        "This field must be unique."
    ]
}

This container pulled just fine

docker pull registry.company.com/docker-cache/library/golang:latest
latest: Pulling from docker-cache/library/golang
7d98d813d54f: Already exists
da802df85c96: Pull complete
7aadc5092c3b: Pull complete
b09bd0f8c72b: Pull complete
2ac1f1163629: Pull complete
f29d27dc8b21: Pull complete
4f4fb700ef54: Pull complete
Digest: sha256:ad5c126b5cf501a8caef751a243bb717ec204ab1aa56dc41dc11be089fafcb4f
Status: Downloaded newer image for registry.company.com/docker-cache/library/golang:latest
registry.company.com/docker-cache/library/golang:latest

This one does not pull

docker pull registry.company.com/docker-cache/pulp/pulp-web:3.66.0
Error response from daemon: repository registry.company.com/docker-cache/pulp/pulp-web not found: name unknown: Repository not found.

The 500 error

Oct 29 16:44:02 172.18.0.1 docker-pulp_api-2[2051]: pulp [3014e1050aab43a39116aa704c8446e7]: django.request:WARNING: Bad Request: /pulp/api/v3/remotes/container/pull-through/
Oct 29 16:44:02 172.18.0.1 docker-pulp_api-2[2051]: ('pulp [3014e1050aab43a39116aa704c8446e7]: ::ffff:172.18.0.12 - admin [29/Oct/2024:21:44:02 +0000] "POST /pulp/api/v3/remotes/container/pull-through/ HTTP/1.0" 400 39 "-" "HTTPie/2.6.0"',)
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]: pulp [ea372ef5b7414c65af625adfe8d6c9f8]: django.request:ERROR: Internal Server Error: /pulp/api/v3/remotes/container/pull-through/
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]: Traceback (most recent call last):
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     response = get_response(request)
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     response = wrapped_callback(request, *callback_args, **callback_kwargs)
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     return view_func(*args, **kwargs)
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/rest_framework/viewsets.py", line 124, in view
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     return self.dispatch(request, *args, **kwargs)
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     response = self.handle_exception(exc)
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     self.raise_uncaught_exception(exc)
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     raise exc
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     response = handler(request, *args, **kwargs)
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/rest_framework/mixins.py", line 43, in list
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     return self.get_paginated_response(serializer.data)
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 795, in data
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     ret = super().data
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 249, in data
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     self._data = self.to_representation(self.instance)
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 713, in to_representation
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     return [
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 714, in <listcomp>
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     self.child.to_representation(item) for item in iterable
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 538, in to_representation
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     ret[field.field_name] = field.to_representation(attribute)
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/rest_framework/relations.py", line 396, in to_representation
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     url = self.get_url(value, self.view_name, request, format)
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/pulpcore/app/serializers/base.py", line 137, in get_url
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     view_name = self._view_name(obj)
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/pulpcore/app/serializers/base.py", line 125, in _view_name
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     return get_view_name_for_model(obj.get_model_for_pulp_type(obj.pulp_type), "detail")
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:   File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/base.py", line 149, in get_model_for_pulp_type
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]:     return cls._pulp_model_map[pulp_type]
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]: KeyError: 'container.remote'
Oct 29 16:44:28 172.18.0.1 docker-pulp_api-2[2051]: ('pulp [ea372ef5b7414c65af625adfe8d6c9f8]: ::ffff:172.18.0.12 - admin [29/Oct/2024:21:44:28 +0000] "GET /pulp/api/v3/remotes/container/pull-through/ HTTP/1.0" 500 145 "-" "HTTPie/2.6.0"',)

The same 500 error happens during a replicate tasks also.

To Reproduce
n/a

Expected behavior
The container plugin should function as designed.

Additional context
I did not make any changes leading up to this error. I saw it on core 3.63.0 and updated to 3.66.0 to see if it went away but it did not.
@lubosmj
Copy link
Member

lubosmj commented Oct 30, 2024 •
edited
Loading

Was this working before? Did you upgrade to a newer version of pulpcore recently? Did you try to replicate the pull-through remotes and distributions?

@lubosmj
Copy link
Member

lubosmj commented Oct 30, 2024

I wonder how did you end up in this state. I have checked out the main branch from pulpcore and pulp_container and the listing seems to be working fine on a fresh installation:

(venv3) [lmjachky@lmjachky-thinkpadt14gen4 pulpcore]$ http :5001/pulp/api/v3/distributions/container/pull-through/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, POST, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 52
Content-Type: application/json
Correlation-ID: 5dab5a3052574a22a1128ff2372f2127
Cross-Origin-Opener-Policy: same-origin
Date: Wed, 30 Oct 2024 09:20:35 GMT
Referrer-Policy: same-origin
Server: nginx/1.22.1
Vary: Accept
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "count": 0,
    "next": null,
    "previous": null,
    "results": []
}


(venv3) [lmjachky@lmjachky-thinkpadt14gen4 pulpcore]$ http :5001/pulp/api/v3/remotes/container/pull-through/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, POST, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 52
Content-Type: application/json
Correlation-ID: 5ae9a1b76e954d44a4258898799181f1
Cross-Origin-Opener-Policy: same-origin
Date: Wed, 30 Oct 2024 09:20:40 GMT
Referrer-Policy: same-origin
Server: nginx/1.22.1
Vary: Accept
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "count": 0,
    "next": null,
    "previous": null,
    "results": []
}

@PotentialIngenuity
Copy link
Author

PotentialIngenuity commented Oct 30, 2024 •
edited
Loading

I can see we had a successful replication a week ago. I just found it in a broken state yesterday because I went to update pulp itself. It was at 3.63.0 and went to 3.66.0. Both versions were in this broken state. I do see the pull through assets on the replica.

I was able to list the container repos in the catalog. The list is the same between the source and the replica.

{
    "repositories":[
        "docker-cache/balabit/syslog-ng"
        "docker-cache/library/alpine"
        "docker-cache/library/balabit/syslog-ng"
        "docker-cache/library/busybox"
        "docker-cache/library/centos"
        "docker-cache/library/golang"
        "docker-cache/library/hello-world"
        "docker-cache/library/nginx"
        "docker-cache/library/postgres"
        "docker-cache/library/postgres-backup-local"
        "docker-cache/library/prodrigestivill/postgres-backup-local"
        "docker-cache/library/pulp/pulp-minimal"
        "docker-cache/library/pulp/pulp-web"
        "docker-cache/library/rabbitmq"
        "docker-cache/library/redis"
        "docker-cache/library/rockylinux"
        "docker-cache/postgres"
        "docker-cache/prodrigestivill/postgres-backup-local"
        "docker-cache/pulp/pulp-minimal"
        "docker-cache/pulp/pulp-web"
        "docker-cache/redis"
        "docker-cache/zabbix/rabbitmq"
        "docker-cache/zabbix/zabbix-agent"
        "quay-cache/ansible/awx"
        "quay-cache/ansible/awx-ee"
        "quay-cache/ansible/awx-manager"
        "quay-cache/ansible/awx-operator"
        "quay-cache/ansible/awx-task"
        "quay-cache/library/ansible/awx"
        "quay-cache/library/awx"
    ]
}

Interestingly I can pull any container from both source/replica that is not a pulp container.

❯ skopeo list-tags docker://registry.cobaltiron.com/docker-cache/library/pulp/pulp-web
{
    "Repository": "registry.cobaltiron.com/docker-cache/library/pulp/pulp-web",
    "Tags": []
}
❯ skopeo list-tags docker://registry.cobaltiron.com/docker-cache/pulp/pulp-web
FATA[0002] Error listing repository tags: fetching tags list: name unknown: Repository not found.

I was able to get the pulp containers working by deleting the remotes/distributions and then pulling them again.

docker pull registry.company.com/docker-cache/pulp/pulp-web:3.66.0
3.66.0: Pulling from docker-cache/pulp/pulp-web
b2cc5146c9c7: Already exists
782ee99b4ec9: Already exists
28c6947581d3: Already exists
f622eee2e866: Already exists
02ada972a7c9: Already exists
0f0aa4efd0d2: Already exists
346b0e450128: Already exists
fe083a31111c: Already exists
d7e5fb241397: Pull complete
Digest: sha256:e4c880250611e2b867775e1c2c9bb20ca6ed06a033fc25a50f4ccff1340011c9
Status: Downloaded newer image for registry.company.com/docker-cache/pulp/pulp-web:3.66.0
registry.company.com/docker-cache/pulp/pulp-web:3.66.0

I want to try to delete the remote/distribution and recreate them for the docker-cache pull-through but I cant list it to get the href.

@PotentialIngenuity
Copy link
Author

Side question: why do some repositories have library in the name and some do not. Which one should be used to pull?

@PotentialIngenuity
Copy link
Author

Any ideas on how to fix the KeyError: 'container.remote' error? This is still happening on 3.68.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Issue Triage-Needed
Projects
Pulp Container Roadmap
Status: Not Started
Milestone
No milestone
Development

No branches or pull requests
2 participants
@PotentialIngenuity @lubosmj