TektonHub CR not able to clone #2072

Elyytscha · 2024-03-23T14:09:49Z

Expected Behavior

Creates a working Tektonhub with the default tekton catalog

Actual Behavior

Fails with cloning the default tekton catalog repo

Steps to Reproduce the Problem

kubectl apply -f tektonhub.yml

apiVersion: operator.tekton.dev/v1alpha1
kind: TektonHub
metadata:
  name: hub
spec:
  api:
    catalogRefreshInterval: 30m
    secret: tekton-hub-api
  customLogo: {}
  db: {}
  default: {}
  options:
    disabled: false
  targetNamespace: tekton-pipelines

this will create the following configmap

# Secret: name: tekton-hub-api
kind: ConfigMap
apiVersion: v1
metadata:
  name: tekton-hub-api
  namespace: tekton-pipelines
  labels:
    app: tekton-hub-api
    operator.tekton.dev/operand-name: tektoncd-hub
data:
  CATALOGS: |
    - name: tekton
      org: tektoncd
      type: community
      provider: github
      url: https://github.com/tektoncd/catalog
      revision: main
  CATALOG_REFRESH_INTERVAL: 30m
  CATEGORIES: |
    - Automation
    - Build Tools
    - CLI
    - Cloud
    - Code Quality
    - Continuous Integration
    - Deployment
    - Developer Tools
    - Image Build
    - Integration & Delivery
    - Git
    - Kubernetes
    - Messaging
    - Monitoring
    - Networking
    - Openshift
    - Publishing
    - Security
    - Storage
    - Testing
  DEFAULT: |
    scopes:
      - rating:read
      - rating:write
  SCOPES: ''

which will result in the following error

{"level":"info","ts":1711167219.9637597,"caller":"git/git.go:49","msg":"cloning","service":"git","name":"git","path":"/tmp/catalog/tekton"}
{"level":"error","ts":1711167220.0836065,"caller":"git/git.go:149","msg":"git fetch --recurse-submodules=yes origin main : error exit status 128 ;output: ","service":"git","name":"git","stacktrace":"github.com/tektoncd/hub/api/pkg/git.git\n\t/go/src/github.com/tektoncd/hub/api/pkg/git/git.go:149\ngithub.com/tektoncd/hub/api/pkg/git.(*client).Fetch\n\t/go/src/github.com/tektoncd/hub/api/pkg/git/git.go:59\ngithub.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Process\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:235\ngithub.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Run.func1\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:105"}
{"level":"error","ts":1711167220.0947776,"caller":"git/git.go:149","msg":"git pull --recurse-submodules=yes origin : error exit status 1 ;output: ","service":"git","name":"git","stacktrace":"github.com/tektoncd/hub/api/pkg/git.git\n\t/go/src/github.com/tektoncd/hub/api/pkg/git/git.go:149\ngithub.com/tektoncd/hub/api/pkg/git.(*client).Fetch\n\t/go/src/github.com/tektoncd/hub/api/pkg/git/git.go:62\ngithub.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Process\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:235\ngithub.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Run.func1\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:105"}
{"level":"info","ts":1711167220.0948365,"caller":"git/git.go:63","msg":"Failed to pull originerrexit status 1","service":"git","name":"git"}
{"level":"error","ts":1711167220.0962138,"caller":"git/git.go:149","msg":"git checkout main : error exit status 1 ;output: ","service":"git","name":"git","stacktrace":"github.com/tektoncd/hub/api/pkg/git.git\n\t/go/src/github.com/tektoncd/hub/api/pkg/git/git.go:149\ngithub.com/tektoncd/hub/api/pkg/git.(*client).Fetch\n\t/go/src/github.com/tektoncd/hub/api/pkg/git/git.go:65\ngithub.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Process\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:235\ngithub.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Run.func1\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:105"}
{"level":"error","ts":1711167220.0962665,"caller":"catalog/syncer.go:237","msg":"exit status 1clone failed","service":"syncer","action":"process","stacktrace":"github.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Process\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:237\ngithub.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Run.func1\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:105"}
{"level":"info","ts":1711167220.1104074,"caller":"app/gorm_logger.go:76","msg":"UPDATE \"sync_jobs\" SET \"status\"='error',\"updated_at\"='2024-03-23 04:13:40.096' WHERE \"sync_jobs\".\"deleted_at\" IS NULL AND \"id\" = 1","service":"syncer","file":"/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:214","duration":"[14.027ms]","rows":1}

Additional Info

Kubernetes version:

Output of kubectl version:

$ oc version
Client Version: v4.2.0-alpha.0-2008-g286cfa5
Kustomize Version: v5.0.1
Server Version: 4.15.0-0.okd-2024-03-10-010116
Kubernetes Version: v1.28.2-3598+6e2789bbd58938-dirty

Tekton Pipeline version:

v0.53.3
Tekton Operator version:

v0.69.1

installed via OLM and operatorhub

The text was updated successfully, but these errors were encountered:

piyush-garg · 2024-04-25T12:15:29Z

@PuneetPunamiya please take a look at this

PuneetPunamiya · 2024-05-16T11:12:55Z

@Elyytscha I tried to reproduce this issue, but was not able to reproduce it. All pods were up and running.

Also seeing your error it seems like it was a network issue error. Can you try it once if you can still reproduce it ?

Elyytscha · 2024-05-27T19:01:26Z

I can still reproduce it:

apiVersion: operator.tekton.dev/v1alpha1
kind: TektonHub
metadata:
  name: hub
spec:
  api:
    catalogRefreshInterval: 30m
    secret: tekton-hub-api
  customLogo: {}
  db: {}
  default: {}
  options:
    disabled: false
  targetNamespace: tekton-pipelines

{"level":"info","ts":1716835976.1408136,"caller":"catalog/syncer.go:104","msg":"processing the queue","service":"syncer","action":"run"}
{"level":"info","ts":1716835976.142004,"caller":"app/gorm_logger.go:76","msg":"SELECT * FROM \"sync_jobs\" WHERE \"sync_jobs\".\"status\" = 'queued' AND \"sync_jobs\".\"deleted_at\" IS NULL ORDER BY created_at,\"sync_jobs\".\"id\" LIMIT 1","service":"syncer","file":"/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:217","duration":"[1.089ms]","rows":1}
{"level":"info","ts":1716835976.154482,"caller":"app/gorm_logger.go:76","msg":"UPDATE \"sync_jobs\" SET \"status\"='running',\"updated_at\"='2024-05-27 18:52:56.142' WHERE \"sync_jobs\".\"deleted_at\" IS NULL AND \"id\" = 1","service":"syncer","file":"/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:214","duration":"[12.366ms]","rows":1}
{"level":"info","ts":1716835976.1553483,"caller":"app/gorm_logger.go:76","msg":"SELECT * FROM \"catalogs\" WHERE \"catalogs\".\"id\" = 1 AND \"catalogs\".\"deleted_at\" IS NULL","service":"syncer","file":"/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:229","duration":"[0.716ms]","rows":1}
{"level":"info","ts":1716835976.155447,"caller":"git/git.go:49","msg":"cloning","service":"git","name":"git","path":"/tmp/catalog/tekton"}
{"level":"error","ts":1716835976.2589736,"caller":"git/git.go:149","msg":"git fetch --recurse-submodules=yes origin main : error exit status 128 ;output: ","service":"git","name":"git","stacktrace":"github.com/tektoncd/hub/api/pkg/git.git\n\t/go/src/github.com/tektoncd/hub/api/pkg/git/git.go:149\ngithub.com/tektoncd/hub/api/pkg/git.(*client).Fetch\n\t/go/src/github.com/tektoncd/hub/api/pkg/git/git.go:59\ngithub.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Process\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:235\ngithub.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Run.func1\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:105"}
{"level":"error","ts":1716835976.2688093,"caller":"git/git.go:149","msg":"git pull --recurse-submodules=yes origin : error exit status 1 ;output: ","service":"git","name":"git","stacktrace":"github.com/tektoncd/hub/api/pkg/git.git\n\t/go/src/github.com/tektoncd/hub/api/pkg/git/git.go:149\ngithub.com/tektoncd/hub/api/pkg/git.(*client).Fetch\n\t/go/src/github.com/tektoncd/hub/api/pkg/git/git.go:62\ngithub.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Process\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:235\ngithub.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Run.func1\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:105"}
{"level":"info","ts":1716835976.2688737,"caller":"git/git.go:63","msg":"Failed to pull originerrexit status 1","service":"git","name":"git"}
{"level":"error","ts":1716835976.2704506,"caller":"git/git.go:149","msg":"git checkout main : error exit status 1 ;output: ","service":"git","name":"git","stacktrace":"github.com/tektoncd/hub/api/pkg/git.git\n\t/go/src/github.com/tektoncd/hub/api/pkg/git/git.go:149\ngithub.com/tektoncd/hub/api/pkg/git.(*client).Fetch\n\t/go/src/github.com/tektoncd/hub/api/pkg/git/git.go:65\ngithub.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Process\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:235\ngithub.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Run.func1\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:105"}
{"level":"error","ts":1716835976.2705216,"caller":"catalog/syncer.go:237","msg":"exit status 1clone failed","service":"syncer","action":"process","stacktrace":"github.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Process\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:237\ngithub.com/tektoncd/hub/api/pkg/service/catalog.(*syncer).Run.func1\n\t/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:105"}
{"level":"info","ts":1716835976.2864943,"caller":"app/gorm_logger.go:76","msg":"UPDATE \"sync_jobs\" SET \"status\"='error',\"updated_at\"='2024-05-27 18:52:56.27' WHERE \"sync_jobs\".\"deleted_at\" IS NULL AND \"id\" = 1","service":"syncer","file":"/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:214","duration":"[15.865ms]","rows":1}
{"level":"info","ts":1716835976.2876918,"caller":"app/gorm_logger.go:76","msg":"SELECT count(*) FROM \"sync_jobs\" WHERE \"sync_jobs\".\"status\" = 'queued' AND \"sync_jobs\".\"deleted_at\" IS NULL","service":"syncer","file":"/go/src/github.com/tektoncd/hub/api/pkg/service/catalog/syncer.go:186","duration":"[1.065ms]","rows":1}
{"level":"info","ts":1716835976.2877243,"caller":"catalog/syncer.go:191","msg":"queued job count: 0","service":"syncer","action":"next"}

PuneetPunamiya · 2024-06-25T06:53:42Z

Have you upgraded Hub from previous version or did you fresh install Hub ?

Elyytscha · 2024-06-25T15:32:11Z

i freshly installed the hub, it happens everyime i want to install the hub from scratch

PuneetPunamiya · 2024-06-26T08:19:15Z

I tried this again by fresh installing hub, i don't see any issue
I think you have upgraded from a previous version of operator

Elyytscha · 2024-06-26T13:25:46Z

I can confirm with Tekton Operator v0.70.0 its now getting past this step, now it syncs the task repo successfully as I interpret the logs. but it still shows:

should I open a new issue regarding this?

i attached the logs for now over here
tekton-hub-api-7c8f95779-np5h2-tekton-hub-api.log
tekton-hub-ui-5fb499c944-5g7rk-tekton-hub-ui.log

PuneetPunamiya · 2024-06-26T13:41:50Z

Can you hit this url and check if you can see the resources in the api
<API_URL>/v1/resources

Elyytscha · 2024-06-26T15:37:34Z

yes I can, but I think due to your last comment I found the issue: API_URL is empty

Modified ENV
window.config = {
  API_URL: '',
  API_VERSION: 'v1',
  AUTH_BASE_URL: '',
  REDIRECT_URI: '',
  CUSTOM_LOGO_MEDIA_TYPE: '',
  CUSTOM_LOGO_BASE64_DATA: '',
};

https://github.com/tektoncd/hub/blob/main/docs/DEPLOYMENT.md#update-ui-configmap
Could you point me to an example or the docs how its possible to set the API_URL in the TektonHub CR? I think then we can close the issue.

PuneetPunamiya · 2024-07-01T08:38:31Z

If you are installing Hub on openshift in that case the API_URL, AUTH_BASE_URL and REDIRECT_URL should be set by the operator only, can you see the values in configMap i.e. tekton-hub-ui configMap

And if in case you are using it on other k8s cluster then for that we might have to add a support for that where you can provide the url and UI will be served on that url

PuneetPunamiya · 2024-07-01T08:43:41Z

As of today you can add these fields only in the Hub CR https://github.com/tektoncd/operator/blob/main/docs/TektonHub.md

Elyytscha · 2024-07-03T11:06:01Z

If you are installing Hub on openshift in that case the API_URL, AUTH_BASE_URL and REDIRECT_URL should be set by the operator only, can you see the values in configMap i.e. tekton-hub-ui configMap

And if in case you are using it on other k8s cluster then for that we might have to add a support for that where you can provide the url and UI will be served on that url

I use OKD with the original tekton operator, because the tekton operator for openshift is not available on okd.

so i think thats why its not working, i also have to manually create the ingress for UI and API

As of today you can add these fields only in the Hub CR https://github.com/tektoncd/operator/blob/main/docs/TektonHub.md

i can't see where i can configure there the URL, do you mean i can write the API_URL into the db secret?

PuneetPunamiya · 2024-07-03T11:17:48Z

i can't see where i can configure there the URL, do you mean i can write the API_URL into the db secret?

What I meant was as of today it is not supported to add the API_URL

Elyytscha · 2024-07-03T11:52:46Z

i can't see where i can configure there the URL, do you mean i can write the API_URL into the db secret?

What I meant was as of today it is not supported to add the API_URL

Should we open a Bug ticket for this? Because as I see this, without this set, the hub cr is basically unuseable.

We could also think about automatically creating the ingress for API and UI so the API_URL could then be set based on the created ingress

PuneetPunamiya · 2024-07-03T11:56:13Z

Yeah, that makes sense, would you mind to create another issue for that

error418 · 2024-08-27T18:46:23Z

I am also running into this issue when trying to clone a repository using a ssh key.
To gain more information about the issue I exec'd into the container and tried to pull the custom catalog from the command line

$ cd /tmp/catalog/tekton
$ git pull
No user exists for uid 65532
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists

Do not get misdirected by the output claiming to check your access rights. The error seems to be triggered by the missing uid in the image, causing ssh to exit abnormally.

Conclusion

I am suspecting that the missing uid 65532 in the tekton-hub-api container causes the cloning issues.

Solution

One of the two

set user id 65532 for the user hub in the tekton-hub-api image
operator needs to set correct uid used by the image when creating the manifest

Edit

I just confirmed this issue by u0-execing into the container, creating a user with the missing uid and successfully pulling the repository in /tmp/catalog/tekton

Elyytscha added the kind/bug Categorizes issue or PR as related to a bug. label Mar 23, 2024

piyush-garg assigned PuneetPunamiya Apr 25, 2024

jkandasa mentioned this issue May 27, 2024

log rawGit execution error output tektoncd/hub#1731

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TektonHub CR not able to clone #2072

TektonHub CR not able to clone #2072

Elyytscha commented Mar 23, 2024 •

edited

Loading

piyush-garg commented Apr 25, 2024

PuneetPunamiya commented May 16, 2024

Elyytscha commented May 27, 2024

PuneetPunamiya commented Jun 25, 2024

Elyytscha commented Jun 25, 2024

PuneetPunamiya commented Jun 26, 2024

Elyytscha commented Jun 26, 2024

PuneetPunamiya commented Jun 26, 2024

Elyytscha commented Jun 26, 2024 •

edited

Loading

PuneetPunamiya commented Jul 1, 2024

PuneetPunamiya commented Jul 1, 2024

Elyytscha commented Jul 3, 2024

PuneetPunamiya commented Jul 3, 2024

Elyytscha commented Jul 3, 2024

PuneetPunamiya commented Jul 3, 2024

error418 commented Aug 27, 2024 •

edited

Loading

TektonHub CR not able to clone #2072

TektonHub CR not able to clone #2072

Comments

Elyytscha commented Mar 23, 2024 • edited Loading

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Additional Info

piyush-garg commented Apr 25, 2024

PuneetPunamiya commented May 16, 2024

Elyytscha commented May 27, 2024

PuneetPunamiya commented Jun 25, 2024

Elyytscha commented Jun 25, 2024

PuneetPunamiya commented Jun 26, 2024

Elyytscha commented Jun 26, 2024

PuneetPunamiya commented Jun 26, 2024

Elyytscha commented Jun 26, 2024 • edited Loading

PuneetPunamiya commented Jul 1, 2024

PuneetPunamiya commented Jul 1, 2024

Elyytscha commented Jul 3, 2024

PuneetPunamiya commented Jul 3, 2024

Elyytscha commented Jul 3, 2024

PuneetPunamiya commented Jul 3, 2024

error418 commented Aug 27, 2024 • edited Loading

Conclusion

Solution

Edit

Elyytscha commented Mar 23, 2024 •

edited

Loading

Elyytscha commented Jun 26, 2024 •

edited

Loading

error418 commented Aug 27, 2024 •

edited

Loading