1

my GitLab CI job build a example container image with Kaniko with the GitLab runner on K8s. All services are in my LAN behind a static IP. The build process pulls an ArchLinux image from my first Nexus registry service under secure domain A. Then I try do push it to the same Nexus registry into a second snapshot repo under a second domain B. Therefor I use the same Nexus user. If the job try to push, it results in the following error:

$ /kaniko/executor  --context $CI_PROJECT_DIR  --dockerfile $CI_PROJECT_DIR/Dockerfile  --destination ${SNAPSHOT_CONTAINER_REGISTRY}/${IMAGE_NAME}

error checking push permissions -- make sure you entered the correct tag name, 
and that you are authenticated correctly, and try again: 
checking push permission for "B/example-app:dev-proto-kokosbot": 
POST https://B/v2/example-app/blobs/uploads/: 
UNAUTHORIZED: access to the requested resource is not authorized

Even if I grant the cicd user admin rights I can not push over kaniko. With the docker CLI pushing is no problem and I can reach the URL of the error in the browser and it works...

My CI config:

stages:
  - build

variables:
  COMMIT_SHORT_SHA: ${CI_COMMIT_SHA} at ${CI_COMMIT_BRANCH}
  IMAGE_NAME: example-app:${CI_COMMIT_BRANCH}
  
build-job:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo [INFO] Start building container ${IMAGE_NAME}
    - echo [INFO] Target container registry ${SNAPSHOT_CONTAINER_REGISTRY}
    - mkdir -p /kaniko/.docker
    - |
      echo "{\"auths\":{
        \"${CONTAINER_REGISTRY}\":{
          \"auth\":\"${AUTH_REGISTRY}\"
          }
        },
        \"${SNAPSHOT_CONTAINER_REGISTRY}\":{
          \"auth\":\"${AUTH_REGISTRY}\"
          }
        }        
      }" > /kaniko/.docker/config.json
    - less /kaniko/.docker/config.json
    - >
      /kaniko/executor 
      --context $CI_PROJECT_DIR 
      --dockerfile $CI_PROJECT_DIR/Dockerfile 
      --destination ${SNAPSHOT_CONTAINER_REGISTRY}/${IMAGE_NAME}
      --no-push
    - >
      /kaniko/executor 
      --context $CI_PROJECT_DIR 
      --dockerfile $CI_PROJECT_DIR/Dockerfile 
      --destination ${SNAPSHOT_CONTAINER_REGISTRY}/${IMAGE_NAME}

I have no idea where to start to understand the problem, does anyone have a clue?

Greetings, Kokos Bot.

  • GitLab CE
  • Nexus Repository Manager: Free editon

EDIT: Do I understand it correctly that simply the user is not recognized? The user is called CICD and not *UNKOWN.

Nexus log:

nexus_1  | 2021-02-11 06:48:20,719+0000 INFO  [qtp874955081-232] *UNKNOWN org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Enabling session validation scheduler...
nexus_1  | 2021-02-11 06:48:20,739+0000 INFO  [qtp874955081-232] *UNKNOWN org.sonatype.nexus.internal.security.anonymous.AnonymousManagerImpl - Loaded configuration: OrientAnonymousConfiguration{enabled=false, userId='anonymous', realmName='NexusAuthorizingRealm'}
nexus_1  | 2021-02-11 06:48:20,741+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Servicing: GET /repository/dio-container-snapshot/v2/ (https://snapshot.container.ABC.ABC/repository/dio-container-snapshot/v2/)
nexus_1  | 2021-02-11 06:48:20,742+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Parsed path: RepositoryPath{repositoryName='dio-container-snapshot', remainingPath='/v2/'}
nexus_1  | 2021-02-11 06:48:20,742+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Looking for repository: dio-container-snapshot
nexus_1  | 2021-02-11 06:48:20,743+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Repository: RepositoryImpl$$EnhancerByGuice$$765e645e{type=hosted, format=docker, name='dio-container-snapshot'}
nexus_1  | 2021-02-11 06:48:20,743+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Dispatching to view facet: org.sonatype.nexus.repository.view.ConfigurableViewFacet$$EnhancerByGuice$$d744b4fe@6a0c93a4
nexus_1  | 2021-02-11 06:48:20,745+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.HttpResponseSenderSelector - Looking for HTTP response sender: docker
nexus_1  | 2021-02-11 06:48:20,747+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Router - Request: Request{action='GET', path='/v2/', parameters={}, payload=HttpRequestPayloadAdapter{contentType='null', size=-1}, multipart=false}
nexus_1  | 2021-02-11 06:48:20,748+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.logic.AndMatcher - Matching: org.sonatype.nexus.repository.view.matchers.ActionMatcher@2503258 AND LiteralMatcher{literal='/v2/', ignoreCase=true}
nexus_1  | 2021-02-11 06:48:20,748+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.LiteralMatcher - Matching: /v2/=/v2/ ignore-case: true
nexus_1  | 2021-02-11 06:48:20,749+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Starting: Route{matcher=AndMatcher{matchers=[org.sonatype.nexus.repository.view.matchers.ActionMatcher@2503258, LiteralMatcher{literal='/v2/', ignoreCase=true}]}, handlers=[org.sonatype.nexus.repository.view.handlers.TimingHandler@38a76c9d, org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1@64b09cb9, org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18@60e34347, org.sonatype.nexus.repository.security.SecurityHandler@63226ff1, org.sonatype.nexus.repository.view.handlers.HandlerContributor@60e2348c, org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure2@57297d]}
nexus_1  | 2021-02-11 06:48:20,749+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.view.handlers.TimingHandler@38a76c9d
nexus_1  | 2021-02-11 06:48:20,749+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1@64b09cb9
nexus_1  | 2021-02-11 06:48:20,752+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18@60e34347
nexus_1  | 2021-02-11 06:48:20,752+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.security.SecurityHandler@63226ff1
nexus_1  | 2021-02-11 06:48:20,800+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Router - Response: Response{status=Status{successful=false, code=401, message='null'}, payload=StringPayload{size=113, charset=UTF-8, contentType='application/json'}}
nexus_1  | 2021-02-11 06:48:20,801+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Request: Request{action='GET', path='/v2/', parameters={}, payload=HttpRequestPayloadAdapter{contentType='null', size=-1}, multipart=false}
nexus_1  | 2021-02-11 06:48:20,801+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.DefaultHttpResponseSender - Sending response: Response{status=Status{successful=false, code=401, message='null'}, payload=StringPayload{size=113, charset=UTF-8, contentType='application/json'}}
nexus_1  | 2021-02-11 06:48:20,806+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Service completed
nexus_1  | 2021-02-11 06:48:20,820+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Servicing: POST /repository/dio-container-snapshot/v2/example-app/blobs/uploads/ (https://snapshot.container.ABC.ABC/repository/dio-container-snapshot/v2/example-app/blobs/uploads/)
nexus_1  | 2021-02-11 06:48:20,821+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Parsed path: RepositoryPath{repositoryName='dio-container-snapshot', remainingPath='/v2/example-app/blobs/uploads/'}
nexus_1  | 2021-02-11 06:48:20,821+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Looking for repository: dio-container-snapshot
nexus_1  | 2021-02-11 06:48:20,821+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Repository: RepositoryImpl$$EnhancerByGuice$$765e645e{type=hosted, format=docker, name='dio-container-snapshot'}
nexus_1  | 2021-02-11 06:48:20,821+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Dispatching to view facet: org.sonatype.nexus.repository.view.ConfigurableViewFacet$$EnhancerByGuice$$d744b4fe@6a0c93a4
nexus_1  | 2021-02-11 06:48:20,822+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.HttpResponseSenderSelector - Looking for HTTP response sender: docker
nexus_1  | 2021-02-11 06:48:20,822+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Router - Request: Request{action='POST', path='/v2/example-app/blobs/uploads/', parameters={}, payload=HttpRequestPayloadAdapter{contentType='application/json', size=0}, multipart=false}
nexus_1  | 2021-02-11 06:48:20,822+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.logic.AndMatcher - Matching: org.sonatype.nexus.repository.view.matchers.ActionMatcher@2503258 AND LiteralMatcher{literal='/v2/', ignoreCase=true}
nexus_1  | 2021-02-11 06:48:20,823+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.logic.AndMatcher - Matching: org.sonatype.nexus.repository.view.matchers.ActionMatcher@2fb965f0 AND OrMatcher{matchers=[org.sonatype.nexus.repository.view.matchers.token.TokenMatcher@6264781a, org.sonatype.nexus.repository.view.matchers.token.TokenMatcher@3679c546]}
nexus_1  | 2021-02-11 06:48:20,823+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.logic.AndMatcher - Matching: org.sonatype.nexus.repository.view.matchers.ActionMatcher@291c9924 AND OrMatcher{matchers=[org.sonatype.nexus.repository.view.matchers.token.TokenMatcher@4deeef12, org.sonatype.nexus.repository.view.matchers.token.TokenMatcher@c0bea1d]}
nexus_1  | 2021-02-11 06:48:20,823+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.logic.AndMatcher - Matching: org.sonatype.nexus.repository.view.matchers.ActionMatcher@74a04d2f AND org.sonatype.nexus.repository.view.matchers.token.TokenMatcher@39b83f26
nexus_1  | 2021-02-11 06:48:20,823+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.logic.AndMatcher - Matching: org.sonatype.nexus.repository.view.matchers.ActionMatcher@314d17c AND org.sonatype.nexus.repository.view.matchers.token.TokenMatcher@501c8801
nexus_1  | 2021-02-11 06:48:20,824+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.token.TokenMatcher - Matching: /v2/example-app/blobs/uploads/~=TokenParser{pattern=\Q/v2/\E(?<name>.+)\Q/blobs/uploads/\E, variables=[var(name,.+)]}
nexus_1  | 2021-02-11 06:48:20,825+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Starting: Route{matcher=AndMatcher{matchers=[org.sonatype.nexus.repository.view.matchers.ActionMatcher@314d17c, org.sonatype.nexus.repository.view.matchers.token.TokenMatcher@501c8801]}, handlers=[org.sonatype.nexus.repository.view.handlers.TimingHandler@38a76c9d, org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1@64b09cb9, org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18@60e34347, org.sonatype.nexus.repository.security.SecurityHandler@63226ff1, org.sonatype.nexus.repository.view.handlers.HandlerContributor@60e2348c, org.sonatype.nexus.repository.storage.UnitOfWorkHandler@7ba0816b, org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure3@4f48429e]}
nexus_1  | 2021-02-11 06:48:20,825+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.view.handlers.TimingHandler@38a76c9d
nexus_1  | 2021-02-11 06:48:20,825+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1@64b09cb9
nexus_1  | 2021-02-11 06:48:20,825+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18@60e34347
nexus_1  | 2021-02-11 06:48:20,826+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.security.SecurityHandler@63226ff1
nexus_1  | 2021-02-11 06:48:20,827+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Router - Response: Response{status=Status{successful=false, code=401, message='null'}, payload=StringPayload{size=113, charset=UTF-8, contentType='application/json'}}
nexus_1  | 2021-02-11 06:48:20,827+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Request: Request{action='POST', path='/v2/example-app/blobs/uploads/', parameters={}, payload=HttpRequestPayloadAdapter{contentType='application/json', size=0}, multipart=false}
nexus_1  | 2021-02-11 06:48:20,828+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.DefaultHttpResponseSender - Sending response: Response{status=Status{successful=false, code=401, message='null'}, payload=StringPayload{size=113, charset=UTF-8, contentType='application/json'}}
nexus_1  | 2021-02-11 06:48:20,828+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Service completed
Kokos Bot
  • 63
  • 1
  • 6
  • As best I can tell, [`--destination accepts my.registry/my:tag`](https://github.com/GoogleContainerTools/kaniko/blob/v1.5.0/examples/pod.yaml#L11) but [`"auths":{}` wants "https://my.registry/whatever"](https://github.com/GoogleContainerTools/kaniko/tree/v1.5.0#pushing-to-docker-hub) so your `echo` needs to be updated to reference the URL, which will not be the same format as `--destination`; failing that, some [`--verbosity` increase](https://github.com/GoogleContainerTools/kaniko/tree/v1.5.0#--verbosity) is always helpful – mdaniel Feb 11 '21 at 05:12
  • Thank you for your help. I tried it out, but no positive result. Even with trace verbosity no further information are shown by kaniko. But I learned how to tweak logging in Nexus and got an error when the Pipeline try to push. I edited my post. – Kokos Bot Feb 11 '21 at 06:34
  • Ok it was no error. – Kokos Bot Feb 11 '21 at 06:46

1 Answers1

1

I have found the problem. There is one faulty bracket in the Docker config. Unfortunately with the environment variables there are many brackets. Therefore this is not easy noticeable.

The more astonishing it is that Kaniko, with a verbosity of tracing, does not spit out an error...

Kokos Bot
  • 63
  • 1
  • 6