How to set environment variables for the CI/CD in GitLab when using Auto DevOps (with GCP Kubernetes)?

Question

I'm having quite a few moving parts here, so, I'm not sure what's wrong yet. I set up my variable this way:

but during the build phase, I get this failure (it's a Django application):

   raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
   django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

   !     Error while running '$ python manage.py collectstatic --noinput'.
   See traceback above for details.

   You may need to update application code to resolve this error.
   Or, you can disable collectstatic for this application:

   $ heroku config:set DISABLE_COLLECTSTATIC=1

   https://devcenter.heroku.com/articles/django-assets

In case there's a clue somewhere else in the UI that I'm missing, this is the output:

I created a public skeleton application to showcase and debug this problem: https://gitlab.com/pupeno/example/-/jobs/113402553

What am I missing to expose the environment variable to the build process?

I copied the Auto DevOps their .gitlab-ci.yml template and added:

variables:
  # ...
  SECRET_KEY: $SECRET_KEY

as well as hard coding it in case $SECRET_KEY wasn't working:

variables:
  # ...
  SECRET_KEY: "12345678901234567890123456789012345678901234567890"

Also, I added that to the build stage as variables:

build:
  stage: build
  image: docker:stable-git
  services:
  - docker:stable-dind
  script:
    - setup_docker
    - build
  only:
    - branches
  variables:
    SECRET_KEY: "12345678901234567890123456789012345678901234567890"

or export:

build:
  stage: build
  image: docker:stable-git
  services:
  - docker:stable-dind
  script:
    - export SECRET_KEY="12345678901234567890123456789012345678901234567890"
    - setup_docker
    - build
  only:
    - branches

and the variable is still not available to python manage.py.

How do I make it available?

The full log from the build is:

[0KRunning with gitlab-runner 11.4.0-rc1 (1ff344e1)
[0;m[0K  on docker-auto-scale fa6cab46
[0;m[0KUsing Docker executor with image docker:stable-git ...
[0;m[0KStarting service docker:stable-dind ...
[0;m[0KPulling docker image docker:stable-dind ...
[0;m[0KUsing docker image sha256:acfec978837639b4230111b35a775a67ccbc2b08b442c1ae2cca4e95c3e6d08a for docker:stable-dind ...
[0;m[0KWaiting for services to be up and running...
[0;m[0KPulling docker image docker:stable-git ...
[0;m[0KUsing docker image sha256:a8a2d0da40bc37344c35ab723d4081a5ef6122d466bf0a0409f742ffc09c43b9 for docker:stable-git ...
[0;msection_start:1540485937:prepare_script
[0KRunning on runner-fa6cab46-project-8991099-concurrent-0 via runner-fa6cab46-srm-1540485878-b0f2f392...
section_end:1540485939:prepare_script
[0Ksection_start:1540485939:get_sources
[0K[32;1mCloning repository...[0;m
Cloning into '/builds/eligibleai/phoenix'...
[32;1mChecking out 4753fd6d as master...[0;m
[32;1mSkipping Git submodules setup[0;m
section_end:1540485940:get_sources
[0Ksection_start:1540485940:restore_cache
[0Ksection_end:1540485942:restore_cache
[0Ksection_start:1540485942:download_artifacts
[0Ksection_end:1540485943:download_artifacts
[0Ksection_start:1540485943:build_script
[0K[32;1m$ # Auto DevOps variables and functions # collapsed multi-line command[0;m
[32;1m$ setup_docker[0;m
[32;1m$ build[0;m
Logging to GitLab Container Registry with CI credentials...
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

Building Heroku-based application using gliderlabs/herokuish docker image...
Unable to find image 'gliderlabs/herokuish:latest' locally
latest: Pulling from gliderlabs/herokuish
8ee29e426c26: Pulling fs layer
6e83b260b73b: Pulling fs layer
e26b65fd1143: Pulling fs layer
40dca07f8222: Pulling fs layer
b420ae9e10b3: Pulling fs layer
da73fdbc63c8: Pulling fs layer
06627042f798: Pulling fs layer
91591ee9e53a: Pulling fs layer
16bb228b9f74: Pulling fs layer
5ac34e15ac7b: Pulling fs layer
7ff3cba3e710: Pulling fs layer
8be7609fa05c: Pulling fs layer
28b563a2d870: Pulling fs layer
7d2ba817ef0b: Pulling fs layer
40dca07f8222: Waiting
b420ae9e10b3: Waiting
da73fdbc63c8: Waiting
06627042f798: Waiting
91591ee9e53a: Waiting
16bb228b9f74: Waiting
5ac34e15ac7b: Waiting
7ff3cba3e710: Waiting
8be7609fa05c: Waiting
28b563a2d870: Waiting
7d2ba817ef0b: Waiting
6e83b260b73b: Verifying Checksum
6e83b260b73b: Download complete
e26b65fd1143: Verifying Checksum
e26b65fd1143: Download complete
40dca07f8222: Verifying Checksum
40dca07f8222: Download complete
8ee29e426c26: Verifying Checksum
8ee29e426c26: Download complete
b420ae9e10b3: Verifying Checksum
b420ae9e10b3: Download complete
da73fdbc63c8: Verifying Checksum
da73fdbc63c8: Download complete
91591ee9e53a: Verifying Checksum
91591ee9e53a: Download complete
5ac34e15ac7b: Verifying Checksum
5ac34e15ac7b: Download complete
7ff3cba3e710: Verifying Checksum
7ff3cba3e710: Download complete
16bb228b9f74: Verifying Checksum
16bb228b9f74: Download complete
28b563a2d870: Verifying Checksum
28b563a2d870: Download complete
8be7609fa05c: Verifying Checksum
8be7609fa05c: Download complete
7d2ba817ef0b: Verifying Checksum
7d2ba817ef0b: Download complete
06627042f798: Verifying Checksum
06627042f798: Download complete
8ee29e426c26: Pull complete
6e83b260b73b: Pull complete
e26b65fd1143: Pull complete
40dca07f8222: Pull complete
b420ae9e10b3: Pull complete
da73fdbc63c8: Pull complete
06627042f798: Pull complete
91591ee9e53a: Pull complete
16bb228b9f74: Pull complete
5ac34e15ac7b: Pull complete
7ff3cba3e710: Pull complete
8be7609fa05c: Pull complete
28b563a2d870: Pull complete
7d2ba817ef0b: Pull complete
Digest: sha256:d1581d20408be7a7366b6ea74dc5cddc866f4ad4c573a04251272471c0aa2e56
Status: Downloaded newer image for gliderlabs/herokuish:latest
[1G       [1G-----> Python app detected
[1G-----> Installing python-3.6.6
[1G-----> Installing pip
[1G-----> Installing SQLite3
[1G-----> Installing requirements with pip
[1G       Collecting Django==2.1.2 (from -r /tmp/build/requirements.txt (line 1))
[1G       Downloading https://files.pythonhosted.org/packages/32/ab/22530cc1b2114e6067eece94a333d6c749fa1c56a009f0721e51c181ea53/Django-2.1.2-py3-none-any.whl (7.3MB)
[1G       Collecting django-dotenv==1.4.2 (from -r /tmp/build/requirements.txt (line 2))
[1G       Downloading https://files.pythonhosted.org/packages/e4/af/d9175f2b40a9bcd700db2861610d6ed48c2795ffba46c1d6abf25f7c1dea/django_dotenv-1.4.2-py2.py3-none-any.whl
[1G       Collecting psycopg2==2.7.5 (from -r /tmp/build/requirements.txt (line 3))
[1G       Downloading https://files.pythonhosted.org/packages/5e/d0/9e2b3ed43001ebed45caf56d5bb9d44ed3ebd68e12b87845bfa7bcd46250/psycopg2-2.7.5-cp36-cp36m-manylinux1_x86_64.whl (2.7MB)
[1G       Collecting pytz (from Django==2.1.2->-r /tmp/build/requirements.txt (line 1))
[1G       Downloading https://files.pythonhosted.org/packages/52/8b/876c5745f617630be90cfb8fafe363c6d7204b176dc707d1805d1e9a0a35/pytz-2018.6-py2.py3-none-any.whl (507kB)
[1G       Installing collected packages: pytz, Django, django-dotenv, psycopg2
[1G       Successfully installed Django-2.1.2 django-dotenv-1.4.2 psycopg2-2.7.5 pytz-2018.6
[1G       
[1G-----> $ python manage.py collectstatic --noinput
[1G       Environment variables:
[1G       CPLUS_INCLUDE_PATH: /app/.heroku/vendor/include:/app/.heroku/python/include:
[1G       HOSTNAME: 09751c55b3ac
[1G       EXPORT_PATH: /tmp/buildpacks/04_buildpack-python/bin/../export
[1G       CURL_CONNECT_TIMEOUT: 30
[1G       PROFILE_PATH: /tmp/build/.profile.d/python.sh
[1G       HEROKUISH_VERSION: 0.4.4
[1G       BUILDPACK_LOG_FILE: /dev/null
[1G       LIBRARY_PATH: /app/.heroku/vendor/lib:/app/.heroku/python/lib:
[1G       CACHE_DIR: /tmp/cache
[1G       LATEST_37: python-3.7.0
[1G       PYTHONUNBUFFERED: 1
[1G       LATEST_36: python-3.6.6
[1G       DEFAULT_PYTHON_STACK: cedar-14
[1G       LD_LIBRARY_PATH: /app/.heroku/vendor/lib:/app/.heroku/python/lib:
[1G       BIN_DIR: /tmp/buildpacks/04_buildpack-python/bin
[1G       PATH: /app/.heroku/python/bin:/app/.heroku/vendor/bin::/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tmp/buildpacks/04_buildpack-python/vendor/:/tmp/buildpacks/04_buildpack-python/vendor/pip-pop
[1G       ENV_DIR: /tmp/env
[1G       SELF: /bin/herokuish
[1G       RECOMMENDED_PYTHON_VERSION: python-3.6.6
[1G       C_INCLUDE_PATH: /app/.heroku/vendor/include:/app/.heroku/python/include:
[1G       APP_DIR: /app
[1G       PWD: /tmp/build
[1G       BPLOG_PREFIX: buildpack.python
[1G       LANG: en_US.UTF-8
[1G       SELF_EXECUTABLE: /bin/herokuish
[1G       CACHED_PYTHON_STACK: heroku-16
[1G       STACK: heroku-16
[1G       REQUEST_ID: build-16345
[1G       HOME: /app
[1G       SHLVL: 3
[1G       LATEST_2: python-2.7.15
[1G       SQLITE3_VERSION: 3.11.0-1ubuntu1
[1G       PIP_UPDATE: 9.0.2
[1G       PYTHONPATH: .
[1G       BUILD_DIR: /tmp/build
[1G       VENDOR_URL: https://lang-python.s3.amazonaws.com/heroku-16
[1G       WARNINGS_LOG: /tmp/tmp.Iy1gH0Xs0X
[1G       DEFAULT_PYTHON_VERSION: python-3.6.6
[1G       CURL_TIMEOUT: 180
[1G       DEBIAN_FRONTEND: noninteractive
[1G       PKG_CONFIG_PATH: /app/.heroku/vendor/lib/pkgconfig:/app/.heroku/vendor/lib/pkg-config:/app/.heroku/python/lib/pkg-config:
[1G       _: /app/.heroku/python/bin/python
[1G       Traceback (most recent call last):
[1G       File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 204, in fetch_command
[1G       app_name = commands[subcommand]
[1G       KeyError: 'collectstatic'
[1G       During handling of the above exception, another exception occurred:
[1G       Traceback (most recent call last):
[1G       File "manage.py", line 23, in <module>
[1G       execute_from_command_line(sys.argv)
[1G       File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
[1G       utility.execute()
[1G       File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
[1G       self.fetch_command(subcommand).run_from_argv(self.argv)
[1G       File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 211, in fetch_command
[1G       settings.INSTALLED_APPS
[1G       File "/app/.heroku/python/lib/python3.6/site-packages/django/conf/__init__.py", line 57, in __getattr__
[1G       self._setup(name)
[1G       File "/app/.heroku/python/lib/python3.6/site-packages/django/conf/__init__.py", line 44, in _setup
[1G       self._wrapped = Settings(settings_module)
[1G       File "/app/.heroku/python/lib/python3.6/site-packages/django/conf/__init__.py", line 126, in __init__
[1G       raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
[1G       django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.
[1G       
[1G       !     Error while running '$ python manage.py collectstatic --noinput'.
[1G       See traceback above for details.
[1G       
[1G       You may need to update application code to resolve this error.
[1G       Or, you can disable collectstatic for this application:
[1G       
[1G       $ heroku config:set DISABLE_COLLECTSTATIC=1
[1G       
[1G       https://devcenter.heroku.com/articles/django-assets
section_end:1540486016:build_script
[0Ksection_start:1540486016:after_script
[0Ksection_end:1540486017:after_script
[0Ksection_start:1540486017:upload_artifacts_on_failure
[0Ksection_end:1540486018:upload_artifacts_on_failure
[0K[31;1mERROR: Job failed: exit code 1
[0;m

Having the EXACT same problem here. Essentially we need to figure out a way to pass AutoDevOps this command before launching Heroku: `heroku config:set DJANGO_SECRET_KEY=$SECRET_KEY` — Rakaim, Oct 26 '18 at 02:24
@Rakaim: how did you do that? I'm new to GitLab and I'm not sure where that would be configured. — Pablo, Oct 26 '18 at 11:30

score 2 · Answer 1 · answered Nov 26 '18 at 04:20

I ran into a very similar issue on Gitlab Auto DevOps.

I was able to move past this issue by adding the following to the settings.py file.

STATIC_URL = '/static/'
PROJECT_DIR = os.path.dirname(os.path.abspath(__file__))
STATIC_ROOT = os.path.join(PROJECT_DIR, 'static')

Credit:

score 2 · Answer 2 · answered Sep 08 '19 at 11:58

In my case, I had hardcoded the secret key. I changed my code so that it would be read from .env file. I forgot to add the variable in docker-compose.yml like the rest of the variables. Even though I had added the SECRET_KEY variable in Gitlab CI/CD settings, it was not available in the web container. Have a look here: https://stackoverflow.com/questions/57835476/how-to-set-environmental-variables-properly-gitlab-ci-cd-and-docker

score 1 · Answer 3 · answered Jun 12 '19 at 20:07

I am running into the same issues. I appearently do not have enough reputation to comment so I will just answer...

I think this issue stems from the fact that the dind does not pass the environment from where it is running. The docker file then tries to build everything but django is missing the variables needed. I have not found a nice way to get this to work without hard coding the values in the settings. (The answer from Aaron Blythe is not related at all)

score 0 · Answer 4 · answered May 16 '22 at 23:10

This is an older question and I'm aware the functionality wasn't available at the time it was asked, but since this has remained unanswered and others may still come across the post while looking for a solution to their problem:

Version 12.3 of GitLab introduced forwarding of CI/CD variables into the build environment via a separate variable AUTO_DEVOPS_BUILD_IMAGE_FORWARDED_CI_VARIABLES.

So, if you wanted Heroku not to run collectstatic during the build process – the first error referenced in this post –, you would set DISABLE_COLLECTSTATIC to 1 in the CI/CD variables section of the repository and add AUTO_DEVOPS_BUILD_IMAGE_FORWARDED_CI_VARIABLES with the value DISABLE_COLLECTSTATIC as additional variable. See this section of the most recent GitLab docs for more details.

If, on the other hand, you wanted to make certain variables available to the Django project once it is running in your container, you would prefix them with K8S_SECRET_ to have them passed down to it. See this table on CI/CD variables.

How to set environment variables for the CI/CD in GitLab when using Auto DevOps (with GCP Kubernetes)?

4 Answers4