Repeated installation of a package inside docker image

Question

I built a python package called my-package. I have no intension to make it public so installation is mostly through our internal servers. Recently one senior developer built an architecture using docker where the application is hosted and my-package is a dependency.

The problem is in order to test the package, I REPEATEDLY need to COPY my code into docker image, then uninstall old version of package and re-install from the local code.

1. Rebuilding entire image again takes half an hour. - Not an option.
2. Create another Dockerfile FROM existing image and run only specific commands to COPY and install the pip package. - My current solution yet not very efficient.

I am pretty sure the docker users would have come across this issue so need an expert opinion on the most efficient way to handle this.

UPDATE: The Dockerfile

# VERSION 1.8.2
# AUTHOR: Matthieu "Puckel_" Roisil
# DESCRIPTION: Basic Airflow container
# BUILD: docker build --rm -t puckel/docker-airflow .
# SOURCE: https://github.com/puckel/docker-airflow

FROM ubuntu:17.10
MAINTAINER Puckel_

# Never prompts the user for choices on installation/configuration of packages
ENV DEBIAN_FRONTEND noninteractive
ENV TERM linux

# Airflow
ARG AIRFLOW_VERSION=1.8.9
ARG AIRFLOW_HOME=/usr/local/airflow

# Define en_US.
ENV LANGUAGE en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LC_CTYPE en_US.UTF-8
ENV LC_MESSAGES en_US.UTF-8
ENV LC_ALL en_US.UTF-8

ENV MATPLOTLIBRC /etc

RUN set -ex \
    && buildDeps=' \
        python3.6-dev \
        libkrb5-dev \
        libsasl2-dev \
        libssl-dev \
        libffi-dev \
        build-essential \
        libblas-dev \
        liblapack-dev \
        libpq-dev \
        git \
        wget \
    ' \
    && apt-get update -yqq \
    && apt-get dist-upgrade -yqq \
    && apt-get install -yqq --no-install-recommends \
        $buildDeps \
        python3.6 \
        python3.6-tk \
        apt-utils \
        curl \
        netcat \
        locales \
        ca-certificates \
        sudo \
        libmysqlclient-dev \
    && ln -s /usr/bin/python3.6 /usr/bin/python \
    && sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g' /etc/locale.gen \
    && locale-gen \
    && update-locale LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 \
    && useradd -ms /bin/bash -d ${AIRFLOW_HOME} -u 1500 airflow \
    && mkdir ${AIRFLOW_HOME}/logs \
    && wget https://bootstrap.pypa.io/get-pip.py \
    && python get-pip.py \
    && rm -rf get-pip.py \
    && python -m pip install Cython \
    && python -m pip install requests \
    && python -m pip install pytz \
    && python -m pip install pyOpenSSL \
    && python -m pip install ndg-httpsclient \
    && python -m pip install pyasn1 \
    && python -m pip install Flask-OAuthlib \
    && python -m pip install apache-airflow[crypto,celery,postgres,ldap,jdbc,mysql,s3,samba]==$AIRFLOW_VERSION \
    && python -m pip install celery[redis]==4.1.0 \
    && python -m pip install boto3 \
    && python -m pip install pymongo \
    && python -m pip install statsd \   
    && apt-get remove --purge -yqq $buildDeps \
    && apt-get clean \
    && rm -rf \
        /var/lib/apt/lists/* \
        /tmp/* \
        /var/tmp/* \
        /usr/share/man \
        /usr/share/doc \
        /usr/share/doc-base \
    && apt-get autoremove -yqq

The important part is in the end.

ARG CACHEBUST=1

COPY config/matplotlibrc /etc/matplotlibrc
COPY script/entrypoint.sh /entrypoint.sh
COPY script/shell.sh /shell.sh
COPY config/airflow.cfg ${AIRFLOW_HOME}/airflow.cfg

RUN chown -R airflow: ${AIRFLOW_HOME}

RUN pip install matplotlib seaborn xlsxwriter pandas Jinja2
#Add custom PIP repo - THIS IS OF INTEREST
COPY config/pip.conf /etc/pip.conf 
RUN python -m pip install my-package

COPY my-package2 /usr/local/my-package2
# RUN pip uninstall my-package2
RUN python -m pip install /usr/local/my-package2

EXPOSE 8080 5555 8793

USER airflow
WORKDIR ${AIRFLOW_HOME}
ENTRYPOINT ["/entrypoint.sh"]

As you can see, I copy my-package2 from my local machine to the image and run pip install.

1. The image size is getting bigger every time I rebuild the image.
2. Volumes is definitely an option I haven't tried yet. I already make use of script/shell.sh which just has $@. I set that as entry point and run any command I wish to run inside the image without much haggle.
3. I use docker-compose so every time I rebuild with the new tag, I need to update in docker-compose as well. Over the time it gets annoying to do this for a single line change in code.

Answer 1

You will need to share some of your dockerfile so we could understand why it takes so long to install the pip package. If you wish to optimize it, these references might help :

• How to Optimize Your Dockerfile
• Optimizing Your Dockerfile
• Best practices for writing Dockerfiles

An alternate solution is, instead of building an image for testing, just use the package from the host via the Docker parameter of -v /host/directory:/container/directory.

This will let you immediately test your package in the context of the container, so you will only create the production image when the testing is complete.

Much more information can be found, for example : Understanding Volumes in Docker.

---

From your posted dockerfile, it seems that almost all of it is for installing dependencies. For testing, you can create an image where all these dependencies are already installed, then just repeat the last step for installing your application each time for testing.

For readability, you could finally write the dockerfile as multi-stage, for separating dependencies-building from production, and perhaps also to only generate a final minimal production build. The ONBUILD instruction might be useful here.

Only you know what you are trying to achieve and what are your constraints. The above links can serve as a starting-point, and there are many more articles to be found on the subject.