Docker Containers for Deep Learning Experiments

Deep learning is a powerful tool to solve problems in the area of image analysis. The dominant compute platform for deep learning is Nvidia’s proprietary CUDA, which can only be used together with Nvidia graphics cards. The nivida-docker project allows exposing Nvidia graphics cards to docker containers and thus makes it possible to run deep learning experiments in docker containers.

In our department, we use deep learning to solve problems in the area of medical image analysis and use docker containers to offer researchers a unified way to set up experiments on machines with inhomogeneous hardware configurations. By running experiments in docker containers, researches can set up their own custom software environments which often depend on the medical image modality that is being analyzed. Experiments can be archived in a docker registry and easily be moved between computers. Differences in hardware configurations can be hidden through the system configuration of the base system. This way, container environments remain largely the same even across different computers.

Using graphics hardware from docker containers, however, also introduces extra complications: CUDA uses C-like code that is compiled to binaries that are not necessarily compatible between graphics cards. It is also possible, due to the lack of proper hardware virtualization, to crash the Nvidia driver on the base system which will affect all other containers running on the system.

Allowing researchers to define their own runtime environments for their experiments using containers made archiving of experiments more viable. Experiments do not depend on local system configurations anymore and therefore can be moved between systems and expected run. Using graphics hardware from docker containers introduces more complexity, but generally works and should make deep learning experiments more repeatable in the long run.