5 things I learned using docker with bamboo

I decided to try out Docker for Keycrunch acceptance tests because the currently test setup with Bamboo is slow and complicated. To my surprise it was much simpler than I thought. The main tools I was using are Docker, Docker Machine (for Mac) and Docker Compose. And here are five things that I’ve learned.

docker-compose up and docker-compose run is not the same

While you can start your service inside the docker by both commands, docker-compose run will not expose any ports. So to debug, I ended up adding a ssh service and then start the container with docker-compose up.

Update: The document says you can use --service-ports flag for docker-compose run to create the ports, which I have just found out 😀

Use external-link to between system and acceptance test

Here is the docker-compose.yml file in the beginning (simplified version):

web:
  build: .
  ports:
    - "80:80"
    - "443:443"
  links:
    - mysql
    - redis
mysql:
  image: mysql:5.5
  environment:
    MYSQL_ROOT_PASSWORD: docker
redis:
  image: redis:2.6
test:
  build: ../test-system
  links:
    - web

One of the problems I have is that the part of the bootstrap process is in the entry script, and that takes some time to finish. So if I do

docker-compose up test

all the containers will start at the same time and test will be executed while web is still bootstrapping.

I solved it by using seperate docker-compose files and using external-link to connect them together. So for backend system:

web:
  build: .
  ports:
    - "80:80"
    - "443:443"
  links:
    - mysql
    - redis
mysql:
  image: mysql:5.5
  environment:
    MYSQL_ROOT_PASSWORD: docker
redis:
  image: redis:2.6

For test (backend is the project name for previous docker-compose project):

test:
    build: .
    external_links:
        - backend_web_1:web

Then you can easily just do it in two steps, first start backend

docker-compose -p backend up -d

And then do something to check if the bootstrap is finished, if it is run

docker-compose -p qa up

Three other things

  • Dockerfile cannot read files in the parent folder, use volume or move your Dockerfile! :)
  • You will have to watch out for disk usage if the docker images are built on the fly (cleaning up old images constantly)
  • In Dockerfile, install your packages (pip install, npm install, etc) before copy all the files to minimize intermediate container images and make next build faster

 

Set up Askbot project in Codeship

codeship

Update:

Askbot just upgraded their Django-avatar version so we don’t have to compile PIL anymore.

Codeship is a great tool for private continuous integration system. But setting up a project that uses Askbot can be a bit troublesome, since we have to compile PIL to support avatar module in offerQA. Mainly because it is not possible to symblink /usr/include/freetype2 to /usr/include/freetype. But later I realized that pip also searches in ~/.virtualenv/include folder so things became much easier. A local_setting file is also created specifically for Codeship database and Redis. Here is my setup commands:

pip install -r requirements-dev.txt
pip install redis
# Use settings for CI
ln -s local_settings.ci local_settings.py
# Sync your DB for django projects
python manage.py syncdb --noinput
# Run migrations for your django project
python manage.py migrate --noinput

And here is my test commands:

# Running your Django tests
python manage.py test

The local_settings.ci file:

# -*- coding: utf-8 -*-
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': 'default.db',
        'USER': '',
        'PASSWORD': '',
        'HOST': '',
        'PORT': ''
    }
}
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
        'LOCATION': 'offerqa',
        'CACHE_TIMEOUT' : 60,
        'KEY_PREFIX': 'askbot',
    }
}

No Django log printout in Gunicorn

I found a weird problem when I tried to deploy a new project with Gunicorn this weekend. It seems Gunicorn has swallowed all Django printouts. No matter how I set up the project: change the logging settings in Django, add debug flag and set log level, I couldn’t get anything to show up.

After a couple of hours of trying and googling (!!!) I found an interesting comment in Stackoverflow, it seems Gunicorn has changed the default logging to not to print on the console in R19, but it is reverted back in R20. But if you are using R19, you can get Gunicorn to write log to console again by adding:

--log-file=-

flag to the command.

Tips for mocking in Python

Restore patching

I like mocking and patching in unittest. And you shouldn’t forget to restore all the patchings also. I would think it should be put in the TearDown() method in a unittest. But it seems that I was wrong.

From the official document:

…This method will only be called if the setUp() succeeds, regardless of the outcome of the test method…

So if there is an exception in setUp(), the patchings will not be restored. The correct way of doing it should be using the addCleanup() method.

…If setUp() fails, meaning that tearDown() is not called, then any cleanup functions added will still be called.

And example if you use patch() as function.

class MyTest(TestCase):
  def setUp(self):
    patcher = patch('package.module.Class')
    self.MockClass = patcher.start()
    self.addCleanup(patcher.stop)

Mock the base class

I am not sure if you should do that since I haven’t found too many articles on the internet. But I will put it here anyway. The base classes are defined in a class’s __bases__ attribute, but when I tried to patch it like other attribute using:

@mock.patch.object(ExtendedClass, '__bases__', (MockBaseClass,))

I got a TypeError:

TypeError: can't delete ExtendedClass.__bases__

Looks like mock module is trying to remove it before reassign it. I had struggled for a few hours before I found out another simple way to do it:

ExtendedClass.__bases__ = (MockBaseClass,)

The only drawback is that you will have to restore the base class manually afterwards.