Contributing Guide

How to configure developer environment

System configuration

We assume you have a linux based system. Technically everything should be doable using docker on Windows, but we did not try it.

Hosts

You’ll need to set up mapping in: /etc/hosts from olcms.dev.local to localhost IP, for example:

127.0.2.1       olcms.dev.local

Note

It is required for local swift datastore, due to swift internal details container must be resolvable under the same domain/ip on both localhost (so your browser can access it) and inside django container (so django can access it), since localhost won’t work, as well as using raw IP, best way I could thought of was just using a domain.

Docker

You’ll need to install docker (any recent version) and docker-compose (any recent version, tested on 1.17.1).

Also make sure you have rights to call docker command, to check it see if docker info exits successfully

Setting the system up

touch .local.dev.env
./dev.sh build
./dev.sh init
./dev.sh up

Sanity check

Run ./dev.sh ps and see all services are up.

Basic set-up

  • See “Creating user accounts”

Developer environment features

Email Server

In development, it is often nice to be able to see emails that are being sent from your application. For that reason local SMTP server MailHog with a web interface is available as docker container.

With MailHog running, to view messages that are sent by your application, open your browser and go to http://127.0.0.1:8025

Youtube

To work with the upload of videos to youtube service developer needs to create locally .local.dev.env file, next to .dev.env.

In this file add two variables: YOUTUBE_SECRET_JSON and YOUTUBE_OAUTH2_JSON. As values for this variables use data from olcms-secrets project under yoututbe group. Management of the youtube account is done in cellery tasks. To see logs from this operations open logs for celleryworker process in the docker compose. Also remember to set correctly environment variable YT_PLAYLIST_ID with the correct id.

Note

See “Obtaining youtube credentials” on how to obtain youtube credentials.

Run elasticsearch locally

We use different indexing service locally than in production. Locally we use whoosh, and elasticsearch on production. Main reason for this difference is that Elasitc search eats a lot of RAM.

To use elastic search locally follow instructions below:

To start a local cluster using elastic search you can use: docker-compose -f docker-compose-elasticsearch.yml and then usuall up, run django bash (and so on).

You can access elastic search gui, by using: http://localhost:9200/_plugin/gui/#/inspect.

Common developer tasks

Creating users

  • To create normal user account just go create account normally, to verify emails go to http://localhost:8025 where all emails sent from the system are presented.
  • To create superuser just call: ./dev.sh manage createsuperuser.

Running tests

To run all tests call: ./dev.sh test (this takes some time), once you isolate failing tests you may re-run just them: ./dev.sh test www.content_item.

Running linter

Run ./dev.sh lint.

Starting bash shell in the container

Run ./dev.sh bash.

Generate translation files

Start shell in the container ./dev.sh bash and then do following:

  • cd /app
  • ./manage.py makemessages — this will extract strings from all applications.
  • cd www/templates
  • ./manage.py makemessages — this will extract strings from templates

Not so common developement tasks

Debugging selenium tests

If your selenium tests start to fail, here is what you can do:

  1. When you run tests screenshots and page HTML dumps are stored in the screenshots, usually there are more than one screen-shoot per test. File names have following format: test_name-specifier.extension, where specifier describes the screen-shots/page dump. At the end of each test (both for successes and failures) final screen-shots is stored it has final specifier, so the name looks like: www.integration_tests.test_content_item.ContentItemTest.test_add_content_item_with_default_author.final.png.

    Screenshots are easiest way of spotting errors in your tests.

  2. Try adding more screenshots self.save_screenshot("before-submit") at critical moments of failing test.

  3. If the tests are failing locally skip to the next (Running selenium tests on a local browse) section.

    If tests fail only on CI and not on your computer you might try running stress-ng locally this will simulate noisy-neighbours on CI and hopefully enable you to reproduce.

  4. If tests fail on CI, things get harder. Start with running ssh console on our CI instance: Click launch SSH console on the failing job.

  5. Now you can start tests manually (starting with docker-compose -f dev.yaml run django bash).

  6. To pinpoint the issue I usually used pudb (console debugger).

Running selenium tests on a local browser

Easiest way to pinpoint selenium failures is to run selenium tests on your local browser.

You’ll need to:

  1. Download geckodriver and add it to path. You can use this handy script (taken from Dockerfile-dev.

    curl https://ocs-pl.oktawave.com/v1/AUTH_385fff76-290b-43da-b2fc-96b1c08bce24/tools/geckodriver.xz | xzcat > /usr/local/bin/geckodriver  && \
# Verify file integrity (did checksumming myself)
sha256sum /usr/local/bin/geckodriver | grep --quiet 469052c6bf2c4f6e0b07f32d50ba0ff21b20e83d132468ec389050734dd23142 && \
chmod +x /usr/local/bin/geckodriver

  2. There might be some way to run tests in docker, on the local browser but I have no idea how to configure it, so in this case we’ll just run django outside of container.

  3. Create a virtualenv on python 3.5

  4. Install all requirements: pip install -r requirements/local.txt && pip install -r requirements/test.txt

  5. Source proper variables: source compose/django/entrypoint-local.sh.

  6. If you want to use search: ./manage.py rebuild_index.

  7. ./manage.py test. Browser should start, and you’ll see what selenium does.

  8. It is useful to run ./manage.py test in debugging mode and then manually stepping through the code.

Debugging indexes

When you encounter incorrect search results in development enviorment, inspecting index contents can help:

from whoosh.index import open_dir
ix = open_dir("whoosh_index_dir")
print(list(ix.searcher().documents()))

Debugging hanging tests

If selenium hangs you can try to debug it by sending USR1 signal to ./manage.py test, eg. by execing to container and then running: pkill -USR1 -f test. This will dump current stack-trace to stacktrace.log log in your OLCMS folder.

Obtaining youtube credentials

ref:See here <obtain-youtube-credentials>.

How to add a new feature

  1. First get in touch with us e.g. by creating an issue: https://bitbucket.org/olcms/www/issues?status=new&status=open.
  2. Then write the feature.
  3. Make sure to add enough tests too keep coverage 100%.

What to test

For now I want to have 100% coverage on Python code, and good coverage on in the functional Selenium UI tests.

  1. Create unit tests that cover everything, except UI templating stuff.
  2. Explicitly test for permissions — check that users without permission have no access.
  3. Test rest framework. Test permissions there as well.
  4. Create automatic selenium tests for UI, Focus here on pure frontend aspects of the system.

How to test

Django unittest facilities are explained here: https://docs.djangoproject.com/en/1.10/topics/testing/tools/

Integration tests live in www.integration_tests, while unit tests are inside each application.

Selenium

Selenium is slightly harder to use. Start with extending: LiveServerBrowserTestCase, it runs firefox browser using a virtual framebuffer. Then you have access to selenium driver using (unsurprisingly) self.driver.

There are some helper methods in SeleniumUtilsMixin, most notably you’ll use self.login (LiveServerBrowserTestCase extends that mixin) that logs in self.user to OLCMS using the browser.