Deploy

Third party system you’ll need to configure

email

This application sends emails to it’s users, please set-up SMTP account for it (or better yet) sent emails using some dedicated service (mailgun, mailinator*).

Email volume should be reasonably small.

  • Keep in mind GDPR requirements with overseas providers, in wake of

possible Privacy Shield revocation. Also — I’m not a lawyer.

youtube

Video resources upload videos to youtube.

You’ll need to obtain valid youtube credentials. See how to obtain youtube credentials

Swift file storage

Files uploaded to our repository are hosted in OpenStack Swift Service, which is free/open-source system similar to S3, that is it delivers easy-to use file storage service.

You can find providers here: https://www.openstack.org/marketplace/.

Database
Postgres database. You can either deploy your own PostrgreSQL server, add docker image to the docker-compose file you use or just use database from your hosting provider (e.g. RDS).

Configuring swift

Rationale for swift configuration is here

We use SWIFT such way that users can upload files directly there.

To do this we use Swift cluster using two kinds of middleware:

  1. temp url middleware. This middleware allows us to generate temporary urls, that allow user to perform a single HTTP verb (GET or PUT). After timeout these urls become invalid.
  2. CORS middleware, enables Swift to server proper CORS headers.

To configure swift container you’ll need:

  • Container name
  • Authorization information.

In this tutorial I’ll assume that you have swift client software installed, also I’ll refer as swift-authorized to swift client with proper authorization flags e.g. swift -A https://my.swift.service.org/auth/v1.0 -U username -P password.

To mark container as publicly readable you’ll need to issue following command: swift-authorized post container_name -r '.r:*,

To set temp url key you’ll need: swift-authorized post container_name -H "X-Container-Meta-Temp-URL-Key:{temp_url_key}". Temp url key should be a long random string that matches SWIFT_TEMP_URL_KEY setting in the settings.

You’ll also need to update CORS headers:

swift-authorized post container_name \
      "-H 'X-Container-Meta-Access-Control-Allow-Origin:*' " \
      "-H 'X-Container-Meta-Access-Control-Expose-Headers:*' " \
      "-H 'X-Container-Meta-Access-Control-Allow-Headers: *' "

Note that since upload rights will be guarded by temporary urls permissive CORS are not a big problem.

VM requirements

We use 2 VCPU, 2 GB RAM server with 40GB disk drive, and it is enough.

Building images

To build production docker images use following command: ./dev.sh push-prod.

Proper tags are also build whenever someone updates master and preprod branches in our repository.

Prepare environment variables

Meta variables

Following are meta variables used in many different variables:

  • olcms domain — we will use olcms.example.com thought the examples.

  • discourse domains — we will use discourse.example.com thought the examples.

    See here for forum integration here.

Django environment

Variables for django images (in .dj_env file using docker compose deployment below.

Generic settings

  • USE_HTTPS set to 1 if you want to force https (which is good thing)
  • DJANGO_ADMIN_URL: "/admin" — url for administrative interface
  • DJANGO_SETTINGS_MODULE: "config.settings.production" — settings module used
  • DJANGO_SECRET_KEY — set this to random string.
  • DJANGO_ALLOWED_HOSTS: "*" — if you set this to hostname, eg: “olcms.my.project.eu” OLCMS will deny HTTP requests with other host names
  • DJANGO_SECURE_SSL_REDIRECT if true django redirects http to https. In our case nginx also does that.
  • DJANGO_ACCOUNT_ALLOW_REGISTRATION set to 1 allow users to register themselves (and create new content).
  • OLCMS_DOMAIN: olcms.example.com domain for your instance.
  • CELERY_BROKER_URL: redis://redis/0 — in all cases we use built-in redis docker image.

Database settings

  • POSTGRES_HOST — hostname of postgres database
  • POSTGRES_USER — username for postgres user, as well as name of database to use
  • POSTGRES_PASSWORD — postgresql connection password

Swift settings

  • SWIFT_AUTH_URL: https://swift.example.com/auth/v1.0 — Swift authorization url. Consult your provider documentation for details.

    Our settings support (rather old) authorization version 1.0 (as our provider supports only that. Adapting to never auth should be straightforward.

  • SWIFT_USERNAME: Username for swift administrator

  • SWIFT_KEY: Key (password) for swift administrator user

  • SWIFT_CONTAINER_NAME: Swift container to store data in

  • SWIFT_TEMP_URL_KEY: Key to generate temporary upload urls, see Rationale for swift configuration here. This value needs to match swift configuration

  • SWIFT_TEMP_URL_DURATION duration for which temporary urls generated by file upload api will be valid. in seconds.

SMTP settings

SMTP service used to send emails. Configuring OLCMS to use e.g. mailgun API endpoint will require changes to DJANGO_SETTINGS_MODULE.

  • DJANGO_SERVER_EMAIL — From: email for all sent mail.
  • DJANGO_EMAIL_USER — login user for SMTP server
  • DJANGO_EMAIL_HOST — hostname for SMTP server
  • DJANGO_EMAIL_PORT — port for SMTP connection (unsurprisingly)
  • DJANGO_EMAIL_USE_SSL — set to 1 if you want to use “SSL” for SMTP connection
  • DJANGO_EMAIL_PASS — password for DJANGO_EMAIL_USER

Youtube settings

See this section on how to obtain youtube settings.

  • YOUTUBE_OAUTH2_JSON
  • YOUTUBE_SECRET_JSON
  • YT_PLAYLIST_ID

Google analytics settings

  • GOOGLE_ANALYTICS_TRACKING_CODE: "UA-0000001-1" — we support google analytics user tracking, if you wish to use it, obtain google analytics and put it here. If missing user tracking won’t be used.

Forum integration

  • ENABLE_DISCOURSE_FORUM_TRIGGERS — if you set this to 1 olcms integration with discourse. See here for forum integration here.

    If you set to 0 you can omit rest of these variables.

  • DJANGO_DISCOURSE_SSO_URL: https://discourse.example.com/session/sso_login — discourse sso url it will be used to redirect users after single sign on login to discourse forum.

  • DJANGO_DISCOURSE_ROOT_URL: https://discourse.example.com/ — root url for discourse installation, used to install forum “comments” under materials.

  • COMPOSE_DISCOURSE_SSO_SECRET: Discourse SSO secret. Set this to a long random string. Used both by django and discourse image.

Nginx configuration

Please note that the same cert used for olcms service as well as discourse service. We just use wildcard certificate.

  • OLCMS_DOMAIN: olcms.example.com domain for your instance. Please note that this setting is used both in django and nginx configurations
  • NGINX_CERT base64 encoded certificate file in PEM format. To obtain this value use following command (assuming certificate is named my.crt): cat my.crt | base64 -w0. You should get long string starting with LS0tLS1CRUdJTi.
  • NGINX_KEY base64 encoded certificate key in PEM format. To obtain this value use following command (abusing key file is named my.key): cat my.key | base64 -w0. You should get long string starting with LS0tLS1CRUdJTi.

Discourse configuration

We use ‘unsupported’ (by discourse team) method of deploying Discourse — we want:

  • To treat discourse as “more or less” stateless service
  • Be able to deploy discourse automatically
  • To automatically configure discourse using 7 factor app.

Feel free to use our deployment scripts, feel free also to use official docker image for discourse (and of course, feel free to not use discourse at all). In any case integrating stock discourse service should be relatively straightforward.

See forum integration for full information

Discourse misc settings

Discourse settings that should just be left without changes, used by discourse/ruby configuration.

They should be

  • RAILS_ENV: production
  • RUBY_GLOBAL_METHOD_CACHE_SIZE: 131072
  • RUBY_GC_HEAP_GROWTH_MAX_SLOTS: 40000
  • RUBY_GC_HEAP_INIT_SLOTS: 400000
  • RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR: 1.5
  • UNICORN_WORKERS: 3
  • UNICORN_SIDEKIQS: 1

Configuration

  • DISCOURSE_HOSTNAME: discourse.example.com
  • DISCOURSE_REDIS_HOST: redis_disco — redis instance configured for discourse in our compose file.

SMTP configuration, consult discourse documentation if anything is not obvious.

  • DISCOURSE_SMTP_ADDRESS
  • DISCOURSE_SMTP_PORT
  • DISCOURSE_SMTP_USER_NAME
  • DISCOURSE_SMTP_PASSWORD
  • DISCOURSE_SMTP_ENABLE_START_TLS: false
  • DISCOURSE_DB_USERNAME
  • DISCOURSE_DB_PASSWORD
  • DISCOURSE_DB_HOST
  • DISCOURSE_DB_NAME
  • DISCOURSE_REDIS_HOST

Integration with Django

  • COMPOSE_DISCOURSE_SSO_SECRET: Discourse SSO secret. Set this to a long random string. Used both by django and discourse image.
  • COMPOSE_DISCOURSE_SSO_PROVIDER_URL: https://olcms.example.com/discourse/sso sso endpoint
  • DISCOURSE_ENABLE_CORS: true — needed for SSO
  • DISCOURSE_CORS_ORIGIN: "*" — you probably should limit this to prevent security issues.

Misc configuration

Admin details, admin user with following email data will be created, to login via SSO you’ll need to create user with the same e-mail in OLCMS.

  • COMPOSE_DISCOURSE_ADMIN_EMAIL: admin@ourdomain
  • COMPOSE_DISCOURSE_ADMIN_USERNAME: admin
  • COMPOSE_DISCOURSE_ADMIN_PASSWORD: password

Misc configuration:

  • COMPOSE_DISCOURSE_INVITE_ONLY: t — disable self registration (extra caution)
  • COMPOSE_DISCOURSE_LOGIN_REQUIRED: f — don’t require login to read posts — required for comment integration
  • COMPOSE_DISCOURSE_BOOTSTRAP_MODE: f — Disable bootstrap mode.
  • COMPOSE_DISCOURSE_TITLE: Discussion forum for OLCMS — forum title
  • COMPOSE_DISCOURSE_SITE_DESCRIPTION: A description — forum description
  • COMPOSE_DISCOURSE_CONTACT_EMAIL: contact@example.com — contact e-mail

Deployment

  1. Obtain all credentials third-party services described above.

  2. Install docker on a linux server.

  3. Install docker compose.

  4. Install docker compose file from here.

    Please note that images related to discourse forum are optional.

  5. Create environment file

  6. Start compose service.

Docker compose file we use

version: '2'

volumes:
  elastic_data: {}
  videotmp: {}
  discourse_assets: {}
  discourse_shared: {}
  discourse_logs: {}

services:

  elasticsearch:
    image: elasticsearch:2.4
    volumes:
      - elastic_data:/usr/share/elasticsearch/data
    environment:
      - ES_JAVA_OPTS=-Xms350m -Xmx350m
    logging:
      driver: 'json-file'
      options:
        "max-size": "10mb"
        "max-file": "1"

  # Runs migrations during startup.
  django_migrations:
    image: olcms/www:latest
    user: django
    command: /scripts/prod-init.sh
    env_file: .env
    volumes:
      - videotmp:/videotmp
    logging:
      driver: 'json-file'
      options:
        "max-size": "10mb"
        "max-file": "1"

  # Updates elastic search index.
  update_index:
    image: olcms/www:latest
    user: django
    command: /scripts/update-index.sh
    env_file: .env
    logging:
      driver: 'json-file'
      options:
        "max-size": "10mb"
        "max-file": "1"

  # Django application server
  django:
    image: olcms/www:latest
    user: django
    depends_on:
      - redis
      - django_migrations
      - elasticsearch
    volumes:
      - videotmp:/videotmp
    command: /scripts/gunicorn.sh
    env_file: .env
    logging:
      driver: 'json-file'
      options:
        "max-size": "100mb"
        "max-file": "5"

  # Nginx webserver
  nginx:
    image: olcms/nginx:latest
    env_file: .env
    entrypoint: /entrypoint.sh
    command: /start.sh
    depends_on:
      - django
      - discourse
    ports:
      - "0.0.0.0:80:80"
      - "0.0.0.0:443:443"

    logging:
      driver: 'json-file'
      options:
        "max-size": "100mb"
        "max-file": "3"

  #Redis cache
  redis:
    image: redis:3.0-alpine
    logging:
      driver: 'json-file'
      options:
        "max-size": "100mb"
        "max-file": "1"

  # Celery task queue
  celeryworker:
    image: olcms/www:latest
    user: django
    env_file: .env
    ports: []
    depends_on:
     - redis
     - django_migrations
    command: celery -A www.taskapp worker -l INFO --concurrency 2
    volumes:
      - videotmp:/videotmp
    logging:
      driver: 'json-file'
      options:
        "max-size": "100mb"
        "max-file": "3"

  # Celery task queue
  celerybeat:
    image: olcms/www:latest
    user: django
    env_file: .env
    ports: []
    depends_on:
      - redis
      - django_migrations
    command: celery -A www.taskapp beat -l INFO
    volumes:
      - videotmp:/videotmp
    logging:
      driver: 'json-file'
      options:
        "max-size": "100mb"
        "max-file": "2"

  # Separated worker for converting documents to other formats.
  document_converter:
    image: olcms/documentconverter
    links:
      - redis
    environment:
      - WORKER_QUEUE=converter
      - CELERY_BROKER_URL=redis://redis/0
    logging:
      driver: 'json-file'
      options:
        "max-size": "100mb"
        "max-file": "2"

  ##############################################################################
  ## OPTIONAL DISCOURSE IMAGES
  ##############################################################################

  redis_disco:
    image: redis:3.0-alpine
    logging:
      driver: 'json-file'
      options:
        "max-size": "100mb"
        "max-file": "1"

  discourse:
    ports:
      - "8081:80"
    volumes:
      - discourse_assets:/var/www/discourse/public/
      - discourse_shared:/shared/
      - discourse_logs:/var/log
    command: /discourse-composite/run/scripts/discourse.start.sh
    image: olcms/discourse-composite:2018-01-24
    depends_on:
      - redis_disco
      - discourse_migrate
    links:
      - redis_disco
      - discourse_migrate
    env_file: .env
    logging:
      driver: 'json-file'
      options:
        "max-size": "100mb"
        "max-file": "2"

  discourse_migrate:
    volumes:
      - discourse_assets:/var/www/discourse/public/
      - discourse_shared:/shared/
      - discourse_logs:/var/log
    command: /discourse-composite/run/scripts/discourse.migrate.sh
    image: olcms/discourse-composite:2018-01-24
    depends_on:
      - redis_disco
    links:
      - redis_disco
    env_file: .env
    logging:
      driver: "json-file"
      options:
        max-size: "100kb"
        max-file: "5"