Deploy¶
Third party system you’ll need to configure¶
email
This application sends emails to it’s users, please set-up SMTP account for it (or better yet) sent emails using some dedicated service (mailgun, mailinator*).
Email volume should be reasonably small.
- Keep in mind GDPR requirements with overseas providers, in wake of
possible Privacy Shield revocation. Also — I’m not a lawyer.
youtube
Video resources upload videos to youtube.
You’ll need to obtain valid youtube credentials. See how to obtain youtube credentials
Swift file storage
Files uploaded to our repository are hosted in OpenStack Swift Service, which is free/open-source system similar to S3, that is it delivers easy-to use file storage service.
You can find providers here: https://www.openstack.org/marketplace/.
Database
- Postgres database. You can either deploy your own PostrgreSQL server,
add docker image to the
docker-compose
file you use or just use database from your hosting provider (e.g. RDS).
Configuring swift¶
Rationale for swift configuration is here
We use SWIFT such way that users can upload files directly there.
To do this we use Swift cluster using two kinds of middleware:
temp url
middleware. This middleware allows us to generate temporary urls, that allow user to perform a single HTTP verb (GET or PUT). After timeout these urls become invalid.CORS
middleware, enables Swift to server properCORS
headers.
To configure swift container you’ll need:
- Container name
- Authorization information.
In this tutorial I’ll assume that you have swift client software installed, also I’ll refer as
swift-authorized
to swift client with proper authorization flags e.g.
swift -A https://my.swift.service.org/auth/v1.0 -U username -P password
.
To mark container as publicly readable you’ll need to issue following command:
swift-authorized post container_name -r '.r:*
,
To set temp url key you’ll need:
swift-authorized post container_name -H "X-Container-Meta-Temp-URL-Key:{temp_url_key}"
.
Temp url key should be a long random string that matches SWIFT_TEMP_URL_KEY
setting in the settings.
You’ll also need to update CORS headers:
swift-authorized post container_name \
"-H 'X-Container-Meta-Access-Control-Allow-Origin:*' " \
"-H 'X-Container-Meta-Access-Control-Expose-Headers:*' " \
"-H 'X-Container-Meta-Access-Control-Allow-Headers: *' "
Note that since upload rights will be guarded by temporary urls permissive CORS are not a big problem.
VM requirements¶
We use 2 VCPU, 2 GB RAM
server with 40GB
disk drive, and it is enough.
Building images¶
To build production docker images use following command: ./dev.sh push-prod
.
Proper tags are also build whenever someone updates master
and preprod
branches in our repository.
Prepare environment variables¶
Meta variables¶
Following are meta variables used in many different variables:
olcms domain — we will use
olcms.example.com
thought the examples.discourse domains — we will use
discourse.example.com
thought the examples.
Django environment¶
Variables for django images (in .dj_env
file using docker compose deployment
below.
Generic settings¶
USE_HTTPS
set to 1 if you want to force https (which is good thing)DJANGO_ADMIN_URL: "/admin"
— url for administrative interfaceDJANGO_SETTINGS_MODULE: "config.settings.production"
— settings module usedDJANGO_SECRET_KEY
— set this to random string.DJANGO_ALLOWED_HOSTS: "*"
— if you set this to hostname, eg: “olcms.my.project.eu” OLCMS will deny HTTP requests with other host namesDJANGO_SECURE_SSL_REDIRECT
if true django redirects http to https. In our case nginx also does that.DJANGO_ACCOUNT_ALLOW_REGISTRATION
set to 1 allow users to register themselves (and create new content).OLCMS_DOMAIN: olcms.example.com
domain for your instance.CELERY_BROKER_URL: redis://redis/0
— in all cases we use built-in redis docker image.
Database settings¶
POSTGRES_HOST
— hostname of postgres databasePOSTGRES_USER
— username for postgres user, as well as name of database to usePOSTGRES_PASSWORD
— postgresql connection password
Swift settings¶
SWIFT_AUTH_URL: https://swift.example.com/auth/v1.0
— Swift authorization url. Consult your provider documentation for details.Our settings support (rather old) authorization version
1.0
(as our provider supports only that. Adapting to never auth should be straightforward.SWIFT_USERNAME
: Username for swift administratorSWIFT_KEY
: Key (password) for swift administrator userSWIFT_CONTAINER_NAME
: Swift container to store data inSWIFT_TEMP_URL_KEY
: Key to generate temporary upload urls, see Rationale for swift configuration here. This value needs to match swift configurationSWIFT_TEMP_URL_DURATION
duration for which temporary urls generated by file upload api will be valid. in seconds.
SMTP settings¶
SMTP service used to send emails. Configuring OLCMS to use e.g. mailgun API
endpoint will require changes to DJANGO_SETTINGS_MODULE
.
DJANGO_SERVER_EMAIL
—From:
email for all sent mail.DJANGO_EMAIL_USER
— login user for SMTP serverDJANGO_EMAIL_HOST
— hostname for SMTP serverDJANGO_EMAIL_PORT
— port for SMTP connection (unsurprisingly)DJANGO_EMAIL_USE_SSL
— set to 1 if you want to use “SSL” for SMTP connectionDJANGO_EMAIL_PASS
— password forDJANGO_EMAIL_USER
Youtube settings¶
See this section on how to obtain youtube settings.
YOUTUBE_OAUTH2_JSON
YOUTUBE_SECRET_JSON
YT_PLAYLIST_ID
Google analytics settings¶
GOOGLE_ANALYTICS_TRACKING_CODE: "UA-0000001-1"
— we support google analytics user tracking, if you wish to use it, obtain google analytics and put it here. If missing user tracking won’t be used.
Forum integration¶
ENABLE_DISCOURSE_FORUM_TRIGGERS
— if you set this to1
olcms integration with discourse. See here for forum integration here.If you set to
0
you can omit rest of these variables.DJANGO_DISCOURSE_SSO_URL: https://discourse.example.com/session/sso_login
— discourse sso url it will be used to redirect users after single sign on login to discourse forum.DJANGO_DISCOURSE_ROOT_URL: https://discourse.example.com/
— root url for discourse installation, used to install forum “comments” under materials.COMPOSE_DISCOURSE_SSO_SECRET
: Discourse SSO secret. Set this to a long random string. Used both by django and discourse image.
Nginx configuration¶
Please note that the same cert used for olcms service as well as discourse service. We just use wildcard certificate.
OLCMS_DOMAIN: olcms.example.com
domain for your instance. Please note that this setting is used both in django and nginx configurationsNGINX_CERT
base64 encoded certificate file in PEM format. To obtain this value use following command (assuming certificate is namedmy.crt
):cat my.crt | base64 -w0
. You should get long string starting withLS0tLS1CRUdJTi
.NGINX_KEY
base64 encoded certificate key in PEM format. To obtain this value use following command (abusing key file is namedmy.key
):cat my.key | base64 -w0
. You should get long string starting withLS0tLS1CRUdJTi
.
Discourse configuration¶
We use ‘unsupported’ (by discourse team) method of deploying Discourse — we want:
- To treat discourse as “more or less” stateless service
- Be able to deploy discourse automatically
- To automatically configure discourse using 7 factor app.
Feel free to use our deployment scripts, feel free also to use official docker image for discourse (and of course, feel free to not use discourse at all). In any case integrating stock discourse service should be relatively straightforward.
See forum integration for full information
Discourse misc settings¶
Discourse settings that should just be left without changes, used by discourse/ruby configuration.
They should be
RAILS_ENV: production
RUBY_GLOBAL_METHOD_CACHE_SIZE: 131072
RUBY_GC_HEAP_GROWTH_MAX_SLOTS: 40000
RUBY_GC_HEAP_INIT_SLOTS: 400000
RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR: 1.5
UNICORN_WORKERS: 3
UNICORN_SIDEKIQS: 1
Configuration¶
DISCOURSE_HOSTNAME: discourse.example.com
DISCOURSE_REDIS_HOST: redis_disco
— redis instance configured for discourse in our compose file.
SMTP configuration, consult discourse documentation if anything is not obvious.
DISCOURSE_SMTP_ADDRESS
DISCOURSE_SMTP_PORT
DISCOURSE_SMTP_USER_NAME
DISCOURSE_SMTP_PASSWORD
DISCOURSE_SMTP_ENABLE_START_TLS: false
DISCOURSE_DB_USERNAME
DISCOURSE_DB_PASSWORD
DISCOURSE_DB_HOST
DISCOURSE_DB_NAME
DISCOURSE_REDIS_HOST
Integration with Django¶
COMPOSE_DISCOURSE_SSO_SECRET
: Discourse SSO secret. Set this to a long random string. Used both by django and discourse image.COMPOSE_DISCOURSE_SSO_PROVIDER_URL: https://olcms.example.com/discourse/sso
sso endpointDISCOURSE_ENABLE_CORS: true
— needed for SSODISCOURSE_CORS_ORIGIN: "*"
— you probably should limit this to prevent security issues.
Misc configuration¶
Admin details, admin user with following email data will be created, to login
via SSO you’ll need to create user with the same e-mail
in OLCMS.
COMPOSE_DISCOURSE_ADMIN_EMAIL: admin@ourdomain
COMPOSE_DISCOURSE_ADMIN_USERNAME: admin
COMPOSE_DISCOURSE_ADMIN_PASSWORD: password
Misc configuration:
COMPOSE_DISCOURSE_INVITE_ONLY: t
— disable self registration (extra caution)COMPOSE_DISCOURSE_LOGIN_REQUIRED: f
— don’t require login to read posts — required for comment integrationCOMPOSE_DISCOURSE_BOOTSTRAP_MODE: f
— Disable bootstrap mode.COMPOSE_DISCOURSE_TITLE: Discussion forum for OLCMS
— forum titleCOMPOSE_DISCOURSE_SITE_DESCRIPTION: A description
— forum descriptionCOMPOSE_DISCOURSE_CONTACT_EMAIL: contact@example.com
— contact e-mail
Deployment¶
Obtain all credentials third-party services described above.
Install docker on a linux server.
Install docker compose.
Install docker compose file from here.
Please note that images related to
discourse forum
are optional.Create environment file
Start compose service.
Docker compose file we use¶
version: '2'
volumes:
elastic_data: {}
videotmp: {}
discourse_assets: {}
discourse_shared: {}
discourse_logs: {}
services:
elasticsearch:
image: elasticsearch:2.4
volumes:
- elastic_data:/usr/share/elasticsearch/data
environment:
- ES_JAVA_OPTS=-Xms350m -Xmx350m
logging:
driver: 'json-file'
options:
"max-size": "10mb"
"max-file": "1"
# Runs migrations during startup.
django_migrations:
image: olcms/www:latest
user: django
command: /scripts/prod-init.sh
env_file: .env
volumes:
- videotmp:/videotmp
logging:
driver: 'json-file'
options:
"max-size": "10mb"
"max-file": "1"
# Updates elastic search index.
update_index:
image: olcms/www:latest
user: django
command: /scripts/update-index.sh
env_file: .env
logging:
driver: 'json-file'
options:
"max-size": "10mb"
"max-file": "1"
# Django application server
django:
image: olcms/www:latest
user: django
depends_on:
- redis
- django_migrations
- elasticsearch
volumes:
- videotmp:/videotmp
command: /scripts/gunicorn.sh
env_file: .env
logging:
driver: 'json-file'
options:
"max-size": "100mb"
"max-file": "5"
# Nginx webserver
nginx:
image: olcms/nginx:latest
env_file: .env
entrypoint: /entrypoint.sh
command: /start.sh
depends_on:
- django
- discourse
ports:
- "0.0.0.0:80:80"
- "0.0.0.0:443:443"
logging:
driver: 'json-file'
options:
"max-size": "100mb"
"max-file": "3"
#Redis cache
redis:
image: redis:3.0-alpine
logging:
driver: 'json-file'
options:
"max-size": "100mb"
"max-file": "1"
# Celery task queue
celeryworker:
image: olcms/www:latest
user: django
env_file: .env
ports: []
depends_on:
- redis
- django_migrations
command: celery -A www.taskapp worker -l INFO --concurrency 2
volumes:
- videotmp:/videotmp
logging:
driver: 'json-file'
options:
"max-size": "100mb"
"max-file": "3"
# Celery task queue
celerybeat:
image: olcms/www:latest
user: django
env_file: .env
ports: []
depends_on:
- redis
- django_migrations
command: celery -A www.taskapp beat -l INFO
volumes:
- videotmp:/videotmp
logging:
driver: 'json-file'
options:
"max-size": "100mb"
"max-file": "2"
# Separated worker for converting documents to other formats.
document_converter:
image: olcms/documentconverter
links:
- redis
environment:
- WORKER_QUEUE=converter
- CELERY_BROKER_URL=redis://redis/0
logging:
driver: 'json-file'
options:
"max-size": "100mb"
"max-file": "2"
##############################################################################
## OPTIONAL DISCOURSE IMAGES
##############################################################################
redis_disco:
image: redis:3.0-alpine
logging:
driver: 'json-file'
options:
"max-size": "100mb"
"max-file": "1"
discourse:
ports:
- "8081:80"
volumes:
- discourse_assets:/var/www/discourse/public/
- discourse_shared:/shared/
- discourse_logs:/var/log
command: /discourse-composite/run/scripts/discourse.start.sh
image: olcms/discourse-composite:2018-01-24
depends_on:
- redis_disco
- discourse_migrate
links:
- redis_disco
- discourse_migrate
env_file: .env
logging:
driver: 'json-file'
options:
"max-size": "100mb"
"max-file": "2"
discourse_migrate:
volumes:
- discourse_assets:/var/www/discourse/public/
- discourse_shared:/shared/
- discourse_logs:/var/log
command: /discourse-composite/run/scripts/discourse.migrate.sh
image: olcms/discourse-composite:2018-01-24
depends_on:
- redis_disco
links:
- redis_disco
env_file: .env
logging:
driver: "json-file"
options:
max-size: "100kb"
max-file: "5"