Kuma Documentation Release latest Mozilla November 10, 2014
Transcription
Kuma Documentation Release latest Mozilla November 10, 2014
Kuma Documentation Release latest Mozilla November 10, 2014 Contents 1 Development 2 Getting Started 2.1 Installation . . . . . . . . . . . 2.2 Vendor Library . . . . . . . . . 2.3 Kuma via Vagrant . . . . . . . 2.4 Celery and Rabbit . . . . . . . 2.5 Running Kuma with mod_wsgi 2.6 Email from Kuma . . . . . . . 2.7 Search . . . . . . . . . . . . . 2.8 Development . . . . . . . . . . 2.9 Localization . . . . . . . . . . 2.10 The Kuma Test Suite . . . . . . 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 5 8 10 13 14 18 18 20 22 27 i ii Kuma Documentation, Release latest Kuma is the platform that powers MDN (developer.mozilla.org) Contents 1 Kuma Documentation, Release latest 2 Contents CHAPTER 1 Development Code https://github.com/mozilla/kuma Issues http://mzl.la/mdn_backlog (Product) https://github.com/mozilla/kuma/issues?state=open (Engineering) https://prs.paas.allizom.org/mozilla:kuma,kuma-lib,kumascript,mozhacks (PR Queue) Dev Docs https://kuma.readthedocs.org/en/latest/installation-vagrant.html CI Server https://travis-ci.org/mozilla/kuma Mailing list https://lists.mozilla.org/listinfo/dev-mdn IRC irc://irc.mozilla.org/mdndev http://logs.glob.uno/?c=mozilla%23mdndev (logs) Servers http://mzl.la/deployed-mdn (What’s Deployed) https://developer-dev.allizom.org/ (dev) https://developer.allizom.org/ (stage) https://developer.mozilla.org/ (prod) 3 Kuma Documentation, Release latest 4 Chapter 1. Development CHAPTER 2 Getting Started Want to help make MDN great? Here are some options: 1. Contribute to our selenium test suite 2. Contribute to the platform itself. Our contribution guide lists some good first projects and offers direction on submitting code. Contents: 2.1 Installation This page describes the manual installation procedure. If you can, you should set up the vagrant-managed virtual machine instead. 2.1.1 Requirements To run everything and make all the tests pass locally, you’ll need the following things (in addition to Git, of course). • Python 2.6. • setuptools or pip. • MySQL Server and client headers. • Memcached Server and libmemcached. • Elasticsearch 0.90.9. • RabbitMQ. • libxml and headers. • libxslt and headers. • libjpeg and headers. • zlib and headers. • libmagic and headers. • libtidy and headers • Several Python packages. See Installing the Packages. Installation for these is very system dependent. Using a package manager, like yum, aptitude, or brew, is encouraged. 5 Kuma Documentation, Release latest Additional Requirements If you want to use Apache, instead of the dev server (not strictly required but it’s more like our production environment) you’ll also need: • Apache HTTPD Server. • mod_wsgi See the documentation on WSGI for more information and instructions. 2.1.2 Getting the Source First, to follow the instructions from Webdev Bootcamp, fork the project into your own account. Then get the source using: mkdir mdn # you probably want to do this, since you’ll have to create cd mdn # product_details_json/ as a sibling of kuma/ later. git clone [email protected]:<your_username>/kuma.git cd kuma git submodule update --init --recursive 2.1.3 Installing the Packages Compiled Packages There are a small number of compiled packages, including the MySQL Python client. You can install these using pip (if you don’t have pip, you can get it with easy_install pip) or via a package manager. To use pip, you only need to do this: sudo pip install -r requirements/compiled.txt Python Packages All of the pure-Python requirements are available in a git repository, known as a vendor library. This allows them to be available on the Python path without needing to be installed in the system, allowing multiple versions for multiple projects simultaneously. 2.1.4 Configuration Start by creating a file named settings_local.py, and putting this line in it: from settings import * Now you can copy and modify any settings from settings.py into settings_local.py and the value will override the default. Database At a minimum, you will need to define a database connection. An example configuration is: 6 Chapter 2. Getting Started Kuma Documentation, Release latest DATABASES = { ’default’: { ’NAME’: ’kuma’, ’ENGINE’: ’django.db.backends.mysql’, ’HOST’: ’localhost’, ’USER’: ’kuma’, ’PASSWORD’: ’’, ’OPTIONS’: {’init_command’: ’SET storage_engine=InnoDB’}, ’TEST_CHARSET’: ’utf8’, ’TEST_COLLATION’: ’utf8_unicode_ci’, }, } Note the two settings TEST_CHARSET and TEST_COLLATION. Without these, the test suite will use MySQL’s (moronic) defaults when creating the test database (see below) and lots of tests will fail. Hundreds. Once you’ve set up the database, you can generate the schema with Django’s syncdb command: mkdir ../product_details_json ./manage.py syncdb ./manage.py migrate This will generate an empty database, which will get you started! Initializing Mozilla Product Details One of the packages Kuma uses, Django Mozilla Product Details, needs to fetch JSON files containing historical Firefox version data and write them within its package directory. To set this up, just run: ./manage.py update_product_details ...to do the initial fetch. Media If you want to see images and have the pages formatted with CSS you need to set your settings_local.py with the following: DEBUG = True TEMPLATE_DEBUG = DEBUG SERVE_MEDIA = True Setting DEBUG = False will put the installation in production mode and ask for minified assets. In that case, you will need to generate CSS from stylus and compress resource: ./scripts/compile-stylesheets ./manage.py compress_assets Configure Persona Add the following to settings_local.py so that Persona works with the development instance: SITE_URL = ’http://localhost:8000’ PROTOCOL = ’http://’ DOMAIN = ’localhost’ PORT = 8000 2.1. Installation 7 Kuma Documentation, Release latest SESSION_COOKIE_SECURE = False # needed if the server is running on http:// SESSION_EXPIRE_AT_BROWSER_CLOSE = False The SESSION_EXPIRE_AT_BROWSER_CLOSE setting is not strictly necessary, but it’s convenient for development. Secure Cookies To prevent error messages like Forbidden (CSRF cookie not set.):, you need to set your settings_local.py with the following: CSRF_COOKIE_SECURE = False 2.1.5 Testing it Out To start the dev server, run ./manage.py runserver, then open up http://localhost:8000. If everything’s working, you should see the MDN home page! You might need to first set LC_CTYPE if you’re on Mac OS X until bug 754728 is fixed: export LC_CTYPE=en_US 2.1.6 What’s next? See development for further instructions. Some site funcationaly require waffle flags. Waffle flags include: • kumaediting: Allows creation, editing, and translating of documents • page_move: Allows moving of documents • revision-dashboard-newusers: Allows searching of new users through the revision dashboard • events_map: Allows display of map on the events page • elasticsearch: Enables elastic search for site search To create or modify waffle flags, visit “/admin/” and click the “Waffle” link. 2.1.7 Last Steps Setting Up Search See the search documentation for steps to get Elasticsearch search working. 2.2 Vendor Library The vendor library is the easiest way to manage pure-Python dependencies. It contains the source code for all the library Kuma depends on. 8 Chapter 2. Getting Started Kuma Documentation, Release latest 2.2.1 Getting the Vendor Library Getting the vendor library is easy. In your Kuma clone, just type: $ git clone --recursive git://github.com/mozilla/kuma-lib.git vendor Git will clone the repository and all its submodules. 2.2.2 Updating the Vendor Library From time to time we need to update libraries, either for new versions of libraries or to add a new library. There are two ways to do that. The easiest and prefered way is pure git. Using Git Submodules Using git submodules is prefered because it is much easier to maintain, and it keeps the repository size small. Upgrading is as simple as updating a submodule. Updating a Library with Git Submodules If the library is in vendor/src, it was pulled directly from version control, and if that version control was git, updating the submodule is as easy as: $ $ $ $ $ $ cd vendor/src/$LIBRARY git fetch origin git checkout <REFSPEC> cd ../.. git add src/$LIBRARY git ci -m "Updating $LIBRARY" Easy! Just like updating any other git submodule. Adding a New Library with Git Submodules Technically this can be done with pip install --no-install but there’s an even easier method when installing a new library from a git repo: $ $ $ $ $ $ $ cd vendor/src git clone git://<repo> cd ../.. ./addsubmodules.sh vim kuma.pth # Add the new library’s path git add kuma.pth git ci -m "Adding $LIBRARY" Using PyPI Follow the playdoh instructions for non-git based repos, replacing vendor-local with vendor. 2.2. Vendor Library 9 Kuma Documentation, Release latest 2.2.3 Requirements Files We still maintain requirements files in requirements/. Sometimes people will use these to install the requirements in a virtual environment. When you update the vendor repo, you should make sure to update version numbers (if necessary) in the requirements files. 2.3 Kuma via Vagrant Core developers run Kuma in a Vagrant-managed virtual machine so we can run the entire MDN stack. (Django, KumaScript, Search, Celery, etc.) If you’re on Mac OS X or Linux and looking for a quick way to get started, you should try these instructions. At the end, you’ll earn the badge: Note: If you have problems using vagrant, check Troubleshooting below 2.3.1 Install and run everything 1. Install VirtualBox 4.x from http://www.virtualbox.org/ Note: (Windows) After installing Files\Oracle\VirtualBox\VBoxManage.exe; VirtualBox you need to set PATH=C:\Program 2. Install vagrant >= 1.6 using the installer from vagrantup.com 3. Install the vagrant-vbguest plugin. 4. Fork the project. (See GitHub and Webdev Bootcamp) 5. Clone your fork of Kuma and update submodules: git clone [email protected]:<your_username>/kuma.git cd kuma git submodule update --init --recursive 6. Copy a vagrantconfig_local.yaml file for your VM: cp vagrantconfig_local.yaml-dist vagrantconfig_local.yaml 7. Start the VM and install everything. (approx. 30 min on a fast net connection).: vagrant up Note: VirtualBox creates VMs in your system drive. Kuma’s VM is 3GB. If it won’t fit on your system drive, you will need to change that directory to another drive. At the end, you should see: => default: notice: Finished catalog run in .... seconds If the above process fails with an error, check Troubleshooting. 8. Add the hostnames to the end of your hosts file with this shell command: 10 Chapter 2. Getting Started Kuma Documentation, Release latest echo ’192.168.10.55 developer-local.allizom.org mdn-local.mozillademos.org’ | sudo tee -a /etc/h 9. Log into the VM with ssh: vagrant ssh 10. Use foreman inside the VM to start all site services: foreman start You should see output like: 20:32:59 20:32:59 20:32:59 20:32:59 ... web.1 celery.1 kumascript.1 stylus.1 | | | | started started started started with with with with pid pid pid pid 2244 2245 2246 2247 11. Visit the following address in your browser and add an exception for the security certificate if prompted: https://mdn-local.mozillademos.org 12. Go to the homepage: https://developer-local.allizom.org/ 13. You’ve installed Kuma! If you want the badge, send a screenshot of your browser to luke dot crouch at gmail dot com. 2.3.2 Create an admin user You will want to make yourself an admin user to enable important site features. 1. Sign up/in with Persona 2. After you sign in, SSH into the vm and make yourself an admin: vagrant ssh mysql -uroot kuma UPDATE auth_user set is_staff = 1, is_active=1, is_superuser = 1 WHERE username = ’YOUR_USERNAME You should see: Query OK, 1 row affected (0.01 sec) Rows matched: 1 Changed: 1 Warnings: 0 2.3.3 Enable Important Site Features Some site features are controlled using django-waffle. You control these features in the waffle admin. Some site features are controlled using constance. You control these features in the constance config admin panel. Wiki Editing The central feature of MDN is wiki editing. We use a waffle flag called kumaediting to control edits to the wiki. So we can effectively put the site into “read-only” and/or “write-by-staff-only” modes. To enable wiki editing on your MDN vm, add a waffle flag called kumaediting and set “Everyone” to “Yes”. 2.3. Kuma via Vagrant 11 Kuma Documentation, Release latest KumaScript To enable KumaScript (Kuma’s template system): 1. Sign in 2. Visit the constance config admin panel 3. Change KUMASCRIPT_TIMEOUT to 600 4. Click “Save” at the bottom 2.3.4 Create pages You can visit https://developer-local.allizom.org/docs/new to create new wiki pages as needed. Many core MDN contributors create a personal User:<username> page as a testing sandbox. 2.3.5 Developing with Vagrant Edit files as usual on your host machine; the current directory is mounted via NFS at /home/vagrant/src within the VM. Updates should be reflected without any action on your part. • See development for tips not specific to vagrant. • Useful vagrant sub-commands: vagrant vagrant vagrant vagrant vagrant ssh suspend halt up destroy # # # # # Connect to the VM via ssh Sleep the VM, saving state Shutdown the VM Boot up the VM Destroy the VM 2.3.6 Troubleshooting Errors during vagrant up vagrant up starts the virtual machine. The first time you run vagrant up it also provisions the vm - i.e., it automatically installs and configures Kuma software on the vm. We provision the vm with puppet manifests in the puppet/manifests directory. Sometimes we put puppet declarations in the wrong order. Which means some errors can be fixed by simply provisioning the vm again: vagrant provision In some rare occasions you might need to run this multiple times. If you see the same error over and over, please ask for more help. On Ubuntu, vagrant up might fail after being unable to mount NFS shared folders. First, make sure you have the nfs-common and nfs-server packages installed and also note that you can’t export anything via NFS inside an encrypted volume or home dir. If that doesn’t help you can disable nfs by setting the nfs flag in the vagrantconfig_local.yaml file you just created. nfs: false 12 Chapter 2. Getting Started Kuma Documentation, Release latest Note: If you decide to run nfs: false, the system will be a lot slower. There is also the potential of running into weird issues with puppet, since the current puppet configurations do not currently support nfs: false. If you have other problems during vagrant up, please ask for more help. Errors after switching branches • You should occasionally re-run the Puppet setup, especially after updating code with major changes. This will ensure that the VM environment stays up to date with configuration changes and installation of additional services. – On the Host: vagrant provision – Inside the VM: sudo puppet apply /home/vagrant/src/puppet/manifests/dev-vagrant.pp Getting more help If you have more problems using vagrant, please: 1. Paste errors to pastebin 2. email [email protected]. 3. After you email dev-mdn, you can also ask in IRC 2.4 Celery and Rabbit Kuma uses Celery to enable offline task processing for long-running jobs like sending email notifications and rerendering the Knowledge Base. Though Celery supports multiple message backends, we use, and recommend that you use, RabbitMQ. RabbitMQ is an AMQP message broker written in Erlang. 2.4.1 When is Celery Appropriate You can use Celery to do any processing that doesn’t need to happen in the current request-response cycle. Examples are generating thumbnails, sending out notification emails, updating content that isn’t about to be displayed to the user, and others. Ask yourself the question: “Is the user going to need this data on the page I’m about to send them?” If not, using a Celery task may be a good choice. 2.4.2 RabbitMQ Installing RabbitMQ should be installed via your favorite package manager. It can be installed from source but has a number of Erlang dependencies. 2.4. Celery and Rabbit 13 Kuma Documentation, Release latest Configuring RabbitMQ takes very little configuration. # Start the server. sudo rabbitmq-server -detached # Set up the permissions. rabbitmqctl add_user kuma kuma rabbitmqctl add_vhost kuma rabbitmqctl set_permissions -p kuma kuma ".*" ".*" ".*" That should do it. You may need to use sudo for rabbitmqctl. It depends on the OS and how Rabbit was installed. 2.4.3 Celery Installing Celery (and Django-Celery) is part of our vendor library. You shouldn’t need to do any manual installation. Configuring and Running We set some reasonable defaults for Celery in settings.py. These can be overriden either in settings_local.py or via the command line when running manage.py celeryd. In settings_local.py you should set at least this, if you want to use Celery: CELERY_ALWAYS_EAGER = False This defaults to True, which causes all task processing to be done online. This lets you run Kuma even if you don’t have Rabbit or want to deal with running workers all the time. You can also configure the log level or concurrency. Here are the defaults: CELERYD_LOG_LEVEL = logging.INFO CELERYD_CONCURRENCY = 4 Then to start the Celery workers, you just need to run: ./manage.py celeryd This will start Celery with the default number of worker threads and the default logging level. You can change those with: ./manage.py celeryd --log-level=DEBUG -c 10 This would start Celery with 10 worker threads and a log level of DEBUG. 2.5 Running Kuma with mod_wsgi 2.5.1 Requirements • See the installation docs. • Apache HTTP server 14 Chapter 2. Getting Started Kuma Documentation, Release latest • mod_rewrite • mod_headers • mod_expires • mod_wsgi • Not mod_python! It is incompatible with mod_wsgi. 2.5.2 Overview Setting up Kuma to run as a WSGI application is fairly straightforward. You will need to install the requirements and clone the vendor repo as described in the installation docs. There are 3 steps once Kuma is installed: • Set the document root. • Set up aliases. • Some file permissions. • Set up WSGI itself. Apache Modules Most of the Apache modules are part of a default Apache install, but may need to be activated. If they aren’t installed, all of them, including mod_wsgi should be installable via your favorite package manager. WSGI Configuration In the Apache config (or <VirtualHost>) you will need the following: Note that values may be slightly different. DocumentRoot /path/to/kuma/webroot/ <Directory "/path/to/kuma/webroot/"> Options +FollowSymLinks </Directory> Alias /media/ "/path/to/kuma/media/" Alias /admin-media/ \ "/path/to/kuma/vendor/src/django/django/contrib/admin/media/" WSGISocketPrefix /var/run/wsgi WSGIDaemonProcess kuma processes=8 threads=1 \ maximum-requests=4000 WSGIProcessGroup kuma WSGIScriptAlias /k "/path/to/kuma/wsgi/kuma.wsgi" WSGISocketPrefix: May or may not be necessary. It was for me. WSGIDaemonProcess: processes should be set to the number of cores. threads should probably be left at 1. maximum-requests is good at between 4000 and 10000. 2.5. Running Kuma with mod_wsgi 15 Kuma Documentation, Release latest WSGIScriptAlias: Will make Kuma accessible from http://domain/k, and we use rewrites in webroot/.htaccess to hide the /k. This will change soon, and the .htaccess file won’t be necessary. The Alias directives let Kuma access its CSS, JS, and images through Apache, reducing the load on Django. Configuration Most of our settings.py is under version control, but can be overridden in a file called settings_local.py in the base of the app (the same place as settings.py). You can see example settings in docs/settings/settings_local.prod.py: from settings import * DEBUG = False TEMPLATE_DEBUG = False # The default database should point to the master. DATABASES = { ’default’: { ’NAME’: ’kitsune’, ’ENGINE’: ’django.db.backends.mysql’, ’HOST’: ’’, ’USER’: ’’, ’PASSWORD’: ’’, ’OPTIONS’: {’init_command’: ’SET storage_engine=InnoDB’}, }, } # This is used to hash some things in Django. SECRET_KEY = ’replace me with something long and random’ DEBUG_PROPAGATE_EXCEPTIONS = DEBUG 2.5.3 File Permissions To upload files, the webserver needs write access to media/uploads and all its subdirectories. The directories we currently use are: media/uploads media/uploads/avatars media/uploads/images media/uploads/images/thumbnails media/uploads/gallery/images media/uploads/gallery/images/thumbnails media/uploads/gallery/videos media/uploads/gallery/videos/thumbnails media/uploads and its subdirectories should never be added to version control, as they are installation-/contentspecific. Product Details JSON Some people have issues with django-mozilla-product-details and file permissions. The management command manage.py update_product_details writes a number of JSON files to disk, and the webserver 16 Chapter 2. Getting Started Kuma Documentation, Release latest then needs to read them. If you get file system errors from product_details, make sure the files are readable by the webserver (should be by default) and the directory is readable and executable. By default, product_details stores the JSON files in: vendor/src/django-mozilla-product-details/product_details/json This is configurable. If you have multiple web servers, they should share this data. You can set the PROD_DETAILS_DIR variable in settings_local.py to a different path, for example on NFS. 2.5.4 Debugging Debugging via WSGI is a little more interesting than via the dev server. One key difference is that you cannot use pdb. Writing to stdout is not allowed within the WSGI process, and will result in a Internal Server Error. There are three relevant cases for debugging via WSGI (by which I mean, where to find stack traces): Apache Error Page So you’ve got a really bad error and you aren’t even seeing the Kuma error page! This is usually caused by an uncaught exception during the WSGI application start-up. Our WSGI script tries to run all the initial validation that the dev server runs, to catch these errors early. So where is the stack trace? You’ll need to look in your Apache error logs. Where these are is OS-dependent, but a good place to look is /var/log/httpd. If you are using SSL, also check the SSL VirtualHost‘s logs, for example /var/log/httpd/ssl_error_log. With DEBUG=True With DEBUG = True in your settings_local.py, you will see a stack trace in the browser on error. Problem solved! With DEBUG=False With DEBUG = False in your settings_local.py, you’ll see our Server Error message. You can still get stack traces, though, by setting the ADMINS variable in settings_local.py: ADMINS = ( (’me’, ’[email protected]’), ) Django will email you the stack trace. Provided you’ve set up email. 2.5.5 Reloading WSGI WSGI keeps Python and Kuma running in an isolated process. That means code changes aren’t automatically reflected on the server. In most default configurations of mod_wsgi, you can simply do this: touch wsgi/kuma.wsgi That will cause the WSGI process to reload. 2.5. Running Kuma with mod_wsgi 17 Kuma Documentation, Release latest 2.6 Email from Kuma The default settings for Kuma do not send email. If you want to get email, you should double check one thing first: are there any rows in the tidings_watch table? If there are, you may be sending email to real users. The script in scripts/anonymize.sql will truncate this table. Simply run it against your Kuma database: mysql -ukuma -p kuma < scripts/anonymize.sql 2.6.1 Sending Email So now you know you aren’t emailing real users, but you’d still like to email yourself and test email in general. There are a few settings you’ll need to use. First, set the EMAIL_BACKEND. This document assumes you’re using the SMTP mail backend. EMAIL_BACKEND = ’django.core.mail.backends.smtp.EmailBackend’ If you have sendmail installed and working, that should do it. However, you might get caught in spam filters. An easy workaround for spam filters or not having sendmail working is to send email via a Gmail account. EMAIL_USE_TLS = True EMAIL_PORT = 587 EMAIL_HOST = ’smtp.gmail.com’ EMAIL_HOST_USER = ’<your gmail address>@gmail.com’ EMAIL_HOST_PASSWORD = ’<your gmail password>’ Yeah, you need to put your Gmail password in a plain text file on your computer. It’s not for everyone. Be very careful copying and pasting settings from settings_local.py if you go this route. 2.7 Search Kuma uses Elasticsearch to power its on-site search facility. Elasticsearch search gives us a number of advantages over MySQL’s full-text search or Google’s site search. • Much faster than MySQL. * And reduces load on MySQL. • We have total control over what results look like. • We can adjust searches with non-visible content. • We don’t rely on Google reindexing the site. • We can fine-tune the algorithm ourselves. 2.7.1 Installing Elasticsearch Search We currently require Elasticsearch 0.90.9. You may be able to install this from a package manager like yum, aptitude, or brew. If not, you can easily download the source and compile it. Generally all you’ll need to do is: $ cd elasticsearch-0.90.9 $ bin/elasticsearch -f Then run the Kuma search tests: 18 Chapter 2. Getting Started Kuma Documentation, Release latest $ ./manage.py test -s --noinput --logging-clear-handlers search If the tests pass, everything is set up correctly! 2.7.2 Using Elasticsearch Having Elasticsearch installed will allow the search tests to run, which may be enough. But you want to work on or test the search app, you will probably need to actually see search results! The Easy, Sort of Wrong Way The easiest way to start Elasticsearch for testing is: $ cd path/to/elasticsearch-0.90.9 $ bin/elasticsearch -f Then from the Kuma source code path: $ ./manage.py reindex If you need to update the search indexes: $ ./manage.py reindex While this method is very easy, you will need to reindex after any time you run the search tests, as they will overwrite the data files Elasticsearch uses. The Ellaborate, Kinda Proper Way Assuming you’re running the full stack with foreman start (or any other way that makes sure the Celery workers run) there is a better way: • Open the Django admin UI under http://127.0.0.1:8000/admin/search/index/ or https://developerlocal.allizom.org/admin/search/index/ if you’re using Vagrant. • Add a search index by clicking on the “Add index” button in the top right corner, safe it by clicking on the same button in the lower right corner to safe it to the database. • On the search index list view again, select the just created index (the top most) and select “Populate search index with Celery task” from the actions dropdown below. • Once the population is ready the “populated” field with show up as a green checkbox image. You’ll also get an email (probably via the console if you’re developing kuma locally) notifying you of the completion. • To actually enable that newly created search index you have to promote it now. On the search index list view again, select the just created index (the top most) and select “Promote search index” from the actions dropwdown below. • Once the search index is promoted the “promoted” and the “is current index” field will show up as a green checkbox image. The index is now live. Similarly you can also demote a search index and it will automatically fall back to the previously created index (by created date). That helps to figure out issues in production and should allow for a smooth deployment of search index changes. It’s recommmended to keep a few search indexes around just in case. If no index is created in the admin UI the fallback “main_index” index will be used instead. 2.7. Search 19 Kuma Documentation, Release latest Warning: If you delete any of the search indexes in the admin interface they will be deleted on Elasticsearch as well. 2.8 Development We strongly suggest using a Vagrant-managed VM if you can. Or, you can use the manual installation steps. 2.8.1 Running Kuma in Vagrant VM If you are using a Vagrant-managed VM, you can start all Kuma servers and services with: vagrant ssh foreman start 2.8.2 Running Kuma manually If you are using manual installation, you can run the django server with: ./manage.py runserver and the kumascript service with: node kumascript/run.js Note: Before running kumascript, you need to install the node.js fibers module by running npm install fibers. 2.8.3 Log in You can log into the wiki using Persona or via the django admin interface. If you use the admin interface, you can log in as the user you created during installation or on the vagrant VM use login admin with password admin. 2.8.4 Set up permissions Some features are only available to privileged users. To manage permissions use the Auth -> Users section of the django admin interface. 2.8.5 Compiling Stylus Files If you’re updating the Stylus CSS files, you’ll need to compile them before you can see your updates within the browser. To compile stylus files, run the following from the command line: ./scripts/compile-stylesheets The relevant CSS files will be generated and placed within the media/redesign/css directory. You can add a -w flag to that call to compile stylesheets upon save. 20 Chapter 2. Getting Started Kuma Documentation, Release latest 2.8.6 Hacking on bleeding edge features To hack on the features not yet ready for production you have to enable them first. Enable Kumascript Kuma uses a separate nodejs-based service to process templates in wiki pages. Its use is disabled by default, to enable: open the django admin interface and in the Constance section change the value of KUMASCRIPT_TIMEOUT parameter to a positive value (such as 2.0 seconds). 2.8.7 Running the Tests A great way to check that everything really is working is to run the test suite. Django tests If you’re not using the vagrant VM, you’ll need to add an extra grant in MySQL for your database user: GRANT ALL ON test_NAME.* TO USER@localhost; Where NAME and USER are the same as the values in your database configuration. The test suite will create and use this database, to keep any data in your development database safe from tests. Running the test suite is easy: ./manage.py test -s --noinput --logging-clear-handlers Note that this will try (and fail) to run tests that depend on apps disabled via INSTALLED_APPS. You should run a subset of tests: ./manage.py test actioncounters authkeys contentflagging devmo landing kpi kuma For more information, see the test documentation. Kumascript tests If you’re changing Kumascript, be sure to run its tests too. See https://github.com/mozilla/kumascript 2.8.8 Coding Conventions Tests • Avoid naming test files test_utils.py, since we use a library with the same name. test__utils.py instead. Use • If you’re expecting reverse to return locales in the URL, use LocalizingClient instead of the default client for the TestCase class. 2.8. Development 21 Kuma Documentation, Release latest 2.9 Localization Kuma is localized with gettext. User-facing strings in the code or templates need to be marked for gettext localization. We use Verbatim to provide an easy interface to localizing these files. Localizers are also free to download the PO files and use whatever tool they are comfortable with. 2.9.1 Making Strings Localizable Making strings in templates localizable is exceptionally easy. Making strings in Python localizable is a little more complicated. The short answer, though, is just wrap the string in _(). Interpolation A string is often a combination of a fixed string and something changing, for example, Welcome, James is a combination of the fixed part Welcome,, and the changing part James. The naive solution is to localize the first part and the follow it with the name: _(’Welcome, ’) + username This is wrong! In some locales, the word order may be different. Use Python string formatting to interpolate the changing part into the string: _(’Welcome, {name}’).format(name=username) Python gives you a lot of ways to interpolate strings. The best way is to use Py3k formatting and kwargs. That’s the clearest for localizers. The worst way is to use %(label)s, as localizers seem to have all manner of trouble with it. Options like %s and {0} are somewhere in the middle, and generally OK if it’s clear from context what they will be. Localization Comments Sometimes, it can help localizers to describe where a string comes from, particularly if it can be difficult to find in the interface, or is not very self-descriptive (e.g. very short strings). If you immediately precede the string with a comment that starts L10n:, the comment will be added to the PO file, and visible to localizers. Adding Context with msgctxt Strings may be the same in English, but different in other languages. English, for example, has no grammatical gender, and sometimes the noun and verb forms of a word are identical. To make it possible to localize these correctly, we can add “context” (known in gettext as “msgctxt”) to differentiate two otherwise identical strings. For example, the string “Search” may be a noun or a verb in English. In a heading, it may be considered a noun, but on a button, it may be a verb. It’s appropriate to add a context (like “button”) to one of them. Generally, we should only add context if we are sure the strings aren’t used in the same way, or if localizers ask us to. 22 Chapter 2. Getting Started Kuma Documentation, Release latest Plurals “You have 1 new messages” grates on discerning ears. Fortunately, gettext gives us a way to fix that in English and other locales, the ngettext function: ngettext(’singular’, ’plural’, count) A more realistic example might be: ngettext(’Found {count} result.’, ’Found {count} results’, len(results)).format(count=len(results)) This method takes three arguments because English only needs three, i.e., zero is considered “plural” for English. Other locales may have different plural rules, and require different phrases for, say 0, 1, 2-3, 4-10, >10. That’s absolutely fine, and gettext makes it possible. Strings in Templates When putting new text into a template, all you need to do is wrap it in a _() call: <h1>{{ _(’Heading’) }}</h1> Adding context is easy, too: <h1>{{ _(’Heading’, ’context’) }}</h1> L10n comments need to be Jinja2 comments: {# L10n: Describes this heading #} <h1>{{ _(’Heading’) }}</h1> Note that Jinja2 escapes all content output through {{ }} by default. To put HTML in a string, you’ll need to add the |safe filter: <h1>{{ _(’Firefox <span>Help</span>’)|safe }}</h1> To interpolate, you should use one of two Jinja2 filters: |f() or, in some cases, |fe(). |f() has exactly the same arguments as u’’.format(): {{ _(’Welcome, {name}!’)|f(name=request.user.username) }} The |fe() is exactly like the |f() filter, but escapes its arguments before interpolating, then returns a “safe” object. Use it when the localized string contains HTML: {{ _(’Found <strong>{0}</strong> results.’)|fe(num_results) }} Note that you do not need to use |safe with |fe(). Also note that while it may look similar, the following is not safe: {{ _(’Found <strong>{0}</strong> results.’)|f(num_results)|safe }} The ngettext function is also available: {{ ngettext(’Found {0} result.’, ’Found {0} results.’, num_results)|f(num_results) }} 2.9. Localization 23 Kuma Documentation, Release latest Using {% trans %} Blocks for Long Strings When a string is very long, i.e. long enough to make Github scroll sideways, it should be line-broken and put in a {% trans %} block. {% trans %} blocks work like other block-level tags in Jinja2, except they cannot have other tags, except strings, inside them. The only thing that should be inside a {% trans %} block is printing a string with {{ string }}. These are defined in the opening {% trans %} tag: {% trans user=request.user.username %} Thanks for registering, {{ user }}! We’re so... hope that you’ll... {% trans %} Strings in Python NB: Whenever you are adding a string in Python, ask yourself if it really needs to be there, or if it should be in the template. Keep logic and presentation separate! Strings in Python are more complex for two reasons: 1. We need to make sure we’re always using Unicode strings and the Unicode-friendly versions of the functions. 2. If you use the ugettext function in the wrong place, the string may end up in the wrong locale! Here’s how you might localize a string in a view: from tower import ugettext as _ def my_view(request): if request.user.is_superuser: msg = _(u’Oh hi, staff!’) else: msg = _(u’You are not staff!’) Interpolation is done through normal Python string formatting: msg = _(u’Oh, hi, {user}’).format(user=request.user.username) ugettext supports context, too: msg = _(’Search’, ’context’) L10n comments are normal one-line Python comments: # L10n: A message to users. msg = _(u’Oh, hi there!’) If you need to use plurals, import the function ungettext from Tower: from tower import ungettext, ugettext as _ n = len(results) msg = ungettext(’Found {0} result’, ’Found {0} results’, n).format(n) Lazily Translated Strings You can use ugettext or ungettext only in views or functions called from views. If the function will be evaluated when the module is loaded, then the string may end up in English or the locale of the last request! (We’re tracking 24 Chapter 2. Getting Started Kuma Documentation, Release latest down that issue.) Examples include strings in module-level code, arguments to functions in class definitions, strings in functions called from outside the context of a view. To localize these strings, you need to use the _lazy versions of the above methods, ugettext_lazy and ungettext_lazy. The result doesn’t get translated until it is evaluated as a string, for example by being output or passed to unicode(): from tower import ugettext_lazy as _lazy PAGE_TITLE = _lazy(u’Page Title’) ugettext_lazy also supports context. It is very important to pass Unicode objects to the _lazy versions of these functions. Failure to do so results in significant issues when they are evaluated as strings. If you need to work with a lazily-translated string, you’ll first need to convert it to a unicode object: from tower import ugettext_lazy as _lazy WELCOME = _lazy(u’Welcome, %s’) def my_view(request): # Fails: WELCOME % request.user.username # Works: unicode(WELCOME) % request.user.username 2.9.2 Getting the Localizations Localizations are not stored in this repository, but are in Mozilla’s SVN: http://svn.mozilla.org/projects/mdn/trunk/locale You don’t need the localization files for general development. However, if you need them for something, they’re pretty easy to get: $ cd kuma $ svn checkout https://svn.mozilla.org/projects/mdn/trunk/locale (Alternatively, you can do yourself a favor and use: $ git svn clone -r HEAD https://svn.mozilla.org/projects/mdn/trunk/locale if you’re a git fan.) 2.9.3 Updating the Localizations Updating strings is easy. But when we add or update strings, we need to update Verbatim templates and PO files for localizers. If you commit changes to SVN without updating Verbatim, localizers will have merge head-aches. 1. Check out the localizations (See get-localizations) 2. Run the following in the virtual machine (see installation-vagrant): $ python manage.py extract 2.9. Localization 25 Kuma Documentation, Release latest 3. Commit the POT file. If you used svn checkout above: $ cd locale $ svn up $ svn ci -m "MDN string update YYYY-MM-DD" If you used git svn clone above: $ $ $ $ $ cd locale git svn fetch git add -A git commit -m "MDN string update YYYY-MM-DD" git svn dcommit Note: You need verbatim permissions for the following. If you don’t have permissions, email groovecoder or mathjazz to do the following ... 4. Go to the MDN templates on Verbatim 5. Click ‘Update all from VCS’ 6. ssh to sm-verbatim01 (See L10n:Verbtim on wiki.mozilla.org) 7. Update all locales against templates: sudo su verbatim cd /data/www/localize.mozilla.org/verbatim/pootle_env/Pootle POOTLE_SETTINGS=localsettings.py python2.6 manage.py update_against_templates --project=mdn -v 2 2.9.4 Adding a new Locale Adding a new locale is also easy. 1. Check out the localizations (See get-localizations) 2. Follow the “Add locale” instructions on wiki.mozilla.org. 3. Update your locale repo to get the new locale: $ cd locale $ svn up 4. Add the locale to MDN_LANGUAGES in settings.py 5. Verify django loads new locale without errors by visiting the locale’s home page. E.g., https://developerlocal.allizom.org/ml/ 6. BONUS: Use podebug to test a fake translation of the locale: $ cd locale $ podebug --rewrite=bracket templates/LC_MESSAGES/messages.pot ml/LC_MESSAGES/messages.po $ ./compile-mo.sh . Restart the django server and re-visit the new locale to verify it shows “translated” strings in the locale. 26 Chapter 2. Getting Started Kuma Documentation, Release latest 2.10 The Kuma Test Suite Kuma has a fairly comprehensive Python test suite. Changes should not break tests–only change a test if there is a good reason to change the expected behavior–and new code should come with tests. 2.10.1 Running the Test Suite If you followed the steps in the installation docs, then all you should need to do to run the test suite is: ./manage.py test However, that doesn’t provide the most sensible defaults. Here is a good command to alias to something short: ./manage.py test -s --noinput --logging-clear-handlers The -s flag is important if you want to be able to drop into PDB from within tests. Some tests will fail. See Running a Subset below for running the subset that is expected to pass. Some other helpful flags are: -x: Fast fail. Exit immediately on failure. No need to run the whole test suite if you already know something is broken. --pdb: Drop into PDB on an uncaught exception. (These show up as E or errors in the test results, not F or failures.) --pdb-fail: Drop into PDB on a test failure. This usually drops you right at the assertion. Running a Subset You can run part of the test suite by specifying the apps you want to run, like: ./manage.py test kuma You can also exclude tests that match a regular expression with -e: ./manage.py test -e "search" To run the subset of tests that should pass: ./manage.py test actioncounters contentflagging devmo landing kuma See the output of ./manage.py test --help for more arguments. The Test Database The test suite will create a new database named test_%s where %s is whatever value you have for settings.DATABASES[’default’][’NAME’]. Make sure the user has ALL on the test database as well. When the schema changes, you may need to drop the test database. You can also run the test suite with FORCE_DB once to cause Django to drop and recreate it: FORCE_DB=1 ./manage.py test -s --noinput --logging-clear-handlers 2.10. The Kuma Test Suite 27 Kuma Documentation, Release latest 2.10.2 Adding Tests Code should be written so it can be tested, and then there should be tests for it. When adding code to an app, tests should be added in that app that cover the new functionality. All apps have a tests module where tests should go. They will be discovered automatically by the test runner as long as the look like a test. 2.10.3 Changing Tests Unless the current behavior, and thus the test that verifies that behavior is correct, is demonstrably wrong, don’t change tests. Tests may be refactored as long as its clear that the result is the same. 2.10.4 Removing Tests On those rare, wonderful occasions when we get to remove code, we should remove the tests for it, as well. If we liberate some functionality into a new package, the tests for that functionality should move to that package, too. 28 Chapter 2. Getting Started