Common tasks

Running the tests

You can run flake8, isort and the py.test suite inside the Vagrant VM, using:

vagrant ~/treeherder$ ./

Or for more control, run each tool individually:

  • py.test:

    vagrant ~/treeherder$ py.test tests/
    vagrant ~/treeherder$ py.test tests/log_parser/
    vagrant ~/treeherder$ py.test tests/etl/ -k test_ingest_builds4h_jobs

    To run all tests, including slow tests that are normally skipped, use:

    vagrant ~/treeherder$ py.test --runslow tests/

    For more options, see py.test --help or

  • flake8:

    vagrant ~/treeherder$ flake8

    NB: If running flake8 from outside of the VM, ensure you are using the same version as used on Travis (see requirements/dev.txt).

  • isort (checks the Python import style):

    To run interactively:

    vagrant ~/treeherder$ isort

    Or to apply all changes without confirmation:

    vagrant ~/treeherder$ isort --apply

    NB: isort must be run from inside the VM, since a populated (and up to date) virtualenv is required so that isort can correctly categorise the imports.

Profiling API endpoint performance

On our development (vagrant) instance we have django-debug-toolbar installed, which can give information on exactly what SQL is run to generate individual API endpoints. Just navigate to an endpoint (example: http://localhost:8000/api/repository/) and you should see the toolbar to your right.

Add a new Mercurial repository

To add a new repository, the following steps are needed:

  • Append new repository information to the fixtures file located at treeherder/model/fixtures/repository.json

  • Load the file you edited with the loaddata command:

    vagrant ~/treeherder$ ./ loaddata repository
  • Restart any running gunicorn/celery processes.

For more information on adding a new GitHub repository see Add GitHub repository.

Building the docs locally

  • Either vagrant ssh into the VM, or else activate a virtualenv on the host machine.

  • From the root of the Treeherder repo, run:

    > pip install -r requirements/docs.txt
    > make -C docs html
  • The built docs can then be found inside docs/_build/html/.

Sharing UI-only changes with others using GitHub Pages

It’s possible to share UI-only changes with others (for prototyping/testing) using GitHub Pages. This is recommended over pushing a custom branch to stage, unless the feature requires that you be logged into Treeherder (which won’t work cross-domain).

To do this:

  • Fork the Treeherder repository to your own GitHub account.

  • Create a gh-pages branch locally based on the feature branch you wish to test, that is configured to point at production’s API. eg:

    git checkout (your feature branch)
    git checkout -b gh-pages
    SERVICE_DOMAIN= yarn build
    git add -f dist/
    git commit -m "Add dist directory containing built UI"
  • Push the gh-pages branch to your Treeherder fork.

  • Tell people to visit: https://<your-username>

Updating package.json

  • Always use yarn to make changes, not npm, so that yarn.lock remains in sync.
  • Add new packages using yarn add <PACKAGE> --no-bin-links (yarn.lock will be automatically updated).
  • After changes to package.json use yarn install --no-bin-links to install them and automatically update yarn.lock.
  • For more details see the Yarn documentation.

Note: To work around symlink issues for Windows hosts, use --no-bin-links with any command that adds/modifies packages. Whilst this is technically unnecessary with non-Windows hosts, it’s still recommended since otherwise your local changes might inadvertently rely on node_modules/.bin/ symlinks that won’t exist in a newly created Vagrant environment. Unfortunately yarn doesn’t yet support setting this option via the global yarn config, otherwise we could just enable it by default.

Releasing a new version of the Python client

  • Determine whether the patch, minor or major version should be bumped, by inspecting the client Git log.

  • File a separate bug for the version bump.

  • Open a PR to update the version listed in

  • Use Twine to publish both the sdist and the wheel to PyPI, by running the following from the root of the Treeherder repository:

    > pip install -U twine wheel
    > cd treeherder/client/
    > rm -rf dist/*
    > python sdist bdist_wheel
    > twine upload dist/*
  • File a Release Engineering::Buildduty bug requesting that the sdist and wheel releases (plus any new dependent packages) be added to the internal PyPI mirror. For an example, see bug 1236965.