Common tasks

Running the tests

You can run flake8, isort and the py.test suite inside the Vagrant VM, using:

vagrant ~/treeherder$ ./

Or for more control, run each tool individually:

  • py.test:

    vagrant ~/treeherder$ py.test tests/
    vagrant ~/treeherder$ py.test tests/log_parser/
    vagrant ~/treeherder$ py.test tests/etl/ -k test_ingest_builds4h_jobs

    To run all tests, including slow tests that are normally skipped, use:

    vagrant ~/treeherder$ py.test --runslow tests/

    For more options, see py.test --help or

  • flake8:

    vagrant ~/treeherder$ flake8

    NB: If running flake8 from outside of the VM, ensure you are using the same version as used on Travis (see requirements/dev.txt).

  • isort (checks the Python import style):

    To run interactively:

    vagrant ~/treeherder$ isort

    Or to apply all changes without confirmation:

    vagrant ~/treeherder$ isort --apply

    NB: isort must be run from inside the VM, since a populated (and up to date) virtualenv is required so that isort can correctly categorise the imports.

Profiling API endpoint performance

On our development (vagrant) instance we have django-debug-toolbar installed, which can give information on exactly what SQL is run to generate individual API endpoints. Just navigate to an endpoint (example: http://localhost:8000/api/repository/) and you should see the toolbar to your right.

Add a new Mercurial repository

To add a new repository, the following steps are needed:

  • Append new repository information to the fixtures file located at treeherder/model/fixtures/repository.json

  • Load the file you edited with the loaddata command:

    vagrant ~/treeherder$ ./ loaddata repository
  • Restart any running gunicorn/celery processes.

For more information on adding a new GitHub repository see Add GitHub repository.

Building the docs locally

  • Either vagrant ssh into the VM, or else activate a virtualenv on the host machine.

  • From the root of the Treeherder repo, run:

    > pip install -r requirements/docs.txt
    > make -C docs html
  • The built docs can then be found inside docs/_build/html/.

Sharing UI-only changes with others using GitHub Pages

It’s possible to share UI-only changes with others (for prototyping/testing) using GitHub Pages. This is recommended over pushing a custom branch to stage, unless the feature requires that you be logged into Treeherder (which won’t work cross-domain).

To do this:

  • Fork the Treeherder repository to your own GitHub account.

  • Create a gh-pages branch locally based on the feature branch you wish to test, that is configured to point at production’s API. eg:

    git checkout (your feature branch)
    git checkout -b gh-pages
    SERVICE_DOMAIN= yarn build
    git add -f dist/
    git commit -m "Add dist directory containing built UI"
  • Push the gh-pages branch to your Treeherder fork.

  • Tell people to visit: https://<your-username>

Updating package.json

  • Always use yarn to make changes, not npm, so that yarn.lock remains in sync.
  • Add new packages using yarn add <PACKAGE> --no-bin-links (yarn.lock will be automatically updated).
  • After changes to package.json use yarn install --no-bin-links to install them and automatically update yarn.lock.
  • For more details see the Yarn documentation.

Note: To work around symlink issues for Windows hosts, use --no-bin-links with any command that adds/modifies packages. Whilst this is technically unnecessary with non-Windows hosts, it’s still recommended since otherwise your local changes might inadvertently rely on node_modules/.bin/ symlinks that won’t exist in a newly created Vagrant environment. Unfortunately yarn doesn’t yet support setting this option via the global yarn config, otherwise we could just enable it by default.

Releasing a new version of the Python client

  • Determine whether the patch, minor or major version should be bumped, by inspecting the client Git log.

  • File a separate bug for the version bump.

  • Open a PR to update the version listed in

  • Use Twine to publish both the sdist and the wheel to PyPI, by running the following from the root of the Treeherder repository:

    > pip install -U twine wheel
    > cd treeherder/client/
    > rm -rf dist/*
    > python sdist bdist_wheel
    > twine upload dist/*
  • File a Release Engineering::Buildduty bug requesting that the sdist and wheel releases (plus any new dependent packages) be added to the internal PyPI mirror. For an example, see bug 1236965.

Hide Jobs with Tiers

To hide jobs we use the job’s tier setting. Jobs with tier of 3 are hidden by default. There are two ways to set a job to be hidden in Treeherder:

  • TaskCluster - Edit the task definition to include the tier setting in the Treeherder section.
  • BuildBot - You must get the signature hash of the job from the UI and add that signature hash to the file in the Treeherder repo. To get the signature, click the job and then click the sig link in the Job Details Panel. That will place the signature hash in the filter field.