Loading buildbot data

In order to start ingesting data, you need to turn on a celery worker with a ‘-B’ option. In this way the worker can run some scheduled tasks that are defined in treeherder.config.settings.CELERYBEAT_SCHEDULE. Here is a brief description of what each periodic task will do for you:

Retrieves and store all the latest pushes (a.k.a. resultsets) from the available repositories. You need to have this running before you can start ingestiong job data. No pushes, no jobs.
Retrieves and store buildbot pending jobs using RelEng buildapi service
Same as before, but for running jobs
Same as before, but it collects all the jobs completed in the last 4 hours.

The following is a data flow diagram which can help to understand better how these tasks are used by treeherder