Loading Pulse data¶
By default, running the Docker container with
docker-compose up will ingest data
autoland repo using a shared Pulse Guardian user. You can configure this the following ways:
- Specify a custom set of repositories for which to ingest data
- Create a custom Pulse User on Pulse Guardian
Custom list of Repositories¶
Set the environment variable of
Visit Pulse Guardian, sign in, and create a Pulse User. It will ask you to set a username and password. Remember these as you'll use them in the next step. This is recommended, because using the default value MAY cause you to miss some data, if it was already ingested by another user Unfortunately, Pulse doesn't support creating queues with a guest account.
If your Pulse User was username:
foo and password:
bar, your Pulse URL
Be sure you do NOT use quotes when setting the value of PULSE_URL. Otherwise, you may get an
KeyError: 'No such transport: '
On your localhost set PULSE_URL as follows, subsituting the url above:
See Starting a local Treeherder instance for more info.
Advanced Celery Configuration¶
If you only want to ingest the Pushes and Tasks, then the default will do that for you. But if you want to do other processing like parsing logs, etc, then you can specify the other queues you would like to process.
Open a new terminal window. To run all the queues do:
docker-compose run backend celery -A treeherder worker --concurrency 1
You will see a list of activated queues. If you wanted to narrow that down, then note which queues you'd like to run and add them to a comma-separated list. For instance, to only do Log Parsing:
celery -A treeherder worker -Q log_parser,log_parser_fail --concurrency 1
To post data to your own Pulse exchange, you can use the
management command. This command takes the
payload_file. The payload file must be a
JSON representation of
a job as specified in the YML Schema.
Here is a set of example parameters that could be used to run it:
./manage.py publish_to_pulse mozilla-inbound.staging amqp://treeherder-test:firstname.lastname@example.org:5672/ ./scratch/test_job.json
You can use the handy Pulse Inspector to view messages in your exchange to
test that they are arriving at Pulse the way you expect. Each exchange has its
own inspector that can be accessed like so: