New Jenkins Jobs Tutorial

[seantalts]

Hey all,

I’ve been switching the Jenkins jobs over to pipelines. There are a few things that are changing now as a result from a user interface perspective.

Repo build status

You can now see a repo’s build status split out among branches and PRs:

and

Please ignore the “downstream tests” job status - this is the job that is triggered by downstream builds that want to test that e.g. a commit to the math repo works upstream with both CmdStan and Stan.

Job stages

Job stages are now shown in the new pipeline UI like so:

(You get to this page by clicking on the develop from the previous page)
You can see the logs here by clicking on a stage:

Debugging a build failure

I really like the pipeline steps view for debugging:

In this view you can click on the individual steps for just their console output:

For parallel build sections (many of them are now parallelized), the UI is pretty bad. So you will see all of the sections that were executed in parallel as having failed simultaneously in any of the graphical UIs. If the high level Console Output doesn’t give you any clues, the only way I’ve found to diagnose which stage actually failed is to click on each of them in the Pipeline Steps view and look for the one that doesn’t have the console output “Sending interrupt signal to process”.

The general job’s console output sometimes also has additional error messages, found here:

Retrying a job

Before, we would try to figure out the correct incantation to ask Jenkins to retest from the comments. While convenient if one remembered the correct incantation, often you had to click through and make sure it triggered anyway. Now the only way to retrigger a job is to go to the job page and click Build.

If you don’t see a little circular “Replay” option in the new UI in this area (it’s missing from this screenshot as well):

Then you should exit the new UI by pressing the door icon. Then you’ll see something like this:

If you click on PR-833 here in the top left, you will now see a Build with Parameters button:

This will bring up a screen where you can input parameters. You only need to adjust these if you’re trying to test a specific branch combination across repos; otherwise leave them as their default values.

(I’ll add more text on this screen soon).

This will trigger the build and automatically update the PR’s status on github.

If there are any questions please ask them here! Thanks everyone.

[Bob_Carpenter]

@seantalts

Thanks everyone.

I think we should be thaking you. So, thank you. You don’t realize how many moving pieces there are until they get linearized as images!

[mitzimorris]

this is great - helpful hint - remember to log in to Jenkins

[syclik]

I’m gonna sticky this thread for a while on the Developers category. Looks like we have some issues with the pipelines for Math right now.

[seantalts]

Also, if there are issues where Jenkins isn’t running tests or they’re failing spuriously (or you think Jenkins is somehow broken), please post an issue to github for the repo involved, assign me and the “continuous integration” label if it’s not already posted. Thanks!

[seantalts]

I’ve installed a new UI called “Blue Ocean” that you may see now. I think it makes diagnosing job failures easier, but you might want to go back to the old version for something that isn’t as good in the new one yet. To do that you can click on the door-based exit-like icon in the top right of every page:

[seantalts]

Jobs for branches requiring specific upstream branches

This follows “Retrying a job” above, more or less, but here is more detail.

1. Click on the Jenkins job link in the PR.
2. If you’re in the Blue Ocean view, click the little door in the upper right (See above post)
3. Click from the current run back to the PR job (in the screenshot, PR-832):
4. Click “Build with Parameters”:
5. Put in the upstream PRs to test against:
   
   
6. Press Build!