Compound Assignment Reference Semantics and Gradient Calculations

General
1

This is a two-parter:

  1. What are the reference semantics for a compound assignment in Stan? E.g., in C++, x += y modifies x in-place. Does Stan retain this behavior this behavior, or is the expression expanded to x = x + y during compilation?

  2. If compound operators do modify in-place, does this lead to restrictions on usage? E.g., I could see in-place operations on parameters causing issues with autodiff - (earlier versions of?) Tensorflow mostly did not allow in-place operations on any variables for which gradients were computed.

This question arose when writing a custom log-probablity function; I ended up using the expanded (not compound) alternative to be safe, but I’m curious now and couldn’t find much on the topic.

2

Sorry this didn’t get answered. I guess people were leaving it for me. For questions about the language, you can ping me or @WardBrian.

  1. x += y and x = x + y are functionally synonymous in C++ and Stan. It’s a bit subtle on the C++ side because the += form is considered primitive and best practice is to implement x + y in terms of +=.

  2. No, it does not lead to restrictions on usage. Stan uses dynamic autodiff like PyTorch rather than the static autodiff of TensorFlow (though we built Stan before either of those packages existed, we debated which approach to use as both were known in the existing autodiff literature at the time).

The details are in our autodiff paper, which is a bit out of date, but has the details of how += works:

The C++ code is actually pretty simple, too.

You can always look at how a Stan program is compiled by inspecting the .hpp file generated. Here’s an example.

parameters {
  real y;
  real z;
}
transformed parameters {
  real x = y;
  x += z;
}

I compiled using CmdStanPy:

>>> import cmdstanpy as csp
>>> csp.CmdStanModel(stan_file = 'pluseq.stan')
17:56:52 - cmdstanpy - INFO - compiling stan file /Users/bcarpenter/temp2/pluseq.stan to exe file /Users/bcarpenter/temp2/pluseq
17:57:32 - cmdstanpy - INFO - compiled model executable: /Users/bcarpenter/temp2/pluseq
CmdStanModel: name=pluseq
	 stan_file=/Users/bcarpenter/temp2/pluseq.stan
	 exe_file=/Users/bcarpenter/temp2/pluseq
	 compiler_options=stanc_options={}, cpp_options={}

The .hpp file lives in the same directory. Opening it up, you can see this:

      x = y;
      current_statement__ = 4;
      x = (x + z);

So it looks like it’s getting expanded out. I’m not sure why as our C++ API supports the += form with the right behavior. Maybe @WardBrian will know why code is being generated that way.

3

This recently came up in another post: Intended behavior of vector[array_idx] += vector?

The short answer is there is no reference semantics. In cases where C++ would, we end up creating a hard copy to avoid it.

The code is generated that way in the scalar case because of how it is required to be when non-scalars are involved, and it is easier to have it be uniformly handled on our end.

1 Like
4

Thanks both! So a += (with reference semantics) exists for the var class in the C++ library, but it’s almost never used when compiling from a += for parameters in the Stan language because naive usage would create issues for non-scalar parameters. Am I getting that right?