Hello!

My question: How to efficiently calculate the layer values in a neural net?

I was playing around with fitting (simple) neural nets in Stan. I started with the code provided here. However, I wanted a bit more flexibility in modelling the hidden layers. For that I modified the above code and took some inspiration from Neal (1996).

My first approach is the function above `calc_u`

, but I thought it could not be the most efficient thing. (My problem is basically how to add the bias term efficiently.) Therefore, I tried the alternative `calc_u_alt`

, which seemed a bit more elegant. However, when running and comparing the two, the first one is faster. Iâ€™m not a programmer, so for me itâ€™s not so obvious why that is. Is it because it basically just does the matrix multiplication once per layer?

Is there actually a way to improve the calculations?

```
functions {
// Calculate values for layers
matrix calc_u(matrix X, matrix weights, vector bias) {
matrix[rows(X), rows(weights)] first;
matrix[rows(X), rows(weights)] res;
first = X*weights;
for (j in 1:rows(weights))
res[,j] = bias[j] + first[,j];
return res;
}
// Calculate values for layers (alternative)
matrix calc_u_alt(matrix X, matrix weights, vector bias) {
matrix[rows(X), cols(weights)] res;
for(j in 1:cols(weights))
res[,j] = X*weights[,j] + bias[j];
return res;
}
}
```

Thanks in advance for any help!

Cheers,

Max

PS: Since itâ€™s my first time posting, I canâ€™t include the R file to run the model here, sorry!

simple_nn.stan (2.7 KB)