# Separable logistic regression performance optimizations

**URL:** <https://discourse.mc-stan.org/t/separable-logistic-regression-performance-optimizations/35628>\
**Category:** Modeling\
**Tags:** logistic-regression, performance, fitting-issues\
**Created:** [June 24, 2024, 5:47pm UTC](https://discourse.mc-stan.org/t/separable-logistic-regression-performance-optimizations/35628 "2024-06-24T17:47:30Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paul\_Jones](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/paul_jones/32/17565_2.png) [@Paul\_Jones](https://discourse.mc-stan.org/u/Paul_Jones)\
**Post date:** [June 24, 2024, 5:47pm UTC](https://discourse.mc-stan.org/t/separable-logistic-regression-performance-optimizations/35628/1 "2024-06-24T17:47:31Z")

</div>

I’m experiencing significant performance slowdowns in STAN when solving logistic regression problems with separable or near-separable data.

Performance is important, so we have implemented a few optimizations. We use [QR decomposition](https://mc-stan.org/docs/stan-users-guide/regression.html#QR-reparameterization.section) to reduce the dependence between parameters, and we also exploit [sufficient statistics](https://mc-stan.org/docs/stan-users-guide/efficiency-tuning.html#exploiting-sufficient-statistics) to reduce the time we spend computing the gradient. These optimizations have a noticeable impact on performance, but we’re still struggling in a few corner cases.

Here is our current model.

```stan
data {
  int<lower=0> num_predictors; // Number of predictors
  int<lower=0> num_original_predictors; // Number of predictors *before* qr decomposition
  int<lower=1> num_rows; // Number of rows
  matrix[num_rows, num_predictors] predictors;
  array[num_rows] int<lower=0> num_successes;
  array[num_rows] int<lower=1> count; // Number of repetitions
  matrix[num_original_predictors, num_predictors] r_star_inverse;
}
parameters {
  vector[num_predictors] theta_beta_tilde;
}
transformed parameters {
  vector[num_original_predictors] beta = r_star_inverse * theta_beta_tilde;
}
model {
  theta_beta_tilde ~ student_t(9, 0, 1);

  target += binomial_logit_lupmf(num_successes | count, predictors * theta_beta_tilde);
}

```

However, our model can run over 10-100x slower (3 hours vs 5 min) when there is [separability](https://mc-stan.org/docs/stan-users-guide/regression.html#separability) or near-separability. Separability means that one or more of the predictors will diverge to infinity. Only the inclusion of a prior will ensure that the problem is well-defined.

However, even when our model is well-defined, the shape of the posterior seems to give STAN some difficulties.

 ![Screenshot 2024-06-24 at 9.40.28 AM](https://canada1.discourse-cdn.com/flex030/uploads/mc_stan/original/3X/6/5/65c55d17faae5ad32b1a54499c591e2caec80ced.png)  
 ![Screenshot 2024-06-24 at 9.41.05 AM](https://canada1.discourse-cdn.com/flex030/uploads/mc_stan/original/3X/6/9/69f56062cfb342996e9077d75198b9a94490f341.jpeg)  
 ![Screenshot 2024-06-24 at 9.41.17 AM](https://canada1.discourse-cdn.com/flex030/uploads/mc_stan/original/3X/4/f/4f2d3d968f5387d868a3ac5dc751e13610924e16.jpeg)

It seems that we have a funnel shape posterior, that is [well-known](https://www.martinmodrak.cz/2018/02/19/taming-divergences-in-stan-models/) to cause STAN problems.

I would love to hear approaches for how I can speed up model solving when there is separability.

Here are some ideas that I’ve had:

## Transforming the parameters

My guess as to why model-fitting is so slow is that curvature of the posterior along `theta_beta_tilde[1]` is high variable. STAN is spending equal time sampling from a small area with high likelihood and a much larger area with lower likelihood. However, I’m not sure what a good transform would be. Further, because the curvature is coming from the data and not the model, we’d have to numerical compute a transform.

## Special-casing Separability

Maybe there’s another model that we can be using when we detect the separability? Is so, what would this be?

## Adjusting Priors

Are there priors that would provide better shaped posteriors? I’ve played with various degrees of freedom for the Student t prior (including normal), and while some run faster than others, I haven’t yet found one that was “fast”

## Adjusting Hamiltonian Monte Carlo fitting

Are there parameters that I can provide to STAN that would help fit this type of data better?

# How you can help

Eager to hear what you all think of these ideas, or to hear what other ideas might help.

---

<div class="post-metadata">

**Author:** ![jsocolar](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jsocolar/32/2486_2.png) [@jsocolar](https://discourse.mc-stan.org/u/jsocolar)\
**Post date:** [June 25, 2024, 6:13pm UTC](https://discourse.mc-stan.org/t/separable-logistic-regression-performance-optimizations/35628/2 "2024-06-25T18:13:30Z")

</div>

I’d be interested to hear what others say, but also just want to ask: What is your inferential goal in cases of complete separability?

You’re willing to entertain adjusting your priors, but in the case of separability the posterior can be quite sensitive to the choice of priors, and therefore in order for the posterior to be meaningful you might have to make a really considered choice of priors.

If you want to compute some particular posterior (based on some particular prior) then we should be thinking about whether a good reparameterization is available. But if you don’t really care about accurately computing any particular posterior (based on a particular prior), it might be worth asking in the first place whether you really need accurate MCMC inference for cases of separability.

---

<div class="post-metadata">

**Author:** ![Paul\_Jones](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/paul_jones/32/17565_2.png) [@Paul\_Jones](https://discourse.mc-stan.org/u/Paul_Jones)\
**Post date:** [June 25, 2024, 11:14pm UTC](https://discourse.mc-stan.org/t/separable-logistic-regression-performance-optimizations/35628/3 "2024-06-25T23:14:12Z")

</div>

> [@jsocolar](#):
>
> If you want to compute some particular posterior (based on some particular prior) then we should be thinking about whether a good reparameterization is available. But if you don’t really care about accurately computing any particular posterior (based on a particular prior), it might be worth asking in the first place whether you really need accurate MCMC inference for cases of separability.

That’s a fantastic question. I run this model on a wide range of datasets, and I don’t know a priori which ones have separability. Since there are only a few of them, I mainly just want something that works well enough.

More to the point, we have many datasets where the predictors, X, are the same, but the outcomes, y, differ. Rarely, the y's are a function of the X's and that’s when we get separability.

Here’s a more concrete example. Suppose that I add a predictor to X such that the outcome, y\_1, has complete separability with respect to X. Then, I can use information about the posterior, \theta\_1 to make qualitative inferences about the posterior, \theta\_2 of the outcome, y\_2. This is especially true when the outcome is y\_2 is close to the outcome y\_1.

So, I really do care about accuracy, but I’m willing to trade off a bit of accuracy if it gets me more performance.

> Blockquote  
> then we should be thinking about whether a good reparameterization is available.

Yes, I agree that this is the best path.

I’ve written code to find the mode of the posterior. This point has zero gradient and a negative definite Hessian. I was wondering about trying to find a reparameterization that makes the third derivative (in the direction of `theta_beta_tilde[1]` in the images) zero. Or perhaps another way of thinking about it, making the [skewness](https://en.wikipedia.org/wiki/Skewness) of `theta_beta_tilde[1]` zero
