# Bulk/Tail ESS Warnings in Bradley-Terry Model

**URL:** <https://discourse.mc-stan.org/t/bulk-tail-ess-warnings-in-bradley-terry-model/25243>\
**Category:** Modeling\
**Created:** [November 11, 2021, 11:23pm UTC](https://discourse.mc-stan.org/t/bulk-tail-ess-warnings-in-bradley-terry-model/25243 "2021-11-11T23:23:31Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![rja](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/rja/32/10478_2.png) [@rja](https://discourse.mc-stan.org/u/rja)\
**Post date:** [November 11, 2021, 11:23pm UTC](https://discourse.mc-stan.org/t/bulk-tail-ess-warnings-in-bradley-terry-model/25243/1 "2021-11-11T23:23:31Z")

</div>

I am trying to estimate individual abilities from the results of head-to-head comparisons similar to [Bob Carpenter](https://www.rpubs.com/dkarwosk12345/560307) and [Andrew Gelman’s](https://statmodeling.stat.columbia.edu/2014/07/13/stan-analyzes-world-cup-data/) examples. I’m consistently getting Bulk ESS and Tail ESS errors on the ability parameters and was curious if the board had any suggestions. I’ve tried both centered and non-centered parameterizations as well as adding more iterations with no luck.

The stan code is below.

```stan
data {
  int<lower = 1> N; //number of observations
  int<lower = 1> I; //number of individuals
  int<lower = 0, upper = 1> y[N]; //
  int<lower = 1, upper = I> id_1[N];
  int<lower = 1, upper = I> id_2[N];
}
parameters {
  real<lower = 0> sigma;
  vector[I] alpha;
}
model{
  alpha ~ normal(0, sigma);
  sigma ~ normal(4, 2);
  y ~ bernoulli_logit(alpha[id_1] - alpha[id_2]);
}

```

And here is code to simulate data and fit the model

```
library(dplyr)
library(rstan)

n <- 100
sigma <- 4

id <- 1:n

ability <- rnorm(n, 0, sigma)

ability_df <- data.frame(id, ability)

matchup_grid <- expand.grid(id_1 = id, id_2 = id)
matchup_grid <- matchup_grid[matchup_grid$id_1 > matchup_grid$id_2,]

model_dat <- matchup_grid %>%
  inner_join(ability_df, by = c('id_1' = 'id')) %>%
  rename(ability_1 = ability) %>%
  inner_join(ability_df, by = c('id_2' = 'id')) %>%
  rename(ability_2 = ability)

stan_dat <- list(N = nrow(model_dat),
                 I = n,
                 y = model_dat$id_1_preferred,
                 id_1 = model_dat$id_1,
                 id_2 = model_dat$id_2)

model_dat$id_1_preferred <- rbinom(nrow(model_dat), 1, arm::invlogit(model_dat$ability_1 - model_dat$ability_2))

mod_sim_cen <- stan_model(model_code = stan_code_cen)
fit_sim_cen <- sampling(mod_sim_cen, stan_dat, iter = 2000, chains = 4, cores = 4)

```

---

<div class="post-metadata">

**Author:** ![andre.pfeuffer](https://avatars.discourse-cdn.com/v4/letter/a/e480ec/32.png) [@andre.pfeuffer](https://discourse.mc-stan.org/u/andre.pfeuffer)\
**Post date:** [November 12, 2021, 5:58am UTC](https://discourse.mc-stan.org/t/bulk-tail-ess-warnings-in-bradley-terry-model/25243/2 "2021-11-12T05:58:42Z")

</div>

Is there a reason, why do you specify?

> [@rja](#):
>
> `sigma ~ normal(4, 2);`

Try `sigma ~ normal(0, 2);`

You may also use:

```no-highlight
{
  vector[I] alpha_n = alpha * sigma;
  alpha ~ std_normal();
  sigma ~ normal(0, 2);
  y ~ bernoulli_logit(alpha_n[id_1] - alpha_n[id_2]);
}

```

---

<div class="post-metadata">

**Author:** ![maw](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@maw](https://discourse.mc-stan.org/u/maw)\
**Post date:** [December 8, 2021, 4:59pm UTC](https://discourse.mc-stan.org/t/bulk-tail-ess-warnings-in-bradley-terry-model/25243/3 "2021-12-08T16:59:59Z")

</div>

Using Andre’s reparameterization I am still getting the bulk ess and tail ess issues.

The bizarre behavior I found is that if you change the line that gets the matchup grid to sample only 25% of the data:

`matchup_grid <- matchup_grid[matchup_grid$id_1 > matchup_grid$id_2,] %>% sample_frac(.25)`

It gets rid of the ess issues. Why would removing a significant portion of sample size help the model?

---

<div class="post-metadata">

**Author:** ![andre.pfeuffer](https://avatars.discourse-cdn.com/v4/letter/a/e480ec/32.png) [@andre.pfeuffer](https://discourse.mc-stan.org/u/andre.pfeuffer)\
**Post date:** [December 9, 2021, 4:09am UTC](https://discourse.mc-stan.org/t/bulk-tail-ess-warnings-in-bradley-terry-model/25243/4 "2021-12-09T04:09:04Z")

</div>

```
data {
  int<lower = 1> N; //number of observations
  int<lower = 1> I; //number of individuals
  int<lower = 0, upper = 1> y[N]; //
  int<lower = 1, upper = I> id_1[N];
  int<lower = 1, upper = I> id_2[N];
}
parameters {
  real<lower = 0> sigma;
  vector[I] alpha;
  real intercept;
}
model{
  alpha ~ normal(0, sigma);
  sigma ~ lognormal(0, 0.5);
  intercept ~ std_normal();
  y ~ bernoulli_logit(alpha[id_1] - alpha[id_2] + intercept);
}

```

```
> fit$cmdstan_diagnose()
Processing csv files: /tmp/RtmpB1EblK/bradleyterry-202112091102-1-737976.csv, /tmp/RtmpB1EblK/bradleyterry-202112091102-2-737976.csv

Checking sampler transitions treedepth.
Treedepth satisfactory for all transitions.

Checking sampler transitions for divergences.
No divergent transitions found.

Checking E-BFMI - sampler transitions HMC potential energy.
E-BFMI satisfactory.

Effective sample size satisfactory.

Split R-hat values satisfactory all parameters.

Processing complete, no problems detected.

```

Changed to `sigma ~ normal(0, 2)` to `lognormal(0, 0.5);` to keep sigma away from zero.  
Added `intercept` parameter.

---

<div class="post-metadata">

**Author:** ![maw](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@maw](https://discourse.mc-stan.org/u/maw)\
**Post date:** [December 9, 2021, 4:40pm UTC](https://discourse.mc-stan.org/t/bulk-tail-ess-warnings-in-bradley-terry-model/25243/5 "2021-12-09T16:40:58Z")

</div>

Interesting…I’m still getting bulk ess issues.

As for the intercept, why would that help sample better? How would that parameter be interpreted in the model.

Are you using cmdstan instead of RStan?

---

<div class="post-metadata">

**Author:** ![andre.pfeuffer](https://avatars.discourse-cdn.com/v4/letter/a/e480ec/32.png) [@andre.pfeuffer](https://discourse.mc-stan.org/u/andre.pfeuffer)\
**Post date:** [December 10, 2021, 5:21am UTC](https://discourse.mc-stan.org/t/bulk-tail-ess-warnings-in-bradley-terry-model/25243/6 "2021-12-10T05:21:22Z")

</div>

I’m using `cmdstanr/cmdstan 2.28.2`.  
Using `rstan`: If you increase the number of iterations , the warnings disappear.  
The intercept terms helps to have the overall `alpha`'s too be centered around zero.

---

<div class="post-metadata">

**Author:** ![maw](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@maw](https://discourse.mc-stan.org/u/maw)\
**Post date:** [December 13, 2021, 4:23pm UTC](https://discourse.mc-stan.org/t/bulk-tail-ess-warnings-in-bradley-terry-model/25243/7 "2021-12-13T16:23:19Z")

</div>

Any thoughts on why rstan vs cmdstan would be different?

In the model the alphas represent the latent talent between the two players. What would the intercept mean as an adjustement to the difference in talent level? especially if we are centering alpha around zero?

---

<div class="post-metadata">

**Author:** ![andre.pfeuffer](https://avatars.discourse-cdn.com/v4/letter/a/e480ec/32.png) [@andre.pfeuffer](https://discourse.mc-stan.org/u/andre.pfeuffer)\
**Post date:** [December 14, 2021, 3:18am UTC](https://discourse.mc-stan.org/t/bulk-tail-ess-warnings-in-bradley-terry-model/25243/8 "2021-12-14T03:18:14Z")

</div>

> [@maw](#):
>
> Any thoughts on why rstan vs cmdstan would be different?

cmdstanr is a more basic interface than rstan. These are diagnostic warnings based on some measurements. I am certain you can do those checks in cmdstanr.

> [@maw](#):
>
> What would the intercept mean as an adjustement to the difference in talent level?

It can be interpreted as home advantage.  
But there is more reason for adding an intercept:  
[when-is-it-ok-to-remove-the-intercept-in-a-linear-regression-model](https://stats.stackexchange.com/questions/7948/when-is-it-ok-to-remove-the-intercept-in-a-linear-regression-model)