# WAIC for probit model confirmation or troubleshooting?

**URL:** <https://discourse.mc-stan.org/t/waic-for-probit-model-confirmation-or-troubleshooting/1115>\
**Category:** Modeling\
**Tags:** loo\
**Created:** [July 1, 2017, 4:33pm UTC](https://discourse.mc-stan.org/t/waic-for-probit-model-confirmation-or-troubleshooting/1115 "2017-07-01T16:33:02Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jeremy.koster](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jeremy.koster/32/5756_2.png) [@jeremy.koster](https://discourse.mc-stan.org/u/jeremy.koster)\
**Post date:** [July 1, 2017, 4:33pm UTC](https://discourse.mc-stan.org/t/waic-for-probit-model-confirmation-or-troubleshooting/1115/1 "2017-07-01T16:33:02Z")

</div>

Hi everyone,

I’m analyzing some dyadic data with a probit model and thinking about calculating WAIC for model selection using built-in functions in _loo_. I think that the following code in “generated quantities” will produce the **log\_lik** values that feed into _loo_, but it’d be helpful to get some confirmation of that because my first attempt returned a moderately higher value than expected (compared against a simpler model). I’m debugging other aspects of the code now, but thinking that it’d be good to confirm the **log\_lik** portion of the code as well. Suggestions are appreciated. Thanks, Jeremy

```
model_code_b <- "
data{
    int N;
    int N_i__j_ID;
    int N_ij_dyads;
    int y[N];
    int i_ID[N];
    int j_ID[N];
    int ij_dyad_ID[N];
}

parameters{
    real a;
    
    // Node level effects cov matrix
    matrix[2,N_i__j_ID] z_i__j_ID;
    vector<lower=0>[2] sigma_i__j_ID;
    cholesky_factor_corr[2] L_Rho_i__j_ID;

    vector[N_ij_dyads] v_ij_dyad_ID;
    real<lower=0> sigma_ij_dyad_ID;
}

transformed parameters{
	// Define all matrices
	matrix [N_i__j_ID,2] v_i__j_ID;
	
	{
	v_i__j_ID = (diag_pre_multiply ( sigma_i__j_ID, L_Rho_i__j_ID ) * z_i__j_ID)';
	}
}

model{
    vector[N] p;
    
    //priors
    a ~ normal( 0 , 5 );
    
    to_vector (z_i__j_ID) ~ normal (0,1);
    
    v_ij_dyad_ID ~ normal( 0 , 1 );
    sigma_ij_dyad_ID ~ exponential (1);
    
    sigma_i__j_ID ~ exponential (1);
    
    L_Rho_i__j_ID ~ lkj_corr_cholesky(4);

// LIKELIHOOD
    for ( i in 1:N ) {
        
        p[i] = 
        a
       	+ v_i__j_ID [i_ID[i],1] 
       	+ v_i__j_ID [j_ID[i],2]
       	+ v_ij_dyad_ID [ij_dyad_ID[i]] * sigma_ij_dyad_ID
       	;
      	}
      	
      	// update target
      	y ~ bernoulli (Phi(p));
      	}
      	
generated quantities{
	matrix[2,2] Rho_i__j_ID;
	vector[N] log_lik;
	Rho_i __j_ID = L_Rho_i__ j_ID * L_Rho_i__j_ID';

for ( i in 1:N ) {
       real p;
	
	        p = 
		a
       	+ v_i__j_ID [i_ID[i],1] 
       	+ v_i__j_ID [j_ID[i],2]
       	+ v_ij_dyad_ID [ij_dyad_ID[i]] * sigma_ij_dyad_ID
       	;
	log_lik[i] = bernoulli_lpmf ( y[i] | Phi(p) );
	
	}
}
"
```

---

<div class="post-metadata">

**Author:** ![bgoodri](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bgoodri/32/4451_2.png) [@bgoodri](https://discourse.mc-stan.org/u/bgoodri)\
**Post date:** [July 1, 2017, 5:48pm UTC](https://discourse.mc-stan.org/t/waic-for-probit-model-confirmation-or-troubleshooting/1115/2 "2017-07-01T17:48:00Z")

</div>

> [@jeremy.koster](#):
>
> I’m analyzing some dyadic data with a probit model and thinking about calculating WAIC for model selection using built-in functions in loo.

Code seems fine, but computing WAIC is not. Use LOOIC from loo() instead.

---

<div class="post-metadata">

**Author:** ![jeremy.koster](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jeremy.koster/32/5756_2.png) [@jeremy.koster](https://discourse.mc-stan.org/u/jeremy.koster)\
**Post date:** [July 1, 2017, 6:04pm UTC](https://discourse.mc-stan.org/t/waic-for-probit-model-confirmation-or-troubleshooting/1115/3 "2017-07-01T18:04:28Z")

</div>

Thanks for the tip, Ben. I went ahead and gave this a try and for other users wishing to do likewise, I found the help documentation to be helpful: help(loo)

It shows how to extract the log likelihoods from the stan model.

```
log_likb <- extract_log_lik(mb)
loo(log_likb)

```

My discipline is just getting familiar with WAIC, so LOOIC will be something new for most. Is there a canonical reference that details the advantages of LOOIC?

---

<div class="post-metadata">

**Author:** ![bgoodri](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bgoodri/32/4451_2.png) [@bgoodri](https://discourse.mc-stan.org/u/bgoodri)\
**Post date:** [July 1, 2017, 6:39pm UTC](https://discourse.mc-stan.org/t/waic-for-probit-model-confirmation-or-troubleshooting/1115/4 "2017-07-01T18:39:08Z")

</div>

Same as the one in the documentation for the loo package:  
[http://www.stat.columbia.edu/~gelman/research/published/loo\_stan.pdf](http://www.stat.columbia.edu/~gelman/research/published/loo_stan.pdf)

---

<div class="post-metadata">

**Author:** ![avehtari](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/avehtari/32/5935_2.png) [@avehtari](https://discourse.mc-stan.org/u/avehtari)\
**Post date:** [July 2, 2017, 6:31pm UTC](https://discourse.mc-stan.org/t/waic-for-probit-model-confirmation-or-troubleshooting/1115/5 "2017-07-02T18:31:10Z")

</div>

> [@bgoodri](#):
>
> Same as the one in the documentation for the loo package:  
> [http://www.stat.columbia.edu/~gelman/research/published/loo\_stan.pdf](http://www.stat.columbia.edu/~gelman/research/published/loo_stan.pdf)

Andrew’s preprint mentions “To appear”, but in case you want to cite it, it has appeared in Statistics and Computing, 27(5):1413–1432.

Aki