# Fail to run due to relatively high number of missing observations

**URL:** <https://discourse.mc-stan.org/t/fail-to-run-due-to-relatively-high-number-of-missing-observations/3369>\
**Category:** Modeling\
**Created:** [February 21, 2018, 8:02pm UTC](https://discourse.mc-stan.org/t/fail-to-run-due-to-relatively-high-number-of-missing-observations/3369 "2018-02-21T20:02:16Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Panagiotis\_Arsenis](https://avatars.discourse-cdn.com/v4/letter/p/ce7236/32.png) [@Panagiotis\_Arsenis](https://discourse.mc-stan.org/u/Panagiotis_Arsenis)\
**Post date:** [February 21, 2018, 8:02pm UTC](https://discourse.mc-stan.org/t/fail-to-run-due-to-relatively-high-number-of-missing-observations/3369/1 "2018-02-21T20:02:16Z")

</div>

Hello,

referring to a previous post ([Missing data in a 2PL (IRT) model - #37 by Panagiotis\_Arsenis](http://discourse.mc-stan.org/t/missing-data-in-a-2pl-irt-model/1211/37)), we currently have the following Stan code:

> functions {  
> real rsm(int y, real theta, real beta, vector kappa) {  
> vector[rows(kappa) + 1] unsummed;  
> vector[rows(kappa) + 1] probs;  
> unsummed = append\_row(rep\_vector(0, 1), theta - beta - kappa);  
> probs = softmax(cumulative\_sum(unsummed));  
> return categorical\_lpmf(y + 1 | probs);  
> }  
> real rsm\_rng(real theta, real beta, vector kappa) {  
> vector[rows(kappa) + 1] unsummed;  
> vector[rows(kappa) + 1] probs;  
> unsummed = append\_row(rep\_vector(0, 1), theta - beta - kappa);  
> probs = softmax(cumulative\_sum(unsummed));  
> return categorical\_rng(probs);  
> }  
> }  
> data {  
> int\<lower=1\> I; // # items  
> int\<lower=1\> J; // # persons  
> int\<lower=1\> N; // # observations  
> int\<lower=1\> N\_mis; // # missing observations  
> int\<lower=1, upper=I\> ii[N]; // item for n  
> int\<lower=1, upper=J\> jj[N]; // person for n  
> int\<lower=0\> y[N]; // response for n  
> }  
> transformed data {  
> int m; // # steps  
> m = max(y);  
> }  
> parameters {  
> vector[I] beta;  
> vector[m-1] kappa\_free;  
> vector[J] theta;  
> real\<lower=0\> sigma;  
> }  
> transformed parameters {  
> vector[m] kappa;  
> kappa[1:(m-1)] = kappa\_free;  
> kappa[m] = -1\*sum(kappa\_free);  
> }  
> model {  
> beta ~ normal(0, 3);  
> target += normal\_lpdf(kappa | 0, 3);  
> theta ~ normal(0, sigma);  
> sigma ~ exponential(.1);  
> for (n in 1:N)  
> target += rsm(y[n], theta[jj[n]], beta[ii[n]], kappa);  
> }  
> generated quantities {  
> vector[N\_mis] y\_mis;  
> for (n in 1:N\_mis)  
> y\_mis[n] = rsm\_rng(theta[jj[n]], beta[ii[n]], kappa);  
> }

The issue here is that when N\_mis \> N, Stan will come back with this:

> Exception: : accessing element out of range. index 7455 out of range; expecting index to be between 1 and 7454; index position = 1jj (in ‘model52b1217b4c\_1e9f8627a1bbd76d859458272d0dfc57’ at line 54)

We are not sure how to deal with this. Any suggestions?

Panos

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [February 21, 2018, 11:07pm UTC](https://discourse.mc-stan.org/t/fail-to-run-due-to-relatively-high-number-of-missing-observations/3369/2 "2018-02-21T23:07:35Z")

</div>

I’m just speculating here:

Are `y_mis[n]` and `y[n]` someway related?

would this return the same goal if you randomly pick an integer between `1` to `N` each round and use that as a `n`.

---

<div class="post-metadata">

**Author:** ![mike-lawrence](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mike-lawrence/32/59_2.png) [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)\
**Post date:** [February 21, 2018, 11:33pm UTC](https://discourse.mc-stan.org/t/fail-to-run-due-to-relatively-high-number-of-missing-observations/3369/3 "2018-02-21T23:33:30Z")

</div>

The problem is that jj and ii are of length N but in the generated quantities you’re indexing into them up to N\_miss, which as you say is \> N.

---

<div class="post-metadata">

**Author:** ![Panagiotis\_Arsenis](https://avatars.discourse-cdn.com/v4/letter/p/ce7236/32.png) [@Panagiotis\_Arsenis](https://discourse.mc-stan.org/u/Panagiotis_Arsenis)\
**Post date:** [February 23, 2018, 6:38pm UTC](https://discourse.mc-stan.org/t/fail-to-run-due-to-relatively-high-number-of-missing-observations/3369/4 "2018-02-23T18:38:19Z")

</div>

Well, both `N_mis` and `N` come from the same data set. So, what is not `N_mis` (NA, missing data), it is `N`.

---

<div class="post-metadata">

**Author:** ![Panagiotis\_Arsenis](https://avatars.discourse-cdn.com/v4/letter/p/ce7236/32.png) [@Panagiotis\_Arsenis](https://discourse.mc-stan.org/u/Panagiotis_Arsenis)\
**Post date:** [February 23, 2018, 7:44pm UTC](https://discourse.mc-stan.org/t/fail-to-run-due-to-relatively-high-number-of-missing-observations/3369/5 "2018-02-23T19:44:40Z")

</div>

I see, this sounds right. If I simply re-index `ii` and `jj` as `ii[N_mis]` and `jj[N_mis]` I get the following:

> Error in new\_CppObject\_xp(fields$.module, fields$.pointer, …) :  
> Exception: mismatch in dimension declared and found in context; processing stage=data initialization; variable name=ii; position=0; dims declared=(10285); dims found=(7454) (in ‘model79fbfba1a\_d0f40f95b93802bc6d0fcad905f25cf3’ at line 24)

A bit more insight here if possible?

---

<div class="post-metadata">

**Author:** ![syclik](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/syclik/32/6_2.png) [@syclik](https://discourse.mc-stan.org/u/syclik)\
**Post date:** [February 23, 2018, 9:04pm UTC](https://discourse.mc-stan.org/t/fail-to-run-due-to-relatively-high-number-of-missing-observations/3369/6 "2018-02-23T21:04:41Z")

</div>

The error message is telling you that the data you passed in is the wrong size. I’ll try to break it down.

- “mismatch in dimension declared and found”: it’s telling you that the size you declared and what it found were different.

- “processing stage=data initialization”: it’s telling you this is when it’s initializing data, i.e. reading data from R / Python / command line

- “variable name=ii”: check ii

- “position = 0”: this might throw you off and is unimportant for this error. It’s telling you which dimension of the array that the problem was found.

- “dims declared=(10285)”: in your program, it’s expecting ii to be length 10285

- “dims found=(7454)”: in your data, check the size of ii. I bet it’s length 7454

The error messages contain a lot of info if you unpack them.

---

<div class="post-metadata">

**Author:** ![Bob\_Carpenter](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bob_carpenter/32/9230_2.png) [@Bob\_Carpenter](https://discourse.mc-stan.org/u/Bob_Carpenter)\
**Post date:** [February 27, 2018, 7:41am UTC](https://discourse.mc-stan.org/t/fail-to-run-due-to-relatively-high-number-of-missing-observations/3369/7 "2018-02-27T07:41:14Z")

</div>

It’s telling you which line in the model is causing problems:

> [@Panagiotis\_Arsenis](#):
>
> at line 24)

That’s always a good place to start debugging. (Just thought I’d call that out explicitly in addition to @syclik’s comments.

---

<div class="post-metadata">

**Author:** ![Panagiotis\_Arsenis](https://avatars.discourse-cdn.com/v4/letter/p/ce7236/32.png) [@Panagiotis\_Arsenis](https://discourse.mc-stan.org/u/Panagiotis_Arsenis)\
**Post date:** [February 28, 2018, 11:19am UTC](https://discourse.mc-stan.org/t/fail-to-run-due-to-relatively-high-number-of-missing-observations/3369/8 "2018-02-28T11:19:01Z")

</div>

Thank you for this helpful information.