PYSTAN on Windows Error message: PARSER EXPECTED: whitespace to end of file

#1

This is on windows 10. PYSTAN was installed using conda which installed python 2.18.00 and libpython_2,1 . ( Incidentally installing pystan with pip fails with " NameError: name ‘logger’ is not defined" ) but at least conda install works. Anyway…

The program is :

import os
import pystan
sm = pystan.StanModel(file=“C:/Users/N903193/Documents/Claims/model_np_utf8.txt”)

When I run this python program via Spyder I get the following error:

ValueError: Failed to parse Stan model ‘anon_model_b0ce2e73ab82ec9cfa804312d1756e6e’. Error message:
PARSER EXPECTED: whitespace to end of file.
FOUND AT line 1:
data {
int<lower=0> N;
int<lower=0> J; # number of counties
int<lower=0,upper=J> county[N];
vector[N] y;
}
parameters {
vector[J] a; # county intercepts
real mu_a;
real<lower=0> sigma_a;
real<lower=0> sigma_y;

}
transformed parameters {
vector [N] y_hat;
}
model {

//vector[N] y_hat1;
real y_hat1;
mu_a ~ normal(0, 1);
sigma_a ~ cauchy(0, 2.5);
sigma_y ~ cauchy(0, 2.5);
a ~ normal (mu_a, sigma_a);
for (i in 1:N) y_hat1[i] = a[county[i]];
// y_hat[1] = a[county[1]];

y ~ normal(y_hat1, sigma_y);
}

The input file contents are as described in the error message. I created the file in windows notepad and saved it wikth UTF-8 encoding.

When I use an input file created by windows notepad and save with ANSI encoding or use a python three quotes string with the input program I get a different error

ValueError: Failed to parse Stan model ‘anon_model_fcc9006ee9df993afd6bf34d447ecdda’. Error message:
SYNTAX ERROR, MESSAGE(S) FROM PARSER:

Warning (non-fatal): Comments beginning with # are deprecated. Please use // in place of # for line comments.
Warning (non-fatal): Comments beginning with # are deprecated. Please use // in place of # for line comments.
Unknown variable: real
variable “real” does not exist.
error in ‘model_np_ansi.txt’ at line 21, column 5

19: 	       
20: //vector[N] y_hat1;
21: real	y_hat1;

If I save the code in windows notepad as a unicode or a unicode bigendian file I get this error :
UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xfe in position 0: invalid start byte

#2

When you saved the file as UTF-8, Notepad probably added a byte-order mark at the beginning of the file, which confused PyStan. (Offhand, I think Stan files are supposed to be pure ASCII, not UTF-8.)

When I tried compiling (after deleting the byte-order mark), I got a different final error:

Left-hand side indexing incompatible with variable.
Too many indexes, expression dimensions=0, indexes found=1
  error in 'foo.stan' at line 25, column 25
  -------------------------------------------------
    23: sigma_y ~ cauchy(0, 2.5);
    24: a ~ normal (mu_a, sigma_a);
    25: for (i in 1:N) y_hat1[i] = a[county[i]];
                                ^
    26: // y_hat[1] = a[county[1]];
  -------------------------------------------------

Take a closer look at your declaration for y_hat1. Can you see what you did wrong?

#3

Thanks, yes I can see what was wrong with yhat1, it was declared as just a scalar real, not an array one. Anyway… this path I took isn’t the real issue… I corrected the program etc… and I can make THIS following program compile ok, whether I use windows notepad stored it in ANSI or using Notepad++ storing it as .txt … The program that compiles ok is :

Blockquote
data {
int<lower=0> N;
int<lower=0> J; # number of counties
int<lower=0,upper=J> county[N];
vector[N] y;
}
parameters {
vector[J] a; # county intercepts
real mu_a;
real<lower=0> sigma_a;
real<lower=0> sigma_y;

}
transformed parameters {
// NOTE LOCATION - its in transformed parameters BOTH the declaration AND the assignment.
// The assignment cannot be in the model block either
vector [N] y_hat;
for (i in 1:N) { y_hat[i] = a[county[i]]; }

}
model {

mu_a ~ normal(0, 1);
sigma_a ~ cauchy(0, 2.5);
sigma_y ~ cauchy(0, 2.5);
a ~ normal (mu_a, sigma_a);

y ~ normal(y_hat, sigma_y);
}

BUT, if I move the declaration of y_hat and its assignment into the model block compilation fails . Both the declaration and the assignment have to be in the transformed parameter block for a successful compilation. The failing compilation gives error :
Error message:
SYNTAX ERROR, MESSAGE(S) FROM PARSER:

Warning (non-fatal): Comments beginning with # are deprecated. Please use // in place of # for line comments.
Warning (non-fatal): Comments beginning with # are deprecated. Please use // in place of # for line comments.
Unknown variable: vector
variable “vector” does not exist.
error in ‘nplus_move_to_model.txt’ at line 20, column 9

18: 
19: // NOTE LOCATION - its in model
20:   vector [N] y_hat;
            ^
21:   for (i in 1:N)  { y_hat[i] = a[county[i]]; }

The full program is :
data {
int<lower=0> N;
int<lower=0> J; # number of counties
int<lower=0,upper=J> county[N];
vector[N] y;
}
parameters {
vector[J] a; # county intercepts
real mu_a;
real<lower=0> sigma_a;
real<lower=0> sigma_y;

}
transformed parameters {

}
model {

// NOTE LOCATION - its in model
vector [N] y_hat;
for (i in 1:N) { y_hat[i] = a[county[i]]; }

mu_a ~ normal(0, 1);
sigma_a ~ cauchy(0, 2.5);
sigma_y ~ cauchy(0, 2.5);
a ~ normal (mu_a, sigma_a);

y ~ normal(y_hat, sigma_y);
}

I don’t recall this issue for stan under Linux last year, not under windows 2 years or so ago…

#4

I can’t reproduce your problem. I successfully compiled the version of your Stan program where y_hat is declared in the model block. I can do that with both PyStan 2.18 and RStan 2.17.4.

Offhand, the only possibility that I can think of is that there may be some invisible character at the top of the model block in your original Stan program that doesn’t appear in the copy-and-pasted version shown in this forum.

#5

Hi, there is something weird with Anaconda PyStan. See last message here https://github.com/stan-dev/pystan/issues/414

Use pip install pystan and try again.

#6

I’m on windows 10, with a anaconda 3 platform and pystan installed by conda. How about you ?

Once I move those declarations and assignments into the transformed parameters block things seem to work, and the results make sense too but who knows what will happen with more complicated models.

#7

yes, I’d read issue 414 which is why I was farting around with UTF-8 files etc…eventually getting round to doing the full workaround suggested in the issue ( workaround : declaration AND assignment in the transformed parameters block).

" NameError: name ‘logger’ is not defined"

I guess I’ll have to debug that. Unless somebody has some ideas about it ?

#8

Hi, please make an issue in github so we can go through and debug installation process with pip.

Maybe make sure you don’t have a folder named logging where you do your pip installation.

#9

OK so I went to recreate the problem with a pip install of pystan and instead it installed successfully ! The difference from a week ago when it gave me that error : " NameError: name ‘logger’ is not defined" ???
Today, it needed pyHamCrest installed so I installed that first with pip.
I am installing off the net with

pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org pystan

Separately, yes with the pip installed version OK so I went to recreate the problem with a pip install of pystan and instead it installed successfully ! The difference from a week ago when it gave me that error : " NameError: name ‘logger’ is not defined" ???
Today, it needed pyHamCrest installed so I installed that first with pip.
I am installing off the net with

pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org pystan

Separately, yes with the pystan installed with pip I no longer have to put declarations and assignments in the transformed parameters block, it can live in the model block just fine. FWIW, here’s the output from the pip installation done today :

pip install pystan
Collecting pystan
Downloading https://files.pythonhosted.org/packages/6f/a1/fdf8906311733b7894aad1395d4d43afbbc79ca7573a07200eb105c5938c/pystan-2.18.0.0.tar.gz (17.5MB)
100% |████████████████████████████████| 17.5MB 2.2MB/s
Requirement already satisfied: Cython!=0.25.1,>=0.22 in c:\users\n903193\anaconda3\lib\site-packages (from pystan) (0.28.5)
Requirement already satisfied: numpy>=1.7 in c:\users\n903193\anaconda3\lib\site-packages (from pystan) (1.15.1)
Building wheels for collected packages: pystan
Running setup.py bdist_wheel for pystan … done
Stored in directory: C:\Users\N903193\AppData\Local\pip\Cache\wheels\4b\4c\27\f6d6b3ad1c8b6007911b3d033a2c98265e4d09a9c8c87970c6
Successfully built pystan
Installing collected packages: pystan
Successfully installed pystan-2.18.0.0
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the ‘python -m pip install --upgrade pip’ command.