Custom Probability distribution

Modeling
1

How do i use custom probability distribution as a prior distribution in modelling

2

This depends on the language you want to use for coding your custom distribution.
For Stan language, see the manual on defining probability functions.
For C++, see this post on interfacing with external functions.

3

Had used the Stan language to to define the custom probability distribution, but I want to use the user defined distribution as a prior distribution in a random effect modeling

4

If you used Stan to define the density, you can use it as a prior. For example,

functions {
  real foo_lpdf(real x, real lam) { return normal_lpdf(x | 0, lam); }
}
parameters {
  real alpha;
}
model {
  alpha ~ foo(3);  // custom prior
  ...
}