How do i use custom probability distribution as a prior distribution in modelling

This depends on the language you want to use for coding your custom distribution.

For Stan language, see the manual on defining probability functions.

For C++, see this post on interfacing with external functions.

Had used the Stan language to to define the custom probability distribution, but I want to use the user defined distribution as a prior distribution in a random effect modeling

If you used Stan to define the density, you can use it as a prior. For example,

```
functions {
real foo_lpdf(real x, real lam) { return normal_lpdf(x | 0, lam); }
}
parameters {
real alpha;
}
model {
alpha ~ foo(3); // custom prior
...
}
```