Unfortunately, I don’t think there is an easy and straightforward solution. One hack would be to define a predictor is_neutral that would be 1 when the response is neutral and 0 otherwise. You could then use is_neutral in your formula by itself or even as interaction, so say Category:is_neutral would end up adding another effect of Category only for neutral responses. This way, you get additional flexibility in the model only for the neutral category. Interpreting such coefficients directly would however be quite tricky.
A better way would be to define a custom “5 - inflated ordinal” response family that would have one additional parameter for the degree of 5 inflation (that you can then put predictors on). This is semi-advanced, but has been done before, see: Ordinal Mixture Model w/ Set Outcome for One Component - #20 by mike-lawrence (the code there is unfortunately incomplete) and there is some semi-related discussion at Test zero inflated Graded response model
Does that make sense?