by reading your post I have to believe the bolt is too long or wrong diameter or your sealing washers are too thin.

I have stacked washers before..people say not to but I haven't had any failures.