With everything in and tightened I would find it hard to believe that the shaft would move enough to pop the shoes out. I would have no fear if your confident that the column is bolted in tight and both split rings are in at the upper shaft bearing.

Keep an eye on it. Your pot coupler cap will work off first if there would be any play. Btw, I don't have a pin in mine and there is no play.