Ok, this sounds crazy but lots of lots of experience with the threaded stop nut style cables has showed me something I don't really understand.
Align the swivel exactly like the manufacturers instructions indicate. Now, do your adjusting at the cable end, not the swivel.