Yes, alum and cleanliness are an illusive goal in welding alum. Everything, as you suggest, that comes in contact with the area, only seems to contaminate the weld. On other threads, I've suggested even dedicated alum tig gloves, to keep the tig rod clean that you handle. I think there is something to welding used/weathered alum castings with non visible porosity concerns, it seldom/never matches the results of when welding new alum, no matter how clean one tries to get the weld area..

edit: Lacquer thinner has always been my favorite alum cleaner, it seems to have a wider range of solvents, since the exact contaminates are not always known, although acetone is promoted as a stronger solvent?