Just do the rocker adjustment. They stay in adjustment for so long, do it now and you won't have to worry about it for a long time.

I tried what you're thinking once, thought "oh ya, I didn't touch the exhaust rockers or valves, just played with the intakes so I can just bolt my rockers back on and only adjust the intake rockers not the exhaust rockers." Want to know what happened? Some of the exhaust rockers were far enough out of spec now that two of the exhaust valves didn't close all the way, resulting in zero compression on those cylinders! I didn't make any other changes and this was on a 4cyl engine with an overhead cam with manual lash adjusters. No lifters or pushrods even!