I have an 03 as well and went through this exercise last year. Mine turned out to be injectors, which is the last thing you want to replace $$$.

Do you have old lift pump attached to the fuel filter? If so, it may be time to upgrade. Its probably not the problem though. If its hard to start but then running well, its likely the rail pressure is not high enough to fire the injectors when cranking. I saved some notes from my research that may point you in the right direction:

Before trying to start, disconnect the electrical connection from the FCA (Fuel Control Actuator), it is the only electrical connection on your CP3 injection pump. Just squeeze the tab to release and pull, it will come apart.

Now try and start, if it starts you have a low rail pressure condition, by disconnecting the FCA, you have fooled the ECM into believing you have adequate rail pressure. The ECM needs to see a minimum of 3. 5k rail pressure to fire the injector solenoids, by disconnecting the FCA, it by default raises your rail pressure to max pressure. After it starts, if it does, shut it down, you do not want to operate it with the FCA disconnected.

Now you have to figure out why you have low rail pressure, faulty COV, PRV, RP Sensor, Loose cross tubes, bad injector(s) to name a few. I will add a link that will give you a better understanding than my rambling, but the first thing I would do, would be to re-torque the cross tubes. When the engine is cold the tolerance between the cross tube and injector is greater than when warm, thereby allowing more fuel to bypass and lowering rail pressure, when the engine is warm the fuel is thinner, so that could also bypass more fuel, kind of a double edged sword.

Injectors are not receiving the the pressure they need. . . . you can replace the Cascade Valve on the CP3 injection pump. . . .

Before you do that, and free of charge(if your handy), retorque your injector cross tubes to 37 ft lbs. You will need a 24mm and a 19mm crows foot, to fit your torque wrench.

This is an easy procedure, and often overlooked, try it 1st.

For hard starting or no start, Don't forget the CR high pressure sensor and/or relief valve. Either one of these malfunctioning can cause this problem.