Well I pulled the rubber line right before the fuel pump and found that the hose fitting that mates the 3/8" tube to the -6AN line has a partially disintegrated rubber piece in it. This would explain the rubber in the filter.

I have already ordered the new Carter pump and new hose/fittings to replace this segment. Hopefully this will cure my issues.

I will also try and blow some compressed air through the hard line to make sure there is nothing blocking it up anywhere.