A tight third round tie really came to life in the second half with four goals in a 13-minute spell.
There was little to choose between the two sides in the opening half when Hereford were content to soak up the home team's pressure.
But keeper Wayne Brown made an important save when tipping over Craig Curran's 25-yard shot.
Calvin Zola missed a great chance for Tranmere early in the second half and then forced Brown to make another excellent stop.
But it was Hereford that went ahead in the 65th minute with skipper Ben Smith volleying home in style after two other shots had been blocked in a goalmouth scramble.
Tranmere switched Curran to a striking role and sent on Mike Jones to play down the right flank.
Curran was unlucky to see a close-range effort deflected over but Tranmere drew level in the 75th minute, Steve Jennings netting off the underside of the bar following Jones' cross.
Just 60 seconds later Hereford were ahead again with a cracking finish by Trevor Benjamin after being set up by Theo Robinson.
But Tranmere refused to lie down and equalised again in the 78th minute when Gareth Taylor headed home Paul McLaren's centre.
Hereford produced the stronger finish but could not get in a blow at home keeper Danny Coyne.
Smith produced a superb display in the United midfield. Brown produced some smart saves and there was a lively display in attack from Benjamin and Robinson.
Curran worked tirelessly for Tranmere and Ben Chorley produced a five-star display in the heart of their defence.