Striker Calvin Zola was shown a straight red card for a two-footed challenge on Michael Collins, but by that time the hosts were already a goal to the good.
Carl Tremarco found the target with a left-foot finish in the fourth minute when set up by Sam Aiston. It was his first league goal.
It then became a backs to the wall task for Tranmere although the Terriers offered little up to half-time.
Pawel Abbott did hit the side-netting with one shot and substitute Gary Taylor-Fletcher fired wide from the corner of the box.
The visitors were more purposeful after the break and were level by the 48th minute. David Graham scored from close range when set up by Taylor-Fletcher following a cross by Dan Smith.
Danny Harrison flashed a shot just over for Tranmere and rather than settling for a point, Rovers boss Brian Little made a statement of intent by sending on striker Delroy Facey.
But they needed a terrific save by Dino Seremet to keep out an Andy Booth header in the 79th minute.
Tranmere then grabbed a shock winner after a quick breakout in the 84th minute, as Facey found space on the edge of the box to plant a right foot shot out of the reach of visiting goalkeeper Phil Senior.