You are right. The driveshaft yoke runs on the bushing. So if the driveshaft moves up and down, side to side, the bushing is bad.
But, heck if you have the trans out and are rebuilding it.....just replace it and the seal. Too much work to do it later.
2005 F150 SuperCrew "King Ranch"
1986 LTD II
1964 Corvair Monza