What you have done when you put the rocker on was not ensure that the hydraulic lifter was sitting correctly on top of the valve stem. It is possible to install everything and have the lifter's seat resting against the top of the valve stem, rather than on it. When you got to start the engine, everything binds up and the transverse load on the rocker caused by the lifter being pushed aside by the valve breaks the lifter.
We have done this throwing the rockers back together in a hurry during a rally, so you're not the only goose out there! We took the lifter out of the broken rocker, threw the tappet cover back on and did half a rally The good news is that the I6 will take this sort of punishment in it's stride. 60km flat out with only 11 lifters in the engine....