They are called 'races'. Yes, they must be replaced with the bearings. It's a set. Bad things and rappid wear can happen if you don't. imo if you've been running it that way and want to do it now you'll have to buy new bearings. If it were mine I'd probably leave it alone and just watch it for wear since you know it wasn't done right. It might not hurt anything but that's definately not the proper way to do the job.
