Most seat fabrics will fluff up, especially in the base. Its like new carpet, you constantly get the fluff until it wears in.
A tip from an upholsterer is to get a sharp razor blade, and carefully shave the fluff from the fabric then vacuum it.
I had the fluff after about 3000k so i shaved it off and vacumed it. You could never tell that it ever fluffed up. My car has now over 10,000km and no signs of wear.
I think once you remove the initial fluffing in the fabric it will be ok.
5 speed or Nothin' Mercury BA Falcon XR6 Manual with 18's, Whiteline Lowered Springs, Adjustable Sway Bars, and Herrod Exhaust......mmmmmh