The silencer does more than just "quiet" the rushing air.
If you remove it, you'll likely not notice any difference in sound
(at least I don't b/c the radio is on!
But, the silencer is a RESTRICTIVE DEVICE to the air box. It is very easy to remove, and by doing so you remove a restriction to air flow.
Remeber the three points to increased performance:
(1) get the air in
(2) burn it
(3) get the exhaust out!
Removing restrictions on the intake and exhaust is a cheap and easy way to get more HP to the rear wheels.