If you want to do it, you can.
My take on the issue has always been that hydrogen effuses so fast through the air, particularly regularly exchanged lab air, that it's extremely difficult to reach 4% hydrogen in your lab air.
At least on Agilent systems, there are a few precautions recommended, including that you don't try to pump down if the carrier has been allowed to build up in the MS for any amount of time(shut it off and let it clear out before pumping down), that you remove the front cover of the MS, and that you LIGHTLY screw down side plate cover to keep it from flying open in the event of an explosion.
The Agilent EPCs will generally shut off if they can't reach the setpoint pressure within a couple of minutes, which also makes things a bit safer.
I have never understood the warning to remove the front cover on the Agilent MS when using Hydrogen, you would think if it is going to explode and break the front glass you would want something there to help catch it.
With Hydrogen carrier in an MS, you are going to want to keep the flow down to about 1ml/minute to be able to maintain proper vacuum, less flow the better with Hydrogen. You can't actually take advantage of the higher flow rate/sharper chromatography that you would with other detectors when using Hydrogen because the extra flow will kill the sensitivity of the MS. So with Hydrogen on MS there should be no difference in safety than using Hydrogen carrier on a GC/FID or GC/ECD if you are at the same split flows on the inlet since you will be normally using lower column flow rates than other detectors. The FID with Hydrogen would burn it away, but any other detector will still vent it to the room unless you use tubing to route it outside or to a hood.