In both cases, the presence of *very* low levels of the buffers in question are unlikely to cause problems.
The issue with phosphate in MS is volatility, with phosphate building up in the ionization source. If you do thorough flush of your system, the amount of non-volatile CRUD coming from your samples will probably far exceed any traces of phosphate left behind.
The issue with UV is absorbance at low wavelength. Unless you are going *very* low (200 nm or below), even 0.1 mM of acetate or formate probably will not give a significant background over trace junk in your acetonitrile.
If the mobile phases are otherwise compatible, a useful rule of thumb is that it takes about 10x the system volume to get reasonably complete washout. I'm paranoid, so I would probably go longer than that. When switching to the UV detector, you can track the washout of formate or acetate by simply monitoring the signal at 205 or 200 nm.