For small injection volumes, it should not make much difference, as the sample will mix with the mobile phase and be at or close to pH 5 when it hits the columnn (i'm assuming the ionic strength is constant).
For larger injection volumes (too large to mix completely with the mobile phase between injector and column), I would expect to see some on-column reconcentration with the pH 4 (at that pH, your protein would be strongly bound; i.e., for constant ionic strength, a pH 4 buffer would be a weak eluant) and possibly some peak broadening and/or shape problems with the pH 6 (at that pH, your protein will be approximately unretained; i.e., for a constant ionic strength, a pH 6 buffer would be a strong eluant for your protein).
All of the above arguments are hypothetical. Remember that your stationary phase can only interact with the *outside* of the protein. At the pI, your protein has a net charge of zero, but depending on the distribution of charged groups and the conformation of the protein, the surface may have either a net positive or negative charge.