Wear on (someone)

To anger or annoy (someone).

His constant complaining is beginning to wear on me.