Let (something) go

To allow something to pass, to do nothing about something.

Although I was angry at my friend’s remark I decided to let it go.