Draw the line (at something)

To set a limit for something, to refuse to consider something.

We have to draw the line somewhere to limit the costs of the party.