Weigh down on (someone)

To worry or depress someone.

The man’s problems at work seem to be weighing down on him.