A millstone around (someone’s) neck

A continual burden or handicap for someone.

The empty store was a millstone around the neck of the small businessman.