Come down in the world

To lose one’s social position.

My father came down in the world when he decided to change jobs.