The other side of the tracks

The poorer/richer section of a town (the opposite place from where you are).

The girl came from the other side of the tracks and was not welcome in some parts of the city.