Is it up or is it down?

Isn`t it strange how we use the words up or down to relate to going anywhere? Instead of just saying “going to”
From where I live I`ve always used the expression “going up the town”. Yet from a different road and direction I find myself saying “going down the town”. No, the town isn`t at the top or bottom of a hill, and I don`t live at the top or bottom of a hill, so why do I use that expression. And others like “down to the beach”, “up the city”, “up to the church”. I don`t know!.
I can remember writing in an essay when I was at primary school (and we`d had to write what we`d done at the weekend or something), writing about going up the town. And when my work came back marked, it had been crossed through as incorrect and a ladder drawn beside that expression. At that age, I couldn`t understand why teacher had drawn a ladder or what was wrong with it, because it was normal language to me to say that. It puzzled me for ages. But, it`s about the only thing I can remember about any primary school essays…. That`s odd too, isn`t it?