As seems to be a custom in Angleland, the Keep, in this case Duke (name)'s, is nowhere near the market or abby of the city; where nearly all the residents live. In fact, it is a ten minute walk from the actual city to the keep. Travelers even must cross a bridge to get to the keep, as it is on a small island.