It is true that the Gondor building is difficult to spot from STREET LEVEL in Bree.

Here is the building from a rooftop in NW Bree, you can see the Prancing Pony on the left in this image, and the Gondorian rooftop on the right.



Also, the Gondor building is a clear feature from in front of the Cat lady's home. The label is simply "Door" for the new Scholar's Hall