Firstly the FOI would not allow the disclosure of any premise as this is a preconception, or thought. As in working on the premise that ..... A very common mistake is to assume that premise is the singular of premises. It isn't. Premises, like sheep, is both singular and plural.
Now to the point about what you have seen, this would not be disclosable, the FOI covers what we do and not what we see. For example the number and type of visits may be covered but not what the premises occupiers business details etc are as this is not a disclosure of anything to do with us. So s21 would still apply - you may disclose what fell into your duties but not other information. This is intended to cover confidential business information, for example you have visited a premises where a new product is under development and they have an extension in progress to allow for its production. You can/should disclose information on the building and works but not about the commercially sensitive product.
Seems quite clear to me and rather worrying if people thought that FOI would allow businesses/individuals to start asking us about their competitors/neighbours. I know they would get a very polite, but sharp, response in the negative!
