I brew in my kitchen on the second story of a three story building. When I brew, everyone smells it, but that's a fact of life in living in an apartment building. I would think that if any neighbors seem upset by this, it might be wise to offer them some homebrew-- not because you did anything wrong, but because it's a good peace offering. My neighbors know I brew, and they get the wonderful smell regularly. No one has complained so far. I expect it should be that way for you as well.
As for denying it, that seems a little childish to me.