I think you answered the main concern. Contact time is important for leaching. I personally don't have an issue using brass in certain areas of my system.
The only other concern is its durability with certain cleaning materials like caustics. Its not as durable as stainless, but in the home setting it is fine to use it.