Question about yeast harvesting from fermenter

I have a question about harvesting yeast from a primary. I will be brewing a seasonal beer in the upcoming days and will be pitching pacman yeast that I am currently building up. I want to build up a nice collection of the yeast and was going to harvest a larger quantity from the upcoming brew but, what I am unsure of is if off flavors will follow the yeast when harvesting.

My upcoming brew will have ginger and a number of spices. If I harvest yeast from that batch can traces of the spices follow the yeast into other beers made?
Or will multiple yeast washing wash out those spices from the yeast?

I would just make double the size if your starter and pitch half and save the other half. That way it's still clean yeast.

Don't even need to double it.

Just save a small amount from the 'clean' portion you are building up (i.e. don't throw the entire starter into the ginger batch) and then build it up again. Easy as pie.