this is one of those "its up to you" questions. adding and removing berries, or any fruit, at different times will yield different results. adding it early may cause the fruit taste to be more subdued while adding near the end may make it stronger. also the amount of berries added will also affect the strength of their taste.
your timing sounds just fine to me. let us know how it turns out.