If the beer is fermenting, there isn't any reason to throw more yeast in it. You move the beer to secondary to get it off the yeast cake and trub to clear up a bit.
I don't know of any real downside, besides having to wait for the yeast to flocculate out and maybe having a yeasty taste to the beer. At this point, though, I think the best thing to do is leave it alone for a total of two weeks (counting the four days in the primary) and then check the s.g.
Most people around here recommend keeping the beer in the primary at least a week before moving it to the secondary tank. If you're not using a secondary, it's good to leave the beer in the primary for about 2 weeks. When it's done fermenting all the available sugars, the yeast "cleans up after itself" and gives a cleaner taste with less off-flavors. Then, it's moved to secondary for clearing. The term "secondary" really is wrong- it should be called the clearing tank or the "bright tank" because you shouldn't expect fermentation to be taking place. I use the 1-2-3 method; 1 week (or more) in the primary, 2 weeks in the clearing tank, and then three in bottles before sampling.
Good beer just can't be rushed. So, it'll take its time and make beer for you. All you need is a little patience.