No problem at all. It is pretty common that folks will leave their beer in the primary for up to a month.
The purpose of using an secondary is to help clear the beer. So you really need at least a week or two in the secondary. It won't hurt to leave it longer, either, especially if it is a big beer or a beer with lots of dark specialty grains.