First, don't swirl it back in. That's the krausen you're talking about. It eventually falls back in on its own and settles at the bottom. It's mostly hop residue, but there is a lot of yeast in it as well. The yeast is supposed to work from the top in ales.
As far as how long you have to wait, don't panic at all. No matter how long it takes, if it's still going, it's fine. It's when your beer peters out after two days and the gravity is still really high that you worry. Even then, there are things you can do. If your gravity was high to begin with, you're fermenting at the low end of your yeasts ideal temperature range, or your yeast wasn't top quality, you'll have a longer ferment time. If it's your yeast, or a high gravity, there's not much you can do but wait or pitch new yeast, which I wouldn't recomend. The temperature is the easiest to fix, and the most likely culprit. In any case, a longer fermentation process sometimes results in a clearer, more mellow, better tasting beer. Don't worry.