A keg can be treated like one giant bottle. In fact, a lot of people here let their beers condition in the keg for weeks or months at a time. So long as you purge the air out of it and keep it tightly sealed with CO2 it is fine.
The only issue you may run into is pressure changes from swapping from warm to cold and trying to fill bottles warm. Depending on how quickly you plan on drinking the growlers you could find it hard to keep them carbed (if at all) if filling at room temp. You'll also have to fiddle with the pressure in order to fill a growler because at room temp you may be looking at letting the keg sit at 20 psi or more to carb up to the right level, and in order to fill a bottle you'll need to purge the gas and go down to like 1 or 2 psi so the beer just barely flows from the tap so you can keep your carbonation, etc. Even then, it's a challenge.
So it can be done and it's fine to do so, but it will just take a little extra work on your part every time you want a beer.