If they are left in there long enough, both liquids will be the same temperature regardless of the volume. However, since you are pitching only a couple of hours after putting it in the fridge, you are probably right to assume that the starter has cooled down more than the wort.
Nevertheless, the yeast are fine. They can handle a pretty big temperature swing without much of a problem. Certainly a bigger swing than the 10-20 F you are subjecting them to.