If it was an extract kit, most of the fermentable sugars come from the extract, not the specialty grains. Usually specialty grains are for color and subtle flavors more than adding to the gravity.
Per pound, most specialty grains will add about 0.005 to the gravity compare to LME's .006 - .008 and DME's ~0.01. In extract kits you usually have 1-2 pounds of grain, so it's only affecting your OG by about 0.005 - 0.01. So if it is an extract kit, with an OG of 1.04, it's likely it either didn't have enough malt extract or you didn't adjust your measurement for temperature. For instance, if you measured the wort at 130F, your OG was actually 1.05 rather than 1.04.
According to the BJCP Style Guidelines, an Oatmeal stout should have an OG between 1.048 and 1.065, so the temperature adjustment would put you right in that range. Otherwise there just wasn't enough extract and you've made a lighter stout
