I haven't made this yet but I've got it saved to make soon. You should be fine, as long as the lactose and DME did get added. Lactose is just unfermentable sugar, and the reason it is added late is (mostly) preference by most people, and by adding the sugar sooner, you changed the hop utilization some.
Is your beer ruined? Not in the least.
Will the IBU's be slightly off? Yes, but probably not enough that it'll make it go from good to bad. If you want to know how it effects in numbers, put the info into beersmith, brewtoad, or brewers friend and check it out.
Cheers!