Try and plan out your brew day in advance, get everything cleaned and sanitized an laid out in preparation. Make a list of things that need to be done and the steps to follow that you can check off as you go and stay organized. It was only your fourth batch so there is still learning going on. After several batches your system will fall in to place and things will get smoother as you become more familiar with everything that goes on.
Yup, this. We followed this list for about a year I think. There is a lot going on so it is very easy to forget things.
Brewing procedure
At least 2 days before brewing check yeast viability
1 Gather ingredients, water, and equipment
2 Open beer, gotta stay hydrated
3 Start steeping water heating, heat to temp then turn to low
A
4 Put specialty grains into bag
5 Steep grains at suggested temp
6 Start boil water heating
7 Put LME jug in warm water to warm up
8 Set up burner
9 When steeping is done pour grains and steep water through strainer into boil pot
10 Heat till almost boiling
11 Remove from heat then add extract
12 Heat to boil and follow brewing instructions
13 Start top off water boiling with 20 minutes left in boil
14 Place wort chiller in pot 15 minutes before the boil ends
15 Cool wort
16 Sanitize fermenter while wort cools
17 Pour wort into fermenter
18 Sanitize yeast and scissors
19 Top off to 5 gallons with cooled water
20 Check specific gravity, add more water if necessary
17 Pitch yeast
18 Put airlock or blow off tube on fermenter and put in a cool dark place
19 Relax, don't worry, have a homebrew - Charlie Papazian