While I have not really been involved in the organizing of a competition, I've judged a couple and entered into a few more. The BJCP has some great resources you can peruse. They have an extensive list of recommendations and requirements. You don't have to host a certified event to read through how they organize their judging, but I bet it will still give you lots of ideas.
I think part of your consideration may be the number of entrants. Also if you wish to continue to judge based on overall taste, the scoring system would probably not fully work for you. But, you might see other things about their competitions that would work well.
I think most competitions I've entered into have stored the beer in a cooler for a few days to chill the beer. The only thing the taste knows is the style and the number of the beer. The number system helps to keep people from knowing who brewed what and keep it fair. When a beer is entered, an organizer writes down the entrant's information and assigns the beer a number, which ties in with the personal information on the organizer's paperwork. The tasters never know who sent what. Along with that plain common bottles are used to further prevent favoritism.
Number of tables depends on how many beers are entered and how many tasters you have. Normally each judge would sample a particular style along with a partner and if the number of entries is small enough, they may do more than one style category.
Normally rinse water is provided and crackers, so you can cleanse your palate between beers.
There are people helping called stewards who bring the bottles of beer to the judges and do the gopher stuff when needed.
There are tons of other things that go into a BJCP competition, so I suggest you read through their requirements and cull out what doesn't suit you, and implement what does.