Either way a 38-40 is a great score! Sometimes it depends on the comp and the judges. I've had beers score lower and medal in one comp and score higher and not medal in a different more competitive comp. Recently I had a beer score a 29 one week and a 42 the next week and move on to overall BOS. Variation happens.
It could be a clerical error, but they might have adjusted the final scores based on the results of the mini best of show round. Sometimes a comp director wants the beers' scores to be in order. If they winners of the mini BOS ended up beating your beer but had a score of 39, then they might have lowered your official score to 38.
Sub categories will compete in a mini BOS to determine the winner for that category. So you can have American Pale Ales competing against browns and ambers in mini BOS. A comp director can also choose to combine categories into larger medal groups if they want to. For example, light lagers could be grouped with pilsners. As far as who advances to mini BOS, it is up to each judging pair. Each pair usually advances their top 1-3 beers. The reason for mini BOS is to account for scoring variation between different judging pairs doing the same category. But, you really shouldn't have, say, stouts in the same grouping as light lagers or pale ales. What was your beer and what were the medal winners?