The reactions do take some time to complete so pH usually gradually declines over 20 minutes and even a bit longer but most of the decrease should have taken place after 20 minutes. To decrease 0.4 or more from that point on would be most unusual and suggests that indeed there was something wrong with the 20 minute reading. Certainly some residual phosphoric acid could throw the reading off. Whenever you get a reading that causes you a raised eyebrow you should repeat it and if is still off a cal check on the meter is in order. An important part of using a pH meter is knowing when not to believe it.
I have had pH readings under 5 in mashes that use sauermalz. I theorize that the lactic acid on the surface of the malt is washed off at strike and is responsible for the low reading. Within a few minutes, however, this acid has a chance to react with bases in the othe malts and the pH goes up starting to level off at around 20 minutes.