I'm gonna go with "They all do that".
Here are five of my esp32s running light camera duty and their response times look every bit as random as yours
Pinging espcam1 [192.168.1.201] with 32 bytes of data:
Reply from 192.168.1.201: bytes=32 time=61ms TTL=255
Reply from 192.168.1.201: bytes=32 time=278ms TTL=255
Reply from 192.168.1.201: bytes=32 time=182ms TTL=255
Reply from 192.168.1.201: bytes=32 time=90ms TTL=255
Pinging espcam2 [192.168.1.202] with 32 bytes of data:
Reply from 192.168.1.202: bytes=32 time=139ms TTL=255
Reply from 192.168.1.202: bytes=32 time=57ms TTL=255
Reply from 192.168.1.202: bytes=32 time=279ms TTL=255
Reply from 192.168.1.202: bytes=32 time=189ms TTL=255
Pinging espcam4 [192.168.1.204] with 32 bytes of data:
Reply from 192.168.1.204: bytes=32 time=139ms TTL=255
Reply from 192.168.1.204: bytes=32 time=34ms TTL=255
Reply from 192.168.1.204: bytes=32 time=248ms TTL=255
Reply from 192.168.1.204: bytes=32 time=164ms TTL=255
Pinging espcam7 [192.168.1.207] with 32 bytes of data:
Reply from 192.168.1.207: bytes=32 time=100ms TTL=255
Reply from 192.168.1.207: bytes=32 time=96ms TTL=255
Reply from 192.168.1.207: bytes=32 time=113ms TTL=255
Reply from 192.168.1.207: bytes=32 time=119ms TTL=255
Pinging espcam8 [192.168.1.208] with 32 bytes of data:
Reply from 192.168.1.208: bytes=32 time=180ms TTL=255
Reply from 192.168.1.208: bytes=32 time=198ms TTL=255
Reply from 192.168.1.208: bytes=32 time=217ms TTL=255
Reply from 192.168.1.208: bytes=32 time=17ms TTL=255
All of my 8266s are used for BT/Serial bridges, ain't gonna get no ping responses from them ever...
Cheers!