I’ve noticed this aswell.
I’m guessing it has something to do with the 20min journey time.
ie, passenger goes to other town via train, train gets held up and the journey time increases (or a bus on the other side of the train?) and it pushes the entire journey time over 20mins, poof, passenger gone.
It might happen on buses/coaches too but it’s not as obvious.