I doubt it being caused by agents / number of player vehicles / lines, since I have deleted all my vehicles and no noticable change to performance. I expected a different result there.
Also, if it where the agent simulation, then the low FPS should persist no matter where I am looking. As you can see, looking at an almost empty field yields very high FPS. A developed town, very low FPS by comparison. All in the same game / save.