TFGM can be dropped anywhere, it doesn’t need to be in the TF root folder. TFGM is maintaining its own library of mods, so when you uninstall a mod from TYF it isn’t lost. All mods are kept in their original format and converted up on install.
UG do recommend TFGM or TFMM as mod managers in their FAQ section
http://www.train-fever.com/modding-2/faq/
But it is all up to you if you want to use one or not. A lot of users find them time saving. You can drag and drop folders with multiple mods at once in TFGM for batch install.
The delay when starting up TF is not related to TFGM, it sounds quite logical that TF has to scan through the mod folder to build the mod list. More mods, more time…