If you're coming from Hakone or Kawaguchi-ko areas, it doesn't make sense to meet the group in Tokyo because you're pretty much at Mt. Fuji already.
Instead, take a bus from either area to Gotenba train station. Buses are frequent and cost effective, but you'll need to talk to your hotel for specifics on time schedules as they change frequently.
The bus to Gotenba from Kawaguchi-ko takes about 1 hour and from Hakone is between 45 minutes to 1 hour, depending on your location within Hakone.
You can select a Gotenba pick-up or drop-off option during checkout when you make a tour reservation. During the in-season 2-day Mt. Fuji tour the group will meet inside the Gotenba train station (right in front of the only convenience store) at 9:00 am. The off season 2-day tour pick up time at the same location is 11:30 am. For more details on the pick up location at Gotenba, click here.