100 million is definitely too many - that was just before the Black Plague.
It's more likely, as was said above, that Middle-earth in the late Third Age was more analagous to Europe at the time of Charlemagne - so, more like 25 or 30 million. Though I doubt Tolkien delved to heavily into demographics.
