The rest of the voters have not followed trump around for his entire life, like you claim to have, so are not absolutely sure trump has never committed rape. A few percent will believe trump committed rape, and will not vote for him because of that. Even more will be unsure, and will err on the side of safety when voting for the President.
In a way, voters who do not vote at all are worse for the Republicans than voters who vote against trump. A voter who votes against trump, can still vote for other Republicans down ballot. A voter who does not vote at all because of trump can not vote for other Republicans.