Pardon me. The self same argument can be made for those that opposed Hitler and the Third Reich. Indifference doesn't win you a pass.
Those whites who were "actively" opposed to slavery were known as abolitionists. The folks that didn't comment don't get to align themselves with the front-liners. Damn.
The moral equivalency is patently ridiculous. Many whites actively worked against slavery, were they somehow complicit? Your argument is total rubbish. It would basically be saying that those Germans who actively worked against the Nazis in underground movements were complicit in the actions of those they acted against. Many whites never had an ancestor here and their families came much later, are they complicit in slavery? What if they left serfdom to come to the US in 1906? Should they be considered equally responsible as slave traders and owners?
The assumption that every white in existence should be "ashamed" of what their ancestors did during the slave trade in the US is flat stupid.
You can be pardoned... But realistically assuming that every white in America now is responsible for what was done in the past is foolish. We don't put people in jail for what their father did, let alone their great-great Grandfather...