Most of the members who have been removed from Congress occurred during the Civil War for those who kept their federal office while also holding positions in the Confederacy. But those members were removed by a 2/3 vote of the House or Senate, not by the Speaker.