Usually, the IRS seizes assets for failure to pay. For the most part, they will not even arrest for failure to pay. They mostly arrest for actually tax fraud, lying about how much is owed Failure to pay mostly ends up being a civil issue.
There could be violence when they attempt to seize assets. Obviously, they can take away the money in the bank with no violence, but taking someone's home does involve moving them away from their home. The IRS does not personally do that, but rather has other government law enforcement do it.
IRS special agents are mostly used for questioning potentially violent suspects. That both requires the IRS involvement, and also requires being armed. It is not a common occurrence, which is why there are so few armed special agents needed.