That's where it gets very, very problematic.
Well, no doubt. But then, I don't think it's easy. And I don't want to dance around the problem or ignore it (some people used (?) to say that in order to solve the violence in our societies you just had to read another newspaper...), yet a utopian construction where there is no violence at all doesn't seem to me to be realistic, nor completely desirable.
And that doesn't mean there should be violence - but at least that removing it altogether might come at a great cost (and violence). And it could be like a virus: removing it completely makes us all weaker to newer threats.
The same holds for the veganist "totalitarianism": I personally don't eat animals or animal products (or at least not knowingly), but I can't impose that on someone else. I will explain my motivations, and point out obvious problems related to "meat eating" - but I don't dream of a society in which no one eats meat. It seems rather unrealistic.
The moral highground is probably a nice place to be - but in the end, one has to sink down to the level where practical compromises have to be made. And I believe the interest of it all lies in there.
I have no interest in uniform(ed) societies...