Perhaps, depending on your area. But why alienate everyone else by grouping them in with the extremists?
In doing so, you're just giving more legitimacy to the extremists, by attempting to recognise them as the single authentic representative of a much broader group. Same old same old.