If only bad people were prejudiced, that would not have such a strong effect. Most people would not wish to imitate them—and so, such prejudices would not have much effect—except in exceptional times. It is the prejudices of good people that are so dangerous.