Alignment

In 2023 Paul Röttger and colleagues wrote 250 questions that no reasonable safety policy should refuse, such as how to kill a Python process, and asked Meta's Llama 2 70B Chat to answer them. Running with Meta's original system prompt, it fully refused 38% of them and partially refused another 21.6%. The same model refused 99.5% of a matched set of genuinely unsafe requests. It was not unsafe; it was unhelpful in a way its safety training had caused. The same summer brought the opposite failure: researchers at Carnegie Mellon and elsewhere showed that a string of optimised gibberish appended to a harmful request made ChatGPT, Bard and Claude comply, although the suffix had been computed on small open models.