Any input to the 2nd LLM is a prompt, so if it sees the user input, then it affects the probabilities of the output.
There’s no such thing as “training an AI to follow instructions”. The output is just a probibalistic function of the input. This is why a jailbreak is always possible, the probability of getting it to output something that was given as input is never 0.
No. Consider a model that has been trained on a bunch of inputs, and each corresponding output has been “yes” or “no”. Why would it suddenly reproduce something completely different, that coincidentally happens to be the input?
Any input to the 2nd LLM is a prompt, so if it sees the user input, then it affects the probabilities of the output.
There’s no such thing as “training an AI to follow instructions”. The output is just a probibalistic function of the input. This is why a jailbreak is always possible, the probability of getting it to output something that was given as input is never 0.
You are wrong: https://stackoverflow.com/questions/76451205/difference-between-instruction-tuning-vs-non-instruction-tuning-large-language-m
Ah, TIL about instruction fine-tuning. Thanks, interesting thread.
Still, as I understand it, if the model has seen an input, then it always has a non-zero chance of reproducing it in the output.
No. Consider a model that has been trained on a bunch of inputs, and each corresponding output has been “yes” or “no”. Why would it suddenly reproduce something completely different, that coincidentally happens to be the input?