So let's say you are a security researcher and, working with a "frontier model", you find a critical bug in a very widely used system X. In doing so, you have provided the model with the missing input (whatever text you entered) to help it extrude the program proving the vulnerability. That model was provided by a company that has every incentive to merge your input into the training set, and you have at best a pinky promise that they won't. How much would you bet they really won't?