As long as the prompt is not encrypted at some point, and I don't think LLMs can run on encrypted prompts, then it can be read.
What if the GPU has a custom bios flash that somehow logs the unencrypted prompts?
As long as the prompt is not encrypted at some point, and I don't think LLMs can run on encrypted prompts, then it can be read.