🔗 Does a url in a prompt steer an llm's output toward its content?

Does the inclusion of a URL in a prompt influence the output of a language model? Paul Kinlan explored this question through experiments on LLM responses to various URLs. He found that URLs can have an impact, but only if their content is part of the model's training data. Significant observations included that JavaScript-heavy sites often produce no recall, while descriptive URLs help guide output effectively. Kinlan demonstrated that opaque URLs generally perform poorly, while some well-known identifiers show high accuracy. This raises concerns about how data is gathered for training models. The exploration reveals a complex interaction between prompts and language model outputs, highlighting the necessity for transparency in LLM training data.