🔍 What is top_p?

top_p controls how many tokens (words or pieces of words) the model considers when generating the next word. It uses a technique called nucleus sampling.

• top_p: 1 ➜ The model considers all possible tokens.
• top_p: 0.8 ➜ The model only considers the top tokens whose cumulative probability adds up to 80%.

🧠 Example:

Let’s say the model is trying to predict the next word. Here’s a simplified list of possible words and their probabilities:

With different top_p settings:

• top_p: 1
  • All of these tokens are included in the selection pool.
  • The model can choose even rare, unexpected, or creative words.
  • Output is more diverse, but sometimes less predictable.
• top_p: 0.8
  • Only the top tokens (happy, joyful, glad) are considered.
  • Output will be more focused, safer, and more expected.

🤔 top_p vs temperature

📌 Best practice: Change either top_p or temperature, not both at the same time (unless you know what you’re doing).

✅ When to use what?