One of the most influential emerging technologies today is Large Language Models or LLMs. LLMs are language models like GPT-3 and GPT-4, which are the models behind ChatGPT. These models are the results of years of research in machine learning and artificial intelligence. The release of such models has proven that technology can become so powerful and can actually help us make our lives easier. That being said, though these models are good and, with time, their performance will only improve, it’s very unlikely that they will be as “smart” as humans are.
When chatGPT was released a few months back, it triggered other companies to release their own language models. Which leads to an important question, what are the factors affecting the quality of performance of such language models? How can we, as users, developers, and students of data science, better understand and evaluate the performance of these models?
As we just mentioned, these models are the results of extensive research. However, we can consider a set of factors if we want to evaluate the performance of a language model.
When interacting with large language models (LLMs) like GPT-3 or GPT-4, several parameters influence the behavior and performance of the model. In this article, we will go over and briefly discuss some of the most important ones:
MODEL SIZE:
This refers to the number of parameters in the model. The larger the model, the better it will be in understanding the nuances of human language and hence can generate more accurate and nuanced responses. Parameters are different variables that developers tune when they are training the model and thus setting up patterns for how it will perform on new data. Though the more parameters a model has, the better it will perform. Unfortunately, the number of parameters may make the model more computationally expensive to run. So, developers must trade between the performance and computation energy required to run a model.
PROMPT:
In the world of LLMs, a prompt is whatever the user gives the model, whether it is a question or an instruction. The simpler, more precise, and more direct a prompt is, the better the results the model will return.
Generally and regardless of the model, the prompt should be clear and precise and give the model enough context to generate the response you want. Since the release of chatGPT, a “new” job role has been created called Promot Engineering. Prompt Engineering is all about crafting and articulating the best prompts to feed LLMs to get the responses we want back from them.
TEMPERATURE:
When you try some generative AIs, you’re sometimes asked how random you want the generated answer to be. Often, temperature is a number between 0 and 1. A higher temperature (closer to 1) will result in more random outputs. While a lower temperature (closer to 0) will make the outputs more deterministic and consistent. A higher temperature might be beneficial for tasks requiring creative or diverse responses. For more factual or specific queries, a lower temperature can be used.
MAX TOKENS:
Token here means the length of text (in the case of LLMs). So, max tokens mean the maximum length of the model’s response. Usually, the larger we set the parameter, the more details and information the model can return. It is also important to know that often, the max tokens refer to the max length of a model’s response and the prompt the user is allowed to give the model. For GPT-3, the max token (or we can call it the model’s limit) is 4096 tokens.
FREQUENCY PENALTY:
This is a simple parameter, and it’s about training the model to not repeat itself, leading to more original responses.
It is also a test of how the model can come up with creative responses that are not repeated to deliver similar ideas.
PRESENCE PENALTY:
This parameter encourages the model to bring up new topics in the conversation. These topics are related to the original topic proposed by the user. This parameter tests the model’s ability to connect related topics. It gives the user a better picture of the topic they are inquiring about.
TOP-P:
Last on today’s list is Top-p. In natural language processing (NLP), there’s a method called nucleus sampling, which is used to generate text. Since LLMs are models mainly developed to generate text, the value of Top-p significantly affects the model’s performance. Meaning a smaller value for top-p will make the output more focused and less random.
WRAP UP:
Large language models are becoming a massive part of our daily lives; they can significantly help automate and make everyday mundane tasks less time and effort-consuming. With time, more and more LLMs will be released to people; some will be free, some won’t, so how can we decide which model is better for whatever application we intend to use it for?
There are different approaches engineers and data scientists use to evaluate language models; these approaches depend on comparing other parameters. We discussed 7 of these parameters in this article.
So, next time you think about which language model to use or wonder how people evaluate and label LLM as “good” or “not-so-good,” I hope the information in this article gave you an idea of how that label was decided.
