With AI assistant technology proliferating, organisations need to know how to choose the right genAI tool for them.
As a result, JetBrains’ Irina Mariasova offers a viewpoint on key questions to ask and metrics to compare.
“Which model is best for you? Do you need just one, or should you mix and match for different jobs? The answer isn’t always obvious,” Mariasova explained.
The AI-powered features inside many JetBrains products can boost productivity and performance.
However, success depends on choosing correctly. For instance, the speed metric can prove crucial. How fast does the model generate responses?
“If one model is slower than another, that’s not necessarily a bad thing,” cautioned Mariasova.
“Some models take extra time because they use a reasoning-based approach, which can lead to more precise answers.”
JetBrains has in-house speed data calculated in tokens per second (TPS).
Meanwhile, organisations should also consider hallucination rate. No AI model is perfect.
“Some models have a higher tendency to generate incorrect or misleading answers. The lower the hallucination rate, the better,” Mariasova said.
GitHub has data on hallucination rates, she added.
Another consideration is context window size. This defines how much code a model can process at once. The larger the context window, the more the AI can ‘remember’ in one go.
This function can be crucial for working on complex projects, Mariasova said.
AI Assistant from JetBrains can blend strengths
In addition, think about coding performance. Reliable benchmarks for rating large language models (LLMs) include the likes of HumanEval+ or ChatBot Arena.
“HumanEval+ measures how well an LLM can solve Python coding problems within a certain number of attempts, with 100 as the maximum,” Mariasova said.
“ChatBot Arena ranks LLMs based on real user feedback (while) Aider’s polyglot benchmark evaluates how well LLMs write and fix code in multiple programming languages.”
She said JetBrains’ multi-model AI Assistant can blend different LLMs’ strengths, including from OpenAI and Google. Click here for data at the time of writing on different LLMs’ performance.
“We work hard to connect you to the best available LLMs as soon as they’re released. The world of LLMs is vast and evolving rapidly, and no single model excels in every aspect,” Mariasova said.
( Photo by Mohamed Nohassi on Unsplash )
