Amar Merabti · notebook FR
← Back to the notebook

Exp. 05 · 2025 · MSc research project supervised by Mohamed Bouadi

Three 7-billion-parameter LLMs, three tests

How good are three open-source LLMs, really, at maths, common sense and code?

128/164
coding problems solved by DeepSeek-R1
61.2%
on HellaSwag for Mistral, the best at common sense
258/500
maths questions where DeepSeek-R1 gives a usable answer
~100×
slower: the price of long reasoning
01 · The protocol

Measure the same thing for everyone

Three 7-billion-parameter models, served locally with Ollama on Kaggle GPUs: LLaMA 2 (Meta), Mistral (Mistral AI) and DeepSeek-R1 distilled into Qwen, built for reasoning. Every raw answer is saved alongside the extracted answer, and metrics are recomputed from those files: results can be checked without rerunning the models.

TestTaskProtocol
GSM8Kgrade-school maths problems, 500 questions8-shot, extraction of the final numeric answer
HellaSwagchoose the most plausible ending among 4, 500 questions0-shot, the model returns an index
HumanEvalcomplete a Python function, 164 problems0-shot, code run in an isolated process against the unit tests (pass@1)
Fig. 1 — The questions come from the original datasets (GSM8K, HellaSwag and HumanEval, MIT licence) and the answers shown are the ones the three models actually produced during my runs: nothing is replayed here. Your turn to do better.
02 · Results

No model wins everywhere

HumanEvalPython code, 164 problems, pass@1
DeepSeek-R1 7B78.0%
Mistral 7B31.7%
LLaMA 2 7B9.1%
HellaSwagcommonsense, 500 questions, 0-shot
DeepSeek-R1 7B51.8%
Mistral 7B61.2%
LLaMA 2 7B24.1%
GSM8Kmaths, 500 questions, 8-shot
DeepSeek-R1 7B47.6%
Mistral 7B44.0%
LLaMA 2 7B5.0%
Fig. 2 — Accuracy on each test, axis from 0 to 100%. The best model on each test is highlighted.

DeepSeek-R1 dominates coding with 128 problems solved out of 164. Mistral is the strongest at common sense and offers the best trade-off between accuracy and response time. LLaMA 2 lags far behind on all three tests.

03 · Behind the score

What a percentage hides

On GSM8K, DeepSeek-R1 scores 47.6%. But its reasoning is so long that it produced a usable answer for only 258 questions out of 500; of those, 92% are correct. The score therefore mixes two very different things: the quality of the reasoning, excellent, and the ability to conclude in the expected format, poor. All for an inference time about 100 times longer than the other two models.

Coding errors tell the same story. Going through the error message of every failure:

ModelFailuresMost frequent error
DeepSeek-R136NameError (27), including 22 where the requested function is missing from the returned code
Mistral112AssertionError (76): the code runs but gives a wrong result; 17 IndentationError
LLaMA 2149AssertionError (90), then NameError (33), 20 of them using List without an import