TL;DR: You can see the results of GitBench here https://gitbench.gitkraken.com

GitBench is our attempt to see how well LLMs handle git-specific tasks. We put together a suite of 17 benchmark categories with a total of 204 specific test cases. We ran these benchmarks against 43 (and counting) different models with all of their supported effort/reasoning levels.

LLMs these days have gotten really good at some of the benchmarks that are available. There are benchmarks like Humanity’s Last Exam that test general knowledge and reasoning. The largest frontier models are best suited to that test because of the huge corpus of training data they have been built on top of. Then there are more coding focused tests like SWE-Bench which comes in several flavors like Verified and Pro and tests a models ability to come up with a similar Pull Request and code change for Issues that have already been resolved on various codebases.

Benchmarks like these help us understand the capabilities and limits of many of the models that exist today. There is speculation that some of the models and creators of models are “benchmaxxing” which is really just the model focusing on solving those specific benchmarks rather than focusing on improving their general intelligence instead. One way we can combat benchmaxxing is by creating more specific benchmarks and so many of them that optimizing for them is too costly to pursue.

Take this Theo tweet for example:

Inspiration

The Theo tweet is actually somewhat recent compared to when we started building GitBench. We tend to use artificialanalysis.ai for a reference when new models come out. This could be for our dev team to understand which models are worth the session limit cost or it could even be about which models we think should be best used for various parts of GitKraken AI, whether that is:

• Commit Message Generation
• Pull Request Generation
• Merge Conflict Resolution
• Automated Rebasing
• Commit Composer
• and more

What we realized, is that there isn’t a specific benchmark that helps us evaluate which models are more capable and efficient than others for these specific things. Of course, we use some evals in the way that Braintrust, Arize, and others mean that term, but to create evals for all of these things is a very involved effort and the results are a lot harder to share.

Another reason we made it was because I thought it sounded like a lot of fun to make. I haven’t really tried my hand at research like this and building out the harness, fixtures, and results web app was a great experience even if an LLM wrote most of the code.

How It’s Made

GitBench started out as just a CLI to run the benchmarks. We needed to output JSON into a format that Artificial Analysis might be able to consume, if if a long shot. But then we realized we needed some kind of web app to truly visualize things.

The CLI is built with python since that seemed like the common choice for things like this in the AI and ML space. I’m not the greatest python dev, but I can get enough done that I felt confident I could review the code as we iterated. Building CLIs in python can be as simple as some argparse functionality, but I have come to expect a bit more of a developer experience from frameworks/libraries in other languages, so we opted for click. While the benchmarks run, we also wanted some nice output to keep up with the progress and status of the individual models.

The web app is its own beast. We wanted to consume the results.json into it, but we needed the data to be explorable as well as statically rendered for SEO which naturally led to using Astro. The graphing is done with recharts and the components are shadcn/ui because their lack of needing a React Context makes them play well with frameworks like Astro.

Openrouter, Oh me-oh-my!

At this point, you might be wondering where we are going to test these models. Well, OpenRouter has access to almost every model you could possibly care about. We couldn’t possible test every single model that OpenRouter provides so we pick and choose which new models we test beyond our initial baseline. Our current config still leaves room for other providers, especially if they support the OpenAI SDK or other compatible SDKs, but we think we are at least covering our bases by choosing OpenRouter.

Happy Little Surprises

We found a bunch of strange edge cases as well as feedback related to how we handled all parts of our workflow. Some of them are obvious in hindsight and others took us completely by surprise.

Surprise 1: JSON Structured Output

I was at a conference recently and I was explaining GitBench to a few people. One person I respect responded to me mentioning that some models are really bad at following directions.They suggested that maybe underperforming models would perform better if given better guidelines. Without spoiling too much, I will say the results are really worth thinking about and we would love to chat more if we could test things better.

If you don’t know what structured output is, you can think of it as a set of instructions you provide to a model that HEAVILY encourage it to spit out a JSON structure that matches what you asked for.

One thing we didn’t predict was some models not supporting structured output in any way. It was pretty rare and seemed isolated to some of the smaller models. Nonetheless, that was a fun problem to adapt to.

Another problem we ran into was that some models would respond with output that looks plausible from a distance until you actually look at it.

Example: { "The response we were expecting" }

That isn’t event valid JSON. That means validating the JSON output is an important evaluation step.

Another JSON issue we ran into was that we were being a little too generic in our specific JSON output. We had a generic "answer" key on the object which actually runs counter to how models create the output. They actually tokenize the keys in a way that they look for relevant tokens in the text output to place into the value part of the JSON object. This meant we needed to be far more specific in all of our benchmarks. Now we explicitly request JSON with keys like "commit", "commit_message", "command", "files", and more.

Surprise 2: LLM as Judge

We started digging into the code and seeing how some of the tests were run. We saw that the commit_messages benchmark/suite was using a string similarity algorithm that could have some interesting false positives and negatives. Luckily, LLMs are pretty good at judging the output of each other, so we employed openai/gpt-oss-120b, google/gemma-4-26b-a4b-it, ibm-granite/granite-4.1-8b to validate the results. This lead to a much more predictable result after averaging the scores between the judges. We chose these models because they are quite cheap but very capable at non-coding tasks.

Surprise 3: Nemotron Super Moonlights as a Danielle Steele Ghostwriter

Actually it’s worse than that. Nemotron wrote some very NSFW-style romance novel content. You probably think that sounds extreme. We agree. Nemotron is just seemingly trained on a very different data set compared to other models. In a tamer run, Nemotron didn’t go down a NSFW route, but it did respond to one of the same prompts with a “Create Your Own Quiz” app architecture and it got REALLY specific about the implementation. We’re talking more than 20x the expected output tokens.

Naturally, we had to redact NSFW content from our reports. We caught it before it became a problem, which was fortunate. But, that meant yet another iteration on the data gathering process that needed to be tested, validated, and then rerun on a larger scale.

Wrapping Up

We hope you found the process and our inspiration for creating GitBench interesting. We also hope that it inspires you to think about how you might want to start evaluating models for the tasks and tools you care about. You are encouraged to take a look at the results over on GitBench and form some opinions. On the site, you can also sign up for our detailed analysis in PDF form. As a bit of a teaser, you might be surprised which models outperform others and which ones don’t meet expectations. You would probably also be surprised about costs and token usage being counterintuitive in some cases. There are some other interesting learnings and takeaways, as well. Thanks for reading.