AI Code Reviews with Open Code Review & Themis – Part 2

In Part 1, I wrote about why AI code reviews make sense as a first pass & why the existing tools left me unimpressed. This part is about what worked for me and how I implemented it.

Open Code Review

Couple of weeks ago Alibaba released their code review tool named Open Code Review (OCR) – something they’ve been using on their own codebases for last 2 years. Now available as an open source tool, OCR is written in Go. It reads the git diff & produces structured, line level review findings. The part I like – it speaks the OpenAI chat completions protocol (and the Anthropic protocol too), so you can plug in any inference provider. OpenAI, Anthropic, OpenRouter, Amazon Bedrock, a self hosted gateway, whatever. Your model, your API key, your bill. No middleman marking up inference & no guessing what is actually running under the hood.

So far I’m liking OCR – it does a pretty good job. But two things I learned quickly:

1. The choice of model matters. A lot.
2. Whether thinking/reasoning is enabled matters as well. By default OCR keeps thinking disabled, you’ll need to enable it in config specifically.

Don’t forget to turn the thinking on!

Providers don’t agree on how reasoning is requested, so there is no universally safe default – you have to configure it explicitly in your provider’s dialect. For OpenRouter, which normalizes this across vendors:

		
			{"reasoning": {"effort": "high"}}		
	

For a GLM / DashScope style endpoint:

		
			{"thinking": {"type": "enabled"}}		
	

Don’t assume a model is using its deeper reasoning mode just because it supports one. Check your provider’s request format & set it explicitly – otherwise you’re paying for a thinking model & getting its reflexes.

GPT 5.6 Luna vs GLM 5.2 – let’s see the numbers!

To see how much the model actually matters, I ran the same review with two models via OpenRouter, both with reasoning set to high:

1. openai/gpt-5.6-luna-pro
2. z-ai/glm-5.2

The codebase under review was a new project I’m working on – about 3000 lines of code spread out in Go & Bash scripts (more on it below).

So almost similar in speed & volume – which itself is a good reminder that the number of comments is not a measure of review quality. The difference comes in quality of the feedback.

openai/gpt-5.6-luna-pro did the better job. It ignored some context in code written as code comments – to explain why certain things have been done in a specific way – and flagged them anyway, but the overall quality was still good.

z-ai/glm-5.2 was the mirror image – it did not ignore the context in code comments but overall its quality was a bit sloppy in a few places.

For fun, I then fed both review artefacts to Claude Fable 5 (on xhigh reasoning) & asked it which review is better. Fable’s verdict:

Close, but I’d give the GLM one a slight edge on substance – while trusting GPT 5.6-Luna more as an unattended reviewer.

That might sound contradictory but it describes the trade off rather well. If a developer is going to read every comment carefully then the review with a bit more substance might win. If the reviewer is going to run unattended on every PR then consistency & a lower false positive rate matter more. For CI, I’d pick GPT 5.6 Luna based on this test.

This is one project, one PR range & one configuration – not a scientific benchmark. Results will shift with language, codebase, rules & model updates. I plan to test more models over time.

What about the cost?

A note on those numbers. The GPT 5.6 Luna price benefits from a 50% discount OpenAI has running at present – promos are temporary, so don’t treat one run as a permanent price benchmark.

GLM 5.2 being an open weight model means there is room to shop around – different providers on OpenRouter compete on inference pricing & run their own promos (StreamLake has 50% off on GLM 5.2 right now, for example). You can even self host it, though then the hardware & ops cost becomes your problem. Proprietary models like GPT 5.6 give you far less room – there are only 4 providers (OpenAI, AWS, GCP & Azure; GCP is more or less a router similar to OpenRouter while AWS & Azure host the models on their own infra), so you won’t see much price variation there.

Either way – well under a dollar for a full review of a 3000 line codebase. That is the economics which makes “AI first pass on every PR” viable.

Automating code reviews in PRs with Themis ⚖️

OCR is a CLI tool. To make it useful on every pull request, it needs to be wired into your CI – run on the right trigger, read the right diff, post findings as inline PR comments & not spam the PR on every push. That glue is what I built and I’ve made it available as an open source project.

Themis – a composite GitHub Action built in Go, named after the Greek goddess of fair judgement.

Themis can run OCR on all your PRs & publish the findings as inline review comments – with GitHub suggestion blocks you can apply in one click, deduplicate comments based on content fingerprinting, comment budgeting, .themisignore review exclusions & an optional severity based merge gate. And it inherits OCR’s provider agnostic nature – use whichever inference provider & model you like. You control your AI billing entirely; there is no per-seat code review subscription layered on top of the inference cost. And no middlemen adding their mark-up on the AI inference you use.

Getting started is a single workflow file:

		
			name: Code Review

on:
  pull_request:
    types: [opened, synchronize]
  pull_request_target:
    types: [labeled]

permissions:
  contents: read
  pull-requests: write
  actions: read

jobs:
  review:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: coolamit/themis@latest
        with:
          llm-url: https://openrouter.ai/api/v1
          llm-api-key: ${{ secrets.THEMIS_LLM_API_KEY }}
          llm-model: openai/gpt-5.6-luna-pro
          llm-extra-body: '{"reasoning":{"effort":"high"}}'		
	

That’s it. No checkout step (Themis performs its own), no Node.js, no npm install – just two static Go binaries (Themis & OCR). Same repo PRs get reviewed automatically & you can apply the themis-review label to review any PR – including forks – on demand. Label triggered runs verify that the labeler has write access or above; the label alone is never trusted.

Why not just use OCR’s own action?

Fair question – OCR ships an official composite action of its own. Themis is not a competitor to it; it exists because I did not quite agree with how a few things were implemented there and I’ve made my take available as OSS in case its useful to others. The differences which mattered to me:

• Deduplication which never hides anything. Push to a PR & the whole PR gets re-reviewed – a later push can make earlier code buggy without touching it again (a changed function contract, a removed guard). To keep that from turning into comment spam, each finding carries an invisible content fingerprint – a hash of the file, the flagged code & the category rather than line numbers or the LLM’s wording. Already posted findings are recognized even when lines drift or the LLM re-words its review. And when the fingerprint misses, a positional fallback demotes the finding to the summary comment instead of silently dropping it – nothing is ever hidden, which also means the severity gate always sees every finding.
• Comment budgeting. At most 25 findings go inline per run (configurable), with critical findings filled first & allowed to exceed the cap. Overflow folds into a summary comment with links into the diff. Nothing gets dropped.
• A severity merge gate, off by default. Set fail-on-severity: high & any new finding at or above that threshold fails the check with a distinct exit code. Off by default deliberately – severity is LLM assigned, so run in report only mode first, observe the reviews for a while & grant veto power only after you trust the model. A red check on every PR which gets comments just trains people to ignore CI.
• Quiet PRs stay quiet. No new findings, no comment. Docs only PRs skip green without ever invoking the LLM. Nothing erodes trust in a review bot faster than “LGTM! 🎉” spam on every push.
• Fork PRs handled safely. Fork code is never executed – the PR head is read from git objects only, never checked out. And credential pre-flight means a misconfigured key fails before a paid review starts, while fork PRs without secrets skip cleanly instead of going red.

Where a human is still needed

An AI reviewer does not understand business intent unless that intent exists somewhere it can read. It does not attend product meetings, does not remember why that ugly workaround was necessary six months ago & does not know that an apparently harmless API change breaks an important integration. It can also be confidently wrong – my test above showed even good models flagging things which a human would immediately recognize as intentional.

So I would not let an AI review be the only approval for high risk changes – authentication & permissions, payments, destructive migrations, cryptography & security boundaries, public API compatibility, large architectural moves. Even with the severity gate on, treat it as one more CI signal – like tests & static analysis – not a substitute for engineering judgement.

One more thing worth remembering – unless the model runs inside your own environment, your changed code is sent to the inference provider. Themis removes the hosted code review vendor from the path but it does not magically make a third party model API private. Pick your provider & data retention settings according to the sensitivity of the repo.

Closing remarks!

An AI Code Review is not a magic bullet and it will not give best results in every situation. But the pieces have finally come together to make it practical – open source review engines like Open Code Review, open weight models like GLM 5.2, Deepseek V4 Flash (and proprietary models like GPT 5.6-Luna & Grok 4.5) create real price competition & inference routers like OpenRouter, Amazon Bedrock, etc. which let you swap models with a one line change.

My current view is fairly simple. For an important project, use AI as the first reviewer & a human as the final one. For a project where a human review is not going to happen anyway, a well configured AI review is considerably better than no review at all. And do not grant an AI reviewer veto power until you’ve tested the model on your own codebase & learnt what kind of mistakes it makes – the model matters, the reasoning configuration matters and the pipeline around the model matters just as much.

Themis is my attempt at wiring these pieces together into something you can drop into a repo in 2 minutes. Its Apache 2.0 licensed – give it a spin on one of your repos, start in report only mode, see what your model of choice catches & tune from there. Issues & PRs are welcome. 😀