Rafal Kochanowski

Embedding models benchmark for code duplication detection

Published: August 20, 2026

Introduction

This research is supplementary material for my code duplication detection tool, Slopo. The goal is to provide data for answering two major questions:

1. What kind of duplicate embedding models are able to detect?
2. Which model is best for this task?

The script used for analysis, all code snippets, and more technical details are available on the dedicated GitHub repository. You can run and verify it yourself, add more models, adapt, and include it in your research work.

Result data are available on separate pages for easier interactive evaluation:

• Gaps between cases - gap between high and low similarity cases, and gaps between single cases focusing on one aspect of similarity.
• Similarity per case - average cosine similarity for each model on each test case.
• Score consistency - standard deviation showing how consistently each model scores across pairs within a case.

Model labels contain the model name with dimensions used in the test. Labels with the | suffix are variants explained in the article.

Results summary

Winners:

1. Jina AI - jina-code-embeddings-0.5b with task code2code.query (available via API) and jina-embeddings-v2-base-code (small model for running locally).
2. Voyage AI - voyage-4-large (general-purpose model), but not voyage-code (both 3 and 4 scored poorly).

Use 256 dimension vectors.

Models from OpenAI, Google Gemini, Cohere are less suitable for use in code duplication detection tools.

Methodology

Models are evaluated on how good they are at separating duplicated code from not duplicated, including adversarial cases.

The data set includes 19 cases, each containing 10 code pairs, giving 190 data points per model. Pairs within a case vary by language, domain, algorithm, problem, etc. For each case, the average cosine similarity and the population standard deviation are reported.

13 cases expect high similarity and 6 low similarity. This split reflects what we consider a duplicate. Note that this benchmark doesn't use the Type-1/2/3/4 clone taxonomy from the code-clone literature.

For each model, average similarity for high and low cases is calculated separately. The gap between them gives a number used for comparing models. In practice, we want our tool to report similar code snippets above the configured threshold, discarding everything below, so the wider the gap, the more precise the results.

Cosine similarity should not be compared between models directly, because their effective ranges differ, which is visible in the data.

Cases design

The cases are designed to evaluate a single aspect, keeping other variables unchanged. Evaluation of what code does is isolated from evaluation of surface similarity. A good model encodes the code's behavior in the vector, while weak ones can only compare text, and this benchmark separates them.

A few interesting examples:

• The high/different-algorithm case includes code pairs solving the same problem with different algorithms. Nothing except language syntax is shared. No shared boilerplate, different unrelated variable names, different control flow.
• The high/code-and-text case directly targets the model's understanding of code. It compares code with a description in plain English. Behavior is described instead of data structures or control-flow mechanisms so it fits the duplicate detection task. No shared vocabulary, texts don't mention variable names. The snippets are not well-known algorithms that the model could easily recognize.
• The low/same-boilerplate case evaluates how recognizable framework boilerplate influences similarity. The non-boilerplate part is completely unrelated, no shared vocabulary, domain, problem.
• The low/unrelated case is a base case telling how low the model can score when the only similarity is the same programming language and similar code size. No shared vocabulary, domain, problem.

Adversarial cases

Some cases are designed to penalize reliance solely on surface similarities.

For example, the low/same-identifiers case (unrelated code with overlapping variable names) is a low similarity case. Reported similarity will be high since much of the text is the same, narrowing the gap between high and low cases. The model that gets tricked less (better one) reports lower similarity on this case, increasing the gap.

How it differs from MTEB

This benchmark is more focused than massive. Each case is a task intended to evaluate one aspect of duplicate detection. Fewer code snippets are used, but each is carefully designed and manually reviewed.

Model selection

The selection criteria were: include code-focused models and big popular providers used widely in the USA and Europe. Access to the model should be possible without having to deploy it myself on expensive infrastructure.

That's why I included OpenAI, Google Gemini, Cohere, and Voyage AI, which also offers code-focused models. Jina AI seems more niche but offers code-focused models available via API and for local running.

Ranking

An essential part of the data is included in the article. Model variants (reduced vector size, task type) are skipped. To browse full data, open links from the Introduction.

The gap between avg(high) and avg(low) cases

If you are looking for a single metric to compare models, here it is:

(text-embedding-3-large and text-embedding-3-small are OpenAI's models)

A comparison of gaps between two related cases reveals more nuances that averages hide.

Matching code with description

The code description column is a difference between high/code-and-text and low/code-and-text cases, telling how well the model understands code. The high/code-and-text pairs code with its description. low/code-and-text reuses the same data but pairs code with unrelated descriptions.

Different algorithm solving the same problem

This targets a common question: Can embedding models recognize that two different algorithms solve the same problem?

The code behavior column is a difference between cases: high/different-algorithm (same problem solved with different algorithm) and low/unrelated (unrelated code as baseline). It tells how well the model can separate related implementations from unrelated code.

Impact of boilerplate

When unrelated code snippets share the same framework scaffolding or well-known pattern, they can be similar for embedding models. This introduces noise in a typical codebase analysis. Good models are better at recognizing such patterns, but this is penalized in this benchmark.

The boilerplate impact column is a difference between low/same-boilerplate and low/unrelated cases. It shows the effect of boilerplate compared to an unrelated code baseline. Lower gaps are better here, so they are sorted in ascending order.

Impact of identifier names

Two identical code snippets with only renamed variable names should be considered duplicates, but they are not for embeddings. Identifiers are very important for embedding models.

The identifier insensitivity column is a difference between high/renamed-identifiers (same code, half of identifiers renamed) and low/same-identifiers (different code, overlapping identifiers). A higher gap is better. Negative value means that overlapping identifiers in unrelated code are scored higher than identical code with different identifiers.

What can be detected

This research can only tell what kinds of duplication can fall above or below typical (practical) similarity thresholds when they are isolated. In a real codebase, multiple aspects contribute to similarity at once. The codebase has similar vocabulary, domain, problems it solves, patterns, naming conventions, identifiers - all of them increase similarity for the whole code, both duplicated and unrelated.

Differences scored as near-identical

• Different formatting of the same code (high/formatting).
• Same snippet with a small (about 15%) addition of new code (high/small-addition and high/small-additions-large - this variant has 4x larger code snippets).
• Same snippet with some statements reordered without affecting results (high/reordered-statements).
• Simple refactoring like code block extracted to function (high/refactored-structure).

Different algorithm

Code snippets implementing solutions differently (high/different-algorithm) won't be detected as duplicates if there are no other similarity hints, but this is an edge case.

For example, a similarity of 0.6506 below our threshold of 0.92 doesn't mean that the model is unable to recognize what the code does. 0.6506 is still much higher than 0.1059 for unrelated code, and this gap is what matters. Snippets with different algorithms may share the same variable names and boilerplate, increasing similarity. Also, both algorithms may be more alike than the deliberately distinct ones in the evaluation set.

Cross-language similarity

In general, embeddings of the same code rewritten to a different language are similar. They are more similar for language pairs with similar syntax (high/lang-similar-syntax) than different (high/lang-different-syntax). Syntax similarity has a noticeable impact.

If you analyze a codebase using several languages, don't expect that cross-language duplicates will be reported, unless this is the same implementation. For example, TypeScript code and its variant transpiled to JavaScript should be easily detected (high/transpiled).

Renamed literals

Different literals like returned texts, error messages, string variables, numbers have a noticeable impact. However, this is a borderline case: it lies around the typical threshold, and there are large differences between models.

Renamed variable names

A copy of code with renamed variables or other identifiers to something unrelated won't be detected as a duplicate.

Vector size

Models were evaluated using the default (usually maximum) vector size. Some of them have an additional variant with vector size reduced to 256 dimensions.

The results show that vector size reduction has no impact on quality, and the difference is in a range of noise. Often, a small vector gives even slightly better results.

There is no trade-off here. A 256-dim vector is a clear choice. It improves duplication detection tool performance, where calculating the similarity between all code snippet combinations is CPU and RAM intense on larger codebases.

(Note that jina-embeddings-v2-base-code supports only a single vector size)

Model providers

Voyage AI

Voyage is known for code-specialized embedding models, so they were evaluated more thoroughly. Results surprise in two ways.

Coding models they offer are not suitable for duplication detection. Previous voyage-code-3 and recently released voyage-code-4 perform similarly poorly. voyage-code-4 is optimized for "retrieval required by coding agents" and evaluated for this purpose. They are simply intended for different use cases.

Another surprise is how well their newest general-purpose model performs in this benchmark. voyage-4-large is comparable with code-specialized Jina's models. voyage-4-lite also gives decent results, which are adequately reduced with its size. Previous voyage-3-large is a completely different story.

Jina AI

Jina's code-specialized models perform best, so they were broadly evaluated, including variants.

jina-code-embeddings comes with two sizes: 0.5b and 1.5b. There is not much difference between them. The smaller one is slightly more suitable based on this benchmark.

Since jina-code-embeddings-0.5b with a 256-dim vector is the best combination, I tried to push it further and tested different task types than default. Two variants are included:

• Task code2code.passage, jina-code-embeddings-0.5b@256|c2cP in data files. It improves results slightly in a noise range.
• Task code2code.query, jina-code-embeddings-0.5b@256|c2cQ in data files. It gives noticeable improvement in benchmark data, making it a winning combination.

jina-embeddings-v2-base-code is an older and small model suitable for running locally, also on CPU. In terms of duplicate detection, it achieves surprisingly good results. The variant jina-embeddings-v2-base-code@768|local is for a local model run with Ollama, the other one is for the API. I included both to confirm that local gives the same results.

Looking at Voyage's results, we may wonder if Jina's general-purpose model would perform better. I checked this, and the jina-embeddings-v4 would be the weakest model in ranking.

Google Gemini

Google's newest gemini-embedding-2 model is the worst performing one in this benchmark. They claim improvement in the MTEB (Code) benchmark, but this may be a similar case like with Voyage AI.

This model supports task types, and the sentence similarity task with the use case "Semantic similarity" gave hope. This variant is included in data with the gemini-embedding-2@3072|sim label, and it sets a new bottom in the ranking. I also tried the code retrieval task, but there was not much difference (it's not included).

Cohere

Cohere's embed-v4.0 model was evaluated with task type clustering as most relevant for similarity check. It's the next provider after voyage-4-large in the main ranking, so you may wonder if it's an alternative. In code understanding metrics, it's near the bottom, so the two aren't really comparable.

Choosing best model

Provider's API

Choose the company you prefer: Jina AI or Voyage AI. They differ in terms of billing, pricing, account handling, rate limits, performance.

Jina's models give better results in this benchmark, but it doesn't justify sticking to a provider you don't like or have problems with. It's better to have an alternative, and voyage-4-large is a good one.

For Jina's API, jina-code-embeddings-0.5b is preferred over jina-code-embeddings-1.5b also because of service availability. The status page reports problems with 1.5b, while 0.5b has no downtimes.

If both providers suit you, and you plan to use embedding-based duplicate detection extensively in your work, I recommend comparing them. Their models prioritize different kinds of similarities, so one of them may be a better fit for your codebase and objectives, regardless of which one performs better in the benchmark.

Self-hosted solutions

If you prefer or need to rely on self-hosted models, Jina gives a few options.

• The jina-embeddings-v2-base-code is a lightweight model on an Apache-2.0 license, giving good results.
• The jina-code-embeddings-0.5b and 1.5b models can be downloaded but are licensed on CC BY-NC (non-commercial use), excluding them from local model recommendations. However, the Elastic license should allow commercial use.
• All models are available on cloud provider's marketplaces and can be used without restrictions.

(Information based on the "CC BY-NC License Self-Check" section on the Jina website)