Two developers run the same assistant inside the same editor, on the same repository, in the same week, and they reach opposite verdicts on it. The accuracy of the model explains neither verdict.
Trust in an in-IDE assistant is task-shaped. The JetBrains Research review of a systematic review of 90 studies published in Empirical Software Engineering reports that trust falls for high-stakes production work and complex problems, and rises for routine code and proofs of concept, and that GitHub Copilot is the subject of 36 of those 90 studies. A large part of what the industry calls knowledge about AI in the IDE is knowledge about one assistant, in one interaction paradigm, during the period it had the field to itself.
That is why this page is a checklist. A published ranking scores attributes that survived contact with the author's repository. Your repository determines which attributes matter. Comparison pays when the attribute you are comparing is one your team already pays for in time.
Checks to run before the comparison starts
Each statement is either true of your situation or false of it. Two false answers mean the comparison is premature.
- You can name the attribute that costs you time, in one sentence, without naming a product. Build feedback takes four minutes is an attribute. The editor feels old is an impression.
- More than one person's daily work depends on the answer. A single developer changing editor needs no comparison document.
- The decision is reversible inside one sprint. Installing an editor is reversible. A team-wide licence, a rewritten continuous integration job and a rewritten plugin each leave a migration behind them.
- The candidate set contains between two and five products. Above five, you are browsing.
- You have a repository that reproduces the hard part of your work, with the legacy files and the awkward build step left where they are.
- You have one sentence that says what would make you keep what you have. Without that line, every trial confirms the newer tool.
- Someone who did not propose the switch owns the measurement.
- The comparison has a deadline and a decision that follows it.
Checks to run against any published comparison chart
- The chart names the product version and the test date. An IDE is a moving target. A comparison published in 2004 against Visual Studio and Eclipse describes a plugin API both products have since replaced.
- The chart names the host operating system. The EE Times article by Jeff Womble noted in 2004 that Eclipse, built on Java, runs its graphical user interface on many hosts, which mattered because Linux hosts were gaining ground among embedded developers. Host support remains a first-order attribute and most charts still omit it.
- The chart distinguishes the editor from the compiler. In Visual Studio Code, language support, debugging depth and diagnostics arrive through extensions. The review at luisllamas.es describes it as a lightweight editor first, closer to a text editor than a classic IDE, with a debugger it calls "not the best in the world". A chart that scores Visual Studio Code is scoring an extension set.
- The chart gives a measured pass rate with a confidence interval. A single percentage without a spread has no weight.
- The chart names the tasks that ran and the repositories they came from.
- The chart names the tools the underlying evidence covers. The JetBrains Research review reports that 36 of the 90 studies it examined look at one assistant.
- The chart's attributes are ones you can measure in an afternoon with a stopwatch.
- The chart says what it leaves out.
- The chart was published after the last release you plan to use.
- The chart prices the version you would pin. A pinned release with no security updates is a liability with a date on it.
One argument for Eclipse from 2004 turned on which party controls the version a tools vendor provides. Womble noted that a vendor could standardise on a particular Eclipse release and distribute it with its own tools, while the road map of the proprietary alternative stayed closed and its release schedule fell outside any customer's control. That is a versioning argument, and it survives the move to subscription editors and extension marketplaces.
The GeeksforGeeks set of IDEs for programmers gives ten tools three bullet points each: a use case, a feature set and a strength. It measures no value on any axis. That format answers the question of what exists, and it ends there.
More than twenty years ago, someone asked the Coderanch forum whether an IDE comparison chart existed. Axel Janssen answered that the IDEs had too many features for a chart to give a reasonable overview, and that one had to try them. No evidence from the intervening two decades has invalidated that.
One short entry per candidate, by name
- Each candidate requires a value for comparison based on shared attributes. Take each value from the vendor's own documentation and your trial, and keep the wording identical across entries.
- Visual Studio Code: lightweight editor first; language support, debugging depth and diagnostics arrive through extensions; free and cross-platform; the extension set is the unit you trial.
- Visual Studio: full IDE for .NET, C# and C++; heavy install; graphical designer, deployment and packaging tools; Windows-first, with Linux, Android and macOS targets.
- IntelliJ IDEA: IDE for Java and Kotlin with support for Scala and Groovy; completion and refactoring depth are the claim; community and paid editions differ in framework support.
- PyCharm: IDE for Python; Django and Flask integration; built-in test runner and debugger; a narrow language surface by design.
- Eclipse: open-source platform on a Java runtime; host portability across operating systems; a plugin architecture that lets a vendor pin one release; rigid build directory rules that complicate legacy trees.
- NetBeans: open-source and cross-platform; Java, JavaScript and PHP through modules; simple enough for a learner.
- Android Studio: official IDE for native Android work, built on the IntelliJ platform; visual layout editor, emulator and device profiles.
- Cursor: editor built around an in-IDE assistant; the model behind the assistant changes without notice; pin the version you trial and note that version in the sheet.
- Jupyter Notebook: interactive document environment for Python analysis; code, output and prose in one file; no build system and no merge gate.
- Code::Blocks: lightweight open-source IDE for C and C++; multiple compilers; minimal project tooling.
Checks during a trial on your own repository
Run every candidate against the same set, on the same repository, in the same week.
- The trial repository contains the file your team edits most often, taken from production history.
- Every candidate runs the same ten tasks, chosen before the first install.
- You log time to first successful test run, measured from launch to a green test line.
- You log how many diagnostics the editor surfaces that your existing linter misses.
- You log what happens when the tool is wrong. Partial correctness is the common failure: a suggestion plausible in structure and wrong in a detail. The JetBrains Research review reports that verifying suggestions, refining prompts and reworking generated code absorbs a large share of a developer's time.
- You log who can read and merge the shared configuration file. If one person understands it, the trial measures that person.
- You log whether the build runs headless, with no display attached.
- You log install size and process count at idle on the oldest machine on the team.
- You log which version you can pin, and who determines when it moves.
- You log the exit cost, meaning what has to be rewritten if you leave.
The sheet is one file with one column per candidate and one row per check. A candidate that fails a row is out, and the row stays visible in the file so the next comparison inherits it.
Checks for an AI agent evaluated inside the IDE
The IDE-Bench benchmark from AfterQuery ran 15 models across 80 tasks drawn from 8 repositories never published to the internet, spanning C, C++, Java, JavaScript, TypeScript and Python, over 6,000 runs. Its harness gives an agent 17 tools in 5 categories and caps a run at 100 iterations inside a Docker container. IDE-Bench is the first benchmark to correlate agent-reported intent with project-level modifications on uncontaminated code.
Five checks follow from the results.
- Does the tool loop close? On a substantial share of tasks the limiting factor is whether the agent can execute a clean tool loop and converge without drifting. Claude Haiku 4.5, a speed-oriented model, performed nearly as well as Sonnet and better on some tasks.
- Does the first attempt resolve the task? The IDE-Bench results give Claude Sonnet 4.5 a first-attempt resolution rate of 87.50 per cent and DeepSeek V3.2 a rate of 31.25 per cent, with V3.2 reaching 71.25 per cent by the fifth attempt. Where latency or spend caps retries, the first-attempt rate governs selection.
- Can the agent give a reason for each edit? IDE-Bench requires an explanation parameter on every tool call, which is what makes trajectory analysis possible. A harness that logs the call and the reasoning hands you a review artefact.
- Does the harness isolate the run? Each task starts in a fresh container built from the repository's Dockerfile, which is what stops one task's leftovers from flattering the next.
- Does the grader hold back the answer? IDE-Bench removes the reference patch from the environment before deployment and blocks the agent from reading the test files, so the agent reasons from the task description alone.
A public harness exists if you would rather automate the whole comparison: the ide-ai-benchmark project drives Cursor, Windsurf and Visual Studio Code through the same scenarios and generates a cross-IDE comparison report. Read its scenario files before you trust its scores. Its categories are code generation, performance and quality, workflow behaviour, and model capability, and each category weights a different part of your working week.
