Hermes Security plugin separates code evidence from test proof

Nous Research announced an official Security plugin for Hermes Agent on November 8 (opens in new tab), bringing repository review to chat, the command line and Hermes Desktop. It maps what an attacker can reach, divides code among parallel reviewers, and checks candidate bugs against their surrounding implementation. Users can separately allow small reproductions to run in a temporary repository copy.

The consequential distinction is in the report: a finding supported by source code is different from one confirmed by an executed test. Rejected candidates and inconclusive investigations remain visible, alongside code the reviewers did not finish reading. That gives developers something more useful to inspect than a severity label alone. It does not establish that the model reliably finds security bugs, and the repository's own evaluation records show why that question remains open.

The package is hermes-security, version 0.1.1 (opens in new tab). The November 8 post is its announcement, not its first appearance in source: the official repository history already includes the plugin before that date (opens in new tab).

Follow the attacker, then divide the code

The review begins with entry points and trust boundaries: which inputs come from an attacker, which identities the application trusts, and where those inputs acquire authority. Folder-level reviewers inspect the code while a cross-scope reviewer looks for connections that a local reading could miss. Candidate findings then receive further source review and attack-path analysis, with test confirmation available when the user permits execution. The published workflow (opens in new tab) centers that reachability question rather than stopping at a suspicious operation.

Review scope needs attention. Standard scans focus on production source, configuration, CI, infrastructure definitions and migrations. They normally exclude tests, documentation, examples, generated files and other non-production material, with an exception for committed secrets. The standard planner (opens in new tab) sizes work at roughly 4,000 lines per packet and budgets 90 area packets. Larger repositories are prioritized by risk signals; unfinished work can be reassigned, but remaining gaps stay in the report.

Deep scans widen the review beyond those standard exclusions and default to three independent passes. They still have stopping conditions and a budget, not a guarantee of exhaustive analysis. The deep-scan implementation (opens in new tab) also permits stopping after two consecutive passes find no new root cause. “No findings” therefore needs to be read together with the coverage record.

What a finding actually proves

The plugin's evidence labels include “Needs review,” “Supported by code,” “Confirmed by test,” “Not an issue” and “Couldn't confirm.” Severity is a separate field. A high-impact hypothesis can remain unconfirmed; a precise source citation does not automatically demonstrate runtime exploitability. These are explicitly separate states in the implementation (opens in new tab).

There is a concrete check behind the citations. When a worker submits a code-review candidate, the submission boundary (opens in new tab) checks that cited paths and line ranges belong to the inventory and that excerpts match the recorded source snapshot. Workers cannot supply their own runtime-success authority. This helps catch invented evidence, although a genuine excerpt can still support a mistaken interpretation.

Consider a hypothetical tenant-authorization review. An invoice endpoint looks up a record using a request-supplied identifier. A reviewer should trace the authenticated identity, the lookup and any tenant check before concluding that another customer's invoice is exposed. A shared helper might enforce the restriction elsewhere. If so, rejecting the candidate with that counter-evidence is useful work, not a result to hide.

If the source supports the missing-boundary claim, it can become “Supported by code.” With testing authorized, a reproduction could compare the suspected unauthorized access with a matching control where the boundary is enforced. The receipt logic (opens in new tab) requires completed positive and negative controls, the expected difference between them, and cleanup before granting runtime-confirmed status. That records a successful paired reproduction under the submitted plan; it does not independently establish that the plan proves the alleged vulnerability. A proposed test that never executes remains NOT_RUN, which proves neither safety nor exploitability.

A temporary copy is not a sandbox

Test confirmation is optional because it crosses a different trust boundary. The local runner (opens in new tab) copies the repository and runs commands as the user's account. It removes environment variables whose names look like credentials, retaining the normal home directory and toolchain environment. Resource limits and available network isolation do not turn that into a hostile-code filesystem sandbox.

For an unfamiliar repository, permission to read code should not be confused with permission to execute it. The plugin's README makes that warning explicit: only allow tests on code you trust. Filtering variables named like tokens or passwords does not remove every way a process running as you can access sensitive material.

Testing a running application requires a separate user-issued grant. The documented default covers one exact origin for one hour and up to 200 requests; the security toolset does not expose a grant-creation tool. The user-command parser (opens in new tab) exposes the grant, while the runner restricts redirects to the same origin and charges each hop against the request budget. The app-testing panel's Stop control (opens in new tab) revokes that grant early. This is an explicit authorization step, not a general license to probe a deployment.

The report's suggested patches are previews, not automatic edits. A separately bundled fix-finding workflow (opens in new tab) requires an explicit request to fix a selected finding. Asking for an audit does not authorize remediation.

The recorded evaluations leave a substantial gap

The repository publishes two different kinds of evidence. Its deterministic tests exercise boundaries such as citation rejection, report completeness and forged runtime claims. Those checks can establish behavior for the tested cases. They do not measure whether a model discovers real vulnerabilities. The evaluation document (opens in new tab) expressly separates them.

The saved deterministic output (opens in new tab) also uses obsolete gate names and a test selector absent from the pinned tree. Its green results do not establish that the current evaluation harness (opens in new tab) passed unchanged.

Its recorded model work is less reassuring. Five development runs used 26 public synthetic vulnerable/fixed cases. Across the runs, reported precision ranged from 0.818 to 0.917 and recall from 0.692 to 0.846. The first run matched nine of 11 reports and found nine of 13 expected roots. Exact severity agreement was zero in every run, as was completed-case coverage: all 130 scans were sealed but partial. The archived scorer output (opens in new tab) records the failed overall quality gates.

Those results are not a production detection rate. The evaluation describes development work beginning October 5, before the plugin's November addition to the official repository, rather than a fresh test of this announced package. It also identifies fixture defects, exact-anchor matching disagreements and a mismatch between expected severity and the review policy. Partial coverage reflected unfinished standards-review requirements, not an absence of all code review. Those complications limit what the failures say about deployment performance, but they do not turn the gates into passes.

High diagnostic agreement on the partial results likewise does not substitute for passing the official agreement gates, which scored zero because the cases lacked completed runs. The fixtures were not a blind holdout. More importantly for reproducibility, the evaluation says the sealed development bundles were pruned from scratch storage, so the retained score files can no longer be re-scored against those bundles. A genuinely independent holdout and human root-cause adjudication remain necessary before drawing broad quality conclusions.

Local reports still have external data flows

Scans and reports live in the active Hermes profile, with Markdown, JSON, CSV and SARIF exports. Local storage does not mean local analysis: the configured model provider receives the code it reviews. Teams handling proprietary code need to evaluate that provider relationship before sending a repository through the workflow. The plugin's data disclosure (opens in new tab) states the distinction directly.

The default dependency check adds other network activity. It downloads a pinned OSV-Scanner release, checks its SHA-256, and sends package names and versions to advisory services, including api.osv.dev and api.deps.dev. Runtime lifecycle checks use endoflife.date. The dependency-check implementation (opens in new tab) supports the HERMES_SECURITY_DEPENDENCY_CHECK=off opt-out. This is not a fully offline workflow merely because the Python package has no runtime Python dependencies.

Dependency alerts also deserve a different reading from traced code bugs. The dependency-finding implementation (opens in new tab) can group known advisories and end-of-life runtimes into one source-supported, low-confidence finding while explicitly leaving affected-function reachability untraced. That is a material exception to the README's claim that an advisory alone does not produce a finding. A version match is evidence of dependency risk, not proof that an attacker can reach the vulnerable behavior in this application.

What developers get now

The Apache-2.0 plugin requires Hermes 0.21.5 or later and Python 3.11 or later. Its installation documentation (opens in new tab) calls for a restart after installation. The Desktop page starts disabled and must be enabled under Capabilities → Plugins; remote Desktop connections need installation through Desktop's dialog to fetch the page as well.

The report's OWASP and ASVS mappings organize the review; they are not certification.

For a team assessing the plugin, the sensible starting point is a repository it can share with the configured provider, with test execution disabled until the code is trusted. Then inspect the evidence and coverage before acting on the severity list. The report structure makes those distinctions available. The next evidence the product needs is a reproducible discovery evaluation that shows how reliably the reviewer fills them in.