Arvados 2.6.3 Release Notes

June 8, 2023

The Arvados team is pleased to announce Arvados 2.6.3. This release includes a number of important bugfixes and performance improvements, particularly when Crunch is running a large number of containers. We recommend that new and existing installations of 2.6.2 or earlier upgrade to 2.6.3. See Upgrading Arvados for upgrade instructions. For a full list of features and improvements introduced in the 2.6 Arvados release, see the 2.6.0 release notes.


crunch-run now retries all of its Arvados API calls, waiting up to ten minutes in the worst case, to provide improved resiliency when the API server is busy or unresponsive. #20540

All Arvados API calls in crunch-run now select only the fields they need, improving performance. #20541

crunch-run logs more progress information as it uploads output files to Keep. This is intended to help alleviate concerns that a container is hung after the main process has finished, and help identify cases where containers upload more than intended. #20561

The Docker watchdog timeouts in crunch-run have been increased to accommodate specific situations where the watchdog command itself becomes unresponsive. #20595

Cloud Dispatcher

arvados-dispatch-cloud supports a new configuration setting Containers.CloudVMs.InstanceInitCommand. Administrators can write custom shell in this setting, and it will run after Crunch’s own dispatch script. #20520

arvados-dispatch-cloud supports a new configuration flag Containers.CloudVMs.DeployPublicKey to decide whether or not it authorizes its own SSH key for newly created compute nodes. You can set this flag to false if you provision the dispatcher’s SSH key another way, like preinstalling it on the compute node image. #20485

arvados-dispatch-cloud no longer terminates instances or unlocks containers when it reaches its internal request concurrency limit. #20511

arvados-dispatch-cloud no longer terminates instances when there are more supervisor containers running than the cluster’s configured SupervisorFraction allows. #20511

Fixed a bug in arvados-dispatch-cloud where supervisor containers in the Running state were not correctly counted against the cluster’s configured SupervisorFraction. #20511


Python SDK client constructors like arvados.api() now accept a num_retries argument that sets a floor for for all Arvados API requests made with this client. This lets clients define their preferred retry limit once, rather than passing it to each API call. #12684

Related to that change, client constructors will ensure that logs for googleapiclient.http appear when you first fetch the discovery document to let you know about early problems that are being retried. If you want to handle logging yourself, just ensure googleapiclient.http or a parent logger has a log handler installed before you construct Arvados API clients. #20613

When you issue an Arvados API request through the Go SDK, and do not save the result, the SDK automatically adds the select argument to your request set to ["uuid"]. This minimizes the size of the API response, improving performance. #20541

There are a number of changes to the way arvados.Client in the Go SDK retries requests (#20511):

  • The client now starts with a limit of 8 concurrent requests that grows as those requests succeed, rather than starting with unlimited concurrency that gets tamped down on failure.
  • When used by an internal service, the maximum request concurrency is 25% of the cluster’s configured MaxConcurrentRequests.
  • Fixed a bug where the concurrency limit was increased more than intended after a successful request.

Workbench 2

The browser title is now updated to reflect the current navigation path. #19369

The “Subprocesses” and “All Processes” lists will no longer refresh continuously in the background. #20449

Certain permission denied errors are now handled correctly so they no longer crash the application. #20538

CLI tools

When a container fails, arvados-cwl-runner selects different logs to report based on whether the container failed to start, reported its own failure, or was terminated by a signal. The wording of some log messages has been clarified as well. All these changes are intended to help users diagnose problems faster. #20531

The default number of retries for Arvados API requests in all Python tools has been increased to 10, to provide improved resiliency when the API server is unresponsive. In the worst case, a single API request that keeps failing may be retried for about 35 minutes. #12684


The Arvados API controller now drops the If-None-Match header from requests forwarded to the Rails API server. Arvados does not support this header, but a security vulnerability was recently discovered in Rails’ support for it (CVE-2023-22795). Dropping the header defends against exploits without affecting users. #20545


A new page in the Administration Guide documents the /_inspect/requests API endpoint of many Arvados services. #20229

Administrators can set Containers.Logging.LimitLogBytesPerJob to zero in their configuration to disable realtime container logging. This has long been supported, and is now documented in the configuration reference. #20433