Arvados 2.3.0 Release Notes

October 27, 2021

The Arvados team is pleased to announce Arvados 2.3.0. This is a major upgrade, with many new features as well as bugfixes. We recommend that new and existing installations of 2.2.2 or earlier upgrade to 2.3.0. See Upgrading Arvados for upgrade instructions.

Major Features

Initial LSF support

This new release adds support for dispatching containers to Spectrum LSF clusters, allowing the integration of Arvados to additional HPC environments.

For more information please check our documentation page.

Initial Singularity support

This major release adds initial support for running containers with the Singularity Runtime.

We’ve tested this version against Singularity 3.7.4. For more information please check the documentation.

Improved Storage Classes support

This release features expanded support for storage classes. Storage classes enable clients to control which storage backend volumes are used to store data associated with each collection. This makes it possible to manage heterogenous storage with different performance/price/capacity/robustness characteristics.

Arvados applications now support uploading data directly to specific storage classes. This is described in the the user guide. If a storage class is not specified, it will use the default storage classes assigned by the administrator.

Keepstore services now parallelize data write operations when a data block is assigned to multiple storage classes. The administrator can also prioritize which storage classes are preferred for read operations. Please refer to the admin documentation for more information.

Improvements

Workbench2 Webshell

Workbench2 now integrates the browser-based shell node access on its own instead of redirecting to the older Workbench1. This includes support for session auto-logout.

Workbench2 collection’s file browser

The new collection file browser allows navigating through more significant collections faster by loading partial file listings on demand instead of requesting the entire collection at once. This is most noticeable on collections with many directories and/or files.

Group-sync tool improvements

This release’s group syncing tool arvados-sync-groups allows case insensitive username matching as described on the admin guide.

Collection manifest handling

Collection manifest signing performance and caching are notably improved. This will be most noticeable when dealing with really big data collections.

Other changes

Workbench2 fixes

The pagination display now works correctly when showing empty tables. Upload speed indication is now fixed showing MB/s instead of KB/s.

Data access fixes

WebDAV caching fix improves performance on huge collections. Fixed occasional mount point deadlock bugs in the PySDK FUSE driver (arv-mount).

Deprecated database indexes

Removed deprecated full text search database indexes. These were unused and sometimes interfered with creating collections with very large numbers of files and directories.

Error reporting fixes

Request-ID propagation fixes improve error traceability and debugging. Keepproxy spurious 413 error fix avoids future debugging confusion.

Python SDK fixes

Fixed the socket read timeout issues on collection create calls. Upgraded python-rsa dependency to address the security advisory CVE-2020-13757. Upgraded ruamel.yaml & google-api-python-client dependencies that were causing issues to some users.

Misc fixes

Cached user records handling in a satellite cluster auto-resolves username collisions. Removed unused users#update_uuid endpoint. Fixed container locking issue & improved logging on dispatcher.