Arvados 2.1.0 Release Notes

October 13, 2020

The Arvados team is pleased to announce Arvados v2.1.0. This is a major upgrade, with many new features. Some older components have also been removed, superseded by newer components. Please also review the upgrade notes.

A complete list of issues resolved in this release can be found on the developer site.

Major Features

Support for CWL 1.2

This release adds support for v1.2 of the Common Workflow Language standard, which adds conditional execution of workflow steps to the language among other features. Details can be found in the CWL v1.2 Changelog.

Faster permission updates

Adding or removing permissions now only updates users and groups that are directly affected, instead recomputing the permission table from scratch. For large databases, this dramatically improves the performance of updates such as adding a new project.

S3 Compatible API for accessing files in Keep

Alongside the existing WebDAV support, Keep-Web now provides an S3-compatible API for accessing files in Keep over HTTP.

New S3 driver for Keep

Historically, Arvados Keep has used the goamz driver to talk to S3-compatible services. This release adds a new driver based on aws-sdk-go-v2. The new driver can improve read performance by 50-100% but it has not had as much production use. See the S3 backend documentation for information on how to enable it.

Many new Prometheus metrics for the Arvados cloud dispatcher

Arvados-dispatch-cloud, the component that handles dynamic scaling of compute in cloud environments, now sports a number of new Prometheus metrics. The new metrics allow insight in container wait times, slowdowns due to cloud account quota, node boot outcomes, how long it takes for a compute node to become available, and more.

Workbench 2 support for large collections

Using Workbench 2, users can now browse collections of up to 50,000 files without significant performance degradation.

SaltStack formulas for installing/managing Arvados

The Arvados team is now publishing the SaltStack formulas that we use to install and manage our own Arvados clusters in order to make deployment of Arvados easier. SaltStack is a Python-based, open-source devops tool for event-driven IT automation, remote task execution, and configuration management, similar to Ansible or Puppet.

Support for OpenID Connect

Arvados now supports using the OpenID Connect protocol to authenticate users.

Expanded PAM support

Arvados now supports the use Portable Authentication Modules (PAM) in two ways:

For users of shell nodes, PAM makes it possible to log in with an Arvados username and use an Arvados token as a password.
For log in to Arvados Workbench, use PAM to validate the username and password (by default, local Unix accounts on the API server).

Webshell

Arv-copy improvements

The arv-copy tool can now copy projects and the collections, workflows, and subprojects they contain. arv-copy now also infers the source cluster from the uuid, and uses the current Arvados environment as the default destination cluster.

Other changes

The “devise-omniauth” SSO server is no longer supported

Authentication methods have moved into the Arvados Controller component. Arvados installations using the “devise-omniauth” SSO server will continue to work, but cluster admins should plan to migrate to one of the built-in authentication methods as soon as possible. See setting up web based login for details.

The “devise-omniauth” SSO server authentication flow is deprecated and will be removed in the next major release.

The old PAM driver written in Python has been removed

It has been replaced by a new driver written in Go. The distribution package name for the new driver is libpam-arvados-go. When upgrading, the PAM configuration will need to be tweaked, see the documentation for details.

NodeManager is no longer supported or packaged

NodeManager has been removed entirely from this release, having been superseded by arvados-dispatch-cloud since the Arvados 2.0 release.

Python 2.7 is no longer supported

Minimum supported Python version is now 3.5. We no longer publish Python 2 based distribution packages for our Python components. There are equivalent packages based on Python 3, but their names are slightly different. If you were using the Python 2 based packages, you can install the Python 3 based package for a drop in replacement:

apt remove python-arvados-fuse && apt install python3-arvados-fuse
apt remove python-arvados-python-client && apt install python3-arvados-python-client
apt remove python-arvados-cwl-runner && apt install python3-arvados-cwl-runner
apt remove python-crunchstat-summary && apt install python3-crunchstat-summary
apt remove python-cwltest && apt install python3-cwltest

Ruby 2.5 or higher required

The minimum supported Ruby version is now 2.5.

Debian 9 (“stretch”) and Ubuntu 16.04 (“xenial”) support deprecated

The Arvados 2.1 release series will be the last with support for Debian 9 and Ubuntu 16.04.