Arvados 3.2.0 Release Notes
November 3, 2025
The Arvados team is pleased to announce Arvados 3.2.0. This release introduces support for web services in containers, external service credentials, and collection downloads as ZIP files, as well as a new Workbench dashboard and bug fixes throughout the platform. We recommend that existing installations of 3.1.2 or earlier upgrade to 3.2.0. See Upgrading Arvados for instructions.
This release of Arvados is dedicated to Peter Amstutz. Peter led the charge to develop many of the headline features in this release and organized much of the implementation work. His influence on the Arvados architecture and CWL standard will be felt for a long time to come. We miss you Peter.
Release Highlights
Service Containers
Arvados 3.2.0 introduces service containers. Users can describe web services that run from their CWL workflow steps or API container requests. Arvados will let users connect to those web services while the container is running. This can be useful to run interactive analysis on data in Keep or connect to diagnostics for a long-running analysis step.
Workbench lets you connect to these services right from the top of the process page. Custom clients can also get the service URLs from the API. There are options to control who can access the services.
Our user guide has an end-to-end demonstration of how to use service containers. Service containers require the cluster administrator to do some setup described in the upgrade notes.
#23051, #23026, #23044, #22859, #22760, #22777, #22996, #22581, #22613, #23023, #23215
External Service Credentials
Arvados 3.2.0 introduces a credentials store. Credentials can store arbitrary secrets for resources and APIs outside Arvados and use them to access data for compute on demand.
We are using this to make it easier to analyze data in S3. When you submit a workflow to Arvados that uses s3 URLs in the inputs, the workflow runner will automatically search Arvados’ credentials for an access token to use to download that data to Keep on your behalf. This should eliminate the need to manually transfer data from S3 to Keep in most cases. Our user guide explains how to set up and use S3 access tokens.
We’ve built the credentials API to be extensible and avoid leaking secrets. You are welcome to use it in your own Arvados tooling. We’d love to hear your ideas for other kinds of credentials we can officially support.
#20650, #22680, #22819, #22820, #23181, #23210, #23211
Collection ZIP File Download
Keep lets you download multiple files or all the files from one collection as a single ZIP file. This streamlines your ability to download and analyze an entire set of outputs or logs. You can get a ZIP file from the collection file view in Workbench or from the WebDAV API.
New Workbench Navigation
When you log into Workbench, you’ll be greeted with our new dashboard. This puts your favorites, recently viewed projects, and recent workflow runs front and center, giving you instant access to the projects you’re working on right now.
We’ve also streamlined navigation by giving collections, processes, and workflows a tab-based view just like projects have.
#22959, #23177, #22793, #23063, #23080
Deployment Changes
Arvados 3.2.0 introduces support for installation on AlmaLinux, Rocky Linux, and RHEL 9. #21362
For clusters installed on AlmaLinux, Rocky Linux, or RHEL 8, Arvados 3.2.0 requires packages from the 8.8 release. Make sure your servers have at least this version before you upgrade Arvados. #23096
Arvados 3.2 no longer supports Debian 11 “bullseye” or Ubuntu 20.04 “focal.” Arvados 3.1 supports Debian 12 “bookworm” and Ubuntu 22.04 “jammy.” You can upgrade your Arvados cluster to one of those releases, then proceed to upgrade Arvados. #22864
Other Improvements
Workbench
Users can upload entire folders to collections. The folder’s internal structure is mirrored in the collection. #19378
Users may choose the “default” project tab in their preferences. This tab is shown first when opening any project. #22394
Multiselect checkboxes in data tables have smarter and more consistent behavior throughout Workbench. #21746, #23154, #23114
“Copy UUID” is available in the action toolbar and context menu for all kinds of resources. #23208
Cancelling a process using the main cancel button or context menu requires confirmation to prevent accidental cancellations. #23161
The action toolbar and context menus for Groups are more consistent with other pages. The Edit Group dialog works as it should. #22813, #22814
The workflow progress bar was hard to follow and has been replaced. Open the “Status” filters on the Subprocesses tab to to limit what’s listed and see a count of subprocesses by status. #22845
Improved the performance of the All Processes page when it shows many running containers.
#23117, #23083
The All Processes page more consistently fits in the browser window. #22860
When viewing container logs, if you disabled word wrap, the toolbar would move to the end of the log viewport and usually out of view. Now it stays put. #23039
Skeleton-style placeholders indicate when data tables are loading. #22360
Fixed several bugs in the collection file rename dialog. #23180
Breadcrumbs are accurate after trashing an item. #22618
Filtering a project with items selected resets item selection instead of entering an inconsistent state. #23009
Fixed a bug where closing the details side panel interfered with the user’s selected item(s). #22787
Empty dispatch messages are filtered from the process logs panel. #22784
Improved the consistency of workflow input alignment and spacing. #22936
Pagination properly resets when navigating across different pages. #23194, #23060
Workbench screenshots throughout the documentation have been modernized. #22464
Workflows and Compute
Arvados supports expressions in the WorkReuse requirement of CWL workflows. #23227
The SLURM dispatcher supports GPU-enabled containers. #23076
Fixed a race condition that could cause FUSE directory listings to become inconsistent over time. This could cause containers to fail when they mounted data from Keep into the output directory. #23136
Fixed a bug where containers would not be able to connect to Keep when a compute node’s external interface address was in the CGNAT range 100.64.0.0/10. #23174
Fixed a file descriptor leak in crunch-run that could cause containers to fail with a “too many open files” error if a Workbench tab was open and following log messages for a long time. #22931
Fixed a memory leak in crunch-run that could cause containers to fail when copying outputs to Keep. Improved performance for containers that have many mounts inside output_path that refer to a single collection. #22827
The cloud dispatcher disables the retry logic in AWS SDK v2 in favor of its own. #23178
The cloud dispatcher correctly distinguishes whether an “insufficient capacity for node type X” cloud provider error applies to a preemptible or non-preemptible instance. #23196
The cloud dispatcher supports connecting to compute nodes using ED25519 SSH keys. #22935
SDKs and Developer Tools
We’ve added a contrib directory to the Arvados source code. This contains components that the Arvados team helps maintain but are not part of “core” Arvados. The Java and R SDKs have moved here, along with the arvbash shell tools. #23064
The Java SDK uses keep-web for all data transfer to better support Keep’s full set of features. The original support for the Keep API has been removed from the SDK. Thanks to Krzysztof Majewski for the contribution. #23077, GitHub PR #298
API Updates
Stricter validations are applied to several resource fields. For example, the API will reject a container request with misspelled keys in mounts, or environment variable names containing NUL or =. This should help users shorten debugging cycles with clearer error messages. #23152
Improved storage efficiency for files uploaded via WebDAV. Previously, uploading many small files via WebDAV would result in a large manifest referencing many small blocks. Now smaller data blocks are merged while the upload proceeds to optimize later collection access. #22321
Uploading multiple files concurrently to a single collection via keep-web’s S3-compatible API is race-safe. #22791
Fixed a race condition that could cause keep-web to decline some collection updates with 409 Conflict. #22824
Fixed a file descriptor leak in keep-web when another process is using the same block cache directory. Fixed a bug that caused keep-web to fail a request and deadlock subsequent requests when reaching its limit of open cache files. #23078
Fixed a crash in the websocket server when the properties attribute of a log entry was empty or invalid. #23132
In the future we plan to make API workflow records a thin wrapper around collections. Arvados 3.2.0 adds some fields to workflows to support this, but as documented, they are not usable yet. #21074, #23062
Cluster Administration
The workflow that reports cluster activity is easier to use: it comes with full documentation and Docker build instructions. The user documentation recommends running the report this way. #23191, #22216, #23136, #23214
Container shell access for admin users is now enabled by default to make it easier to diagnose workflow issues on new deployments. #22816
The user setup notification email is now disabled by default. This way, the default configuration is to never send email. Administrators can enable options as desired after they confirm the server has a local mail delivery agent configured. #22703
The SLURM dispatcher supports a SbatchGPUArgumentsList setting, analogous to the LSF dispatcher’s BsubGPUArguments. #23110
The SLURM dispatcher SbatchArgumentsList and SbatchGPUArgumentsList settings are templates like the analogous LSF settings. Refer to the upgrade notes if you customized SbatchArgumentsList. #23091
The new contrib/arvados-bootstrap package provides scripts to export Arvados data to disk and import it to a new cluster. The arv-federation-migrate script that used to be included with the Python SDK has been moved into this package. #22868
Arvados 3.2.0 includes an Ansible playbook to install an Arvados cluster. It is not yet suitable for production deployments because it does not set up supporting infrastructure and services most large clusters want. But it is now the preferred way to install a simple single-host cluster for testing and we plan to continue development until it replaces the Salt installer. #22238, #23041, #23027, #22866, #22945, #23040, #23036
arvados-client diagnostics reports the name of each service endpoint it checks. #22781
Removed an erroneous diagnostics error (“unsupported protocol scheme ""”) when the webshell service is not configured. #22828
The Salt installer uses a more reliable and performant approach to update self-signed TLS certificates on a cluster. #23168
The Salt installer configures Loki with roles to write to S3 rather than a configured access key. This improves the security of Loki’s storage. #23160
Fixed a bug that caused the Salt installer to install multiple versions of PostgreSQL in some environments, causing cluster setup to fail. #23010
The Salt installer deploys a new signing key for packages from postgresql.org. #22972
The Salt installer sets up apt package pins to help ensure Arvados stays at the desired version. #22778
The Salt installer configures the pool and queue size for the Rails API Passenger service. #23202
The Salt installer can configure Grafana with SMTP settings to send alerts. #23203
The documentation to set up Google as a login provider has been modernized. Thanks to Zoë Ma for the contribution. #23013
Fixed a bug where the Rails API server package would skip a database setup step in certain circumstances. #22704, #23004
Arvados Development
Arvados 3.2 no longer includes the arvbox Docker image and associated tooling. The arvados-server install subcommand has also been removed from this release. If you were using arvbox in demo mode, consider installing on a Debian-based virtual machine with our single-node Ansible installer. If you were using arvbox or arvados-server install for development, our Hacking Prerequisites wiki describes how to install development dependencies using Ansible. #23012, #22436
Arvados Python packages are built using the build module. #20311, #22691
CWL conformance and integration tests are run through pytest. Dependencies are implemented as test fixtures so they can run without arvbox. #23075, #23006, #23198
Many Docker images that we use for development and testing are built with Ansible to improve consistency across environments. #22957, #22958, #23115
Many Docker images that we use for development and deployment are based on Debian 12. Common build functionality has moved into one shared build script. #23094, #23052, #21390, #21389
We put more guardrails on our database structure file to help ensure it works on all the platforms we support. #22840
Unified Workbench’s item name rendering code. #23179
Smaller test improvements throughout are #23025, #22817, #22666, #23180, and #23241.
Dependency Upgrades
We’ve upgraded our minimum version of many dependencies, including Ansible, Bundler, CUDA, Go, Node, Python, and React. The full list: #23097, #23038, #23033, #22998, #22792, #22928, #23104, #23101, #23098, #22978, #22652, #22974, #22920, #23158, #23145, #23144, #23143, #22849, #22689, #22970, #22922, #23213