Research IT

Savio Cluster: Login/compute nodes and Slurm issue

Over the past few days, we’ve seen intermittent issues where SSH connections to the login nodes have hung or been unusually slow. We have been working to mitigate the issue, and we tested a newer kernel version. That change introduced MUNGE/Slurm authentication incompatibilities, leading to job submission failures and error messages when running Slurm commands. We are reverting to the previous kernel, and job submission and login node stability should be restored by this afternoon. As a longer-term improvement, we are adding an additional login node to the current login pool, with the goal of making it available by early next week. We plan to continue expanding the login pool. We also acknowledge ongoing issues with jobs hanging on compute nodes and with Open OnDemand sessions failing or hanging. Our team is working with VAST support to identify the root cause and resolve the problem as quickly as possible.