Qualified Free Access to Advanced Compute Resources with NSF's Jetstream2 and ACCESS

• 08/12/24

Hancock: Absolutely. I think that is one of our key missions, and our calling to help with — including the workforce development aspects of that whole picture.

This is already a very active area beyond our efforts. There are specific calls and other resources, such as the National AI Research Resource call. More specifically, NSF has put out a "dear colleague" letter that invites people to apply for modest funding to support some of this education and workforce development.

And we are here not only to support individual researchers and educators, but also to help the operators that may want to provide similar infrastructure at their institutions.

Grush: How could advances in cyberinfrastructure affect the future of how Jetstream2 provides access to resources for those who need them?

Hancock: Great question. This is something quite unique about the Jetstream2 project compared with other supercomputing or cluster deployments: Often times you'll see a project deployed well at the outset, but that's the way it is, ongoing, with only some minor revisions, for many years.

Whereas with Jetstream and Jetstream2, we specifically and purposefully embrace the changeable nature of piloting new services. All the software environments and interfaces that are on top of a system and updated as the community needs have been adapted since we first started Jetstream.

With Jetstream and Jetstream2, we specifically and purposefully embrace the changeable nature of piloting new services.

For example, we didn't think there was going to be a large demand for API interfaces or for jumping into the back end of cloud computing. But we found that about half of the people coming to the system were interested in that sort of access, so we enabled that very quickly.

We've also partnered with other projects to provide new interfaces. For example, we were early in deploying the ability for people to use Jupyter notebooks — and now JupyterHubs — with orchestrated environments.

And in gesturing today through interfaces like Exosphere as well as CACAO, these are tools that can be orchestrated with larger deployments of resources, whether that's dozens of virtual machines or deploying through Kubernetes or other specific technologies.

It wasn't clear when we started the Jetstream project what technologies would take off. Docker Swarm was probably the most popular at the time in addition to Apache Mesos. And now, Kubernetes and containers are everywhere. So, we've been able to meet all of that in terms of thinking ahead to what's next.

Of course, we don't very often really know what's next. This is where the folks in the classroom or lab may come to us with ideas or challenges that they have. And that's why we have advisory boards with our partners, and forms and surveys to watch for what the next pilot needs may be. And we watch for what we can do to continue to develop and make that process easier. We'll continue to push to evolve these types of things, because the demand to evolve is our hope.

We have advisory boards with our partners, and forms and surveys to watch for what the next pilot needs may be… The demand to evolve is our hope.

Grush: What makes that kind of evolution possible for a project like this? What's the "secret sauce" for Jetstream2's success in this evolution, and its success in general?

Hancock: It's because of a great team and project partners. We have partners on this project from the University of Hawaii, University of Arizona, Arizona State University, University of Texas/Texas Advanced Computing Center, Cornell, the San Diego Supercomputer Center, and of course, Indiana University. Such partners and advisors really make these large-scale projects work, and I think that together we can all truly magnify the capabilities of any one institution.

[Editor's note: Jetstream2 logo and image courtesy of Jetstream2 and Indiana University. With permission.]