Knowledge is important to HPC, and ensuring your simulations have the information they want, once they want it, is vital. Nevertheless, knowledge can come from many sources and should be consumed by varied assets. Being able so as to add extra and several types of storage choices to your cluster makes this knowledge extra available on your work.

Since its launch, Amazon FSx has aimed to give you extra choices to launch, run, and scale feature-rich, cost-effective storage powered by your alternative of file programs. AWS ParallelCluster helps by permitting integration with these current file system decisions, giving you a similar flexibility so you possibly can higher design your HPC storage.

AWS ParallelCluster model 3.2 introduces assist for 2 new Amazon FSx file system varieties (Amazon FSx for NetApp ONTAP and Amazon FSx for OpenZFS). It additionally lifts the restrict on the variety of Amazon FSx and Amazon EFS file system mounts you possibly can have in your cluster.

By rising file system entry choices, your HPC workloads on AWS could have extra routes to entry the information they want with out you having to do the onerous work. In right this moment’s article we are going to clarify this to you intimately.

What’s new?

New file programs

ParallelCluster already helps Amazon Elastic File System (EFS), Amazon Elastic Block Retailer (EBS), and Amazon FSx for Luster. On this launch, we added assist for FSx file programs for NetApp ONTAP and FSx for OpenZFS.

Various kinds of file programs have particular traits that make them extra appropriate for various knowledge varieties and workflows. For instance, Amazon FSx for OpenZFS is easy and highly effective shared file storage primarily based on OpenZFS and provides ultra-high velocity at low price. You have most likely used OpenZFS for its effectivity and efficiency options like copy-on-write that allows on the spot snapshots, built-in knowledge resiliency, and its adaptive alternative cache, all constructed into the file system. You now have the selection to make use of the file system that most closely fits your wants, with out worrying about incompatibilities in ParallelCluster.

Extra file programs

Previous to this launch, ParallelCluster might solely assist mounting one in every of every filesystem kind (for instance, an EFS mount and an FSx for Luster). This pressured you to consolidate your knowledge storage, requiring you to additional plan your general HPC storage configurations, as a result of these restricted attachment factors.

With this ParallelCluster 3.2, now you can mount as much as 20 Amazon FSx file programs and as much as 20 Amazon EFS file programs. These mounts are for present file programs, the place your knowledge already exists. They aren’t managed by ParallelCluster, so no knowledge motion is required. This additionally signifies that they persist once you delete your cluster, providing you with extra management over decoupling the cluster infrastructure from the information. Person guides for every file system kind doc creation and administration greatest practices (FSx for Luster, FSx for NetApp ONTAP, and FSx for OpenZFS).

Collectively, these two options add a big degree of flexibility for storage options inside ParallelCluster.

Utilizing these new file programs

The cluster configuration YAML syntax for multi-filesystem mounts has not modified with this new launch – you possibly can merely specify extra entries within the SharedStorage part of the configuration file. ParallelCluster will create the cluster with the required storage mounted and able to use. Right here is an instance storage configuration with the newly added entries for FSx for NetApp ONTAP and FSx for OpenZFS…

Learn the complete weblog to be taught extra. Reminder: You may be taught rather a lot from AWS HPC engineers by subscribing to and following the AWS HPC weblog channel.

Learn the complete weblog to be taught extra. Reminder: You may be taught rather a lot from AWS HPC engineers by subscribing to the HPC Tech Brief YouTube channel and following the AWS HPC Weblog channel.

Supply :

Leave A Reply