Middlebury

Difference between revisions of "High Performance Computing (HPC)"

Line 36: Line 36:
 
*Stata  
 
*Stata  
  
=== Guidelines ===
+
 
 +
= Guidelines =
 +
<div class="page" data-loaded="true" data-page-number="2" style="width: 816px;  height: 1056px"><div class="textLayer" style="width: 816px;  height: 1056px">
 +
== <span style="left: 96px;  top: 782.312px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.824511)">Expectations and Support for Users</span> ==
 +
 
 +
<span style="left: 96px;  top: 801.832px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.810211)">All HPC users will be expected to accept the standard Middlebury Code of Cond</span><span style="left: 611.477px;  top: 801.832px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.806441)">uct relating to </span><span style="left: 96px;  top: 821.352px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.803394)">information and technology as well as a general set of best practices specific to the cluster. </span><span style="left: 96px;  top: 840.872px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.80777)">These will be posted on the HPC wiki page. Additionally, faculty</span><span style="left: 510.664px;  top: 840.872px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.814345)">who have little or no </span><span style="left: 96px;  top: 860.392px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.802291)">experience using a shared computing cluster are strongl</span><span style="left: 459.344px;  top: 860.392px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.806746)">y urged to participate in the periodic </span><span style="left: 96px;  top: 879.912px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.801862)">training sessions offered by ITS staff and HPC</span><span style="left: 388.34368px;  top: 879.9122916666668px;  font-size: 16px;  font-family: sans-serif">-</span><span style="left: 393.242px;  top: 879.912px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.825238)">affiliated faculty.</span>
 +
 
 +
== <span style="left: 96px;  top: 96.2323px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.79626)">Cluster Use Principles:</span> ==
 +
 
 +
<span style="left: 96px;  top: 135.272px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.808918)">The use of the Ada cluster is governed by all the policies that apply to Middlebury’s Information </span><span style="left: 96px;  top: 154.792px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.82345)">Technology (</span><span style="left: 179.125px;  top: 154.792px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.85502)">[http://www.middlebury.edu/about/handbook/policies-for-all/appropriate-use/info-tech http://www.middlebury.edu/about/handbook/policies]</span>[http://www.middlebury.edu/about/handbook/policies-for-all/appropriate-use/info-tech <span style="left: 536.0860799999999px;  top: 154.79229166666664px;  font-size: 16px;  font-family: sans-serif">-</span><span style="left: 540.984px;  top: 154.792px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.859345)">for</span><span style="left: 559.8828799999999px;  top: 154.79229166666664px;  font-size: 16px;  font-family: sans-serif">-</span><span style="left: 564.781px;  top: 154.792px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.838247)">all/appropriate</span><span style="left: 663.1014399999999px;  top: 154.79229166666664px;  font-size: 16px;  font-family: sans-serif">-</span><span style="left: 96px;  top: 174.312px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.851039)">use/info</span><span style="left: 150.20313599999997px;  top: 174.31229166666685px;  font-size: 16px;  font-family: sans-serif">-</span><span style="left: 155.102px;  top: 174.312px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.812782)">tech</span>]<span style="left: 183.594px;  top: 174.312px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.811393)">) and the following principles:</span>
 +
 
 +
#<span style="left: 148.19987199999997px;  top: 213.35229166666682px;  font-size: 16px;  font-family: sans-serif">T</span><span style="left: 155.997px;  top: 213.352px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.806925)">he Ada cluster supports the researc</span><span style="left: 386.38px;  top: 213.352px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.810673)">h and educational missions of Middlebury College. </span><span style="left: 148.2px;  top: 232.872px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.808359)">Users agree to only run computational jobs related to those missions. For example, </span><span style="left: 148.2px;  top: 252.392px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.805739)">cryptocurrency mining for financial gain or commercial use of the cluster is not </span><span style="left: 148.2px;  top: 271.912px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.825685)">appropriate.</span>
 +
#<span style="left: 148.2px;  top: 310.952px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.796828)">The Ada cluster is a share</span><span style="left: 313.239px;  top: 310.952px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.808733)">d resource. Running computations that consume large </span><span style="left: 148.2px;  top: 330.472px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.810242)">portions of the cluster for extended periods (including consuming large portions of the </span><span style="left: 148.2px;  top: 349.992px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.802936)">available disk space) could prevent others from using this community resource. Exercise </span><span style="left: 148.2px;  top: 369.512px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.806379)">care in how you use the Ada </span><span style="left: 335.614px;  top: 369.512px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.811188)">cluster to be respectful of other community members’ </span><span style="left: 148.2px;  top: 389.352px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.800285)">interest in using the system.</span>
 +
#<span style="left: 148.2px;  top: 447.912px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.812039)">You are entirely responsible for any data you place on the cluster. You agree that your </span><span style="left: 148.2px;  top: 467.432px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.805831)">data management practices are in accordance with Middlebury’s policies and any </span><span style="left: 148.2px;  top: 486.952px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.81297)">app</span><span style="left: 172.676px;  top: 486.952px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.803173)">licable regulations or agreements, e.g. HIPAA, data use agreements, etc.</span>
 +
#<span style="left: 148.2px;  top: 525.992px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.800411)">The Ada cluster is intended for data analysis not data storage. Data is not backed up. </span><span style="left: 148.2px;  top: 545.512px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.808924)">Data that is no longer needed should be promptly deleted to ensure there is sufficient </span><span style="left: 148.2px;  top: 565.032px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.778102)">disk s</span><span style="left: 183.684px;  top: 565.032px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.8071)">pace for everyone.</span>
 +
#<span style="left: 148.2px;  top: 604.072px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.808222)">You agree to respect the privacy of other users, e.g. by not exploring directories owned </span><span style="left: 148.2px;  top: 623.592px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.803037)">by other users even if those directories are accessible to you.</span>
 +
#<span style="left: 148.2px;  top: 662.632px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.810545)">You are expected to report any security incidents or abuse to ITS immediately. </span><span style="left: 148.2px;  top: 682.152px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.804608)">Examples of security incidents include but are not limited to: unauthorized access or </span><span style="left: 148.2px;  top: 701.672px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.806521)">use, compromised accounts </span><span style="left: 333.21548799999994px;  top: 701.6722916666668px;  font-size: 16px;  font-family: sans-serif">-</span><span style="left: 341.731px;  top: 701.672px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.803259)">including “shared” login credentials, and misuse of data.</span>
 +
 
 +
<span style="left: 96px;  top: 740.712px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.804327)">Users whose behavior runs counter to these principles may be asked by cluster adm</span><span style="left: 641.414px;  top: 740.712px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.811668)">inistrators </span><span style="left: 96px;  top: 760.232px;  font-size: 16px;  font-family: sans-serif;  transform: scaleX(0.810677)">to leave the cluster.</span>
 +
</div> </div>
  
 
=== [[High_Performance_Computing_(HPC)/Training|Training]] ===
 
=== [[High_Performance_Computing_(HPC)/Training|Training]] ===
  
 
=== Questions ===
 
=== Questions ===

Revision as of 11:53, 20 July 2021

 

Overview

High Performance Computing (HPC) is the aggregation of computing power and memory to perform complex calculations in parallel, increasing the speed and efficiency of computer simulations and data analysis. In 2018, a collaboration of faculty in the social and natural sciences and ITS staff successfully secured a $150,000 grant from the National Science Foundation to build Middlebury's first HPC cluster. Dubbed "Ada" in honor of Ada Lovelace, the famed 19th century mathematician, the cluster is a tool intended to support the research efforts of faculty who rely on access to expanded computing resources. We continue to add to our collaboration as resources become available.

This wiki describes the cluster structure and how to use it. The cluster is a shared resource, so we use queuing software (called Slurm) to manage job processing and to ensure fair access. Below are basic instructions for logging in to the cluster, accessing the queue and writing scripts to work efficiently and within best practices for a shared computing resource.

Cluster users must include an acknowledgement of NSF funding in any published research, as quoted below:

"This material is based upon work supported by the National Science Foundation under Grant No. 1827373.”

Please email the principal investigator, Professor Amy Yuen, with publication information for grant reporting purposes.

 

Access

A mananging group of faculty and staff have developed (policies) for various types of users. All users must agree to these policies and submit this (form) before obtaining access. The working group periodically offers training sessions for students and faculty interested in learning how to access the cluster and work with the queueing software. Users may indicate interest in these training sessions using this (form).

 

Hardware

The HPC cluster consists of 17 computer nodes with a cumulative total of 556 processors. It includes 14 nodes with 96GB of RAM each and one additional node with 768GB of RAM. In addition, the HPC cluster has a dedicated graphics processing unit (GPU) with 96GB of RAM, along with a storage node with 60TB of hard drive storage.

 

 

Software

  • MATLAB
  • R
  • Stata


Guidelines

Expectations and Support for Users

All HPC users will be expected to accept the standard Middlebury Code of Conduct relating to information and technology as well as a general set of best practices specific to the cluster. These will be posted on the HPC wiki page. Additionally, facultywho have little or no experience using a shared computing cluster are strongly urged to participate in the periodic training sessions offered by ITS staff and HPC-affiliated faculty.

Cluster Use Principles:

The use of the Ada cluster is governed by all the policies that apply to Middlebury’s Information Technology (http://www.middlebury.edu/about/handbook/policies-for-all/appropriate-use/info-tech) and the following principles:

  1. The Ada cluster supports the research and educational missions of Middlebury College. Users agree to only run computational jobs related to those missions. For example, cryptocurrency mining for financial gain or commercial use of the cluster is not appropriate.
  2. The Ada cluster is a shared resource. Running computations that consume large portions of the cluster for extended periods (including consuming large portions of the available disk space) could prevent others from using this community resource. Exercise care in how you use the Ada cluster to be respectful of other community members’ interest in using the system.
  3. You are entirely responsible for any data you place on the cluster. You agree that your data management practices are in accordance with Middlebury’s policies and any applicable regulations or agreements, e.g. HIPAA, data use agreements, etc.
  4. The Ada cluster is intended for data analysis not data storage. Data is not backed up. Data that is no longer needed should be promptly deleted to ensure there is sufficient disk space for everyone.
  5. You agree to respect the privacy of other users, e.g. by not exploring directories owned by other users even if those directories are accessible to you.
  6. You are expected to report any security incidents or abuse to ITS immediately. Examples of security incidents include but are not limited to: unauthorized access or use, compromised accounts -including “shared” login credentials, and misuse of data.

Users whose behavior runs counter to these principles may be asked by cluster administrators to leave the cluster.

Training

Questions

Powered by MediaWiki