Here are the rules of the game.  Ignorance of the rules is not an acceptable excuse for bad behavior.

• If you are not familiar with HPC scheduling and the various parameters that determine which job goes next, Dr. Tarbox gave a presentation at the Biomedical Informatics Seminar on March 15^th which gives an introduction to HPCs and scheduling that might be helpful.
• The easiest way to access Grace is through her web-based, Open OnDemand portal, found at https://portal.hpc.uams.edu.  Open OnDemand allows you to get a web-based command line interface with Grace, manipulate files in your home directory, and submit and monitor job scripts. In the future Open OnDemand will allow for interactive jobs, for example using Jupyter notebooks or RStudio.  It is a work in progress.
• You may need to log into Grace at least once by ssh to set your password before using Open OnDemand.  It may be possible to manually reset your password via Open OnDemand by launching a terminal window and using the "passwd" command at the prompt.
• Many researchers access Grace by using ssh directed to login.hpc.uams.edu.  Note that the login node is only accessible inside the UAMS firewall.  If you need to access the login node (e.g. via ssh) from outside of the UAMS network you will need to first set up a VPN connenction into the UAMS network.
• The login node is not meant for computation.  Computationally intensive programs should be run as jobs using the queueing system.  Users caught running computationally extensive programs (including containers) on the login node without permission from the system administrators may have their privileges on Grace suspended temporarily until after they have a conversation with a system administrator.  The system administrators may kill computationally extensive programs running on the login node without warning.  Compiling programs is allowed, if you have no facilities on your personal computer for compiling programs.
• Users may not login (e.g. use ssh) directly into any of the computational or management nodes without permission from a system administrator.  Users caught doing so may have the Grace accounts temporarily suspended until they have a conversation with the system administrator.  Users should only log into the login node, or utilize the Open OnDemand portal, nothing else.
• If you need interactive access to a computational node, please submit an interactive job, and be prepared to wait until a node becomes free.  Interactive jobs, while allowed, are discouraged, and excessive use of interactive jobs may trigger a conversation with a system administrator.
• The system administrators do have the right to look into your directories and jobs as needed to assure the smooth operation of Grace.  They may suggest potential improvements or changes to the way you work, either to help your jobs be better "HPC Citizens" or to improve their efficiency.  Since Grace is a shared resource, it is important to use the system gracefully (pun intended).
• Over the October 2, 2020 weekend Grace was switched from the Moab/Torque scheduler/resource manager to slurm.  The slurm website includes a table that shows the mapping between slurm commands and the Torque commands.  Some, but not all, of the old Torque commands (e.g. qsub) are emulated in slurm, but may have somewhat different behavior from the Torque variant.  Switching to slurm, makes Grace operations more consistent with HPC services at the University of Arkansas in Fayetteville (e.g. the Pinnacle cluster).
• The scheduler/resource manager works best if your wallclock, memory, and processing core requests are close to real.  If you are unsure what to use, please try one or two small sample jobs, and then check what the actual runtime, memory, and core usage were, and use that to guide your estimates.  Note that your job priorities are penalized if you consistently overestimate these values.
• The compute nodes are outfitted with the OpenHPC libraries.  Many options require issuing a module command to load appropriate versions of software libraries before use.
• If you need a special software configuration (i.e. the OpenHPC libraries are not enough) we recommend using Singularity containers.  Most Docker containers can run under Singularity, and there are a lot of Singularity and Docker containers available on their respective hubs.  If you can’t find a suitable container, you can choose a base container, and then configure it to your liking on your local workstation before loading the container to the HPC.
• Each HPC user gets a home directory, which is visible to all of the compute nodes and the login node, with a 1 TB quota.  Users who consistently exceed their 1 TB quota without permission from the system administrators may find that they can no longer run jobs due to a lack of disk space.
• Users may temporarily store data in the /scratch directory, which is shared across all nodes.  Note that data in the /scratch directory is not considered permanent, and could be purged of older data at any time.  So do not count on keeping things in /scratch for more than 14 days after a job ends.  Hence the strong recommendation that users utilize the data staging options in Moab to copy input data to /scratch before the job starts, then copy output data to outside storage (e.g. the research NAS, the Research Object Storage System (ROSS), the user’s personal computer, or a cloud storage provider) at the end of a job. 
• Each compute job also has the option to use local scratch storage (/tmp or /local-scratch) on each node.  Note that local scratch areas are not shared between nodes (i.e. they are on a disk local to a particular compute node).  Their big advantage over /scratch is that accessing local scratch storage incurs no network overhead, so can be faster than /scratch if a program is doing lots of small, random reads and writes.  The shared /scratch area may be faster for long sequential reads and writes due to the large block size and parallelism on the shared storage system.  If you use local scratch, be sure to delete what you put into it when done.
• A good habit to get into is to stage (move) data into a scratch area (local or shared, as needed) prior to the start of a job, and to copy data out of the scratch areas after the end of the job.  
• There is a shared area of library files (files common to a number of projects) currently called /storage.  (We may rename it.)  Please do not store personal files there.  At some point in the future, /storage will be made read-only, and personal files will be moved to the user’s home directory, potentially impacting that user’s quota and ability to run jobs. 
• Projects or labs may contact the HPC Administrators to set up shared project-specific area accessible by all compute nodes.  There may be a fee for this service, dependent on the amount of storage reserved (i.e. the quota requested).
• UAMS has bulk campus storage that is available to researchers for a fee.  The research campus storage facilities are accessible to the login node, as well as the data transfer nodes that can used by the data staging scripts, but are not accessible to the compute nodes (i.e. cannot be accessed inside a compute job).  The bulk campus storage is not designed for I/O intensive access by the compute nodes, hence a user must stage the data to/from bulk campus storage and Grace's cluster storage for use inside a compute job.
• There are three categories of processors, selectable via feature requests when submitting jobs, in Grace:
• Xeon Phi (phi) – ideal for massively parallel operations that have lots of threads.  Each Xeon Phi node has 64 physical cores and either 384 GB of memory (80 of the Xeon Phi nodes), or 192 GB of memory (16 of the nodes).  Although the clock speed of each individual processor is slower than a regular xeon, since there are more than double the number of cores on a Xeon Phi compared to regular Xeon, appropriately configured multi-threaded jobs can run faster on the Xeon Phi.  A program optimized for Xeon Phi can gain significant speed advantage over regular Xeon.  Optimized programs also typically will run faster on regular Xeon as well, though the speedup is not as dramatic.
• Regular Xeon (xeon) – use where single thread performance is critical.  Each regular Xeon node has 28 physical cores and 128 GB of memory. 
• Nvidia Tesla GPU nodes (gpu) – for jobs that can capitalize on GPU performance.  Each node has 24 regular Xeon cores, 128 GB of memory, and a pair of 12GB P100 GPUs.  We have Nvidia CUDA 10 drivers installed on the GPU nodes.  The GPU nodes are in a special class (a.k.a. queue) named “gpu”, separate from the other nodes.  So when submitting jobs that depend on the GPU, make sure to select the “gpu” class, not the default “batch” class.  Otherwise your job could sit in limbo asking for GPU resources on nodes that don’t have them.
• Generally, if your program is not geared specifically for one of the above categories, it is best to just let the scheduler decide (i.e. don’t put in a feature request), so that the job gets routed to the first available processor.
• Although the upper limit to the number of jobs that Moab/Torque can comfortably manage is large, it is not infinite.  Please check with a system administrator before submitting a bolus of more than 2,000 jobs, so that the system administrator can check to make sure sufficient resources are available.  If you are submitting a large number of related jobs, please consider using a Job Array to submit the entire set at once, rather than submitting all of the jobs individually.  Job Arrays are much easier to administer and monitor, and often are more efficient than submitting individual jobs.  
• If you are having problems with Grace, please let the system administrators know.  They cannot fix things that they do not know about.
• The HPC system administration staff certainly is willing to answer questions and to assist with small problems, but do not have the bandwidth to do major work for you.  If you need extensive assistance, there may be a need to set up a research partnership or project, e.g. with faculty or staff in the Department of Biomedical Informatics, or in the UAMS IT department.  Obviously such partnerships or projects often require a financial arrangement to cover people's time.
• You should never put sensitive information, such as fully identified patient records, onto Grace or any of her storage systems.  Grace is not considered a HIPAA or FERPA repository.
• Similarly, you should never attempt to access sensitive data on the UAMS network via Grace.  Such activity would be reported, and could result in disciplinary actions.  If you attempt unauthorized access to data protected by HIPAA or FERPA regulations, there could also be criminal charges brought by the government.  Bottom line – just don't do it.
• You agree to maintain the confidentiality of any information that you may encounter that does not belong to you.  If the data is not yours, you may not disclose it to anyone.