The ARC cluster: Difference between revisions

From Italian ARC
Jump to navigation Jump to search
No edit summary
 
(72 intermediate revisions by 4 users not shown)
Line 1: Line 1:
The ARC is equipped with a dedicated computer server connected with a high-speed optical fiber network to the outside world, allowing fast data transfer (10 Gbit/sec).
The ARC is equipped with a dedicated computer server connected with a high-speed optical fiber network to the outside world, allowing fast data transfer (10 Gbit/sec).


Presently we have 110 Tb of diskspace and one 15 nodes cluster (112 cores) dedicated to the ARC.
Presently we have 200 Tb of diskspace and one 26 nodes cluster (112 cores) dedicated to the ARC.


ALMA and CASA users can request access to the server and the disk space by sending an e-mail to ''arc-support at ira.inaf.it'' indicating the reason for the request.  
ALMA and CASA users can request access to the server and the disk space by sending an e-mail to ''arc-support at ira.inaf.it'' indicating the reason for the request.  
Line 33: Line 33:
Once you have obtained an '''ARC''' account at '''IRA''', you can access the computer cluster nodes from everywhere through host  '''scheduler.ira.inaf.it.'''.
Once you have obtained an '''ARC''' account at '''IRA''', you can access the computer cluster nodes from everywhere through host  '''scheduler.ira.inaf.it.'''.
Using graphical applications on the cluster is possible through  remote X access.
Using graphical applications on the cluster is possible through  remote X access.
The accessible working nodes are listed in the table below. Never submit workloads to arcserv (control node) and arcnas1 (storage) as this can slow down the entire cluster.
The accessible working nodes are listed in the table below.
You can  enter a node for interactive work by typing:
You can  enter a node for interactive work by typing:


<source lang="bash">ssh -tX scheduler.ira.inaf.it</source>
<source lang="bash">ssh -X ''<node>''.ira.inaf.it</source>
 
You need to change directory to access your home on the cluster:
 
<source lang="bash"> cd /iranet/homesarc/username </source>


''Useful tip: by typing 'hostname' you can know on which node you are''
''Useful tip: by typing 'hostname' you can know on which node you are''


Access is done via a Torque/Maui scheduler that redirects your job on the less-loaded node.
[http://ganglia.ira.inaf.it/?r=hour&cs=&ce=&c=unspecified&h=&tab=m&vn=&hide-hf=false&m=load_one&sh=1&z=small&hc=4&host_regex=%5Earc&max_graphs=0&s=by+name Here] you can find some statistics about resources consumption on the arcblXX nodes.


'''Jobs on the cluster are limited to a duration of 168 hours'''.
== Using ARC storage ==


[http://ganglia.ira.inaf.it/?c=ARC_Blades&m=load_one&r=hour&s=by%20name&hc=4&mc=2 Here] you can find some statistics about resources consumption on the arcblXX nodes.
You need to change directory to access your ARC storage on the cluster:


== Executing programs ==
<source lang="bash"> cd /iranet/homesarc/''username'' </source>


You can execute programs in two ways:
On each node you can also use local disks on path /local/work/username (if allocated) and /local/scratch


in interactive mode - your command is immediately executed on the less-loaded node and standard input, output and error are linked to your terminal.
/local/scratch will be emptied on every reboot
 
or scheduling a pbs job - submit a job file ([http://beige.ucs.indiana.edu/I590/node35.html here is a guide])
 
== Copying files to from the ARC storage ==
<source lang="bash">
# from storage...
scp user@arcbl01.ira.inaf.it:/remote/path /local/path


# to storage
Beware that local disks have '''no redundancy at all''', never leave important data on them
scp /local/path user@arcbl01.ira.inaf.it:/remote/path
</source>


== Mounting ARC storage on you workstation ==
=== Mounting ARC storage on you workstation ===


On IRA workstations ARC home filesystem can be accessed on ''/iranet/homesarc''
On IRA workstations ARC home filesystem can be accessed on ''/iranet/homesarc''
Line 86: Line 72:
then, as user
then, as user


<source lang="bash">sshfs arcbl01.ira.inaf.it:/iranet/homesarc/yourhome /your/local/mount/point/</source>
<source lang="bash">sshfs storage.alma.inaf.it:/iranet/homesarc/yourhome /your/local/mount/point/</source>


By omitting /remote/path you can mount you home directory. i.e.:
By omitting /remote/path you can mount you home directory.  


Be aware that this method is suboptimal for heavy input/output loads. Running disk-intensive applications directly on the arc cluster will result in a file access speed 10-50 times faster.
Be aware that this method is suboptimal for heavy input/output loads. Running disk-intensive applications directly on the arc cluster will result in a file access speed 10-50 times faster.
Line 94: Line 80:
== Software packages available ==
== Software packages available ==


Software available on ARC cluster could be listed typing the command ''setup-help''
Software available on ARC cluster could be listed typing the command ''setup-help'' or on [https://www.ira.inaf.it/wiki/index.php/Arcsoft this page]


== Computing Nodes ==
{| class="wikitable"
{| class="wikitable"
|-
|-
! Software package !! setup command !! launch command !! notes
! Name !! RAM !! CPU !! Cores !!  Clock !! Data Net !! Work Disk !! Scratch Disk !! scheduler !! groups !! notes
|-
|-
| CASA ||casapy-setup|| casapy || data reduction package http://casa.nrao.edu/
|arcbl01||256G||Intel Xeon E5-1650 v4||6/12||3600||10GbE||15T||57G||N||arc-staff, arc-vlbi, arc-f2f||
|-
|-
| Miriad ||miriad-setup|| miriad || data reduction package http://www.atnf.csiro.au/computing/software/miriad/
|arcbl17||64G||AMD Ryzen 7 1800X||8/16||3600||1GbE||3,5TB|| ||N||arc-staff, arc-vlbi||
|-
|-
|aips||aips-setup|| || http://www.aips.nrao.edu/index.shtml
|arcbl18||64G||Intel Xeon E3-1275 v6||4/8||3800||10GbE||22T||57G||N||arc-staff, arc-vlbi||Data transfer node
|-
|-
|analysis utils||analysisUtils-setup|| ||
|arcbl19||64G||Intel Xeon E3-1275 v6||4/8||3800||10GbE||11T||57G||N||arc-staff, arc-vlbi, arc-f2f, arc-user||NFS server
|-
|-
|analytic infall||analytic_infall-setup||
|arcbl20||256G||Intel Xeon E5-1650 v4||6/12||3600||10GbE||11T||65G||N||arc-staff, arc-vlbi||
|-
|-
|astron||astron-setup|| ||
|arcbl21||64G||Intel Xeon E3-1275 v6||4/8||3800||10GbE||11T||57G||N||arc-staff, arc-vlbi||
|-
|-
|Coyote library||coyote-setup|| ||
|arcbl22||256G||Intel Xeon E5-1650 v4||6/12||3600||10GbE||11T||65G||N||arc-staff, arc-vlbi, arc-f2f||
|-
|-
|fits Viewer||fv-setup|| ||
|arcbl23||256G||Intel Xeon E5-1650 v4||6/12||3600||10GbE||11T||65G||N||arc-staff, arc-vlbi||VM
|-
|-
|GCC Compiler||gcc-setup||
|arcbl24||256G||Intel Xeon E5-1650 v4||6/12||3600||10GbE||11T||65G||N||arc-staff, arc-vlbi||
|-
|-
|Gildas||gildas-setup|| ||http://www.iram.fr/IRAMFR/GILDAS/
|arcbl25||256G||Intel Xeon E5-1650 v4||6/12||3600||10GbE||11T||65G||N||arc-staff, arc-vlbi, arc-f2f||
|-
|-
|Healpix||healpix-setup|| ||http://healpix.jpl.nasa.gov/
|arcbl26||256G||Intel Xeon E5-1650 v4||6/12||3600||10GbE||11T||65G||N||arc-staff, arc-vlbi, arc-f2f||
|-
|-
|IDL||idl-setup|| ||http://www.exelisvis.com/ProductsServices/IDL.aspx
|arcbl27||64G||Intel Xeon E3-1275 v6||4/8||3800||10GbE||15T||57G||N||arc-staff, arc-vlbi||
|-
|-
|heasoft||heasoft-setup|| ||http://heasarc.nasa.gov/lheasoft/
|arcbl28||256G||Intel Xeon E5-1650 v4||6/12||3600||10GbE||15T||57G||N||arc-staff, arc-vlbi||
|-
|-}
|QA2||qa2-setup|| ||
|-
|Ratran||ratran-setup|| ||http://www.sron.rug.nl/~vdtak/ratran/frames.html
|-
|Starlink||starlink-setup|| ||http://starlink.jach.hawaii.edu/starlink
|}


== Nodes ==
== Storage Nodes ==
{| class="wikitable"
{| class="wikitable"
|-
|-
! name !! ram !! cores !! scheduler !! speed !! notes
! Name !! RAM !! CPU !! Cores !!  Clock !! Data Net !! RAID !! Space !! Storage !! export
|-
|arcbl01||32G||16||N||3500||
|-
|arcbl02||8G||8||Y||2100||
|-
|arcbl03||8G||8||Y||2100||
|-
|arcbl04||8G||8||Y||2100||
|-
|arcbl05||8G||8||Y||2100||
|-
|arcbl06||8G||8||Y||2100||
|-
|arcbl07||8G||8||Y||2100||
|-
|arcbl08||8G||8||Y||2100||
|-
|arcbl09||8G||8||Y||2100||
|-
|arcbl10||32G||16||N||3500||
|-
|-
|arcbl11||8G||8||N||2100||
|arcnas2||32G||Intel Xeon Silver 4108||8/16||1800||10GbE||ARC-1883IX-24||91||12x10TB (HGST HUH721010ALE600) RAID6||/lustre/arcfs0/ost3
|-
|-
|arcbl12||16G||8||N||2100||
|arcnas3||32G||Intel Xeon Silver 4108||8/16||1800||10GbE||ARC-1883IX-24||72,8||12x8TB (HGST HUH728080AL5200) RAID6 ||/lustre/arcfs0/ost0
|-
|-
|arcbl13||16G||4||N||2800||  
|arcnas4||16G||Intel XeonE5-2603v3||6/6||1600||10GbE||ARC-1284ML-24||36,4T||12x4TB (WDC WD4000F9YZ-09N20L1) RAID6||/lustre/arcfs0/ost1
|-
|-
|arcbl17||16G||4||N||2800||  
| || || || || || || ||91T||12x10TB (ST10000NM0086-2AA101) RAID6||/lustre/arcfs0/ost2
|-
|-
|arcbl18||64G||4||N||2800||  
|arcnas5||32G||Intel Xeon E5-2640 v4||10/20||2400||10GbE||Broadcom / LSI MegaRAID SAS-3 3108||255G||2x255GB RAID1||/lustre/arcfs0/mdt0
|}
|}

Latest revision as of 15:49, 26 January 2023

The ARC is equipped with a dedicated computer server connected with a high-speed optical fiber network to the outside world, allowing fast data transfer (10 Gbit/sec).

Presently we have 200 Tb of diskspace and one 26 nodes cluster (112 cores) dedicated to the ARC.

ALMA and CASA users can request access to the server and the disk space by sending an e-mail to arc-support at ira.inaf.it indicating the reason for the request.

Blade.jpg

User policy

New ARC users can access the Italian ARC node computing facilities by requesting a face-2-face visit (ALMA users only, through the ALMA Helpdesk) or by visiting the ARC node in Bologna (for any data-reduction-related issue to be solved in collaboration with the ARC staff). In both the cases they are requested to send an e-mail to help-desk@alma.inaf.it indicating the reason for the request.

Please notice that the request for a new account for a new requesting user implies that he/she (and/or his/her collaborators) visits the ARC for an induction on the ARC facilities usage and on issues related to data reduction with CASA both for ALMA or any other telescope. If the request is positively evaluated the visit details will be arranged via e-mail.

The account will guarantee the usage of the facilities and the support for 6 months.

Once the account expires the access to the data will be suspended and, after 1 month of quarantine ALL DATA WILL BE REMOVED. Only one gentle reminder will be send on account expiration. Extensions of the account duration period could be considered on request (via e-mail). No visit is needed in case of account renewal.

The ARC members support is guaranteed for any ALMA-related issue. For data-reduction-related issues that do not involve ALMA, the support (other than the technical support in the usage of the ARC computing facilities) is limited to the knowledge/experience and availability of the ARC members.

The same rules apply also to IRA staff members. IRA collaborators with temporary positions (i.e. students) can have an account for the entire duration of their position.

To ensure a well-balanced load on the cluster nodes please follow instructions about accessing the computer cluster

Queries can be issued via e-mail to help-desk@alma.inaf.it

Users will be automatically added to the arc-cluster-users@ira.inaf.it mailing list that will be used for any communication from our side.

Accessing the computer cluster

Once you have obtained an ARC account at IRA, you can access the computer cluster nodes from everywhere through host scheduler.ira.inaf.it.. Using graphical applications on the cluster is possible through remote X access. The accessible working nodes are listed in the table below. You can enter a node for interactive work by typing: 

ssh -X ''<node>''.ira.inaf.it

Useful tip: by typing 'hostname' you can know on which node you are

Here you can find some statistics about resources consumption on the arcblXX nodes.

Using ARC storage

You need to change directory to access your ARC storage on the cluster: 

 cd /iranet/homesarc/''username''

On each node you can also use local disks on path /local/work/username (if allocated) and /local/scratch

/local/scratch will be emptied on every reboot

Beware that local disks have no redundancy at all, never leave important data on them

Mounting ARC storage on you workstation

On IRA workstations ARC home filesystem can be accessed on /iranet/homesarc

On your laptop ARC filesystems can be seamlessly accessed with fuse-sshfs:

as root, install the package sshfs 

# on RedHat/Centos/ScientificLinux
yum install fuse-sshfs

# on Debian/Ubuntu
apt-get install sshfs

then, as user 

sshfs storage.alma.inaf.it:/iranet/homesarc/yourhome /your/local/mount/point/

By omitting /remote/path you can mount you home directory.

Be aware that this method is suboptimal for heavy input/output loads. Running disk-intensive applications directly on the arc cluster will result in a file access speed 10-50 times faster.

Software packages available

Software available on ARC cluster could be listed typing the command setup-help or on this page

Computing Nodes

Storage Nodes

Name RAM CPU Cores Clock Data Net Work Disk Scratch Disk scheduler groups notes
arcbl01 256G Intel Xeon E5-1650 v4 6/12 3600 10GbE 15T 57G N arc-staff, arc-vlbi, arc-f2f
arcbl17 64G AMD Ryzen 7 1800X 8/16 3600 1GbE 3,5TB N arc-staff, arc-vlbi
arcbl18 64G Intel Xeon E3-1275 v6 4/8 3800 10GbE 22T 57G N arc-staff, arc-vlbi Data transfer node
arcbl19 64G Intel Xeon E3-1275 v6 4/8 3800 10GbE 11T 57G N arc-staff, arc-vlbi, arc-f2f, arc-user NFS server
arcbl20 256G Intel Xeon E5-1650 v4 6/12 3600 10GbE 11T 65G N arc-staff, arc-vlbi
arcbl21 64G Intel Xeon E3-1275 v6 4/8 3800 10GbE 11T 57G N arc-staff, arc-vlbi
arcbl22 256G Intel Xeon E5-1650 v4 6/12 3600 10GbE 11T 65G N arc-staff, arc-vlbi, arc-f2f
arcbl23 256G Intel Xeon E5-1650 v4 6/12 3600 10GbE 11T 65G N arc-staff, arc-vlbi VM
arcbl24 256G Intel Xeon E5-1650 v4 6/12 3600 10GbE 11T 65G N arc-staff, arc-vlbi
arcbl25 256G Intel Xeon E5-1650 v4 6/12 3600 10GbE 11T 65G N arc-staff, arc-vlbi, arc-f2f
arcbl26 256G Intel Xeon E5-1650 v4 6/12 3600 10GbE 11T 65G N arc-staff, arc-vlbi, arc-f2f
arcbl27 64G Intel Xeon E3-1275 v6 4/8 3800 10GbE 15T 57G N arc-staff, arc-vlbi
arcbl28 256G Intel Xeon E5-1650 v4 6/12 3600 10GbE 15T 57G N arc-staff, arc-vlbi
Name RAM CPU Cores Clock Data Net RAID Space Storage export
arcnas2 32G Intel Xeon Silver 4108 8/16 1800 10GbE ARC-1883IX-24 91 12x10TB (HGST HUH721010ALE600) RAID6 /lustre/arcfs0/ost3
arcnas3 32G Intel Xeon Silver 4108 8/16 1800 10GbE ARC-1883IX-24 72,8 12x8TB (HGST HUH728080AL5200) RAID6 /lustre/arcfs0/ost0
arcnas4 16G Intel XeonE5-2603v3 6/6 1600 10GbE ARC-1284ML-24 36,4T 12x4TB (WDC WD4000F9YZ-09N20L1) RAID6 /lustre/arcfs0/ost1
91T 12x10TB (ST10000NM0086-2AA101) RAID6 /lustre/arcfs0/ost2
arcnas5 32G Intel Xeon E5-2640 v4 10/20 2400 10GbE Broadcom / LSI MegaRAID SAS-3 3108 255G 2x255GB RAID1 /lustre/arcfs0/mdt0
Retrieved from "https://www.alma.inaf.it/index.php?title=The_ARC_cluster&oldid=4170"