0

We have installed torque on a dual Xeon (26 core, 52 available in hyperthreading). The node is configured with np=104. If I launch a MPI calculation in command line, I get near 100% cpu usage :

%Cpu(s): 53.9 us, 44.6 sy, 0.0 ni, 1.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

However, if I launch the same calculation with this torque submit file :

#!/bin/bash

#PBS -l walltime=20:00:00:00

#PBS -l nodes=1:ppn=104

#PBS -q batch

#PBS -N QE_test

cd $PBS_O_WORKDIR

/usr/lib64/openmpi/bin/mpirun -np 104 /opt/qe-6.3/bin/pw.x -inp scf.in > scf.out

The cpu usage is about 50% :

%Cpu(s): 32.5 us, 22.9 sy, 0.0 ni, 44.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

Do you have an idea why?

pbsnodes -a

servername

state = free

np = 104

ntype = cluster

status = rectime=1540890927,varattr=,jobs=,state=free,netload=? 0,gres=,loadave=0.00,ncpus=52,physmem=199919700kb,availmem=193132384kb,totmem=199919700kb,idletime=343335,nusers=0,nsessions=0,uname=Linux servername 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64,opsys=linux

mom_service_port = 15002

mom_manager_port = 15003

Francois Dion
  • 21
  • 2
  • Francois, Are you using the same node for torque server and compute? When you launch the mpi from command line it will run it in the local node. Through the torque pbs script it will launch it on the configured node. Can you provide some more information about your server and node setup? – Tux_DEV_NULL Oct 30 '18 at 09:49
  • Yes, I use the same node for torque server and compute node. Here is the output of the node : pbsnodes -a servername state = free np = 104 ntype = cluster status = rectime=1540890927,varattr=,jobs=,state=free,netload=? 0,gres=,loadave=0.00,ncpus=52,physmem=199919700kb,availmem=193132384kb,totmem=199919700kb,idletime=343335,nusers=0,nsessions=0,uname=Linux servername 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64,opsys=linux mom_service_port = 15002 mom_manager_port = 15003 – Francois Dion Oct 30 '18 at 13:17
  • Did you try with different -np values and observe different behavior? Also not sure about how the ppn is getting interpreted here? Did you try without it? I am guessing when running from torque for some reason it cannot assign the slots. Also look into using mpi host file and see if that fixes the issue. – Tux_DEV_NULL Oct 30 '18 at 14:47
  • I solved the problem. First I disabled hypertreading (https://serverfault.com/questions/235825/disable-hyperthreading-from-within-linux-no-access-to-bios). The mpi in command line show near 100% usage, when it was 50% usage and 50% system with hyperthreading. Secondly, I downgraded torque for a version without numa support (from torque-4.2.10-10.el7.x86_64 to torque-4.2.10-5.el7.x86_64). After that, the pbsnodes -a command show ncpus=52 when it was 26 with numa support. Now I get same result with mpirun -np 52 in torque. Thank you for your help :) – Francois Dion Oct 30 '18 at 18:36

1 Answers1

2

I solved the problem. First I disabled hypertreading (Disable hyperthreading from within Linux (no access to BIOS)). The mpi in command line show near 100% usage, when it was 50% usage and 50% system with hyperthreading. Secondly, I downgraded torque for a version without numa support (from torque-4.2.10-10.el7.x86_64 to torque-4.2.10-5.el7.x86_64). After that, the pbsnodes -a command show ncpus=52 when it was 26 with numa support. Now I get same result with mpirun -np 52 in torque.

Francois Dion
  • 21
  • 2