This one follows my previous article focusing on setting up torque system. However, it is found that torque 2.6.1 in Ubuntu system is out of date and not working properly. To circumvent this problem, I decide to move to torque/maui for better schedule efficiency.
http://www.adaptivecomputing.com/support/download-center/torque-download/
It is also noticed that adaptive computing is not maintaining torque and mari any more. which means bugs will not be cleaned. The ultimate solution for the system really is to move to slurm or sun grid system.
First, Download torque and maui from their websites:
maui has to be installed after torque installation
error 1:
pbs_mom: symbol lookup error: pbs_mom: undefined symbol: log_mutex
solution:
echo '/usr/local/lib' > /etc/ld.so.conf.d/torque.conf
ldconfig
error 2:
socket_connect_unix failed: 15137
qstat: cannot connect to server (null) (errno=15137) could not connect to trqauthd
solution: make sure
trqauthd is running with pbs_mom
error 3: at the client
pbs_mom
pbs_mom: symbol lookup error: pbs_mom: undefined symbol: dis_getc
error 4 at the client
./torque-mom start
* Starting Torque Mom torque-mom
/usr/sbin/pbs_mom: symbol lookup error: /usr/sbin/pbs_mom: undefined symbol: dis_getc
...fail!
pbs_mom: symbol lookup error: pbs_mom: undefined symbol: dis_getc
solution:
ldd /usr/local/sbin/pbs_mom
linux-vdso.so.1 => (0x00007fff9f7ff000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f2abbbed000)
libtorque.so.2 => /usr/local/lib/libtorque.so.2 (0x00007f2abb2f6000)
libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f2abaf99000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2abad7c000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f2abab74000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f2aba873000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2aba577000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f2aba361000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2ab9fa1000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2ab9d9d000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f2ab9b86000)
/lib64/ld-linux-x86-64.so.2 (0x00007f2abbe0b000)
the solution so far is to resintall the torque 5.0.1, 2015-05-27
it takes the whole morning to fix it
this happens again 2015-09-28
this file is located in
/usr/local/sbin/pbs_mom
just run it should be ok
dis_getc is the old package from apt-get
first: remove the torque in apt repo : apt-get remove torque-mom
now if run pbs_mom you wiil see
./pbs_mom
pbs_mom: LOG_ERROR::No such file or directory (2) in chk_file_sec, Security violation with "/var/spool/torque/checkpoint" - /var/spool/torque/checkpoint cannot be lstat'd - errno=2, No such file or directory
then reinstall torque-5.0.1-1_4fa836f5
torque-package-clients-linux-x86_64.sh --install
torque-package-mom-linux-x86_64.sh --install
question 1:
limit the maximum processes per user
http://docs.adaptivecomputing.com/maui/6.2throttlingpolicies.php
install pam torque
libtool --finish /lib64/security
/lib64/security/ is the place where pam files are located
/etc/security/access.conf give access to anyone you wish to give
set maui to limit the jobs and process per user
USERCFG[DEFAULT] MAXPROC=64 MAXJOB=5 #working
#GROUPCFG[useraid] MAXJOB[USER]=5 # not working
#CLASSCFG[batch] MAXJOB[USER]=5 working
CLASSCFG[batch] MAXJOB[USER]=5 MAXPROC=64 # not working
Working solution to use pam to prevent user from logging into compute nodes
give some users into compute nodes while others staying outside
versions: torque-5.0.1-1_4fa836f5 maui-3.3.tar.gz
in the tutorial given by official maui http://docs.adaptivecomputing.com/torque/3-0-5/3.4hostsecurity.php
it says
1. first configure torque with ./configure --with-pam
2.
/etc/pam.c/sshd.
account required
pam_pbssimpleauth.so
account required
pam_access.so
and
3.
In
/etc/security/access.conf make sure all users who access the compute node are added to the configuration.This is an example which allows the users root, george, allen, and michael access.
-:ALL EXCEPT root george allen michael torque:ALL
However, I found this method is too strong, specifically, none of root george allen can log into compute node.
my solution:
1. do not need to resinstall torque with ./configure --with-pam
2. put
account required pam_access.so
into /etc/pam.d/sshd
which means pam_access has to be considered for each ssh login
3. put
-:ALL EXCEPT root szhang czhang storres torque:ALL
into /etc/security/access.conf
now only szhang czhang root can log into compute nodes
I think this idea is working and understandable. because at the moment all the submission is done by pbs_mom which is running under root, so pam_pbssimpleauth.so doesn't have to take into effect.
reload maui
just restart it. it wont affect the queue
pkill maui && qterm -t quick && sleep 5&& /usr/local/maui/sbin/maui && pbs_server && ps aux |grep maui
showres working
showres -n
checkjob 810 working
checknode macondo01 % very good feedback
showgrid AVGXFACTOR
showstats
mbal this will kill maui!!!!!!!!!!!!!!!
mdiag same as diagnose
I still didn't get the idea of maxnode. does it mean all job for one person has to go to one perticular node?
mjobct
ERROR: corrupt command received
mclient
ERROR: unknown command: 'mclient'
mprof
USAGE ERROR: (tracefile not specified)
mstat
ERROR: command 'mstat' args not handled
ERROR: service 36 not handled
ERROR: Service[36] 'mstat' not implemented
showbf
backfill window (user: 'czhang' group: 'useraid' partition: ALL) Sun Jan 18 15:25:07
231 procs available for 7:11:35:38
175 procs available for 21:18:13:37
118 procs available for 40:14:55:01
62 procs available for 40:21:06:15
diagnose -j | grep -o -P '(?<=job \047).*(?=\047 utilizes more procs than)
# this line can find out all the job where warnings comes out.
diagnose -j
Name State Par Proc QOS WCLimit R Min User Group Account QueuedTime Network Opsys Arch Mem Disk Procs Class Features
381 Running DEF 1 DEF 10:00:00:00 1 1 cwang useraid uq-Civil 00:49:21 [NONE] [NONE] [NONE] >=0 >=0 NC0 [batch:1] [NONE]
569 Running DEF 1 DEF 25:00:00:00 1 1 pzhang useraid uq-Civil 00:49:21 [NONE] [NONE] [NONE] >=0 >=0 NC0 [batch:1] [NONE]
WARNING: job '569' utilizes more procs than dedicated (10.35 > 1)
650 Running DEF 1 DEF 41:16:00:00 1 1 mgholami useraid uq-Civil 00:49:20 [NONE] [NONE] [NONE] >=0 >=0 NC0 [batch:1] [NONE]
WARNING: job '650' utilizes more procs than dedicated (13.00 > 1)
651 Running DEF 1 DEF 41:16:00:00 1 1 mgholami useraid uq-Civil 00:49:20 [NONE] [NONE] [NONE] >=0 >=0 NC0 [batch:1] [NONE]
WARNING: job '651' utilizes more procs than dedicated (10.28 > 1)
669 Running DEF 1 DEF 41:16:00:00 1 1 mgholami useraid uq-Civil 00:49:19 [NONE] [NONE] [NONE] >=0 >=0 NC0 [batch:1] [NONE]
WARNING: job '669' utilizes more procs than dedicated (14.00 > 1)
671 Running DEF 1 DEF 25:00:00:00 1 1 pzhang useraid uq-Civil 00:49:21 [NONE] [NONE] [NONE] >=0 >=0 NC0 [batch:1] [NONE]
WARNING: job '671' utilizes more procs than dedicated (9.57 > 1)
672 Running DEF 1 DEF 25:00:00:00 1 1 pzhang useraid uq-Civil 00:49:21 [NONE] [NONE] [NONE] >=0 >=0 NC0 [batch:1] [NONE]
WARNING: job '672' utilizes more procs than dedicated (7.80 > 1)
\047 octal ascii represent single quote
diagnose -j | grep -o -P '(?<=than dedicated \050).*(?=>)'
\047 octal ascii represent 'left bracket'
adse=$(diagnose -j | grep -o -P '(?<=than dedicated \050).*(?=>)')
store result into adse
if [ "$a" != "$b" ]
then
echo "$a is not equal to $b."
echo "(string comparison)"
# "4" != "5"
# ASCII 52 != ASCII 53
fi
#!/bin/bash
x=5.0
y=3.0
#ans= $(( $x + $y |bc ))
#ans=$(echo $x + $y |bc )
#ans=$(echo $x / $y |bc -l ) # this ends up with good result
#ans=$(echo $x / $y |bc ) # this does not give good result
#ans=$(python -c "print $x / $y") # this one is also ok but format is a problem
#ans=$(python -c "print( "%.2f" %($x / $y) ) ") #failed
#alpha=`echo "$a/100" | bc -l | awk '{printf("%06.2f", $1);}'`
ans=`echo "$x/$y" | bc -l | awk '{printf("%6.4f", $1);}'`
echo "$x / $y = $ans"
maui starts off to be deprecated. use
Sun Grid Engine (SGE, rock cluster uses this Oracle Grid Engine) or slurm instead.
it feels to me that the soft hard limit only works for the groups not rather for users
/usr/local/maui
http://www.physics.oregonstate.edu/cluster_install
Problem 2016-01-12:
once running trqauthd
trqauthd: symbol lookup error: trqauthd: undefined symbol: debug_mode
this happens for the server, the server has been runing for a few days. once trqauthd is killed, it can not reboot, properly.
root@macondo03:/home/users/uqczhan2# trqauthd
trqauthd: symbol lookup error: trqauthd: undefined symbol: debug_mode
root@macondo03:/home/users/uqczhan2# pbs_server
pbs_server: symbol lookup error: pbs_server: undefined symbol: job_log_mutex
root@macondo03:/home/users/uqczhan2# pbs_mom
pbs_mom: symbol lookup error: pbs_mom: undefined symbol: log_mutex
root@macondo03:/home/users/uqczhan2# which trqauthd
/usr/local/sbin/trqauthd
root@macondo03:/home/users/uqczhan2# pbs_
pbs_demux pbs_mom pbs_restart pbs_sched pbs_server pbs_track
root@macondo03:/home/users/uqczhan2# pbs_sched
pbs_sched: symbol lookup error: pbs_sched: undefined symbol: log_mutex
root@macondo03:/home/users/uqczhan2# pbs_restart
Cannot connect to default server host 'macondo03' - check pbs_server daemon.
qterm: could not connect to server '' (1) Operation not permitted
torque-package-server-linux-x86_64.sh
we get pbs_sched pbs_server qschedd qserverd
./torque-package-mom-linux-x86_64.sh --install
Installing TORQUE archive...
Done.
root@macondo03:/home/user/uqczhan2/czhang/Downloads/torque-5.0.1-1_4fa836f5# ls /usr/local/sbin
momctl pbs_demux pbs_mom pbs_sched pbs_server qnoded qschedd qserverd
solution:
ldd trqauthd
linux-vdso.so.1 => (0x00007ffcf55e1000)
libtorque.so.2 => /usr/local/lib/libtorque.so.2 (0x00007f365ed33000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f365eb16000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f365e816000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f365e458000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f365e250000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f365df54000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f365dd3e000)
/lib64/ld-linux-x86-64.so.2 (0x00007f365f62a000)
today problem resolved again:
infact fds model gets the system hangs. it changes the address of libtorque.so.2 and so trqauthd is not working.
solution: i have removed everything associated with FDS in .bashrc (from LD_LIBRARY_PATH). and check ldd trqauthd. the right one should be the same as the ones above.
also after the restore, there is a bit problem in restart pbs_mom pbs_server and pbs_sched .
solution:
first, apt-get remove torque-mom torque-server torque-sched, make sure the torque in apt system is not installed.
second, reinstall torque 5.0.1 by configure, make make install.
run one by one.
the below are the errors appears when running pbs_mom pbs_server pbs_sched.
pbs_mom
pbs_mom: LOG_ERROR::No such file or directory (2) in chk_file_sec, Security violation with "/var/spool/torque/checkpoint" - /var/spool/torque/checkpoint cannot be lstat'd - errno=2, No such file or directory
for pbs_server and pbs_sched, once running it, it doesn't show as a process in the system.
as long as reinstall torque 5.0.1 problem get resolved. 2016-01-12
problem
pbsnodes
pbsnodes: Server has no node list MSG=node list is empty - check 'server_priv/nodes' file
cd /var/spool/torque/server_priv
