SAGE Routine to Auto Generate LSF Files
The following creates the necessary LSF job script as well as the control file of tasks to submit a (single-program, multiple-data) job on Lonestar.
def OutputLSFFiles(intro, function, outro, parameters, cores, jobname, queue='normal', htime=48, mtime=00, email='', start=False, end=False, workdir='$WORK/data'): """ INPUT: function -- string parameters -- list cores -- positive integer jobname -- string queue -- string htime, mtime -- integers, htime <= 48, mtime < 59 email -- string start, end -- boolean workdir - string OUTPUT: jobname.lsf -- LSF batch file to submit job to Lonestar jobname.txt -- text file containing tasks for Lonestar to be run in parallel """ if os.path.exists('%s.lsf'%(jobname)) or os.path.exists('%s.txt'%(jobname)): print 'Files already exist. Overwriting.' launcher = open('%s.lsf'%(jobname), 'w') tasks = open('%s.txt'%(jobname), 'w') print >> launcher, '#!/bin/csh\n# ---- SETUP PARAMETERS ----' print >> launcher, 'module load launcher' print >> launcher, 'setenv EXECUTABLE $TACC_LAUNCHER_DIR/launcher' print >> launcher, 'setenv CONTROL_FILE %s.txt'%(jobname) print >> launcher, 'setenv WORKDIR %s'%(workdir) print >> launcher, '#BSUB -n %s'%(cores) print >> launcher, '#BSUB -W %02i:%02i'%(htime, mtime) print >> launcher, '#BSUB -J %s'%(jobname) print >> launcher, '#BSUB -o out.%J' # %J is the Job ID, an integer print >> launcher, '#BSUB -e err.%J' print >> launcher, '#BSUB -q %s'%(queue) if not email == '': print >> launcher, '#BSUB -u %s'%(email) if start: print >> launcher, '#BSUB -B' if end: print >> launcher, '#BSUB -N' print >> launcher, '\n# ---- ERROR CHECKING ----' print >> launcher, 'if ( ! -e $WORKDIR ) then\n echo " "' print >> launcher, ' echo "Error: unable to change to working directory."' print >> launcher, ' echo " $WORKDIR"' print >> launcher, ' echo " "\n echo "Job not submitted."\n exit' print >> launcher, 'endif' print >> launcher, '\nif ( ! -f $EXECUTABLE ) then\n echo " "' print >> launcher, ' echo "Error: unable to find launcher executable $EXECUTABLE."' print >> launcher, ' echo " "\n echo "Job not submitted."\n exit' print >> launcher, 'endif' print >> launcher, '\nif ( ! -f $WORKDIR/$CONTROL_FILE ) then\n echo " "' print >> launcher, ' echo "Error: unable to find input control file $CONTROL_FILE."' print >> launcher, ' echo " "\n echo "Job not submitted."\n exit' print >> launcher, 'endif' print >> launcher, '\n# ---- SUBMIT JOB ----' print >> launcher, 'cd $WORKDIR/\necho " LSF_SUBMIT_DIR: $LS_SUBCWD"' print >> launcher, 'echo " WORKING DIR: $WORKDIR/"' print >> launcher, '\npam -g 1 parametric_wrapper $EXECUTABLE $CONTROL_FILE' print >> launcher, 'echo "pam -g 1 parametric_wrapper $EXECUTABLE $CONTROL_FILE"' print >> launcher, 'echo " "\necho " Parameteric Job Complete"\necho " "' for i in range(len(parameters)): cmd = intro + function + "(*" + str(parameters[i]) + ")" + outro print >> tasks, '$WORK/sage-3.0.3/sage -c "%s"'%(cmd) # needs to be fixed
Run the routine on some SAGE function, e.g.
OutputLSFFiles(intro='M=', function='ModularSymbols', outro='; S=M.cuspidal_submodule(); new=S.new_submodule(); D=new.decomposition(); print D', parameters=[[N,2] for N in prime_range(11,20)], cores=4, jobname='demojob', queue='development', email='sbutt@math.utexas.edu', htime=00, mtime=10, start=True, end=True, workdir='$WORK/demo-data')
Note that intro and outro are just concatenated with function(parameters) so take care in setting them. This command produces a jobname.txt file consisting of
$WORK/sage-3.0.3/sage -c "M=ModularSymbols(*[11, 2]); S=M.cuspidal_submodule(); new=S.new_submodule(); D=new.decomposition(); print D" $WORK/sage-3.0.3/sage -c "M=ModularSymbols(*[13, 2]); S=M.cuspidal_submodule(); new=S.new_submodule(); D=new.decomposition(); print D" $WORK/sage-3.0.3/sage -c "M=ModularSymbols(*[17, 2]); S=M.cuspidal_submodule(); new=S.new_submodule(); D=new.decomposition(); print D" $WORK/sage-3.0.3/sage -c "M=ModularSymbols(*[19, 2]); S=M.cuspidal_submodule(); new=S.new_submodule(); D=new.decomposition(); print D"
The generated output to stdout found in out.%J is
LSF_SUBMIT_DIR: /work/utexas/math/sbutt/sage-3.0.3 WORKING DIR: /work/utexas/math/sbutt/demo-data/ -------------------------------------------------- TACC Launcher -> 4 processors allocated. TACC Launcher -> 4 total tasks to complete. -------------------------------------------------- TACC Launcher -> Iniating task 1 on c45-209 ($WORK/sage-3.0.3/sage -c "M=ModularSymbols(*[11, 2]); S=M.cuspidal_submodule(); new=S.new_submodule(); D=new.decomposition(); print D") TACC Launcher -> Iniating task 4 on c45-209 ($WORK/sage-3.0.3/sage -c "M=ModularSymbols(*[19, 2]); S=M.cuspidal_submodule(); new=S.new_submodule(); D=new.decomposition(); print D") TACC Launcher -> Iniating task 3 on c45-209 ($WORK/sage-3.0.3/sage -c "M=ModularSymbols(*[17, 2]); S=M.cuspidal_submodule(); new=S.new_submodule(); D=new.decomposition(); print D") TACC Launcher -> Iniating task 2 on c45-209 ($WORK/sage-3.0.3/sage -c "M=ModularSymbols(*[13, 2]); S=M.cuspidal_submodule(); new=S.new_submodule(); D=new.decomposition(); print D") [ ] [ Modular Symbols subspace of dimension 2 of Modular Symbols space of dimension 3 for Gamma_0(11) of weight 2 with sign 0 over Rational Field ] [ Modular Symbols subspace of dimension 2 of Modular Symbols space of dimension 3 for Gamma_0(19) of weight 2 with sign 0 over Rational Field ] [ Modular Symbols subspace of dimension 2 of Modular Symbols space of dimension 3 for Gamma_0(17) of weight 2 with sign 0 over Rational Field ] Job parametric_wrapper /opt/lsf/bin/TaskStarter -p c45-209:47000 -c /opt/lsf/conf -a X86_64 /opt/apps/launcher/launcher-1.2/ /launcher demojob.txt TID HOST_NAME COMMAND_LINE STATUS TERMINATION_TIME ===== ========== ================ ======================= =================== 00000 c45-209 /opt/apps/launch Done 06/25/2008 18:55:57 00001 c45-209 /opt/apps/launch Done 06/25/2008 18:55:57 00002 c45-209 /opt/apps/launch Done 06/25/2008 18:55:57 00003 c45-209 /opt/apps/launch Done 06/25/2008 18:55:56 pam -g 1 parametric_wrapper /opt/apps/launcher/launcher-1.2//launcher demojob.txt Parameteric Job Complete
Move the control file jobname.txt to the work directory (default is $WORK/data) and then run the command
bsub < jobname.txt
to submit the job to Lonestar.
Notes
- The routine currently only handles the simplest of SAGE functions: at the very most, you can run one function which has variable parameters. The function must be of the form:
f1(fixed paramters); ...; fn(fixed parameters); g(*[variable parameters]); h1(fixed parameters); ...; hm(fixed parameters)
- We can do the basic computation of modular symbols though. Namely we can run things like (as above):
M=ModularSymbols(*[N, 2]); S=M.cuspidal_submodule(); new=S.new_submodule(); D=new.decomposition();
Note that the timing information that is also produced in our code is missing.
- Building SAGE on Lonestar still has some issues (but did build for me with some help from Michael Abshoff)
- Probably makes sense to adapt auto-generation to use files in order to avoid such long command-line arguments.