Debugging Lab - TACC User Portal

Debugging Lab Antonio Gómez-­‐Iglesias [email protected] Texas Advanced Compu?ng Center Setup – X11 •  Login to Stampede: -  ssh -X [email protected]
•  Untar the lab files: –  cd
–  tar zxvf ~train00/dbg_lab.tgz
•  Change directories and ls to see the files: –  cd dbg_lab
–  ls
GNU DEBUGGING LAB Find a bug with gdb In this example we will use gdb to debug a code that crashes. •  First, you should be on a compute node when debugging with gdb %
%
idev -p normal -A TRAINING-HPC
idev -p normal -A TG-TRA140011
•  AMer your session begins, compile the deadlock example: % icc -g -o divcrash ./divcrash.c
•  Run the code % ./divcrash
What happened? # C example
Find a bug with gdb Now, we’ll run with gdb •  Use gdb to load the executable: % gdb ./divcrash
•  Kick off the run (gdb) run
•  Where does the code crash? •  What line was it on? •  Set a breakpoint at the line that the code crashed on •  Step through each itera?on using the con&nue command •  Print the value of x and y at each breakpoint (this will not work for codes compiled with ifort with the default gdb) DDT DEBUGGING LAB Finding a deadlock with DDT In this example we will use DDT to debug a code that deadlocks (you need X11, so ssh -X). •  Exit from your idev session •  Compile the deadlock example:
% mpicc -g -O0 -o deadlock ./deadlock.c
#C example
% mpif90 -g -O0 -o deadlock ./deadlock.f90 #F90
•  Load the DDT module: % module load ddt
•  Start up DDT: % ddt ./deadlock
DDT Run Window If you are using X11, you should see the default Run window with the executable name in the Applica?on text box. Don’t click submit yet! We need to configure: •  MPI type •  Number of processes •  Memory Debugging •  Queue Submission Parameters •  Advanced Op?ons Configure DDT: Op?ons • 
Choose the correct version of MPI –  mvapich 2 –  Intel MPI • 
Leave the default MPI (MVAPICH 2) • 
Leave Debugger on the Automa?c seang Configure DDT: Tasks + Memory Debugging • 
Set the number of tasks to 8 • 
Enable memory debugging • 
Enter Memory Debugging Pane Configure DDT: Memory Debugging • 
Ensure the library is set to C/Fortran • 
If your code is C++, reset this to C++ • 
If ddt crashes with memory debugging enabled, try seang guard pages Configure DDT: Batch Submission This step is specific to this lab. Normally, you will not have to change this. • 
Open the Configure pane for batch submission Configure DDT: Batch Submission • 
Update the Submission template file entry to point to the custom template file in the local directory This template file has the reserva?on command for the batch script created by ddt. • 
Edit the queue parameters Configure DDT: Queue Parameters • 
• 
Choose the “normal” queue Set your project code -­‐ for this training class use: TRAINING-HPC
or TG-TRA140011
• 
The reserva?on text box is specific to this custom template • 
Select the OK bujon DDT: Job Queuing Add any necessary arguments to the program (none for the example) Click the Submit bujon. A new window will open: The job is submijed to the specified queue. An automa?cally refreshing job status window appears. The debug session will begin when the job starts. DDT: The debug session Process controls Source navigator Code pane Stack view, input/output, breakpoints, etc. Variable pane Evalua?on pane Automa?cally stops at the first command past MPI_Init DDT: Program Hangs The output we expect does not appear in the Stdout window. No ac?ve communica?on between procs. Pause execu?on to analyze the program status (top leM). DDT: Stacks On the bojom leM window select the Stacks view. Note that since this version of MPI was compiled with debug informa?on, the hal?ng point is within the MPI library DDT: Stacks Move to the top of the stack to determine which lines of your code are at the hal?ng point. All processors seem to be stuck on a MPI_Send(). DDT: What’s Wrong? This is a classic deadlock situa?on. All tasks are engaged in a blocking send, but there are no tasks receiving. Rearrange the order of communica?ons to remove the deadlock.