How to Profile the MINOS Offline Code

Here are the basic steps one must take to build a static, profilable version of the system:

1. Install a custom copy of ROOT
2. Modify and build ROOT for static linking.
3. Install a test distribution
4. Modify and build the test distribution
5. Link rootLib.a, roota, statbin
6. Run the code!
7. Use gprof to see output.
 
 

1. Install a custom copy of ROOT

cd $ROOTSYS
cvs update

2. Modify and build ROOT for static linking.

Make a new file called "MyConfig.mk" in the $ROOTSYS directory and put these lines in it.

override CFLAGS += -g -pg
override CXXFLAGS += -g -pg
override CINTCXXFLAGS += -g -pg
override CINTCFLAGS += -g -pg
override LDFLAGS += -g -pg

Run gmake.  While ROOT is merrily building away, continue with the rest.

3. Install a test distribution

Make a new test directory, and add ALL the packages.
 

newrel -t development prof
cd prof
srt_setup -a
perl
        foreach $f (glob("$SRT_PUBLIC_CONTEXT/*")) {
       $basename = ($f =~ m!^(.*)/(.*)!)[1];             # Finds basename of #f use [0] to find path
       system("addpkg -h $basename");
       }

4. Modify and build the test distribution

export $ENV_CXXFLAGS = -pg

5. Link rootLib.a, roota, statbin

# MakeStatic.mk
# Custom Makefile
# Nathaniel Tagg, Oxford
# May 23, 2001
#
# Using a complete, compiled SRT test distribution, this makefile will build a
# root static library, a root static executable, and a static MINOS binary
# containing as many classes as I could stuff in.

# This makefile requires that the SRT test release be _completely_ built and UNCLEANED,
# with both static and shared libraries.  It also requires that ROOT be a complete,
# uncleaned build as well.
# (These builds contain the CINT-created object files, which must be manually
#  linked in in order to work. Don't ask me why; even if they exist in the libraries,
#  they are needed in the final link step to be incorporated by the linker.)
 

# All the object files needed to build a ROOT executable (except main.o)
STATICROOT_OBJ = \
        $(ROOTSYS)/html/src/G*.o \
        $(ROOTSYS)/histpainter/src/G*.o \
        $(ROOTSYS)/treeplayer/src/G*.o \
        $(ROOTSYS)/treeviewer/src/G*.o \
        $(ROOTSYS)/x3d/src/G*.o \
        $(ROOTSYS)/postscript/src/G*.o \
        $(ROOTSYS)/physics/src/G*.o \
        $(ROOTSYS)/eg/src/G*.o \
        libRoot.a \
        globalsetup.o

# All the object files that go into building the libRoot.a library
ROOTLIB_OBJ = \
  $(ROOTSYS)/base/src/*.o \
  $(ROOTSYS)/cint/src/*.o \
  $(ROOTSYS)/clib/src/*.o \
  $(ROOTSYS)/cont/src/*.o \
  $(ROOTSYS)/eg/src/*.o \
  $(ROOTSYS)/g3d/src/*.o \
  $(ROOTSYS)/gpad/src/*.o \
  $(ROOTSYS)/graf/src/*.o \
  $(ROOTSYS)/gui/src/*.o \
  $(ROOTSYS)/hist/src/*.o \
  $(ROOTSYS)/histpainter/src/*.o \
  $(ROOTSYS)/html/src/*.o \
  $(ROOTSYS)/matrix/src/*.o \
  $(ROOTSYS)/meta/src/*.o \
  $(ROOTSYS)/minuit/src/*.o \
  $(ROOTSYS)/net/src/*.o \
  $(ROOTSYS)/physics/src/*.o \
  $(ROOTSYS)/postscript/src/*.o \
  $(ROOTSYS)/proof/src/*.o \
  $(ROOTSYS)/rint/src/*.o \
  $(ROOTSYS)/tree/src/*.o \
  $(ROOTSYS)/treeplayer/src/*.o \
  $(ROOTSYS)/treeviewer/src/*.o \
  $(ROOTSYS)/unix/src/*.o \
  $(ROOTSYS)/x11/src/*.o \
  $(ROOTSYS)/x3d/src/*.o \
  $(ROOTSYS)/zip/src/*.o

# List of package names, listed in _dependency order_!
# This ordering came from a custom perl script that uses
ALLPACKAGES = \
BubbleSpeak \
CandEventSR \
CandFitTrackSR \
CandTrackSR \
CandStripSR \
CandSliceSR \
CandShowerSR \
CandClusterSR \
RecoBase \
IoModules \
Islands \
DeMux \
Demo \
Candidate \
CandVtx \
CandEvent \
CandDigit \
UgliGeometry \
RerootExodus \
Record \
RawData \
Plex \
Persistency \
Navigation \
Lattice \
JobControl \
DatabaseInterface \
CandData \
BField \
Validity \
MINF_Classes \
USER_Classes \
TOOL_Classes \
NumericalMethods \
MinosObjectMap \
LeakChecker \
DynamicFactory \
Conventions \
Algorithm \
Registry \
REROOT_Classes \
RDBC \
MessageService \

# Packages to remove:
DONTUSE = BField CandEvent CandFitTrackSR

PACKLIST = $(filter-out $(DONTUSE),$(ALLPACKAGES))

# The sum total object file list.
TOTOBJLIST = $(foreach pkg,$(PACKLIST),tmp/$(SRT_SUBDIR)/$(pkg)/lib$(pkg)-shared/*.o)

all: libRoot.a roota statbin

myobj =
 

#mymain = $(ROOTSYS)/main/src/rmain.o
mymain = JobControl/main/JobCmain.cxx
 

statbin: $(STATICROOT_OBJ) $(TOTOBJLIST) MakeStatic.mk
        g++ \
        -m486 -g -pg \
        -I/home/tagg/root/include \
        -I . \
        $(TOTOBJLIST) \
        $(STATICROOT_OBJ) \
        $(mymain) \
        -L /usr/X11R6/lib \
        -lXpm \
        -lX11 -lm -ldl -rdynamic   \
        -o $@
 

# Static rootlib
libRoot.a: $(ROOTLIB_OBJ)
        ar rv libRoot.a $^

# Static root
roota: libRoot.a $(STATICROOT_OBJ)
        echo 'int  G__globalsetup() {}' >globalsetup.c
        cc -c globalsetup.c
        g++ -pg -o roota \
        $(ROOTSYS)/main/src/rmain.o \
        $(STATICROOT_OBJ) \
        -L /usr/X11R6/lib \
        -lXpm \
        -lX11 -lm -ldl -rdynamic

This makefile can be edited to exclude packages by setting the DONTUSE variable.
The $(mymain) variable is used to determine what your interface is (a la JobControl, or the familiar, evil old ROOT prompt.)

Notice how this works: it simply loots the $(ROOTSYS) and SRT test directories for *.o files and includes them.  It has been my experience that there is no other reasonably easy way of doing a static link.   There is a note here by Rene Bruns describing that this is what he uses for ROOT.

See below for a discussion how this method works.

When your ROOT and prof/ builds are done, you're ready to make your static binaries:

make -f MakeStatic.mk

This might take a LONG time.. it has to build and resolve complete symbol tables for all of ROOT and MINOSSOFT.

You should now have three files in your prof/ directory you can use: libRoot.a, roota, and statbin.
libRoot.a -> static ROOT libs.
roota        -> A static version of root
statbin     -> A static version of demojob

6. Run the code!

However, when the code has finished, you should have a file named gmon.out in the working directory. Hurrah!

Note that the code must finish execution normally: the profilng code must write out the data on program exit, so a program must not SEGV or get killed before it has completed normally.

Note that testers running independent modules can make their own test beds by linking their object files into the ROOT libraries, as is done with the roota target.
 

7. Use gprof to see output.

and look at the gprof.out file.  The interpetation of this file is not trivial, particularly when recursion is involved: see http://www.ibiblio.org/pub/gnu/Manuals/gprof-2.9.1/gprof.html for details on interpretation.
 
 
 

Discussion

Ideally, the right way to do this would be to build static libraries as counterparts to the  MINOS and ROOT *.so libraries built by SRT and the ROOT makefiles.  SRT already has latent capacity to build static libraries, so why not use it?   It would save a lot of searching for files at link-time.

However, this doesn't really work very well. The reason is that the linker treats a lib.a file and an object file differently.  These two commands are treated differently:
Case A:
    gcc foo.o bar.o

Case B:
    ar rv libbar.a bar.o
    gcc foo.o libbar.a

In case A, all symbols in foo.o and bar.o are linked. Period.  In case B, however, the linker takes only those symbols in libbar.a that are undefined AND needed after foo.o has been linked.

It's the phrase 'and needed' that is crucial.  Certain symbols do not appear to be needed to the linker.  Examples are static objects that are not explicitly called from main(), or symbols that are used only when a dymanic factory builds them, or objects that are created by the ROOT command line.   There are options that force the linker to load such symbols, but every such symbol must be named individually on the command line.  Say, every symbol in every ROOT G__*.o file, and every MINOS *Cint.o file, and every file that has a JOBMODULE preprocessor macro, etc, etc.

The clear method is to just load every symbol explictly, by using only object files, and loading all the files individually.  (I use a single static library for a lot of the root objects, just because ROOT convieniently seperates their files for this purpose.)  This scheme is easy to keep up-to-date, since one only needs to add directories with *.o files that you want.  The down side is that you have to keep every *.o file (no gmake clean!) and linking takes forever and hits the hard drive so hard your platters turn into jelly.   But it works.  Functionality before optimization!
 

May 25, 2001.  Nathaniel Tagg,  University of Oxford n.tagg1@physics.ox.ac.uk