Nathaniel's notes

This page is intended as a set of working notes for doing profiling/static linking/documentation of the MINOS offline software.

Making static libraries:

May 24, 2001: Improving and interpreting results.

- Went back and recompiled ROOT again.. this time with -g and -pg flags to ensure complete debugging info.
- Now using JobController to run the session rather than command line. The job I'm using is:

/Path/Create Demo \
Input::Get \
RerootToRawDataModule::Get \
DigitListModule::Get \
DigitListModule::Reco \
DigiPairListModule::Reco \
CutDigiPairListModule::Reco \
SuperSliceListModule::Reco \
MSTClusterListModule::Reco \
ThruMuonListModule::Reco \
FitThruMuonListModule::Reco \
GoodThruMuonListModule::Reco \
ThruMuonAnalysisModule::Ana \
UserAnalysis::Ana \
CheezyDisplayModule::Ana \
EventDump::Ana
/DigiPairListModule/Set ListIn candmcdigitlist
/CutDigiPairListModule/Set ADC 30;
/CutDigiPairListModule/Set TDC 100
/SuperSliceListModule/Set ListIn candcutdigipairlist
/MSTClusterListModule/Set DistMax 50
/MSTClusterListModule/Set HoleMax 0
/MSTClusterListModule/Set PlaneMax 3
/ThruMuonListModule/Set DiffMax 20
/GoodThruMuonListModule/Set ListIn candfitthrumuonlist
/GoodThruMuonListModule/SetStraight MaxBad 20
/GoodThruMuonListModule/SetGeom DistMin 0
/GoodThruMuonListModule/SetGeom NPlnMin 6
/ThruMuonAnalysisModule/Set ListIn candgoodthrumuonlist

/Demo/Input/Add /home/tagg/root_files/gm_far_cosmic_1_hapr01.root
/Demo/Run Nin 10
/ThruMuonAnalysisModule/Plot NDigitSpec

- Looked at results from run. Here's a sample:

Flat profile:
Each sample counts as 0.01 seconds.
%   cumulative   self              self     total
time   seconds   seconds    calls ms/call ms/call name
29.00     15.53    15.53   345118     0.04     0.09 TNodeX::GetNodeToDepth(char const *, int)
9.77     20.76     5.23 59775441     0.00     0.00 TListIter::Next(void)
8.66     25.40     4.64 59994642     0.00     0.00 TIter::operator()(void)
5.45     28.32     2.92       13   224.62   224.62 TObject type_info function
5.14     31.07     2.75 61171346     0.00     0.00 TNamed::GetName(void) const
4.61     33.54     2.47 59751805     0.00     0.00 TObject::TestBit(unsigned int) const
3.42     35.37     1.83        3   610.00   684.97 TNodeX type_info function
3.17     37.07     1.70                             UgliSteelPlnNode type_info function
2.93     38.64     1.57                             UgliScintPlnNode type_info function
1.08     39.22     0.58   520946     0.00     0.00 TNode::UpdateTempMatrix(double const *, double const *, double, double, double, double *, double *, double *)
0.54     39.51     0.29   346095     0.00     0.00 TNode::GetNode(char const *) const
0.49     39.77     0.26 7440373     0.00     0.00 basic_string<char, string_char_traits<char>, __default_alloc_template<true, 0> >::rep(void) const
0.47     40.02     0.25 4119225     0.00     0.00 basic_string<char, string_char_traits<char>, __default_alloc_template<true, 0> >::length(void) const

ggest culprit for absorbing CPU time seems to be TNodeX::GetNodeToDepth, which is uses noticable time. The TList and TListIter classes are hit pretty heavy, but I doublt we can get away from that. (We may want fragile, optimized iterators in particular places though.)

I don't know why TNodeX is so heavy. Here's the call sequence stuff:

                             59751805             TNodeX::GetNodeToDepth(char const *, int) [18]
                7.75    7.70 172225/345118      UgliGeometry::GetStripNode(PlexStripEndId) [17]
                7.78    7.73 172893/345118      UgliGeometry::GetScintPlnNode(PlexPlaneId) [24]
[18]    57.8   15.53   15.43 345118+59751805 TNodeX::GetNodeToDepth(char const *, int) [18]
                4.62    5.28 59752253/59994642     TIter::operator()(void) [28]
                2.70    0.00 60096923/61171346     TNamed::GetName(void) const [56]
                2.47    0.00 59751805/59751805     TObject::TestBit(unsigned int) const [57]
                0.11    0.16 344898/636949      TIter::TIter(TCollection const *, unsigned char) [102]
                0.05    0.03 344898/637639      TIter::~TIter(void) [212]
                             59751805             TNodeX::GetNodeToDepth(char const *, int) [18]

I made a small attempt to make an optimized version of strcmp() in TNodeX, which I thought might be the culprit. Some success there: I managed to reduce 'self seconds' time for TNodeX::GetNodeToDepth() down to 14.14 seconds from 15.53.. If I believe gprof, this means an increase in overall job speed of about 3%.

Whoopee.

May 23, 2001: Everything seems to work!

The mods I describe below (May 22) work to create a true executable, but they fail to run a real job: creation of a JobController fails because the JobCModules are unregistered. This is because the static objects (created by the JOBMODULE macro in JobCModuleRegistry.h) are not generated unless the object files in question are explicitly linked in.

Because there are a dozen or so of these modules, it seems useless to try to continue in this vein. My original strategy was to let SRT build the static libraries and then just use them, but to do this I would either have to:
a) explicitly include all object files with static objects created in them, or
b) use the -u linker option to make these symbols undefined (forcing the loading of the .o file inside the .a library)

So, I've gone back to a previous attempt: namely, just build a big mess o' object files and smush them into an executable, like olives in a press.

Here's the new makefile:

# MakeStatic.mk
# Custom Makefile
# Nathaniel Tagg, Oxford
# May 23, 2001
#
# Using a complete, compiled SRT test distribution, this makefile will build a
# root static library, a root static executable, and a static MINOS binary
# containing as many classes as I could stuff in.
# This makefile requires that the SRT test release be _completely_ built and UNCLEANED,
# with both static and shared libraries. It also requires that ROOT be a complete,
# uncleaned build as well.
# (These builds contain the CINT-created object files, which must be manually
# linked in in order to work. Don't ask me why; even if they exist in the libraries,
# they are needed in the final link step to be incorporated by the linker.)

# All the object files needed to build a ROOT executable (except main.o)
STATICROOT_OBJ = \
        $(ROOTSYS)/html/src/G*.o \
        $(ROOTSYS)/histpainter/src/G*.o \
        $(ROOTSYS)/treeplayer/src/G*.o \
        $(ROOTSYS)/treeviewer/src/G*.o \
        $(ROOTSYS)/x3d/src/G*.o \
        $(ROOTSYS)/postscript/src/G*.o \
$(ROOTSYS)/physics/src/G*.o \
$(ROOTSYS)/eg/src/G*.o \
libRoot.a \
globalsetup.o
# All the object files that go into building the libRoot.a library
ROOTLIB_OBJ = \
$(ROOTSYS)/base/src/*.o \
$(ROOTSYS)/cint/src/*.o \
$(ROOTSYS)/clib/src/*.o \
$(ROOTSYS)/cont/src/*.o \
$(ROOTSYS)/eg/src/*.o \
$(ROOTSYS)/g3d/src/*.o \
$(ROOTSYS)/gpad/src/*.o \
$(ROOTSYS)/graf/src/*.o \
$(ROOTSYS)/gui/src/*.o \
$(ROOTSYS)/hist/src/*.o \
$(ROOTSYS)/histpainter/src/*.o \
$(ROOTSYS)/html/src/*.o \
$(ROOTSYS)/matrix/src/*.o \
$(ROOTSYS)/meta/src/*.o \
$(ROOTSYS)/minuit/src/*.o \
$(ROOTSYS)/net/src/*.o \
$(ROOTSYS)/physics/src/*.o \
$(ROOTSYS)/postscript/src/*.o \
$(ROOTSYS)/proof/src/*.o \
$(ROOTSYS)/rint/src/*.o \
$(ROOTSYS)/tree/src/*.o \
$(ROOTSYS)/treeplayer/src/*.o \
$(ROOTSYS)/treeviewer/src/*.o \
$(ROOTSYS)/unix/src/*.o \
$(ROOTSYS)/x11/src/*.o \
$(ROOTSYS)/x3d/src/*.o \
$(ROOTSYS)/zip/src/*.o
# List of package names, listed in _dependency order_!
# This ordering came from a custom perl script that uses
ALLPACKAGES = \
BubbleSpeak \
CandEventSR \
CandFitTrackSR \
CandTrackSR \
CandStripSR \
CandSliceSR \
CandShowerSR \
CandClusterSR \
RecoBase \
IoModules \
Islands \
DeMux \
Demo \
Candidate \
CandVtx \
CandEvent \
CandDigit \
UgliGeometry \
RerootExodus \
Record \
RawData \
Plex \
Persistency \
Navigation \
Lattice \
JobControl \
DatabaseInterface \
CandData \
BField \
Validity \
MINF_Classes \
USER_Classes \
TOOL_Classes \
NumericalMethods \
MinosObjectMap \
LeakChecker \
DynamicFactory \
Conventions \
Algorithm \
Registry \
REROOT_Classes \
RDBC \
MessageService \
# Packages to remove:
DONTUSE = BField CandEvent CandFitTrackSR
PACKLIST = $(filter-out $(DONTUSE),$(ALLPACKAGES))
# The sum total object file list.
TOTOBJLIST = $(foreach pkg,$(PACKLIST),tmp/$(SRT_SUBDIR)/$(pkg)/lib$(pkg)-shared/*.o)
all: libRoot.a roota statbin
myheaders = JobControl/JobController.h
myobj =

mymain = $(ROOTSYS)/main/src/rmain.o
#mymain = JobControl/main/JobCmain.cxx

statbin: $(STATICROOT_OBJ) $(TOTOBJLIST) MakeStatic.mk
@if [ -r MyDict.cxx ] ; then rm MyDict.cxx ; fi
rootcint MyDict.cxx -c -I./ $(myheaders)
g++ \
-m486 -g -pg \
-I/home/tagg/root/include \
-I . \
$(TOTOBJLIST) \
$(STATICROOT_OBJ) \
MyDict.cxx \
$(mymain) \
-L /usr/X11R6/lib \
-lXpm \
-lX11 -lm -ldl -rdynamic   \
-o $@

# Static rootlib
libRoot.a: $(ROOTLIB_OBJ)
ar rv libRoot.a $^
# Static root
roota: libRoot.a $(STATICROOT_OBJ)
echo 'int G__globalsetup() {}' >globalsetup.c
cc -c globalsetup.c
g++ -pg -o roota \
$(ROOTSYS)/main/src/rmain.o \
$(STATICROOT_OBJ) \
-L /usr/X11R6/lib \
-lXpm \
-lX11 -lm -ldl -rdynamic

May 22, 2001: First successful compile

First, modifications need to be made to SRT to allow static libraries, and to add the profiling commands.

This step is nessessary: modify minossoft/releases/development/SRT_MINOS/special]/pre_standard.mk

LIB_TYPE=both

This step SHOULD make things easier, but it doesn't. This includes the CINT object files in the static libraries, (even though they can't be seen there!)
minossoft/releases/development/SRT_MINOS/SoftRelTools/arch_spec_root.mk

# Now add the object to the list of object files for the
# library. If SoftRelTools internals change this may need to
# change
actual_sharedlib_files += $(cintworkdir)$(CINTOBJECT)
$(SHAREDLIB): $(cintworkdir)$(CINTOBJECT)
$(STATICLIB): $(cintworkdir)$(CINTOBJECT) # N.Tagg, 4/5/2001: allow static libs to have cint object files

Add the compile flag for profiling:
minossoft/releases/development/SRT_MINOS/special/post_standard.mk

override CPPFLAGS += $(ENV_CXXFLAGS) -g -pg

Set up your private test directory (checking out all the packages) and do a complete make. While this is happening....

Next, modify ROOT:
Add a file to the base ROOT directory (i.e. in $ROOTSYS) named MyConfig.mk:override CFLAGS += -pg

override CXXFLAGS += -pg
override CINTCXXFLAGS += -pg
override CINTCFLAGS += -pg
override LDFLAGS += -pg

Then ./configure and make the root distribution.

See Rene Brun's note on building a static ROOT library at http://root.cern.ch/cgi-bin/print_hit_bold.pl/root/roottalk/roottalk00/1958.html?gprof#first_hit
for building a static root executable.

Build the binaries:
This is my custom makefile:

STATICROOT_OBJ = \
        $(ROOTSYS)/html/src/G*.o \
        $(ROOTSYS)/histpainter/src/G*.o \
        $(ROOTSYS)/treeplayer/src/G*.o \
        $(ROOTSYS)/treeviewer/src/G*.o \
        $(ROOTSYS)/x3d/src/G*.o \
        $(ROOTSYS)/postscript/src/G*.o \
$(ROOTSYS)/physics/src/G*.o \
$(ROOTSYS)/eg/src/G*.o \
libRoot.a \
globalsetup.o

#Static root lib.
ROOTLIB_OBJ = \
$(ROOTSYS)/base/src/*.o \
$(ROOTSYS)/cint/src/*.o \
$(ROOTSYS)/clib/src/*.o \
$(ROOTSYS)/cont/src/*.o \
$(ROOTSYS)/eg/src/*.o \
$(ROOTSYS)/g3d/src/*.o \
$(ROOTSYS)/gpad/src/*.o \
$(ROOTSYS)/graf/src/*.o \
$(ROOTSYS)/gui/src/*.o \
$(ROOTSYS)/hist/src/*.o \
$(ROOTSYS)/histpainter/src/*.o \
$(ROOTSYS)/html/src/*.o \
$(ROOTSYS)/matrix/src/*.o \
$(ROOTSYS)/meta/src/*.o \
$(ROOTSYS)/minuit/src/*.o \
$(ROOTSYS)/net/src/*.o \
$(ROOTSYS)/physics/src/*.o \
$(ROOTSYS)/postscript/src/*.o \
$(ROOTSYS)/proof/src/*.o \
$(ROOTSYS)/rint/src/*.o \
$(ROOTSYS)/tree/src/*.o \
$(ROOTSYS)/treeplayer/src/*.o \
$(ROOTSYS)/treeviewer/src/*.o \
$(ROOTSYS)/unix/src/*.o \
$(ROOTSYS)/x11/src/*.o \
$(ROOTSYS)/x3d/src/*.o \
$(ROOTSYS)/zip/src/*.o
# List of package names.
ALLPACKAGES = \
CandEventSR \
CandFitTrackSR \
CandTrackSR \
CandStripSR \
CandSliceSR \
CandShowerSR \
CandClusterSR \
RecoBase \
IoModules \
Islands \
DeMux \
Demo \
Candidate \
CandVtx \
CandEvent \
CandDigit \
UgliGeometry \
RerootExodus \
Record \
RawData \
Plex \
Persistency \
Navigation \
Lattice \
JobControl \
DatabaseInterface \
CandData \
BField \
Validity \
MINF_Classes \
USER_Classes \
TOOL_Classes \
NumericalMethods \
MinosObjectMap \
LeakChecker \
DynamicFactory \
Conventions \
Algorithm \
Registry \
REROOT_Classes \
RDBC \
MessageService \
# Packages to remove:
DONTUSE = BField CandEvent CandFitTrackSR
PACKLIST = $(filter-out $(DONTUSE),$(ALLPACKAGES))
# These packages have no Cint.o file.
DONTCINT = MessageService NumericalMethods Demo IoModules
# Munge the package names into Cint.o file paths
CINTPACKLIST = $(filter-out $(DONTCINT),$(PACKLIST))
CINTLIST = $(foreach pkg,$(CINTPACKLIST),tmp/$(SRT_SUBDIR)/$(pkg)/lib$(pkg)-shared/$(pkg)Cint.o)
# Munge the package names into libBlah.a file paths.
LIBLIST = $(foreach pkg,$(PACKLIST),lib/$(SRT_SUBDIR)/lib$(pkg).a)

default: roota
statbin: $(STATICROOT_OBJ)
g++ \
-m486 -g \
-D_SVID_SOURCE -DUNIX -DLINUX -D__UNIX__ -D__LINUX__ \
-DDATAREP_LITTLE_IEEE -DDATAREP_LITTLE_ENDIAN \
-DDEFECT_NO_IOSTREAM_NAMESPACES -DDEFECT_NO_JZEXT \
-DDEFECT_NO_INTHEX -DDEFECT_NO_INTHOLLERITH -DDEFECT_NO_READONLY \
-DDEFECT_NO_DIRECT_FIXED -DDEFECT_NO_STRUCTURE -DUSE_ROOT \
-D_REENTRANT \
-I/home/tagg/root/include \
$(CINTLIST) \
$(LIBLIST) \
$(STATICROOT_OBJ) \
$(ROOTSYS)/main/src/rmain.o \
-L /usr/X11R6/lib \
-lXpm \
-lX11 -lm -ldl -rdynamic \
-o statbin \

# Static rootlib
libRoot.a: $(ROOTLIB_OBJ)
ar rv libRoot.a $^
# Static root
roota: libRoot.a
echo 'int G__globalsetup() {}' >globalsetup.c
cc -c globalsetup.c
g++ -o roota \
$(STATICROOT_OBJ) \
$(ROOTSYS)/main/src/rmain.o \
l-L /usr/X11R6/lib \
-lXpm \
-lX11 -lm -ldl -rdynamic

QUICK NOTES:

To load all packages into a test release:

write a quick 4-line perl script and run it:

#!/usr/bin/perl -h
foreach $f (glob("$SRT_PUBLIC_CONTEXT/*")) {
$basename = ($f =~ m!^(.*)/(.*)!)[1]; # Finds basename of #f use [0] to find path
system("addpkg -h $basename");
}