m &Ec@sdZdkZdkZdkZdkZdkZdkZeiidddk Z dkl Z l Z l Z l Z lZdklZdkZdZedZeedZdd Zd d d Zd ZdfdYZdZdZdZdZdZdZddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;d<d=d>d?d@dAdBdCdDdEdFdGdHdIdJdKdLdMdNdOdPdQdRdSdTdUdVdWdXdYdZd[d\d]d^d_d`dadbdcdddedfdgdhdidjdkdldmdndodpdqdrdsdtdudvdwdxdydzd{d|d}d~ddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd d d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;d<d=d>d?d@dAdBdCdDdEdFdGdHdIdJdKdLdMdNdOdPdQdRdSdTdUdVdWdXdYdZd[d\d]d^d_d`dadbdcdddedfdgdhdidjdkdldmdndodpdqdrdsdtdudvdwdxdydzd{d|d}d~ddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd d d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;d<d=d>d?d@dAdBdCdDdEdFdGdHdIdJdKdLdMdNdOdPdQdRdSdTdUdVdWdXdYdZd[d\d]d^d_d`dadbdcdddedfdgdhdidjdkdldmdndodpdqdrdsdtdudvdwdxdydzd{d|d}d~ddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd d d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;d<d=d>d?d@dAdBdCdDdEdFdGdHdIdJdKdLdMdNdOdPdQdRdSdTdUdVdWdXdYdZd[d\d]d^d_d`dadbdcdddedfdgdhdidjdkdldmdndodpdqdrdsdtdudvdwdxdydzd{d|d}d~dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddgZ dk!l"Z"dddZ#dZ$dZ%dZ&ddZ'e(djoy,eiei)d ddd g\Z*Z+Wn%ei,j oZ-e'd e-nXxRe*D]J\Z.Z/e.d d fjoe'dne.d d fjo d Znq Wye0ei)d Z1WneGHei2d nXe1ndS(s Test script for exercising various catalog features under load Usage: cd lib/python python Products/ZCatalog/regressiontests/loadmail.py command args where each command has its own command-line arguments that it expects. Note that all of the commands operate on the Zope database, typically var/Data.fs. Note that this script uses the proc file system to get memory size. Many of the commands output performance statisics on lines that look like:: 11.3585170507 0.06 2217781L 7212 where the numbers are: - clock time in seconds - cpu time used by the main thread, in seconds, - Database size growth over the test - Memory growth during the test (if the proc file system is available). Commands: base mbox [max] Build a base database by: - Deleting ../../var/Data.fs - Starting Zope - Adding a top-level folder names 'mail' - Reading up to max messages from the Unix mailbox file, mbox and adding them as documents to the mail folder. index [threshold] Index all of the DTML documents in the database, committing sub-transactions after each threshold objects. threshold defaults to 1000. If the threshold is less than the number of messages, then the size of the temporary sub-transaction commit file is output. inc mbox start end [threads [wait]] Incrementally index messages start to end in unix mailbox mbox. If the threads argument is supplied, then it specifies the number of threads to use. For example, with: python Products/ZCatalog/tests/loadmail.py inc mbox 0 200 2 One thread indexes messages 0 to 99 and another thread indexes messages 100 to 199. If wait is specified, then after each document is indexed, the thread sleeps a random number of seconds between 0 and 2*wait. The default wait is 0.25 seconds. For each thread, a line that looks like:: 3.41 (0, 1) is output, containing: - The cpu time - The number of ZODB transaction conflicts detected when reading - The number of ZODB transaction conflicts detected when committing edit edits deletes inserts threads wait Incrementally edit edits messages from mail. For each message, do a random number of word deletes between 0 and deletes * 2 and do a random number of inserts between 0 and inserts * 2. For each thread, a line that looks like:: 3.41 (0, 1) as described above. catdel Delete the entire catalog in one transaction. This is a fun one for storages that do reference counting garbage collection. pdebug command args Run one of the other commands in the Python debugger. sample suite of tests:: cd lib/python python Products/ZCatalog/regressiontests/loadmail.py base ~/zope.mbox 1000 python Products/ZCatalog/regressiontests/loadmail.py index 100 python Products/ZCatalog/regressiontests/loadmail.py inc ~/python-dev.mbox 0 10 2 python Products/ZCatalog/regressiontests/loadmail.py edit 10 10 10 2 python Products/ZCatalog/regressiontests/loadmail.py catdel Nit.(sstripsfindssplitslowersatoi(squotec Csd}}}}}z}|i}t}t i }t i }t ||}t i |}t i |}|i|}t|}Wd|dj o||||||n|||||fSXdS(s$Do something and measure it's impactN(tNonetttctsizetmemtrtdbtgetSizetVmSizettimetclocktapplytftargstreturnf( RR RRRRRRR((tI/data/zmath/zope/lib/python/Products/ZCatalog/regressiontests/loadmail.pytdos      c CsH|djo|ii}n|djo |i}n|it|d||t|}x|D]} t | } t | d}|djoqjnt| | }|djo d}nt | |d}d} do|djo d} n|djo|id| qjy|i||| WqjqjXqjWdS( Ntfilet:itsubjectttitleitstringtdate(tbodyRtmessagetfptreadtheaderstdesttmanage_addDTMLDocumenttstrtitdocthtstriptfindtltlowertnametvttypetmanage_changePropertiestmanage_addProperty( RRR RRR(R'R!R%R"R)((Rt loadmessages2       ic Csy dk}Wn|i|nX|iii||t||}t|}t i |}d}|i} x| o|djo||joPn|ddjoatoZd}tiidd jo d}ntii|||iftiin|o/|ddjoti|iii nt!|| ||d }|i} qyW||_"HtidS( Niids %s %s tTERMtdumbtemacss %s %s ii(R.R/(#t Products.BTreeFolder.BTreeFoldertProductsRtmanage_addFolderR't BTreeFoldertmanage_addBTreeFoldertgetattrtopentmboxR tmailboxt UnixMailboxtmbR tnextRtmaxtVERBOSEtfmttostenvirontgettsyststdouttwritettelltflusht transactiontcommitt_p_jart_cachetminimizeR,tnumber_of_messages( RR'R7R<R R>R:R R1R((Rtloadmails:        iicCsxdkl}dkl}dkl} dk}dk}d}}d}|i } | ii}t| i}x||jo|o|| d|n|ii} | id}t||} |ddjoatoZd} ti i!dd jo d } n|i"i#| ||i%f|i"i&nt'|}y4t)| | |||| |}|i+i,|Wn)|j o}|d }t.i/nnXyBt.i0|d }|i } | ii}t| i}Wn'|j o|d }t.i/nXd}}} | i1qvWto$|i"i#d ||i%fn|i"i&||fS(N(s ConflictError(ssleep(suniformit Applicationids %s %s R-R.R/s %s %s i(sdumbsemacs(2tZODB.POSExceptiont ConflictErrorR tsleeptrandomtuniformtZope2RBt rconflictst wconflictsR R:R;RRRRtlistRR<twaittDBR6tjartroottappR5R'tmdestR=R>R?R@RARCRDR RERFRtdidR,R!tcattcatalog_objectR(RGtabortRHtclose(R'R:R R<RXRUR\RPRQRR>RZRSR]RRBRVRTR R!RR(R^((RtloadincsZ               $ cCsytidWnnXdk}|i}ttidjottid}nd}t |i t |dtid|fGH|i i dS(Ns../../var/Data.fsiitmaili( R?tunlinkRTR\tlenRBtargvtatoiR<RRYRMRb(RTR<R\((Rtbases  'tREcBstZdZRS(NcOsdS(N((Rtkw((Rtredirects(t__name__t __module__Rl(((RRjscCsHt}d|ig|_|ii||dddgtidS(Nits DTML Document( RjRR\RdtPARENTSR_tmanage_catalogFoundItemsRGRH(R\R((Rtindexfs c CsdtidtrawttexttrandinttndelRRRRftninstchoicetwordstwordRtjoinR_R`R(RGRaRHRb(RRXRRRURR\RRRPRZRR(RVRTRR^R!RRR ((Rtincedit`sL$         cCsdk}dk}ttidttidttid}}}yttid} Wnd} d}n2Xyt tid}Wn d}nX|d}| djo t }n |i }|i } |i}|ii}|ii| i}t}ti}ti} h}g} xt| D]}|i }| djo|i"|d }n |d }dg}xUt%||joAt&i'd|}|i)|pd||<|i*|qqW| i*|||||f|fq?Wx3| D]+\}} }|t,|i t-| |fqWx!| D]\}} }|i"qHWti|}ti| } | i|}t|}|G| G|G|GH|i idS( Niiiiiiif0.25cCs|G|GH|idS(N(RRRR(RRRRRR((RRs cCs |G|GHdS(N(RR(RRRRRR((RRs(.RTRRhRBRgtneditRRRRXRR RRYRR\RdRLRIRbRRR RR RR RtalleditsRRR RRRRRRfRRRteditthas_keyRRRR(RRR\RRRRRRRRRRRRXRRTRRR RRRL((RRsh;                    &  cCsytdti}Wn dSnUXtd|i}|o5titi |ddd}ti |SndS(Ns/proc/%s/statusicCs|d djS(NisVmSize:(R%(R%((Rtsi( R6R?tgetpidR tfiltert readlinesR%RR>R#Rh(R R%((RR s$cCs$dk}tid=|iddS(Nisglobals()[sys.argv[1]]()(tpdbRBRgtrun(R((Rtpdebugs  RocCs6titIJ|oti|IJnti|dS(N(RBtstderrt__doc__tmsgtexittcode(RR((Rtusages t__main__thvthelptverboses-hs--helps-vs --verbose(3RtgetoptR8R RBR?RtpathtinsertRRR#R$R>R&RhturllibtquoteRGR=RRR,RMRcRiRjRrRRRRRRt ZODB.utilstu64RRR RRRmRgtoptsRterrorRtopttargtglobalsR R((RR RRRrR,R#R$RRRRcRRRjR>RRRRRRRR8RBRRhRRRR&RGR=R RRMRiR RR?R((Rt?psn - %    #6   % = ɸ - ?   ,