We are also working on a solution that prevents the deadlock by reordering the parts in our task that are involved in the deadlock. We are quite confident that this reordering approach is the silver bullet. But we have to wait for further tests to be 100% sure.

Subject Preview of TECHNOTE
Company name: IBM

Title *

Server deadlocks at startup when certain extension managers are loaded
Abstract *
This field is displayed on the search results webpage seen by customers, and effects search ranking and relevance.
When running Domino 8.51 with certain configurations the server can get into a semaphore deadlock on startup

Question *

Cause
The reason for this was some I/O performance improvements that were done in Domino 8.51 to have the log.nsf be opened as a “database group” as opposed to opening and closing it every time it needed to be accessed.
A typical NSD output showing the problem would have one thread waiting to lock a semaphore in InitializeNSFExt():

############################################################
### thread 1/3: [nOTAdmin: 0bec: 0bf0]
### FP=0×0011ed54, PC=0×7c82860c, SP=0×0011ece4
### stkbase=0×00130000, total stksize=81920, used stksize=70428
############################################################
[ 1] 0×7c82860c ntdll.KiFastSystemCallRet+0 (110,3e8,0,11ed98)
[ 2] 0×77e61c8d kernel32.WaitForSingleObject+18 (110,3e8,12c7f2c,12c7f00)
@[ 3] 0×600a1363 nnotes.WaitOnNativeSemaphore@16+659 (c7,3e8,0,0)
@[ 4] 0×6000132c nnotes.OSLockSemInt@8+284 (12c7f2c,1)
@[ 5] 0×600011fe nnotes.OSLockSem@4+14 (12c7f00)
@[ 6] 0×60001727 nnotes.OSLockWriteSem@4+55 (12c7efe)
@[ 7] 0×6088cdff nnotes.InitializeNSFExt@4+1343 (0)
@[ 8] 0×60013c67 nnotes.InitializeNSF@0+7 ()
@[ 9] 0×60028c35 nnotes.NetInit@0+3973 ()
@[10] 0×607bdb46 nnotes.LogAddTextVCtx@32+102 (1,4,60d3c7f0,400000,0,11feec,f10f10,ffffffff)
@[11] 0×600d53c6 nnotes.LogAddTextV@28+38 (1,4,60d3c7f0,400000,0,11ff14,f10f10)
@[12] 0×60122f19 nnotes.LogEventTextVExt@20+41 (0,40b090,400000,0,0)
@[13] 0×607bf14a nnotes.LogEventTextV@16+26 (40b090,400000,0,11ff88)
@[14] 0×601f13ac nnotes.AddInLogMessageText+28 (40b090,0,40b088,0)
[15] 0×00401f5c nOTAdmin+8028 (0,40b088,0,ece)
[16] 0×0040b090 nOTAdmin+45200 (53000002,40b0ec68,2cee800,f8a10000)

and a second thread holding the semaphore that the first thread was waiting on and waiting on the semaphore the first thread was holding in NetInit():

############################################################
### thread 1/3: [ nUpdate: 0b68: 0b78]
### FP=0×0012cc34, PC=0×7c82860c, SP=0×0012cbc4
### stkbase=0×00130000, total stksize=24576, used stksize=13372
############################################################
[ 1] 0×7c82860c ntdll.KiFastSystemCallRet+0 (138,3e8,0,12cc78)
[ 2] 0×77e61c8d kernel32.WaitForSingleObject+18 (138,3e8,1c0258c,1c02562)
@[ 3] 0×600a1363 nnotes.WaitOnNativeSemaphore@16+659 (cc,3e8,0,0)
@[ 4] 0×6000132c nnotes.OSLockSemInt@8+284 (1c0258c,1)
@[ 5] 0×600011fe nnotes.OSLockSem@4+14 (1c02562)
@[ 6] 0×60001727 nnotes.OSLockWriteSem@4+55 (1c02560)
@[ 7] 0×600282ab nnotes.NetInit@0+1531 ()
@[ 8] 0×601236db nnotes.BillingGetClass@4+43 (61069524)
@[ 9] 0×608d28e8 nnotes.NSFDbOpenExtended4@48+25784 (1001b55c,0,0,0,0,12efa4,f10f10,ffffffff,0,0,0,0)
@[10] 0×600271b1 nnotes.NSFDbOpenExtended3@48+65 (1001b55c,0,0,0,0,12f084,f10f10,ffffffff,0,0,0,0)
@[11] 0×60074246 nnotes.NSFDbOpenExtended2@44+54 (1001b55c,0,0,0,0,12f0bc,f10f10,ffffffff,0,0,0)
@[12] 0×60092bb6 nnotes.NSFDbOpenExtended@28+54 (1001b55c,0,0,0,0,12f0f0,f10f10)
@[13] 0×600abb4a nnotes.NSFDbOpen@8+26 (1001b55c,12f164)
[14] 0×10002cd2 nnem.SetPASEnabledState+6914 (1,3a4598,3a4270,10001106)
[15] 0×10004f72 nnem.SetPASEnabledState+15778 (10001000,1,1116cc0,7ffd000e)
@[16] 0×6088d88c nnotes.InitializeNSFExt@4+4044 (2dc05)
@[17] 0×60013c67 nnotes.InitializeNSF@0+7 ()
@[18] 0×601f1a2e nnotes.AddInSDKInit@8+30 (1,385244)
@[19] 0×0040c7b2 nUpdate.NotesMain@8+18 (1,385244)

Content *
Until the problem has been resolved customers experiencing this problem can set the notes.ini parameter DISABLE_LOG_DBGROUP=1 on all servers experiencing the problem. This ini setting will cause Domino to operate the way it did in respect to log.nsf prior to 8.51.
*IBMTasks >IBMTopics
Troubleshoot > Problem Resolution

Feedback response number JPAI7XSKDG created by John Paganetti on 11/14/2009