Found this in my inbox today:
“Thanks to help from the IBM development team and support the issue some customers have been experiencing when running custom server tasks and Extension Manager on Domino 8.5.1. has been resolved.
The root cause of this issue was that IBM has introduced some I/O performance improvements for logging with 8.5.1, which unfortunately had the sideeffect that servers were haulting when running custom server tasks … .
But the good news is that IBM introduced a way of turning this new logging off and reverting to the way logging was done in 8.5.0.
So all you have to do is put in an extra line in the notes.ini of the server.
DISABLE_LOG_DBGROUP=1
After that everything works again.This has already been validated. The only adverse effect of this change is that logging will not be faster than it was in 8.5.0. We expect the issue to be fully resolved in the next service release of Domino.”
Is there a technote by any chance with this? We have a number of products that are implemented as server add-ins and extension manager hooks. However, we have yet to see any issues or had customers report issues like this and I’m wondering if there are other factors involved….?
Sounds like a startup problem we had with Domino 8.5.1. Still have to check if DISABLE_LOG_DBGROUP=1 is a workaround for the problem. Will give you feedback then.
A technote would be helpful.
TECHNOTE preview
Subject Preview of TECHNOTE
Company name: IBM
Title *
Server deadlocks at startup when certain extension managers are loaded
Abstract *
This field is displayed on the search results webpage seen by customers, and effects search ranking and relevance.
When running Domino 8.51 with certain configurations the server can get into a semaphore deadlock on startup
Question *
Cause
The reason for this was some I/O performance improvements that were done in Domino 8.51 to have the log.nsf be opened as a “database group” as opposed to opening and closing it every time it needed to be accessed.
A typical NSD output showing the problem would have one thread waiting to lock a semaphore in InitializeNSFExt():
############################################################
### thread 1/3: [nOTAdmin: 0bec: 0bf0]
### FP=0x0011ed54, PC=0x7c82860c, SP=0x0011ece4
### stkbase=0x00130000, total stksize=81920, used stksize=70428
############################################################
[ 1] 0x7c82860c ntdll.KiFastSystemCallRet+0 (110,3e8,0,11ed98)
[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (110,3e8,12c7f2c,12c7f00)
@[ 3] 0x600a1363 nnotes.WaitOnNativeSemaphore@16+659 (c7,3e8,0,0)
@[ 4] 0x6000132c nnotes.OSLockSemInt@8+284 (12c7f2c,1)
@[ 5] 0x600011fe nnotes.OSLockSem@4+14 (12c7f00)
@[ 6] 0x60001727 nnotes.OSLockWriteSem@4+55 (12c7efe)
@[ 7] 0x6088cdff nnotes.InitializeNSFExt@4+1343 (0)
@[ 8] 0x60013c67 nnotes.InitializeNSF@0+7 ()
@[ 9] 0x60028c35 nnotes.NetInit@0+3973 ()
@[10] 0x607bdb46 nnotes.LogAddTextVCtx@32+102 (1,4,60d3c7f0,400000,0,11feec,f10f10,ffffffff)
@[11] 0x600d53c6 nnotes.LogAddTextV@28+38 (1,4,60d3c7f0,400000,0,11ff14,f10f10)
@[12] 0x60122f19 nnotes.LogEventTextVExt@20+41 (0,40b090,400000,0,0)
@[13] 0x607bf14a nnotes.LogEventTextV@16+26 (40b090,400000,0,11ff88)
@[14] 0x601f13ac nnotes.AddInLogMessageText+28 (40b090,0,40b088,0)
[15] 0x00401f5c nOTAdmin+8028 (0,40b088,0,ece)
[16] 0x0040b090 nOTAdmin+45200 (53000002,40b0ec68,2cee800,f8a10000)
and a second thread holding the semaphore that the first thread was waiting on and waiting on the semaphore the first thread was holding in NetInit():
############################################################
### thread 1/3: [ nUpdate: 0b68: 0b78]
### FP=0x0012cc34, PC=0x7c82860c, SP=0x0012cbc4
### stkbase=0x00130000, total stksize=24576, used stksize=13372
############################################################
[ 1] 0x7c82860c ntdll.KiFastSystemCallRet+0 (138,3e8,0,12cc78)
[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (138,3e8,1c0258c,1c02562)
@[ 3] 0x600a1363 nnotes.WaitOnNativeSemaphore@16+659 (cc,3e8,0,0)
@[ 4] 0x6000132c nnotes.OSLockSemInt@8+284 (1c0258c,1)
@[ 5] 0x600011fe nnotes.OSLockSem@4+14 (1c02562)
@[ 6] 0x60001727 nnotes.OSLockWriteSem@4+55 (1c02560)
@[ 7] 0x600282ab nnotes.NetInit@0+1531 ()
@[ 8] 0x601236db nnotes.BillingGetClass@4+43 (61069524)
@[ 9] 0x608d28e8 nnotes.NSFDbOpenExtended4@48+25784 (1001b55c,0,0,0,0,12efa4,f10f10,ffffffff,0,0,0,0)
@[10] 0x600271b1 nnotes.NSFDbOpenExtended3@48+65 (1001b55c,0,0,0,0,12f084,f10f10,ffffffff,0,0,0,0)
@[11] 0x60074246 nnotes.NSFDbOpenExtended2@44+54 (1001b55c,0,0,0,0,12f0bc,f10f10,ffffffff,0,0,0)
@[12] 0x60092bb6 nnotes.NSFDbOpenExtended@28+54 (1001b55c,0,0,0,0,12f0f0,f10f10)
@[13] 0x600abb4a nnotes.NSFDbOpen@8+26 (1001b55c,12f164)
[14] 0x10002cd2 nnem.SetPASEnabledState+6914 (1,3a4598,3a4270,10001106)
[15] 0x10004f72 nnem.SetPASEnabledState+15778 (10001000,1,1116cc0,7ffd000e)
@[16] 0x6088d88c nnotes.InitializeNSFExt@4+4044 (2dc05)
@[17] 0x60013c67 nnotes.InitializeNSF@0+7 ()
@[18] 0x601f1a2e nnotes.AddInSDKInit@8+30 (1,385244)
@[19] 0x0040c7b2 nUpdate.NotesMain@8+18 (1,385244)
Content *
Until the problem has been resolved customers experiencing this problem can set the notes.ini parameter DISABLE_LOG_DBGROUP=1 on all servers experiencing the problem. This ini setting will cause Domino to operate the way it did in respect to log.nsf prior to 8.51.
*IBMTasks >IBMTopics
Troubleshoot > Problem Resolution
Feedback response number JPAI7XSKDG created by John Paganetti on 11/14/2009
First tests showed that this DISABLE_LOG_DBGROUP notes.ini parameter is another workaround that works for us. Delaying the start-up of our server tasks for some seconds is another workaround we are aware of.
We are also working on a solution that prevents the deadlock by reordering the parts in our task that are involved in the deadlock. We are quite confident that this reordering approach is the silver bullet. But we have to wait for further tests to be 100% sure.