DISABLE_LOG_DBGROUP
November 13, 2009 – 7:17 amFound this in my inbox today:
“Thanks to help from the IBM development team and support the issue some customers have been experiencing when running custom server tasks and Extension Manager on Domino 8.5.1. has been resolved.
The root cause of this issue was that IBM has introduced some I/O performance improvements for logging with 8.5.1, which unfortunately had the sideeffect that servers were haulting when running custom server tasks … .
But the good news is that IBM introduced a way of turning this new logging off and reverting to the way logging was done in 8.5.0.
So all you have to do is put in an extra line in the notes.ini of the server.
DISABLE_LOG_DBGROUP=1
After that everything works again.This has already been validated. The only adverse effect of this change is that logging will not be faster than it was in 8.5.0. We expect the issue to be fully resolved in the next service release of Domino.”
4 Responses to “DISABLE_LOG_DBGROUP”
Is there a technote by any chance with this? We have a number of products that are implemented as server add-ins and extension manager hooks. However, we have yet to see any issues or had customers report issues like this and I’m wondering if there are other factors involved….?
By Craig Schumann on Nov 13, 2009
Sounds like a startup problem we had with Domino 8.5.1. Still have to check if DISABLE_LOG_DBGROUP=1 is a workaround for the problem. Will give you feedback then.
A technote would be helpful.
By Reinhold on Nov 16, 2009
TECHNOTE preview
Subject Preview of TECHNOTE
Company name: IBM
Title *
Server deadlocks at startup when certain extension managers are loaded
Abstract *
This field is displayed on the search results webpage seen by customers, and effects search ranking and relevance.
When running Domino 8.51 with certain configurations the server can get into a semaphore deadlock on startup
Question *
Cause
The reason for this was some I/O performance improvements that were done in Domino 8.51 to have the log.nsf be opened as a “database group” as opposed to opening and closing it every time it needed to be accessed.
A typical NSD output showing the problem would have one thread waiting to lock a semaphore in InitializeNSFExt():
############################################################
### thread 1/3: [nOTAdmin: 0bec: 0bf0]
### FP=0×0011ed54, PC=0×7c82860c, SP=0×0011ece4
### stkbase=0×00130000, total stksize=81920, used stksize=70428
############################################################
[ 1] 0×7c82860c ntdll.KiFastSystemCallRet+0 (110,3e8,0,11ed98)
[ 2] 0×77e61c8d kernel32.WaitForSingleObject+18 (110,3e8,12c7f2c,12c7f00)
@[ 3] 0×600a1363 nnotes.WaitOnNativeSemaphore@16+659 (c7,3e8,0,0)
@[ 4] 0×6000132c nnotes.OSLockSemInt@8+284 (12c7f2c,1)
@[ 5] 0×600011fe nnotes.OSLockSem@4+14 (12c7f00)
@[ 6] 0×60001727 nnotes.OSLockWriteSem@4+55 (12c7efe)
@[ 7] 0×6088cdff nnotes.InitializeNSFExt@4+1343 (0)
@[ 8] 0×60013c67 nnotes.InitializeNSF@0+7 ()
@[ 9] 0×60028c35 nnotes.NetInit@0+3973 ()
@[10] 0×607bdb46 nnotes.LogAddTextVCtx@32+102 (1,4,60d3c7f0,400000,0,11feec,f10f10,ffffffff)
@[11] 0×600d53c6 nnotes.LogAddTextV@28+38 (1,4,60d3c7f0,400000,0,11ff14,f10f10)
@[12] 0×60122f19 nnotes.LogEventTextVExt@20+41 (0,40b090,400000,0,0)
@[13] 0×607bf14a nnotes.LogEventTextV@16+26 (40b090,400000,0,11ff88)
@[14] 0×601f13ac nnotes.AddInLogMessageText+28 (40b090,0,40b088,0)
[15] 0×00401f5c nOTAdmin+8028 (0,40b088,0,ece)
[16] 0×0040b090 nOTAdmin+45200 (53000002,40b0ec68,2cee800,f8a10000)
and a second thread holding the semaphore that the first thread was waiting on and waiting on the semaphore the first thread was holding in NetInit():
############################################################
### thread 1/3: [ nUpdate: 0b68: 0b78]
### FP=0×0012cc34, PC=0×7c82860c, SP=0×0012cbc4
### stkbase=0×00130000, total stksize=24576, used stksize=13372
############################################################
[ 1] 0×7c82860c ntdll.KiFastSystemCallRet+0 (138,3e8,0,12cc78)
[ 2] 0×77e61c8d kernel32.WaitForSingleObject+18 (138,3e8,1c0258c,1c02562)
@[ 3] 0×600a1363 nnotes.WaitOnNativeSemaphore@16+659 (cc,3e8,0,0)
@[ 4] 0×6000132c nnotes.OSLockSemInt@8+284 (1c0258c,1)
@[ 5] 0×600011fe nnotes.OSLockSem@4+14 (1c02562)
@[ 6] 0×60001727 nnotes.OSLockWriteSem@4+55 (1c02560)
@[ 7] 0×600282ab nnotes.NetInit@0+1531 ()
@[ 8] 0×601236db nnotes.BillingGetClass@4+43 (61069524)
@[ 9] 0×608d28e8 nnotes.NSFDbOpenExtended4@48+25784 (1001b55c,0,0,0,0,12efa4,f10f10,ffffffff,0,0,0,0)
@[10] 0×600271b1 nnotes.NSFDbOpenExtended3@48+65 (1001b55c,0,0,0,0,12f084,f10f10,ffffffff,0,0,0,0)
@[11] 0×60074246 nnotes.NSFDbOpenExtended2@44+54 (1001b55c,0,0,0,0,12f0bc,f10f10,ffffffff,0,0,0)
@[12] 0×60092bb6 nnotes.NSFDbOpenExtended@28+54 (1001b55c,0,0,0,0,12f0f0,f10f10)
@[13] 0×600abb4a nnotes.NSFDbOpen@8+26 (1001b55c,12f164)
[14] 0×10002cd2 nnem.SetPASEnabledState+6914 (1,3a4598,3a4270,10001106)
[15] 0×10004f72 nnem.SetPASEnabledState+15778 (10001000,1,1116cc0,7ffd000e)
@[16] 0×6088d88c nnotes.InitializeNSFExt@4+4044 (2dc05)
@[17] 0×60013c67 nnotes.InitializeNSF@0+7 ()
@[18] 0×601f1a2e nnotes.AddInSDKInit@8+30 (1,385244)
@[19] 0×0040c7b2 nUpdate.NotesMain@8+18 (1,385244)
Content *
Until the problem has been resolved customers experiencing this problem can set the notes.ini parameter DISABLE_LOG_DBGROUP=1 on all servers experiencing the problem. This ini setting will cause Domino to operate the way it did in respect to log.nsf prior to 8.51.
*IBMTasks >IBMTopics
Troubleshoot > Problem Resolution
Feedback response number JPAI7XSKDG created by John Paganetti on 11/14/2009
By John Paganetti on Nov 16, 2009
First tests showed that this DISABLE_LOG_DBGROUP notes.ini parameter is another workaround that works for us. Delaying the start-up of our server tasks for some seconds is another workaround we are aware of.
We are also working on a solution that prevents the deadlock by reordering the parts in our task that are involved in the deadlock. We are quite confident that this reordering approach is the silver bullet. But we have to wait for further tests to be 100% sure.
By Reinhold on Nov 16, 2009