Tuesday, July 24, 2012

Commit events are not broadcast because no JMS Transport is available

During weblogic server domain creation,if you have enabled RDBMS Security Store and you see these errors during Weblogic Server (10.3.4) startup,

<Jul 31, 2012 7:27:14 PM GMT+10:00> <Error> <Security> <BEA-000000> <Commit events are not broadcast because no JMS Transport is available. Remote L2 caches will be out of sync.> 
<Jul 31, 2012 7:27:14 PM GMT+10:00> <Error> <Security> <BEA-000000> <Commit events are not broadcast because no JMS Transport is available. Remote L2 caches will be out of sync.> 
<Jul 31, 2012 7:27:14 PM GMT+10:00> <Error> <Security> <BEA-000000> <Commit events are not broadcast because no JMS Transport is available. Remote L2 caches will be out of sync.> 
<Jul 31, 2012 7:27:14 PM GMT+10:00> <Error> <Security> <BEA-000000> <Commit events are not broadcast because no JMS Transport is available. Remote L2 caches will be out of sync.> 
<Jul 31, 2012 7:27:14 PM GMT+10:00> <Error> <Security> <BEA-000000> <Commit events are not broadcast because no JMS Transport is available. Remote L2 caches will be out of sync.> 
<Jul 31, 2012 7:27:14 PM GMT+10:00> <Error> <Security> <BEA-000000> <Commit events are not broadcast because no JMS Transport is available. Remote L2 caches will be out of sync.>

...here is the reason why.


The error messages are caused by the server's inability to locate the p13n.security.RDBMSSecurityStoreTopic from the JNDI tree during startup. Contrary to some believe that this can be solved by configuring the "Delivery Failure" parameters such as "Redelivery Delay Override" and "Redelivery Limit" of the Topic, this is really not the case of delivery failure. According to the Oracle Weblogic documentations,


"WebLogic Server security services are always initialized before server resources, applications, and startup classes are deployed. ..."


(Well, the part "because no JMS Transport is available" is not very helpful though it is true that the transport is not yet established at that the time the error occurs. The server assumes that if the resource is registered in the JNDI, then it is deemed to be available).


Provided here a section of the debug trace obtain during server startup, which shows that the JNDILookup has failed to find 'p13n' entry



####<Jul 31, 2012 7:27:14 PM GMT+10:00> <Debug> <JNDIResolution> <appsvr02> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1343726834241> <BEA-000000> <--- failed to find p13n>
####<Jul 31, 2012 7:27:14 PM GMT+10:00> <Debug> <WorkContext> <> <> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1343726834244> <BEA-000000> <get(weblogic.diagnostics.DiagnosticContext)>
####<Jul 31, 2012 7:27:14 PM GMT+10:00> <Debug> <WorkContext> <> <> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1343726834244> <BEA-000000> <get(weblogic.diagnostics.DiagnosticContext)>
####<Jul 31, 2012 7:27:14 PM GMT+10:00> <Error> <Security> <appsvr02> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1343726834244> <BEA-000000> <Commit events are not broadcast because no JMS Transport is available. Remote L2 caches will be out of sync.>

And only at a later time the p13n sub node was created in the JNDI tree.



####<Jul 31, 2012 7:27:15 PM GMT+10:00> <Debug> <JNDIResolution> <appsvr02> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1343726835596> <BEA-000000> <--- failed to find p13n>
####<Jul 31, 2012 7:27:15 PM GMT+10:00> <Debug> <WorkContext> <> <> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1343726835597> <BEA-000000> <get(weblogic.diagnostics.DiagnosticContext)>
####<Jul 31, 2012 7:27:15 PM GMT+10:00> <Debug> <WorkContext> <> <> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1343726835597> <BEA-000000> <get(weblogic.diagnostics.DiagnosticContext)>
####<Jul 31, 2012 7:27:15 PM GMT+10:00> <Debug> <JNDIResolution> <appsvr02> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1343726835597> <BEA-000000> <--- created sub node p13n weblogic.jndi.internal.ServerNamingNode@1f253d6>

About a second later at 1343726835597, the p13n sub node was created in the JNDI tree.

I can't see there is any way to get around this non-issue and the conclusion is that it is safe to ignore the error message because the security service will retry until the JMS resource is established. Even if we try to use an external JNDI service provider (which is not recommended by Oracle), we still need to map the local and the foreign entries but the local entries will not be available after the security services are initialized. AFAIK, there is no 'best-practise' that can remove this startup error because the JNDI tree is simply not populated with the JMS resource reference at the time of security service initialization. Please correct me if I am wrong. Happy to hear from your experience.