TIBCO EMS FT pair on Oracle Database 11gR2 shared storage

Related article: Read this also

In the previous article I wrote about setting up a TIBCO EMS fault tolerant pair that uses NFSv4 shared folder to store shared states. This article describes the step to set up a database-based shared state datastore.

The end-state is depicted in the following deployment diagram.

This is a 2-in-1 article. One part is to set up the required conf files and set up the Oracle Database 11gR2 for as datastore. The other part is to configure 2 separate tibemsd for the FT pairs.

1) Install hibernate package on the host machine
2) Configure tibemsd-db.conf
3) Configure stores-db.conf
4) Create EMS schema using export tool
5) Test it!

1) Install hibernate package

I will use hibernate 3.2.5 from the package hibernate_3.2.5_linux24gl23_x86.zip

1.1 Unzip the package into you temp folder.
1.2 run TIBCOUniversalInstaller-lnx-x86.bin as tibadmin, created earlier in this article.
1.3 Follow the steps in the screens below.

2) Configure tibemsd-db.conf

tibemsd-db.conf is a sample file located in $EMS_HOME/tibco/ems/5.1/sample/config folder. Copy this file from $EMS_HOME/tibco/ems/5.1/sample/config into your shared state directory.

Edit the tibemsd-db.conf and modify the following lines.

users                   = /home/tibadmin/ems_ft_cfg/users.conf
groups                  = /home/tibadmin/ems_ft_cfg/groups.conf
topics                  = /home/tibadmin/ems_ft_cfg/topics.conf
queues                  = /home/tibadmin/ems_ft_cfg/queues.conf
acl_list                = /home/tibadmin/ems_ft_cfg/acl.conf
factories               = /home/tibadmin/ems_ft_cfg/factories.conf
routes                  = /home/tibadmin/ems_ft_cfg/routes.conf
bridges                 = /home/tibadmin/ems_ft_cfg/bridges.conf
transports              = /home/tibadmin/ems_ft_cfg/transports.conf
tibrvcm                 = /home/tibadmin/ems_ft_cfg/tibrvcm.conf
durables                = /home/tibadmin/ems_ft_cfg/durables.conf
channels                = /home/tibadmin/ems_ft_cfg/channels.conf
stores                  = /home/tibadmin/ems_ft_cfg/stores-db.conf

dbstore_classpath       = /home/tibadmin/tibco/components/eclipse/plugins/com.tibco.tpcl.org.hibernate_3.2.5.001/hibernate3.jar:/home/tibadmin/tibco/components/eclipse/plugins/com.tibco.tpcl.org.com.mchange.c3p0_0.9.1.001/c3p0-0.9.1.jar:/home/tibadmin/tibco/ems/5.1/bin/antlr-2.7.6.jar:/home/tibadmin/tibco/ems/5.1/bin/asm-attrs.jar:/home/tibadmin/tibco/ems/5.1/bin/asm.jar:/home/tibadmin/tibco/ems/5.1/bin/cglib-2.1.3.jar:/home/tibadmin/tibco/ems/5.1/bin/commons-collections-2.1.1.jar:/home/tibadmin/tibco/ems/5.1/bin/commons-logging-1.0.4.jar:/home/tibadmin/tibco/ems/5.1/bin/dom4j-1.6.1.jar:/home/tibadmin/tibco/ems/5.1/bin/ehcache-1.2.3.jar:/home/tibadmin/tibco/ems/5.1/bin/jta.jar:/home/tibadmin/tibco/ems/5.1/lib/ojdbc5.jar

dbstore_driver_name     = oracle.jdbc.driver.OracleDriver
dbstore_driver_dialect  = org.hibernate.dialect.Oracle10gDialect
jre_library             = /opt/java/jre/lib/i386/server/libjvm.so
jre_option   = -Xmx256m

3) Configure stores-db.conf

stores-db.conf is a sample stores.conf file located in $EMS_HOME/tibco/ems/5.1/sample/config folder. Copy this file from $EMS_HOME/tibco/ems/5.1/sample/config into your shared state directory.

[$sys.failsafe]
type=dbstore
dbstore_driver_url=jdbc:oracle:thin:emsfs/emsfs@//oradb11g01:1521/orcl
dbstore_driver_username=emsfs
dbstore_driver_password=emsfs

[$sys.meta]
type=dbstore
dbstore_driver_url=jdbc:oracle:thin:emsmeta/emsmeta@//oradb11g01:1521/orcl
dbstore_driver_username=emsmeta
dbstore_driver_password=emsmeta

#store the non failsafe message in NFS drive.
[$sys.nonfailsafe]
  type=file
  file=/home/tibadmin/ems_ft_cfg/datastore/async-msgs.db
  mode=async
  file_crc=disabled

4) Create EMS schema using export tool

TIBCO EMS comes with a tool to create the required database schema for shared states and message storage.

Before you use the tool, ensure the TIBCO EMS host has the required database connectivity. Though you only need the Oracle JDBC java library (ojdbc5.jar) to be in the classpath, I suggest that you install the entire package of Oracle Instant Client for Linux (instantclient-basic-linux32-11.2.0.2.0.zip).

Next, create an oracle db user for each EMS data store of type dbstore.

for meta dbstore

GRANT CONNECT,RESOURCE,UNLIMITED TABLESPACE TO emsmeta IDENTIFIED BY emsmeta;

ALTER USER emsmeta DEFAULT TABLESPACE USERS;

ALTER USER emsmeta TEMPORARY TABLESPACE TEMP;

for failsafe dbstore

GRANT CONNECT,RESOURCE,UNLIMITED TABLESPACE TO emsfs IDENTIFIED BY emsfs ;

ALTER USER emsfs DEFAULT TABLESPACE USERS;

ALTER USER emsfs TEMPORARY TABLESPACE TEMP;

Note: The userid and password must be same is the one specified in the store-db.conf file in step 3.

After you have created the EMS oracle user, execute the tibemsd_util.jar as follow. The util is located under $EMS_HOME/bin where EMS_HOME is /home/tibadmin/tibco/ems/5.1 in my case.

java -jar tibemsd_util.jar -tibemsdconf /home/tibadmin/ems_ft_cfg/tibemsd-db1.conf -createall -export

You should see the following output on the stdout.

drop table ACK_RECS cascade constraints
drop table CONNECTION_RECS cascade constraints
drop table CONSUMER_RECS cascade constraints
drop table CONSUMER_RECS_IMPSELMAP cascade constraints
drop table EMS_HBLOCK_TABLE cascade constraints
drop table EMS_MESSAGES cascade constraints
drop table EMS_SYS_RECORDS_TABLE cascade constraints
drop table PRODUCER_RECS cascade constraints
drop table PURGE_RECS cascade constraints
drop table SESSION_RECS cascade constraints
drop table TXN_RECS cascade constraints
drop table VALID_MSGS_RECORD cascade constraints
drop table VALID_MSGS_RECORD_HELDMSGS cascade constraints
drop table VALID_MSGS_RECORD_HOLDMSGS cascade constraints
drop table VALID_MSGS_RECORD_NOHOLDMSGS cascade constraints
drop table ZONE_RECS cascade constraints
drop sequence hibernate_sequence
create table ACK_RECS (ACK_RECORD_ID number(19,0) not null, CONSUMER_ID number(19,0), SEQNO number(19,0), DISCARD_ACK_FLAG number(1,0), TXN_ID number(19,0), SESS_ID number(19,0), CONN_ID number(19,0), primary key (ACK_RECORD_ID))
create table CONNECTION_RECS (STORE_ID number(19,0) not null, CONNECTION_ID number(19,0), CONNECTION_TYPE number(10,0), CLIENT_ID varchar2(256 char), CONNECTION_USER varchar2(256 char), NEW_TEMP_DESTS_FLAG number(1,0), primary key (STORE_ID))
create table CONSUMER_RECS (STORE_ID number(19,0) not null, CONSUMER_ID number(19,0), SESSION_ID number(19,0), CONNECTION_ID number(19,0), DURABLE_NAME varchar2(255 char), SUBSCRIPTION_NAME varchar2(255 char), CLIENT_ID varchar2(255 char), ZONE_ID number(10,0), SELECTOR_STRING varchar2(4000 char), SELECTOR_BYTES long raw, NO_LOCAL number(1,0), CONNECTION_CONSUMER number(1,0), DESTINATION varchar2(255 char), DEST_TYPE number(3,0), CONSUMER_MSG_SEQNO number(19,0), MULTICAST_FLAG number(1,0), primary key (STORE_ID))
create table CONSUMER_RECS_IMPSELMAP (StoreId number(19,0) not null, HASHMAP_VALUE varchar2(255 char), HASHMAP_KEY varchar2(255 char) not null, primary key (StoreId, HASHMAP_KEY))
create table EMS_HBLOCK_TABLE (id number(19,0) not null, last_update timestamp not null, server_id varchar2(255 char) not null, JDBCURL varchar2(4000 char) not null, primary key (id))
create table EMS_MESSAGES (STORE_ID number(19,0) not null, MESSAGE_SEQNO number(19,0), TYPE number(3,0), PRIORITY number(3,0), DELIVERYMODE number(3,0), REDELIVERED number(3,0), DESTINATION varchar2(255 char), DESTINATION_TYPE number(3,0), EXPIRATION number(19,0), TIMESTAMP number(19,0), REPLYTO varchar2(255 char), REPLYTO_TYPE number(3,0), USERTYPE varchar2(255 char), MSGID varchar2(255 char), CORRELATIONID varchar2(255 char), COMPRESSED number(1,0), CLIENTFLAGS number(19,0), ENCODING varchar2(255 char), ENCODED_PROPERTIES raw(1024), ENCODED_SUPPRESS_CONSIDS raw(1024), TXNID number(19,0), ZONEID number(19,0), ROUTESRC varchar2(255 char), ROUTESEQNO number(19,0), ROUTECONSID number(19,0), PRODUCERID number(19,0), DELETED number(1,0) not null, MESSAGE_SIZE number(19,0), SMALL_MESSAGE_BODY raw(1024), LARGE_MESSAGE_BODY blob, primary key (STORE_ID))
create table EMS_SYS_RECORDS_TABLE (SYSTEM_REC_ID number(19,0) not null, VERSION varchar2(255 char), EMS_START_TIME varchar2(255 char), EMS_UPDATE_TIME varchar2(255 char), primary key (SYSTEM_REC_ID))
create table PRODUCER_RECS (STORE_ID number(19,0) not null, PRODUCER_ID number(19,0), SESSION_ID number(19,0), CONNECTION_ID number(19,0), DESTINATION varchar2(255 char), DEST_TYPE number(3,0), primary key (STORE_ID))
create table PURGE_RECS (PURGE_RECORD_ID number(19,0) not null, DEST_NAME varchar2(255 char), DEST_TYPE number(3,0), SEQ_NO number(19,0), primary key (PURGE_RECORD_ID))
create table SESSION_RECS (STORE_ID number(19,0) not null, SESSION_ID number(19,0), CONNECTION_ID number(19,0), ACKNOWLEDGE_MODE number(10,0), TRANSACTED number(1,0), IS_XA number(1,0), MSGID_PREFIX varchar2(255 char), primary key (STORE_ID))
create table TXN_RECS (TXNREC_STORE_ID number(19,0) not null, SESS_ID number(19,0), TXN_ID number(19,0), RECORD_ID number(19,0), TXN_STATE number(10,0), XID raw(1024), primary key (TXNREC_STORE_ID))
create table VALID_MSGS_RECORD (VALID_MSG_RECORD_ID number(19,0) not null, ACTIVE_HI_SEQ number(19,0), SMALL_MESSAGE_BODY raw(1024), LARGE_MESSAGE_BODY blob, primary key (VALID_MSG_RECORD_ID))
create table VALID_MSGS_RECORD_HELDMSGS (id number(19,0) not null, HELD_MSGS number(19,0), listIndex number(10,0) not null, primary key (id, listIndex))
create table VALID_MSGS_RECORD_HOLDMSGS (id number(19,0) not null, HOLD_MSGS number(19,0), listIndex number(10,0) not null, primary key (id, listIndex))
create table VALID_MSGS_RECORD_NOHOLDMSGS (id number(19,0) not null, NO_HOLD_MSGS number(19,0), listIndex number(10,0) not null, primary key (id, listIndex))
create table ZONE_RECS (STORE_ID number(19,0) not null, ZONE_ID number(10,0), ZONE_NAME varchar2(255 char), ZONE_TYPE number(5,0), primary key (STORE_ID))
alter table CONSUMER_RECS_IMPSELMAP add constraint FKD1B84D7B3B9C4484 foreign key (StoreId) references CONSUMER_RECS
create index DEL_IDX on EMS_MESSAGES (DELETED)
alter table VALID_MSGS_RECORD_HELDMSGS add constraint FK5908794BC44ABBE5 foreign key (id) references VALID_MSGS_RECORD
alter table VALID_MSGS_RECORD_HOLDMSGS add constraint FK6A06C9D5C44ABBE5 foreign key (id) references VALID_MSGS_RECORD
alter table VALID_MSGS_RECORD_NOHOLDMSGS add constraint FK610EB196C44ABBE5 foreign key (id) references VALID_MSGS_RECORD
create sequence hibernate_sequence

Note: For some reasons the required tables are not created in database, probably due to missing/incorrect configuration of hibernate ddl properties.

Anyway, I have converted the output above into a sql script and simply run them with sqlplus.

[oradb11g01 $] sqlplus emsmeta/emsmeta @meta.sql
[oradb11g01 $] sqlplus emsfs/emsfs @fs.sql

This following paragraphs describe the step to setting up a FT pair.

Make 2 copies of the tibemsd-db.conf files made in step 1. Call them tibemsd-db1.conf and tibemsd-db2.conf respectively.

In both files, make sure the stores parameter points to the same conf file.

stores                  = /home/tibadmin/ems_ft_cfg/stores-db.conf

In each files, do the following:

tibemsd-db1.conf

listen = tcp://tibems01:7222
ft_active = tcp://tibems02:7222

tibemsd-db2.conf

listen = tcp://tibems02:7222
ft_active = tcp://tibems01:7222

The config file referencing can be visualised in the following picture.

Start EMS on both servers using tibemsd-db1.conf and tibemsd-db2.conf, in any sequence. The EMS FT pair on Oracle Database 11g R2 are now operational.

tibadmin@tibems02:~/ems_ft_cfg$ startEMS-db.sh 

TIBCO Enterprise Message Service.
Copyright 2003-2010 by TIBCO Software Inc.
All rights reserved.

Version 5.1.5 V3 3/29/2010

2011-04-19 06:02:33.785 Process started from '/home/tibadmin/tibco/ems/5.1/bin/tibemsd'.
2011-04-19 06:02:33.785 Process Id: 1629
2011-04-19 06:02:33.786 Hostname: tibems02
2011-04-19 06:02:33.786 Hostname IP address: 172.16.10.11
2011-04-19 06:02:33.786 Detected IP interface: 127.0.0.1 (lo)
2011-04-19 06:02:33.786 Detected IP interface: 172.16.10.11 (eth0)
2011-04-19 06:02:33.786 Detected IP interface: 192.168.50.11 (eth1)
2011-04-19 06:02:33.786 Reading configuration from '/home/tibadmin/ems_ft_cfg/tibemsd-db2.conf'.
2011-04-19 06:02:34.891 Java Version 1.6.0_24
2011-04-19 06:02:34.904 Server name: 'EMS-SERVER'.
2011-04-19 06:02:34.904 Storage Location: '.'.
2011-04-19 06:02:34.904 Routing is disabled.
2011-04-19 06:02:34.904 Authorization is disabled.
2011-04-19 06:02:37.451 Active server 'tcp://tibems01:7222' not found.
2011-04-19 06:02:37.451 Continuing as active server.
2011-04-19 06:02:37.456 Accepting connections on tcp://tibems02:7222.
2011-04-19 06:02:37.456 Recovering state, please wait.
2011-04-19 06:04:13.561 Store '$sys.failsafe' locked by 'EMS-SERVER'
2011-04-19 06:04:13.561 Creating system record for store '$sys.failsafe' ...
2011-04-19 06:04:13.660 Store '$sys.meta' locked by 'EMS-SERVER'
2011-04-19 06:04:13.660 Creating system record for store '$sys.meta' ...
2011-04-19 06:04:14.342 Server is active.
2011-04-19 06:06:00.481 Backup server 'EMS-SERVER@tibems01' has connected.

tibadmin@tibems01:~/ems_ft_cfg$ startEMS-db.sh 

TIBCO Enterprise Message Service.
Copyright 2003-2010 by TIBCO Software Inc.
All rights reserved.

Version 5.1.5 V3 3/29/2010

2011-04-19 06:05:58.102 Process started from '/home/tibadmin/tibco/ems/5.1/bin/tibemsd'.
2011-04-19 06:05:58.103 Process Id: 10071
2011-04-19 06:05:58.103 Hostname: tibems01
2011-04-19 06:05:58.103 Hostname IP address: 172.16.10.10
2011-04-19 06:05:58.103 Detected IP interface: 127.0.0.1 (lo)
2011-04-19 06:05:58.103 Detected IP interface: 172.16.10.10 (eth0)
2011-04-19 06:05:58.103 Detected IP interface: 192.168.50.10 (eth1)
2011-04-19 06:05:58.103 Reading configuration from '/home/tibadmin/ems_ft_cfg/tibemsd-db1.conf'.
2011-04-19 06:05:58.410 Java Version 1.6.0_24
2011-04-19 06:05:58.419 Server name: 'EMS-SERVER'.
2011-04-19 06:05:58.419 Storage Location: '.'.
2011-04-19 06:05:58.419 Routing is disabled.
2011-04-19 06:05:58.420 Authorization is disabled.
2011-04-19 06:06:00.401 Accepting connections on tcp://tibems01:7222.
2011-04-19 06:06:00.401 Server is in standby mode for 'tcp://tibems02:7222'.

Lastly, I have a Gigabit connection between the EMS host and the DB host, so I will just let the shared state traffic to pass through my MSG_VLAN. For detail of the Laboratory network topology, refer to this article.

Cheers,