In the previous article, I briefly wrote about setting a file-based shared state TIBCO EMS FT pair on Windows platform. This article shows you how to install TIBCO EMS on linux hosts for fault tolerance configuration. The installations will use NFSv4 mounted folder for the shared states.
TIBCO EMS 5.1 is supported on the following Unix/Linux distributions under Intel Architecture.
- Novell SUSE Linux Enterprise 10
- Red Hat Enterprise Linux 4
- Oracle Enterprise Linux 4
- Mac OS X 10.4+
- Solaris 10
This article shows you the installation made under a non-root user. If you need to use multicast you will need to start the daemon as root. I personally do not like to use setuid on the files.
There are 2 possible options when setting up FT pair using NFS storage.
1) Shared binary folder, such as /mountpoint/tibco
2) Independent binary folders on each EMS hosts, such as /home/tibadmin/tibco
In both options, the intances will use a shared NFS mountpoint for the shared states.
I have picked option 2 for my environment because I want to be able to run a single instance of TIBCO EMS if my Openfiler service is unavailable.
The installation package of TIBCO EMS 5.1.5 is TIB_ems_5.1.5_linux26gl23_x86.tar.gz
To install TIBCO EMS 5.1.5, follow the following steps:
1) Create a user account that you will use to install and run your EMS
#mkdir -p /home/tibadmin #groupadd -g 1002 tibadmin #useradd -u 1002 -g tibadmin tibadmin -s /bin/bash -d /home/tibadmin #chown -R tibadmin:tibadmin /home/tibadmin
Note the uid and gid down, we will need to create a user with identical uid and gid on the Openfiler server in the absence of NIS or LDAP. This is a very important step for the NFSv4 idmap to work properly.
Login as tibadmin
#su - tibadmin $
2) Copy the installation package into a temp folder. Make sure it is owned by tibadmin:tibadmin
$sudo chown tibadmin:tibadmin TIB_ems_5.1.5_linux26gl23_x86.tar.gz
3)Unzip the installation package
$gzip -d TIB_ems_5.1.5_linux26gl23_x86.tar.gz
4) Untar the tarball
$tar -xvf TIB_ems_5.1.5_linux26gl23_x86.tar
5) Execute the extracted installation binary
$./TIBCOUniversalInstaller-lnx-x86.bin
Follow the steps in the screens below.
6) Install the libstdc++5 package. This is required because Ubuntu 10.10 comes with libstdc++6 and not libstdc++5. Download the package (google it!) and install as follow.
dpkg -i libstdc++5_3.3.6-20_i386.deb
Repeat step 1 to 5 on another server instance, in my case it is tibems02.
Up to now, we have 2 instances of TIBCO EMS local installations. The next step is to set up the shared storage for the FT pair.
To set up a shared storage for the configuration files and message persistence, I use openFiler NAS and mount the exported directory using NFSv4 client.
1) ssh into the openfiler sever from any host with connectivity to the NFS server.
tibadmin@tibems01:~$ssh root@openfiler root@openfiler's password: Last login: Wed Apr 20 12:40:18 2011 from 192.168.13.101 [root@openfiler ~]#
2) Create tibadmin user on the openfiler server
[root@openfiler ~]# groupadd -g 1002 tibadmin [root@openfiler ~]# useradd -u 1002 -g tibadmin -s /sbin/nologin tibadmin [root@openfiler ~]# id tibadmin uid=1002(tibadmin) gid=1002(tibadmin) groups=1002(tibadmin) [root@openfiler ~]#
3) Setup the shared filesystem exports from openfiler (or on your favourite Linux/UNIX distributions).
4) Edit the /etc/exports file on NFS server. Add fsid=0, you should see something similar to this.
openfiler1:/yoursharedfolderforemsconfig 192.168.50.10 (rw,fsid=0,no_subtree_check,no_root_squash,sync) 192.168.50.11 (rw,fsid=0,no_subtree_check,no_root_squash,sync)
Note 1: 192.168.50.10 and 192.168.50.11 are IP address NAS NIC's of my EMS hosts. Refer to this VMWare ESXi infrastructure for the full picture.
Note 2: The fsid=0 option will will be removed if you change the NFS share configurations from the webgui of openfiler. Just back up a copy of /etc/exports file, in case it is overwritten you always have a copy of it.
Note 3: While it is not the best practice to setup a NFSv4 export without a well thought through pseudo-filesystem, this configuration meets my initial requirement.
If you have time, you should look at setting up the psuedo-fs in this way.
/tibco (fsid=0,ro)
|
-------------------------------
| | |
ems_ft_cfg ems_logging ems_ft_datastore
On the clients (tibems01 and tibems02), the shared folder is mounted with the following mount options, specified in the /etc/fstab file. Note the nfs4 in particular.
$mkdir -p /home/tibadmin/ems_ft_cfg $mount -t nfs4 -o rw openfiler1:/ home/tibadmin/ems_ft_cfg
I am not sure if NFS exported by OpenFiler meets the fault tolerance storage criteria. At least according to the TIBCO EMS User Guide, Chapter 5, the fnctl() POSIX file locking semantic on NFSv4 should work. For compliance to other criteria, a separate test is required.
struct flock fl; int err; fl.l_type = F_WRLCK; fl.l_whence = 0; fl.l_start = 0; fl.l_len = 0; err = fcntl(file, F_SETLK, &fl);
To setup for FT, the next step is to prepare the conf files on both servers (tibems01 and tibems02) and point them to the NFSv4 exported folder mouted under home/tibadmin/ems_ft_cfg.
1) Copy all the files from the sample/config directory (/home/tibadmin/tibco/ems/5.1/samples/config) into /home/tibadmin/ems_ft_cfg
tibadmin@tibems01:/$ cp /home/tibadmin/tibco/ems/5.1/samples/config/* /home/tibadmin/ems_ft_cfg/
2) Create a sub-folder under /home/tibadmin/ems_ft_cfg, name it "datastore" (it becomes "/home/tibadmin/ems_ft_config/datastore")
tibadmin@tibems01:/$ mkdir /home/tibadmin/ems_ft_cfg/datastore
3) Edit /home/tibadmin/ems_ft_cfg/tibemsd.conf, populate the following configuration parameters.
users = /home/tibadmin/ems_ft_cfg/users.conf groups = /home/tibadmin/ems_ft_cfg/groups.conf topics = /home/tibadmin/ems_ft_cfg/topics.conf queues = /home/tibadmin/ems_ft_cfg/queues.conf acl_list = /home/tibadmin/ems_ft_cfg/acl.conf factories = /home/tibadmin/ems_ft_cfg/factories.conf routes = /home/tibadmin/ems_ft_cfg/routes.conf bridges = /home/tibadmin/ems_ft_cfg/bridges.conf transports = /home/tibadmin/ems_ft_cfg/transports.conf tibrvcm = /home/tibadmin/ems_ft_cfg/tibrvcm.conf durables = /home/tibadmin/ems_ft_cfg/durables.conf channels = /home/tibadmin/ems_ft_cfg/channels.conf stores = /home/tibadmin/ems_ft_cfg/stores.conf store = /home/tibadmin/ems_ft_cfg/datastore max_msg_memory = 256MB #uncomment accordingly ft_active = tcp://tibems01:7222 #ft_active = tcp://tibems02:7222 ft_heartbeat = 3 ft_activation = 10 ft_reconnect_timeout = 60
4) Make 2 copies of the tibemsd.conf files. One as tibemsd1.conf, the other as tibemsd2.conf. In tibemsd2.conf, comment out the appropriate ft_active parameter.
#ft_active = tcp://tibems01:7222 ft_active = tcp://tibems02:7222
Summary:
For Primary instance
listen = tcp://tibems01:7222 ft_active = tcp://tibems02:7222
For secondary instance
listen = tcp://tibems02:7222 ft_active = tcp://tibems01:7222
To start the primary and secondary servers.
Login to tibems01 as tibadmin
tibadmin@tibems01:cd ~/tibco/ems/5.1/bin tibadmin@tibems01:~/tibco/ems/5.1/bin$./tibemsd -config /home/tibadmin/ems_ft_cfg/tibemsd1.conf
You should see this.
TIBCO Enterprise Message Service. Copyright 2003-2010 by TIBCO Software Inc. All rights reserved. Version 5.1.5 V3 3/29/2010 2011-04-16 03:27:01.550 Process started from './tibemsd'. 2011-04-16 03:27:01.550 Process Id: 1469 2011-04-16 03:27:01.550 Hostname: tibems01 2011-04-16 03:27:01.551 Hostname IP address: 172.16.10.10 2011-04-16 03:27:01.551 Detected IP interface: 127.0.0.1 (lo) 2011-04-16 03:27:01.551 Detected IP interface: 172.16.10.10 (eth0) 2011-04-16 03:27:01.551 Detected IP interface: 192.168.50.10 (eth1) 2011-04-16 03:27:01.551 Reading configuration from '/home/tibadmin/ems_ft_cfg/tibemsd1.conf'. 2011-04-16 03:27:01.582 Server name: 'EMS-SERVER'. 2011-04-16 03:27:01.582 Storage Location: '/home/tibadmin/ems_ft_cfg/datastore'. 2011-04-16 03:27:01.582 Routing is disabled. 2011-04-16 03:27:01.582 Authorization is disabled. 2011-04-16 03:27:01.681 Active server 'tcp://tibems02:7222' not found. 2011-04-16 03:27:01.681 Continuing as active server. 2011-04-16 03:27:01.686 Accepting connections on tcp://tibems01:7222. 2011-04-16 03:27:01.687 Recovering state, please wait. 2011-04-16 03:27:02.017 Server is active.
Login to tibems02 as tibadmin
tibadmin@tibems02:cd ~/tibco/ems/5.1/bin tibadmin@tibems02:~/tibco/ems/5.1/bin$./tibemsd -config /home/tibadmin/ems_ft_cfg/tibemsd2.conf
You should see this.
TIBCO Enterprise Message Service. Copyright 2003-2010 by TIBCO Software Inc. All rights reserved. Version 5.1.5 V3 3/29/2010 2011-04-16 03:31:45.008 Process started from './tibemsd'. 2011-04-16 03:31:45.009 Process Id: 15288 2011-04-16 03:31:45.009 Hostname: tibems02 2011-04-16 03:31:45.009 Hostname IP address: 172.16.10.11 2011-04-16 03:31:45.009 Detected IP interface: 127.0.0.1 (lo) 2011-04-16 03:31:45.009 Detected IP interface: 172.16.10.11 (eth0) 2011-04-16 03:31:45.009 Detected IP interface: 192.168.50.11 (eth1) 2011-04-16 03:31:45.009 Reading configuration from '/home/tibadmin/ems_ft_cfg/tibemsd2.conf'. 2011-04-16 03:31:45.020 Server name: 'EMS-SERVER'. 2011-04-16 03:31:45.020 Storage Location: '/home/tibadmin/ems_ft_cfg/datastore'. 2011-04-16 03:31:45.020 Routing is disabled. 2011-04-16 03:31:45.020 Authorization is disabled. 2011-04-16 03:31:45.076 Accepting connections on tcp://tibems02:7222. 2011-04-16 03:31:45.076 Server is in standby mode for 'tcp://tibems01:7222'.
Note: One the primary server, it has detected that the secondary server is up and connected.
2011-04-16 03:31:45.075 Backup server 'EMS-SERVER@tibems02' has connected.
Now try to shutdown the primary server, the secondary server will become active.
2011-04-16 03:41:47.082 Connection to primary server 'tcp://tibems01:7222' has been lost.
2011-04-16 03:41:47.083 Server activating on failure of 'tcp://tibems01:7222'.
2011-04-16 03:41:47.083 Server rereading configuration.
2011-04-16 03:41:47.126 Recovering state, please wait.
2011-04-16 03:41:47.135 Server is now active.
Cheers,
No comments:
Post a Comment