TIBCO EMS 5.1.5 Ubuntu Fault Tolerant Pair On NFS

If you'd like to jump the gun and try out the FT pair on Oracle Database, then this article is for you.

In the previous article, I briefly wrote about setting a file-based shared state TIBCO EMS FT pair on Windows platform. This article shows you how to install TIBCO EMS on linux hosts for fault tolerance configuration. The installations will use NFSv4 mounted folder for the shared states.

TIBCO EMS 5.1 is supported on the following Unix/Linux distributions under Intel Architecture.

• Novell SUSE Linux Enterprise 10
• Red Hat Enterprise Linux 4
• Oracle Enterprise Linux 4
• Mac OS X 10.4+
• Solaris 10

Though not officially supported on Ubuntu, the installation of TIBCO EMS 5.1.5 on Ubuntu Linux is very straight forward.

This article shows you the installation made under a non-root user. If you need to use multicast you will need to start the daemon as root. I personally do not like to use setuid on the files.

There are 2 possible options when setting up FT pair using NFS storage.

1) Shared binary folder, such as /mountpoint/tibco
2) Independent binary folders on each EMS hosts, such as /home/tibadmin/tibco

In both options, the intances will use a shared NFS mountpoint for the shared states.

I have picked option 2 for my environment because I want to be able to run a single instance of TIBCO EMS if my Openfiler service is unavailable.

The installation package of TIBCO EMS 5.1.5 is TIB_ems_5.1.5_linux26gl23_x86.tar.gz

To install TIBCO EMS 5.1.5, follow the following steps:

1) Create a user account that you will use to install and run your EMS

#mkdir -p /home/tibadmin
#groupadd -g 1002 tibadmin
#useradd -u 1002 -g tibadmin tibadmin -s /bin/bash -d /home/tibadmin
#chown -R tibadmin:tibadmin /home/tibadmin

Note the uid and gid down, we will need to create a user with identical uid and gid on the Openfiler server in the absence of NIS or LDAP. This is a very important step for the NFSv4 idmap to work properly.

Login as tibadmin

#su - tibadmin
$

2) Copy the installation package into a temp folder. Make sure it is owned by tibadmin:tibadmin

$sudo chown tibadmin:tibadmin TIB_ems_5.1.5_linux26gl23_x86.tar.gz

3)Unzip the installation package

$gzip -d TIB_ems_5.1.5_linux26gl23_x86.tar.gz

4) Untar the tarball

$tar -xvf TIB_ems_5.1.5_linux26gl23_x86.tar

5) Execute the extracted installation binary

$./TIBCOUniversalInstaller-lnx-x86.bin

Follow the steps in the screens below.

6) Install the libstdc++5 package. This is required because Ubuntu 10.10 comes with libstdc++6 and not libstdc++5. Download the package (google it!) and install as follow.

dpkg -i libstdc++5_3.3.6-20_i386.deb

Repeat step 1 to 5 on another server instance, in my case it is tibems02.

Up to now, we have 2 instances of TIBCO EMS local installations. The next step is to set up the shared storage for the FT pair.

To set up a shared storage for the configuration files and message persistence, I use openFiler NAS and mount the exported directory using NFSv4 client.

1) ssh into the openfiler sever from any host with connectivity to the NFS server.

tibadmin@tibems01:~$ssh root@openfiler
root@openfiler's password: 
Last login: Wed Apr 20 12:40:18 2011 from 192.168.13.101
[root@openfiler ~]# 

2) Create tibadmin user on the openfiler server

[root@openfiler ~]# groupadd -g 1002 tibadmin
[root@openfiler ~]# useradd -u 1002 -g tibadmin -s /sbin/nologin tibadmin
[root@openfiler ~]# id tibadmin
uid=1002(tibadmin) gid=1002(tibadmin) groups=1002(tibadmin)
[root@openfiler ~]# 

3) Setup the shared filesystem exports from openfiler (or on your favourite Linux/UNIX distributions).

4) Edit the /etc/exports file on NFS server. Add fsid=0, you should see something similar to this.

openfiler1:/yoursharedfolderforemsconfig 192.168.50.10 (rw,fsid=0,no_subtree_check,no_root_squash,sync) 192.168.50.11 (rw,fsid=0,no_subtree_check,no_root_squash,sync)

Note 1: 192.168.50.10 and 192.168.50.11 are IP address NAS NIC's of my EMS hosts. Refer to this VMWare ESXi infrastructure for the full picture.

Note 2: The fsid=0 option will will be removed if you change the NFS share configurations from the webgui of openfiler. Just back up a copy of /etc/exports file, in case it is overwritten you always have a copy of it.

Note 3: While it is not the best practice to setup a NFSv4 export without a well thought through pseudo-filesystem, this configuration meets my initial requirement.

If you have time, you should look at setting up the psuedo-fs in this way.

                   /tibco (fsid=0,ro)
                            |
             -------------------------------
            |            |                  |
        ems_ft_cfg   ems_logging    ems_ft_datastore

On the clients (tibems01 and tibems02), the shared folder is mounted with the following mount options, specified in the /etc/fstab file. Note the nfs4 in particular.

$mkdir -p /home/tibadmin/ems_ft_cfg
$mount -t nfs4 -o rw openfiler1:/ home/tibadmin/ems_ft_cfg

I am not sure if NFS exported by OpenFiler meets the fault tolerance storage criteria. At least according to the TIBCO EMS User Guide, Chapter 5, the fnctl() POSIX file locking semantic on NFSv4 should work. For compliance to other criteria, a separate test is required.

struct flock fl;
 int err;
 fl.l_type = F_WRLCK;
 fl.l_whence = 0;
 fl.l_start = 0;
 fl.l_len = 0;
 err = fcntl(file, F_SETLK, &fl);

To setup for FT, the next step is to prepare the conf files on both servers (tibems01 and tibems02) and point them to the NFSv4 exported folder mouted under home/tibadmin/ems_ft_cfg.

1) Copy all the files from the sample/config directory (/home/tibadmin/tibco/ems/5.1/samples/config) into /home/tibadmin/ems_ft_cfg

tibadmin@tibems01:/$ cp /home/tibadmin/tibco/ems/5.1/samples/config/* /home/tibadmin/ems_ft_cfg/

2) Create a sub-folder under /home/tibadmin/ems_ft_cfg, name it "datastore" (it becomes "/home/tibadmin/ems_ft_config/datastore")

tibadmin@tibems01:/$ mkdir /home/tibadmin/ems_ft_cfg/datastore

3) Edit /home/tibadmin/ems_ft_cfg/tibemsd.conf, populate the following configuration parameters.

users                   = /home/tibadmin/ems_ft_cfg/users.conf
 groups                  = /home/tibadmin/ems_ft_cfg/groups.conf
 topics                  = /home/tibadmin/ems_ft_cfg/topics.conf
 queues                  = /home/tibadmin/ems_ft_cfg/queues.conf
 acl_list                = /home/tibadmin/ems_ft_cfg/acl.conf
 factories               = /home/tibadmin/ems_ft_cfg/factories.conf
 routes                  = /home/tibadmin/ems_ft_cfg/routes.conf
 bridges                 = /home/tibadmin/ems_ft_cfg/bridges.conf
 transports              = /home/tibadmin/ems_ft_cfg/transports.conf
 tibrvcm                 = /home/tibadmin/ems_ft_cfg/tibrvcm.conf
 durables                = /home/tibadmin/ems_ft_cfg/durables.conf
 channels                = /home/tibadmin/ems_ft_cfg/channels.conf
 stores                  = /home/tibadmin/ems_ft_cfg/stores.conf
   
 store                   = /home/tibadmin/ems_ft_cfg/datastore

 max_msg_memory          = 256MB

 #uncomment accordingly
  ft_active               = tcp://tibems01:7222
 #ft_active               = tcp://tibems02:7222

 ft_heartbeat            = 3
 ft_activation           = 10
 ft_reconnect_timeout    = 60

4) Make 2 copies of the tibemsd.conf files. One as tibemsd1.conf, the other as tibemsd2.conf. In tibemsd2.conf, comment out the appropriate ft_active parameter.

#ft_active               = tcp://tibems01:7222
 ft_active               = tcp://tibems02:7222

Summary:
For Primary instance

listen = tcp://tibems01:7222
ft_active = tcp://tibems02:7222

For secondary instance

listen = tcp://tibems02:7222
ft_active = tcp://tibems01:7222

To start the primary and secondary servers.

Login to tibems01 as tibadmin

tibadmin@tibems01:cd ~/tibco/ems/5.1/bin
tibadmin@tibems01:~/tibco/ems/5.1/bin$./tibemsd -config /home/tibadmin/ems_ft_cfg/tibemsd1.conf

You should see this.

TIBCO Enterprise Message Service.
Copyright 2003-2010 by TIBCO Software Inc.
All rights reserved.

Version 5.1.5 V3 3/29/2010

2011-04-16 03:27:01.550 Process started from './tibemsd'.
2011-04-16 03:27:01.550 Process Id: 1469
2011-04-16 03:27:01.550 Hostname: tibems01
2011-04-16 03:27:01.551 Hostname IP address: 172.16.10.10
2011-04-16 03:27:01.551 Detected IP interface: 127.0.0.1 (lo)
2011-04-16 03:27:01.551 Detected IP interface: 172.16.10.10 (eth0)
2011-04-16 03:27:01.551 Detected IP interface: 192.168.50.10 (eth1)
2011-04-16 03:27:01.551 Reading configuration from '/home/tibadmin/ems_ft_cfg/tibemsd1.conf'.
2011-04-16 03:27:01.582 Server name: 'EMS-SERVER'.
2011-04-16 03:27:01.582 Storage Location: '/home/tibadmin/ems_ft_cfg/datastore'.
2011-04-16 03:27:01.582 Routing is disabled.
2011-04-16 03:27:01.582 Authorization is disabled.
2011-04-16 03:27:01.681 Active server 'tcp://tibems02:7222' not found.
2011-04-16 03:27:01.681 Continuing as active server.
2011-04-16 03:27:01.686 Accepting connections on tcp://tibems01:7222.
2011-04-16 03:27:01.687 Recovering state, please wait.
2011-04-16 03:27:02.017 Server is active.

Login to tibems02 as tibadmin

tibadmin@tibems02:cd ~/tibco/ems/5.1/bin
tibadmin@tibems02:~/tibco/ems/5.1/bin$./tibemsd -config /home/tibadmin/ems_ft_cfg/tibemsd2.conf

You should see this.

TIBCO Enterprise Message Service.
Copyright 2003-2010 by TIBCO Software Inc.
All rights reserved.

Version 5.1.5 V3 3/29/2010

2011-04-16 03:31:45.008 Process started from './tibemsd'.
2011-04-16 03:31:45.009 Process Id: 15288
2011-04-16 03:31:45.009 Hostname: tibems02
2011-04-16 03:31:45.009 Hostname IP address: 172.16.10.11
2011-04-16 03:31:45.009 Detected IP interface: 127.0.0.1 (lo)
2011-04-16 03:31:45.009 Detected IP interface: 172.16.10.11 (eth0)
2011-04-16 03:31:45.009 Detected IP interface: 192.168.50.11 (eth1)
2011-04-16 03:31:45.009 Reading configuration from '/home/tibadmin/ems_ft_cfg/tibemsd2.conf'.
2011-04-16 03:31:45.020 Server name: 'EMS-SERVER'.
2011-04-16 03:31:45.020 Storage Location: '/home/tibadmin/ems_ft_cfg/datastore'.
2011-04-16 03:31:45.020 Routing is disabled.
2011-04-16 03:31:45.020 Authorization is disabled.
2011-04-16 03:31:45.076 Accepting connections on tcp://tibems02:7222.
2011-04-16 03:31:45.076 Server is in standby mode for 'tcp://tibems01:7222'.

Note: One the primary server, it has detected that the secondary server is up and connected.

2011-04-16 03:31:45.075 Backup server 'EMS-SERVER@tibems02' has connected.

Now try to shutdown the primary server, the secondary server will become active.

2011-04-16 03:41:47.082 Connection to primary server 'tcp://tibems01:7222' has been lost.
2011-04-16 03:41:47.083 Server activating on failure of 'tcp://tibems01:7222'.
2011-04-16 03:41:47.083 Server rereading configuration.
2011-04-16 03:41:47.126 Recovering state, please wait.
2011-04-16 03:41:47.135 Server is now active.

Cheers,

Hack - Export existing volume on a physical drive via Openfiler VM

I have a piece of relatively old 150GB HDD with Linux file system, lying around for quite some time. Having installed Windows 7 64-bits as my primary OS, I hardly have a chance to make good use of that disk. To read Linux file system from Windows OS, there are a few good options out there, such as the one by Disk Internal and this opensource one.

But if you are like me, running a bunch of Linux virtual machines on a Windows 7 host, you might find this article helpful.

What we are going to do is to export the entire existing filesystem on this hard disk from the Openfiler virtual machine. But Openfiler uses Logical Volume Manager and does not give you an option to expose existing file systems without destroying the existing partition table. Well, I must say I haven't search hard enough on how to do that officially in Openfiler, but am happy with my workaround, which has been completed in 5 minutes.

This is what we will do.

1) Attach the physical volume to the Openfiler VM.

I assume you know how to figure out which physical drive is the one that you want to attach to your Openfiler VM. Be extremely careful not to attach the wrong one. Your Linux formatted disk drive will not appear in the Windows file explorer. To find out which one is your disk, try device manager, or Google is your friend.

On my Windows host, my linux harddrive is the Disk 3.

Once you have attached your physical disk to the VM, boot it up.

2) Create a share folder in Openfiler under one of your existing shares.

3) Mount the raw device to the new share folder created in step 2.

- Locate your raw device on the Openfiler server by using fdisk command.

[root@openfiler01 physical]# fdisk -l
Disk /dev/hda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000e329f

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1       19457   156288321   83  Linux

To mount,

#mount -t ext4 -o rw /dev/hda1 /mnt/home_shared/sharedvol/physical

Note: My HDD was formatted with ext4 fs.

You might need to re-export your filesystem

#exportfs -rv

Inspect the /etc/exports file in the Openfiler server.

/mnt/home_shared/sharedvol/physical 192.168.0.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)  192.168.19.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)  192.168.50.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)

Go to your client Windows OS (assuming it is on the same network as the Openfiler server, in my case 192.168.50.0/24), map to \\openfiler\home_shared\sharedvol\physical, you should see your existing files and should be able to copy into the drive as Samba mount.

Or from the command prompt

net use drive_letter: \\openfiler\home_shared\sharedvol\physical

Go to your NFS client Linux OS and create a mount point

#mkdir /mnt/physical

Mount with the following options

mount -o rw openfiler:/mnt/home_shared/sharedvol/physical /mnt/physical

Look at the new mounted folder, it works!

root@tibems01:/mnt/physical# df -m .
Filesystem           1M-blocks      Used Available Use% Mounted on
192.168.50.100:/mnt/home_shared/sharedvol/physical
                        150230     78025     64574  55% /mnt/physical
root@tibems01:/mnt/physical# 

Don't forget to make a backup of your openfiler /etc/exports, just in case.

Well, a hack that works and will survive Openfiler boot cycle.

Cheers,

svc: failed to register lockdv1 RPC service (errno 97). lockd: cannot monitor

Alright, you have searched the internet in and out, up and down, plough through the forums and read all the available man pages on earth regarding NFS and still did not find answer you longed for, because they are not directly available/described/discussed whatever it might be. Well, this might help you.

Good thing about linux is that it keeps you at your toes.

When mounting a NFS directory, you might see this error in your /var/log/messages trace.

Apr  1 09:10:45 orasoa1 kernel: [  895.012894] svc: failed to register lockdv1 RPC service (errno 97).
Apr  1 09:25:31 orasoa1 kernel: [ 1778.601154] svc: failed to register lockdv1 RPC service (errno 97).

Or when acquiring a file lock either by some applications, Oracle SOA Admin Server for example, or when simply trying to manually acquire an exclusive lock of a file on NFS.

$flock lockfile -c 'vi dosomeactivities'

You might see these errors.

Apr  1 09:25:57 orasoa1 kernel: [ 1804.313073] lockd: cannot monitor 192.168.50.100
Apr  1 09:33:59 orasoa1 kernel: [ 2285.828349] lockd: cannot monitor 192.168.50.100
Apr  1 09:36:39 orasoa1 kernel: [ 2444.869671] lockd: cannot monitor 192.168.50.100

where "192.168.50.100" is the NAS server IP address (the 'openfiler1' host).

It is probably because you don't have a DNS for your host names or simply because you haven't updated your /etc/hosts file to include the name of your NAS server.

Try this if you had similar problem, I hope it saves your precious time.

On your client host(s), stick this line into your /etc/hosts file(s)

#vi /etc/hosts

nas_server_ip_addr nas_sever_hostname #example 192.168.50.100    openfiler1
nas_client_ip_addr nas_client_hostname     #example 172.16.20.50      orasoa1

Cheers,