Tuesday, October 11, 2011

Setting up a Solaris 11 Express Local Package Repository

As part of the exercise to upgrade my entire infrastructure from ESXi 4 to ESX 5, and adding a few Solaris Global Zones into the mix, I started with setting up a Solaris 11-based utility server. The shared, utility server is envisioned to provide the following services to other nodes in the infrastructure without introducing too many individual servers in the lab environment:

1) Local network IPS repository server for other clients
2) ZFS-based iSCSI target for other Solaris zones
3) ZFS-based NFS for shared storage in Solaris zones
4) LDAP services
5) DNS services
6) and more...

This article will address the topic of setting up a shared repository.

Pre-requisite:
- Solaris 11 Express VM. Download text installer here. I choose VMware workstation, you can obtain the entire VirtualBox image of Solaris 11 Express from Oracle website.
- Solaris 11 Express  full repository CD (sol-11-exp-201011-repo-full.iso). Download from Oracle website.

Concept
A repository is a location where clients publish and retrieve IPS packages. Clients that publish IPS packages are called publisher. A repository contains packages published by a single publisher. A repository is described by a Universal Resource Identifier (URI) such as http URL or a file location. The publisher name of all packages in the Solaris 11 Express full repository CD is 'solaris'. If you create software and publish the packages, you give yourself a publisher name, such as 'mycompany'.

A repository can be either an 'origin' or a 'mirror'.

An IPS package is a collection of software artefacts in a defined format. A package is represented by a Fault Management Resource Identifier (FMRI).

For detailed description of IPS concept, refer to Oracle documentation.

Setting up the IPS repository
In summary, the following components are required in order to setup a network based IPS repository.

- IPS Contents. We will copy all the packages from the Oracle repository CD into a local file system. You can simply create a local directory under the default ZFS pool (rpool) of your Solaris 11 Express installation. Alternatively, you can create a new ZFS pool to host the files.

In this exercise, I used the existing root file system rpool and make a directory under the root mountpoint.

root@sol11util:/rpool# cd /
root@sol11util:/# mkdir -p ips/repo
root@sol11util:/#

Copy all contents under the repo folder in the CD into the /ips/repo

root@sol11util:/# rsync -aP source dest

This is going to take a while as every files under the repo is copied.

After completion, check that the directory is there. You should see under the ips/repo the following files

root@sol11util:/ips/repo# ls -la
total 7
drwxr-xr-x 3 root root   4 2010-11-11 08:40 .
drwxr-xr-x 3 root root   3 2011-10-23 08:33 ..
-rw-r--r-- 1 root root 250 2010-11-11 08:40 pkg5.repository
drwxr-xr-x 3 root root   3 2010-11-11 08:39 publisher

- IPS server. We have the content in /ips/repo now. We need to setup a server to serve the contents of this folder to the clients on the LAN. Solaris 11 Express allows you to manage services with simple commands.

svcs - This command is used to list currently running services managed by the smf (Service Management Facility). To find out if an instance of pkg package server already running, issue the following command

root@sol11util:/tank/repo# svcs \*pkg\*
STATE          STIME    FMRI
disabled        6:48:32 svc:/application/pkg/server:default
disabled        6:48:32 svc:/application/pkg/dynamic-mirror:default
online          6:48:43 svc:/system/pkgserv:default
root@sol11util:/tank/repo#

The /application/pkg/server is not enabled by default in a fresh Solaris 11 installation. This is the service that will serve the client and provide access to the packages published by a publisher.

Let's examine this service by another command called 'svccfg'. The following command line lists all the properties of the application/pkg/server service to the console.

root@sol11util:/tank/repo# svccfg -v -s application/pkg/server listprop

pkg                                                        application
pkg/cfg_file                                               astring 
pkg/content_root                                           astring  usr/share/lib/pkg
pkg/debug                                                  astring 
pkg/file_root                                              astring 
pkg/inst_root                                              astring  /var/pkg/repo
pkg/log_access                                             astring  none
pkg/log_errors                                             astring  stderr
pkg/mirror                                                 boolean  false
pkg/pkg_root                                               astring  /
pkg/port                                                   count    80
pkg/proxy_base                                             astring 
pkg/readonly                                               boolean  true
pkg/socket_timeout                                         count    60
pkg/sort_file_max_size                                     astring 
pkg/ssl_cert_file                                          astring 
pkg/ssl_dialog                                             astring  smf
pkg/ssl_key_file                                           astring 
pkg/threads                                                count    60
pkg/writable_root                                          astring 
pkg_bui                                                    application
pkg_bui/feed_description                                   ustring 
pkg_bui/feed_icon                                          ustring  web/_themes/pkg-block-icon.png
pkg_bui/feed_logo                                          ustring  web/_themes/pkg-block-logo.png
pkg_bui/feed_name                                          ustring  "package repository feed"
pkg_bui/feed_window                                        count    24
pkg_secure                                                 application
pkg_secure/read_authorization                              astring  solaris.smf.read.pkg-server
pkg_secure/ssl_key_passphrase                              astring 
fs                                                         dependency
fs/entities                                                fmri     svc:/system/filesystem/local
fs/grouping                                                astring  require_all
fs/restart_on                                              astring  none
fs/type                                                    astring  service
autofs                                                     dependency
autofs/entities                                            fmri     svc:/system/filesystem/autofs
autofs/grouping                                            astring  optional_all
autofs/restart_on                                          astring  none
autofs/type                                                astring  service
ntp                                                        dependency
ntp/entities                                               fmri     svc:/network/ntp
ntp/grouping                                               astring  optional_all
ntp/restart_on                                             astring  none
ntp/type                                                   astring  service
network                                                    dependency
network/entities                                           fmri     svc:/milestone/network
network/grouping                                           astring  require_all
network/restart_on                                         astring  none
network/type                                               astring  service
manifestfiles                                              framework
manifestfiles/var_svc_manifest_application_pkg-server_xml  astring  /var/svc/manifest/application/pkg-server.xml
startd                                                     framework
startd/duration                                            astring  child
general                                                    framework
general/entity_stability                                   astring  Unstable
start                                                      method
start/exec                                                 astring  "%{pkg/pkg_root}/lib/svc/method/svc-pkg-depot %m"
start/timeout_seconds                                      count    0
start/type                                                 astring  method
stop                                                       method
stop/exec                                                  astring  "%{pkg/pkg_root}/lib/svc/method/svc-pkg-depot %m %{restarter/contract}"
stop/timeout_seconds                                       count    30
stop/type                                                  astring  method
tm_common_name                                             template
tm_common_name/C                                           ustring  "image packaging repository"
As you can see, this is a huge list of service properties. Fortunately, we are only interested in 2 properties, namely the pkg/inst_root and pkg/port. The default values of these properties are /var/pkg/repo and 80 respectively.

We will update the pkg/inst_root to point to /ips/repo (the local copy of the repo folder in the Solaris package CD), and the port number to an unused port 9999.

root@sol11util:/tank/repo# svccfg -v -s application/pkg/server setprop pkg/inst_root=/ips/repo

root@sol11util:/tank/repo# svccfg -v -s application/pkg/server setprop pkg/port=9999

Make sure the props are set.

root@sol11util:~# svccfg -s /application/pkg/server listprop | grep '\(port\)\|\(inst_root\)'

pkg/inst_root                                              astring  /ips/repo/
pkg/port                                                   count    9999

So far so good. We need to start the service.

root@sol11util:~# svcadm enable /application/pkg/server
root@sol11util:~# svcs /application/pkg/server
STATE          STIME    FMRI
online         16:24:28 svc:/application/pkg/server:default

The service is up, good. By now we should be able to use web browser to explore the repository from the network.


To start using this repository, we will need to set the client up to point to this repository. For now, we will set up the host to use the repository it is hosting.

Before doing that, let's see what repository is currently available to the host.

root@sol11util:/usr# pkg publisher

PUBLISHER                             TYPE     STATUS   URI

solaris                  (preferred)  origin   online   http://pkg.oracle.com/solaris/release/

Ok, we currently have an internet based Oracle repository. We are going to replace that with our local (LAN based) one.

root@sol11util:/usr# pkg set-publisher -g http://192.168.12.135:9999 -G http://pkg.oracle.com/solaris/release/ solaris

results in

root@sol11util:/usr# pkg publisher
PUBLISHER                             TYPE     STATUS   URI
solaris                  (preferred)  origin   online   http://192.168.12.135:9999/
root@sol11util:/usr# 


Rebuild the repository index

root@sol11util:~# pkg rebuild-index

Try installing firefox from this default (preferred) repository.

root@sol11util:/usr# pkg install firefox Packages to install: 15 Create boot environment: No Services to restart: 3 DOWNLOAD PKGS FILES XFER (MB) Completed 15/15 5628/5628 34.2/34.2 PHASE ACTIONS Install Phase 6514/6514 PHASE ITEMS Package State Update Phase 15/15 Image State Update Phase 2/2

That is it.

Cheers

Tuesday, April 26, 2011

TIBCO EMS Fault Tolerant Pair - Server Authorization=True

TIBCO EMS User's Guide Chapter 19 Fault Tolerance - Authorization and Fault-Tolerant Servers describes how to setup server authorization in a FT pair. This article tries to compliment the TIBCO official documentation with easy to understand examples and illustrations.

There are at least 3 possible configurations of authorization-enabled FT pairs.

1) Primary and secondary servers share a common users.conf file.
2) Primary and secondary servers each uses separate users.conf file.
3) Primary and secondary servers retrieve user/group from a common LDAP.

Before we look into each option, ensure the server names and server passwords are identical on both servers. Server password is stored in the tibemsd.conf file. Examine the section in tibemsd.conf.

########################################################################
# Server Identification Information.
#
# server:    unique server name
# password:  password used to login into other routed server
########################################################################

server                  = EMS-SERVER
password                =

You must setup the server password on both servers. You can do this on individual server, without needing to run them as pair.

First, start up the first instance (using tibemsd1.conf)

Connect to this instance using tibemsadmin in $EMS_HOME/bin

tibadmin@tibems01:~/tibco/ems/5.1/bin$ ./tibemsadmin

TIBCO Enterprise Message Service Administration Tool.
Copyright 2003-2010 by TIBCO Software Inc.
All rights reserved.

Version 5.1.5 V3 3/29/2010

Type 'help' for commands help, 'exit' to exit:
> 


Connect to tibems01 (login name = admin, password = blank)

> connect tibems01
Login name (admin): 
Password: 
Connected to: tcp://tibems01:7222
tcp://tibems01:7222> 

Set the server password using the following command.

tcp://tibems01:7222> set server password=serverpass
Server parameters have been changed
tcp://tibems01:7222> 

Before you proceed to carry out the same step to tibems02, lets inspect the tibemsd1.conf file. You should see that the mangled password is now there.

########################################################################
# Server Identification Information.
#
# server:    unique server name
# password:  password used to login into other routed server
########################################################################

server                  = EMS-SERVER
password                = $man$o9y3vdC5oxiNRQT6Sb4F+FqKLqM

Now set the server password in tibemsd2.conf, with the exact same password used in tibemsd1.conf.

tibadmin@tibems01:~/tibco/ems/5.1/bin$ ./tibemsadmin 

TIBCO Enterprise Message Service Administration Tool.
Copyright 2003-2010 by TIBCO Software Inc.
All rights reserved.

Version 5.1.5 V3 3/29/2010

Type 'help' for commands help, 'exit' to exit:
> connect tibems02
Login name (admin): 
Password: 
Connected to: tcp://tibems02:7222
tcp://tibems02:7222> set server password=serverpass
Server parameters have been changed
tcp://tibems02:7222> 

Inspect tibemsd2.conf file.

########################################################################
# Server Identification Information.
#
# server:    unique server name
# password:  password used to login into other routed server
########################################################################

server                  = EMS-SERVER
password                = $man$vwtgocxurfOKsIcAiMOwD84fCDc


Note that the mangled password does not need to be identical in both tibemsd1.conf and tibemsd2.conf.

The FT pair are still not operational until we create a user 'EMS-SERVER' in each server. This is where it gets confusing. Not only you set up the server password (stored in the tibemsd.conf files), you also need to create a user called 'EMS-SERVER' with the exactly same password (serverpass). This task is subject to which option we choose and is described in sections below.

Option1: Primary and secondary servers share a common users.conf file (in other words, both tibemsd1.conf and tibemsd2.conf point to a single users.conf file)

Servers using a common users.conf file

Start either one of the instances (I chose tibems01)

Start tibco admin tool and connect.

> connect tibems01
Login name (admin): 
Password: 
Connected to: tcp://tibems01:7222
tcp://tibems01:7222> create user EMS-SERVER password=serverpass
User 'EMS-SERVER' has been created
tcp://tibems01:7222> 

Now inspect the users.conf file.

admin::"Administrator"
EMS-SERVER:$2$og/IEiSh$gp3b15kK67UPbMR3t7NX4h2X:""

The user admin is created by default by the server, without a password.

The EMS-SERVER is the one that we have just created. Note the mangled password.

Option 2: Primary and secondary servers each uses separate users.conf files (say, tibemsd1.conf uses users1.conf and tibemsd2.conf uses users2.conf)

Servers have own users.conf file

Perform what were done in step 1 for both tibems01 and tibems02. You should see the mangled password for EMS-SERVER in both usersN.conf files.

3) Primary and secondary servers retrieve user/group from a common LDAP.

Not covered in this article. We will discuss this when I have my LDAP server deployed.

Happy authorizing...☺

Cheers,

Saturday, April 23, 2011

TIBCO EMS 5.1.5 ESTATUS = 230 com.tibco.tibems.tibemsd.internal.db.EMSDataStoreException

Alright, the FT setup in the previous article worked, but didn't quite right. After the first successful startup and worked as pair, it never did again the second time.

Kept giving out this error.

2011-04-19 13:47:22.004 Recovering state, please wait.
2011-04-19 13:48:23.313 com.tibco.tibems.tibemsd.internal.db.EMSDataStoreException: failed to open store: IO Error: Socket read timed out
2011-04-19 13:48:23.313         at com.tibco.tibems.tibemsd.internal.db.DataStore._checkValidConnParams(DataStore.java:280)
2011-04-19 13:48:23.313         at com.tibco.tibems.tibemsd.internal.db.DataStore.open(DataStore.java:302)
2011-04-19 13:48:23.313         at com.tibco.tibems.tibemsd.internal.db.CallableDataStore.open(CallableDataStore.java:121)
2011-04-19 13:48:23.313         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2011-04-19 13:48:23.314         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
2011-04-19 13:48:23.314         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
2011-04-19 13:48:23.314         at java.lang.reflect.Method.invoke(Method.java:597)
2011-04-19 13:48:23.314         at com.tibco.tibems.tibemsd.internal.Context.invokeMethodInLine(Context.java:308)
2011-04-19 13:48:23.316 SEVERE ERROR: Unable to open store [$sys.failsafe]: [ ESTATUS = 230, ERRSTR = com.tibco.tibems.tibemsd.internal.db.EMSDataStoreException: failed to open store: IO Error: Socket read timed out ]
2011-04-19 13:48:23.317 com.tibco.tibems.tibemsd.internal.db.EMSDataStoreException: failed to open store: IO Error: Socket read timed out
2011-04-19 13:48:23.317         at com.tibco.tibems.tibemsd.internal.db.DataStore._checkValidConnParams(DataStore.java:280)
2011-04-19 13:48:23.317         at com.tibco.tibems.tibemsd.internal.db.DataStore.open(DataStore.java:302)
2011-04-19 13:48:23.324         at com.tibco.tibems.tibemsd.internal.db.CallableDataStore.open(CallableDataStore.java:121)
2011-04-19 13:48:23.324         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2011-04-19 13:48:23.324         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
2011-04-19 13:48:23.324         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
2011-04-19 13:48:23.325         at java.lang.reflect.Method.invoke(Method.java:597)
2011-04-19 13:48:23.325         at com.tibco.tibems.tibemsd.internal.Context.invokeMethodInLine(Context.java:308)
2011-04-19 13:48:23.325 SEVERE ERROR: Unable to open store [$sys.meta]: [ ESTATUS = 230, ERRSTR = com.tibco.tibems.tibemsd.internal.db.EMSDataStoreException: failed to open store: IO Error: Socket read timed out ]
2011-04-19 13:48:23.327 ERROR: Initialization failed: storage for '$sys.meta' not found.
2011-04-19 13:48:23.327 FATAL: Exception in startup, exiting.

After an hour of investigation, it finally narrowed down to oracle jdbc driver problem.

In TIBCO EMS User's Guide, Chapter 10, the documentation did give some clue about RAC.

"The TIBCO Enterprise Message Service server must connect to the Oracle RAC 10g database using the Oracle JDBC OCI driver and TAF configuration."

Why not give it a try? So I went ahead to try to use OCI rather than the thin driver. And it works!

In summary, 3 things needs to be done.

1) Install the Oracle Instant Client 11.2. (instantclient-basic-linux32-11.2.0.2.0.zip)
2) Install libaio package
3) Update the store-db.conf

1) Install the Oracle Instant Client 11.2.
Download the zip file from Oracle website. Unzip all the contents into the $EMS_HOME/lib folder.

2) Install libaio package

root@tibems01:~# apt-get install libaio1

3) Update the stores-db.conf file.

[$sys.failsafe]
type=dbstore
dbstore_driver_url=jdbc:oracle:oci:emsfs/emsfs@oradb11g01:1521:orcl
dbstore_driver_username=emsfs
dbstore_driver_password=emsfs

[$sys.meta]
type=dbstore
dbstore_driver_url=jdbc:oracle:oci:emsmeta/emsmeta@oradb11g01:1521:orcl
dbstore_driver_username=emsmeta
dbstore_driver_password=emsmeta

[$sys.nonfailsafe]
  type=file
  file=/home/tibadmin/ems_ft_cfg/datastore/async-msgs.db
  mode=async
  file_crc=disabled

Start the server pair, with log_trace = DEFAULT, +DBSTORE

Primary (active) server (tibems01)

TIBCO Enterprise Message Service.
Copyright 2003-2010 by TIBCO Software Inc.
All rights reserved.

Version 5.1.5 V3 3/29/2010

2011-04-19 15:35:28.828 Process started from '/home/tibadmin/tibco/ems/5.1/bin/tibemsd'.
2011-04-19 15:35:28.829 Process Id: 11175
2011-04-19 15:35:28.829 Hostname: tibems01
2011-04-19 15:35:28.829 Hostname IP address: 172.16.10.10
2011-04-19 15:35:28.829 Detected IP interface: 127.0.0.1 (lo)
2011-04-19 15:35:28.829 Detected IP interface: 172.16.10.10 (eth0)
2011-04-19 15:35:28.829 Detected IP interface: 192.168.50.10 (eth1)
2011-04-19 15:35:28.829 Reading configuration from '/home/tibadmin/ems_ft_cfg/tibemsd-db1.conf'.
2011-04-19 15:35:28.835 Logging into file '/home/tibadmin/ems01.log'
2011-04-19 15:35:28.836 Trying JVM location: /opt/java/jre/lib/i386/server/libjvm.so
2011-04-19 15:35:28.851 Loaded jvm from /opt/java/jre/lib/i386/server/libjvm.so
2011-04-19 15:35:28.851 Added java virtal machine option -Xmx256m
2011-04-19 15:35:28.851 Added java virtal machine option -Djava.class.path=/home/tibadmin/tibco/ems/5.1/bin/tibemsd.jar
2011-04-19 15:35:28.851 Added java virtal machine option -Xrs
2011-04-19 15:35:28.851 Added java virtal machine option -Dems.com.tibco.tibems.tibemsd.internal.version=5.1.5
2011-04-19 15:35:29.024 Java virtual machine started.
2011-04-19 15:35:29.089 Java Version 1.6.0_24
2011-04-19 15:35:29.095 Java server tasks started.
2011-04-19 15:35:29.108 Server name: 'EMS-SERVER'.
2011-04-19 15:35:29.108 Storage Location: '.'.
2011-04-19 15:35:29.108 Routing is disabled.
2011-04-19 15:35:29.108 Authorization is disabled.
2011-04-19 15:35:30.999 Active server 'tcp://tibems02:7222' not found.
2011-04-19 15:35:30.999 Continuing as active server.
2011-04-19 15:35:30.000 Accepting connections on tcp://tibems01:7222.
2011-04-19 15:35:30.000 Recovering state, please wait.
2011-04-19 15:41:45.330 Store '$sys.failsafe' locked by 'EMS-SERVER'
2011-04-19 15:41:45.347 Store '$sys.meta' locked by 'EMS-SERVER'
2011-04-19 15:41:45.347 Store '$sys.meta': Recovering Zone records...
2011-04-19 15:41:45.398 Store '$sys.meta': Recovered <1> Zone records
2011-04-19 15:41:45.398 Store '$sys.meta': Recovering Connection records...
2011-04-19 15:41:45.414 Store '$sys.meta': Recovered <0> Connection records
2011-04-19 15:41:45.415 Recovering transaction records
2011-04-19 15:41:45.415 Store '$sys.meta': Recovering Transaction records...
2011-04-19 15:41:45.427 Store '$sys.meta': Recovered <0> Transaction records
2011-04-19 15:41:45.427 Store '$sys.failsafe': Recovering Transaction records...
2011-04-19 15:41:45.458 Store '$sys.failsafe': Recovered <0> Transaction records
2011-04-19 15:41:45.458 Recovered all transaction records
2011-04-19 15:41:45.458 Store '$sys.meta': Recovering Session records...
2011-04-19 15:41:45.470 Store '$sys.meta': Recovered <0> Session records
2011-04-19 15:41:45.470 Store '$sys.meta': Recovering Consumer records...
2011-04-19 15:41:45.485 Store '$sys.meta': Recovered <0> Consumer records
2011-04-19 15:41:45.486 Store '$sys.meta': Recovering Producer records...
2011-04-19 15:41:45.499 Store '$sys.meta': Recovered <0> Producer records
2011-04-19 15:41:45.499 Store '$sys.meta': Recovering Purge records...
2011-04-19 15:41:45.511 Store '$sys.meta': Recovered <0> Purge records
2011-04-19 15:42:40.661 Recovering valid msg records...
2011-04-19 15:42:40.661 Store '$sys.meta': Recovering Valid records...
2011-04-19 15:43:13.912 Store '$sys.meta': Recovered <0> Valid records
2011-04-19 15:43:13.912 Store '$sys.failsafe': Recovering Valid records...
2011-04-19 15:43:13.982 Store '$sys.failsafe': Recovered <1> Valid records
2011-04-19 15:43:13.982 Recovered all valid msg records
2011-04-19 15:43:13.983 Store '$sys.meta': Recovering Message records...
2011-04-19 15:43:13.985 Store '$sys.failsafe': Recovering Message records...
2011-04-19 15:43:14.252 Store '$sys.failsafe': Recovered <0> Message records
2011-04-19 15:43:14.260 Store '$sys.meta': Recovered <0> Message records
2011-04-19 15:43:14.280 Recovering Ack records
2011-04-19 15:43:14.280 Store '$sys.failsafe': Recovering Ack records...
2011-04-19 15:43:14.302 Store '$sys.failsafe': Recovered <0> Ack records
2011-04-19 15:43:14.302 Recovered all Ack records
2011-04-19 15:43:14.302 Server is active.

Secondary (standby) server (tibems02)

TIBCO Enterprise Message Service.
Copyright 2003-2010 by TIBCO Software Inc.
All rights reserved.

Version 5.1.5 V3 3/29/2010

2011-04-19 15:42:56.457 Process started from '/home/tibadmin/tibco/ems/5.1/bin/tibemsd'.
2011-04-19 15:42:56.457 Process Id: 2466
2011-04-19 15:42:56.457 Hostname: tibems02
2011-04-19 15:42:56.457 Hostname IP address: 172.16.10.11
2011-04-19 15:42:56.457 Detected IP interface: 127.0.0.1 (lo)
2011-04-19 15:42:56.458 Detected IP interface: 172.16.10.11 (eth0)
2011-04-19 15:42:56.458 Detected IP interface: 192.168.50.11 (eth1)
2011-04-19 15:42:56.458 Reading configuration from '/home/tibadmin/ems_ft_cfg/tibemsd-db2.conf'.
2011-04-19 15:42:56.462 Trying JVM location: /opt/java/jre/lib/i386/server/libjvm.so
2011-04-19 15:42:56.480 Loaded jvm from /opt/java/jre/lib/i386/server/libjvm.so
2011-04-19 15:42:56.480 Added java virtal machine option -Xmx256m
2011-04-19 15:42:56.480 Added java virtal machine option -Djava.class.path=/home/tibadmin/tibco/ems/5.1/bin/tibemsd.jar
2011-04-19 15:42:56.480 Added java virtal machine option -Xrs
2011-04-19 15:42:56.480 Added java virtal machine option -Dems.com.tibco.tibems.tibemsd.internal.version=5.1.5
2011-04-19 15:42:56.647 Java virtual machine started.
2011-04-19 15:42:56.702 Java Version 1.6.0_24
2011-04-19 15:42:56.708 Java server tasks started.
2011-04-19 15:42:56.710 Server name: 'EMS-SERVER'.
2011-04-19 15:42:56.710 Storage Location: '.'.
2011-04-19 15:42:56.710 Routing is disabled.
2011-04-19 15:42:56.711 Authorization is disabled.
2011-04-19 15:43:14.416 Accepting connections on tcp://tibems02:7222.
2011-04-19 15:43:14.416 Server is in standby mode for 'tcp://tibems01:7222'.

Cheers...getting ready for my Easter holiday trip.

Friday, April 22, 2011

TIBCO EMS FT pair on Oracle Database 11gR2 shared storage

Related article: Read this also

In the previous article I wrote about setting up a TIBCO EMS fault tolerant pair that uses NFSv4 shared folder to store shared states. This article describes the step to set up a database-based shared state datastore.

The end-state is depicted in the following deployment diagram.


This is a 2-in-1 article. One part is to set up the required conf files and set up the Oracle Database 11gR2 for as datastore. The other part is to configure 2 separate tibemsd for the FT pairs.

1) Install hibernate package on the host machine
2) Configure tibemsd-db.conf
3) Configure stores-db.conf
4) Create EMS schema using export tool
5) Test it!

1) Install hibernate package

I will use hibernate 3.2.5 from the package hibernate_3.2.5_linux24gl23_x86.zip

1.1 Unzip the package into you temp folder.
1.2 run TIBCOUniversalInstaller-lnx-x86.bin as tibadmin, created earlier in this article.
1.3 Follow the steps in the screens below.

2) Configure tibemsd-db.conf

tibemsd-db.conf is a sample file located in $EMS_HOME/tibco/ems/5.1/sample/config folder. Copy this file from $EMS_HOME/tibco/ems/5.1/sample/config into your shared state directory.

Edit the tibemsd-db.conf and modify the following lines.

users                   = /home/tibadmin/ems_ft_cfg/users.conf
groups                  = /home/tibadmin/ems_ft_cfg/groups.conf
topics                  = /home/tibadmin/ems_ft_cfg/topics.conf
queues                  = /home/tibadmin/ems_ft_cfg/queues.conf
acl_list                = /home/tibadmin/ems_ft_cfg/acl.conf
factories               = /home/tibadmin/ems_ft_cfg/factories.conf
routes                  = /home/tibadmin/ems_ft_cfg/routes.conf
bridges                 = /home/tibadmin/ems_ft_cfg/bridges.conf
transports              = /home/tibadmin/ems_ft_cfg/transports.conf
tibrvcm                 = /home/tibadmin/ems_ft_cfg/tibrvcm.conf
durables                = /home/tibadmin/ems_ft_cfg/durables.conf
channels                = /home/tibadmin/ems_ft_cfg/channels.conf
stores                  = /home/tibadmin/ems_ft_cfg/stores-db.conf

dbstore_classpath       = /home/tibadmin/tibco/components/eclipse/plugins/com.tibco.tpcl.org.hibernate_3.2.5.001/hibernate3.jar:/home/tibadmin/tibco/components/eclipse/plugins/com.tibco.tpcl.org.com.mchange.c3p0_0.9.1.001/c3p0-0.9.1.jar:/home/tibadmin/tibco/ems/5.1/bin/antlr-2.7.6.jar:/home/tibadmin/tibco/ems/5.1/bin/asm-attrs.jar:/home/tibadmin/tibco/ems/5.1/bin/asm.jar:/home/tibadmin/tibco/ems/5.1/bin/cglib-2.1.3.jar:/home/tibadmin/tibco/ems/5.1/bin/commons-collections-2.1.1.jar:/home/tibadmin/tibco/ems/5.1/bin/commons-logging-1.0.4.jar:/home/tibadmin/tibco/ems/5.1/bin/dom4j-1.6.1.jar:/home/tibadmin/tibco/ems/5.1/bin/ehcache-1.2.3.jar:/home/tibadmin/tibco/ems/5.1/bin/jta.jar:/home/tibadmin/tibco/ems/5.1/lib/ojdbc5.jar

dbstore_driver_name     = oracle.jdbc.driver.OracleDriver
dbstore_driver_dialect  = org.hibernate.dialect.Oracle10gDialect
jre_library             = /opt/java/jre/lib/i386/server/libjvm.so
jre_option   = -Xmx256m

3) Configure stores-db.conf

stores-db.conf is a sample stores.conf file located in $EMS_HOME/tibco/ems/5.1/sample/config folder. Copy this file from $EMS_HOME/tibco/ems/5.1/sample/config into your shared state directory.

[$sys.failsafe]
type=dbstore
dbstore_driver_url=jdbc:oracle:thin:emsfs/emsfs@//oradb11g01:1521/orcl
dbstore_driver_username=emsfs
dbstore_driver_password=emsfs

[$sys.meta]
type=dbstore
dbstore_driver_url=jdbc:oracle:thin:emsmeta/emsmeta@//oradb11g01:1521/orcl
dbstore_driver_username=emsmeta
dbstore_driver_password=emsmeta

#store the non failsafe message in NFS drive.
[$sys.nonfailsafe]
  type=file
  file=/home/tibadmin/ems_ft_cfg/datastore/async-msgs.db
  mode=async
  file_crc=disabled


4) Create EMS schema using export tool

TIBCO EMS comes with a tool to create the required database schema for shared states and message storage.

Before you use the tool, ensure the TIBCO EMS host has the required database connectivity. Though you only need the Oracle JDBC java library (ojdbc5.jar) to be in the classpath, I suggest that you install the entire package of Oracle Instant Client for Linux (instantclient-basic-linux32-11.2.0.2.0.zip).

Next, create an oracle db user for each EMS data store of type dbstore.

for meta dbstore
GRANT CONNECT,RESOURCE,UNLIMITED TABLESPACE TO emsmeta IDENTIFIED BY emsmeta;

ALTER USER emsmeta DEFAULT TABLESPACE USERS;

ALTER USER emsmeta TEMPORARY TABLESPACE TEMP;

for failsafe dbstore

GRANT CONNECT,RESOURCE,UNLIMITED TABLESPACE TO emsfs IDENTIFIED BY emsfs ;

ALTER USER emsfs DEFAULT TABLESPACE USERS;

ALTER USER emsfs TEMPORARY TABLESPACE TEMP;

Note: The userid and password must be same is the one specified in the store-db.conf file in step 3.

After you have created the EMS oracle user, execute the tibemsd_util.jar as follow. The util is located under $EMS_HOME/bin where EMS_HOME is /home/tibadmin/tibco/ems/5.1 in my case.

java -jar tibemsd_util.jar -tibemsdconf /home/tibadmin/ems_ft_cfg/tibemsd-db1.conf -createall -export

You should see the following output on the stdout.

drop table ACK_RECS cascade constraints
drop table CONNECTION_RECS cascade constraints
drop table CONSUMER_RECS cascade constraints
drop table CONSUMER_RECS_IMPSELMAP cascade constraints
drop table EMS_HBLOCK_TABLE cascade constraints
drop table EMS_MESSAGES cascade constraints
drop table EMS_SYS_RECORDS_TABLE cascade constraints
drop table PRODUCER_RECS cascade constraints
drop table PURGE_RECS cascade constraints
drop table SESSION_RECS cascade constraints
drop table TXN_RECS cascade constraints
drop table VALID_MSGS_RECORD cascade constraints
drop table VALID_MSGS_RECORD_HELDMSGS cascade constraints
drop table VALID_MSGS_RECORD_HOLDMSGS cascade constraints
drop table VALID_MSGS_RECORD_NOHOLDMSGS cascade constraints
drop table ZONE_RECS cascade constraints
drop sequence hibernate_sequence
create table ACK_RECS (ACK_RECORD_ID number(19,0) not null, CONSUMER_ID number(19,0), SEQNO number(19,0), DISCARD_ACK_FLAG number(1,0), TXN_ID number(19,0), SESS_ID number(19,0), CONN_ID number(19,0), primary key (ACK_RECORD_ID))
create table CONNECTION_RECS (STORE_ID number(19,0) not null, CONNECTION_ID number(19,0), CONNECTION_TYPE number(10,0), CLIENT_ID varchar2(256 char), CONNECTION_USER varchar2(256 char), NEW_TEMP_DESTS_FLAG number(1,0), primary key (STORE_ID))
create table CONSUMER_RECS (STORE_ID number(19,0) not null, CONSUMER_ID number(19,0), SESSION_ID number(19,0), CONNECTION_ID number(19,0), DURABLE_NAME varchar2(255 char), SUBSCRIPTION_NAME varchar2(255 char), CLIENT_ID varchar2(255 char), ZONE_ID number(10,0), SELECTOR_STRING varchar2(4000 char), SELECTOR_BYTES long raw, NO_LOCAL number(1,0), CONNECTION_CONSUMER number(1,0), DESTINATION varchar2(255 char), DEST_TYPE number(3,0), CONSUMER_MSG_SEQNO number(19,0), MULTICAST_FLAG number(1,0), primary key (STORE_ID))
create table CONSUMER_RECS_IMPSELMAP (StoreId number(19,0) not null, HASHMAP_VALUE varchar2(255 char), HASHMAP_KEY varchar2(255 char) not null, primary key (StoreId, HASHMAP_KEY))
create table EMS_HBLOCK_TABLE (id number(19,0) not null, last_update timestamp not null, server_id varchar2(255 char) not null, JDBCURL varchar2(4000 char) not null, primary key (id))
create table EMS_MESSAGES (STORE_ID number(19,0) not null, MESSAGE_SEQNO number(19,0), TYPE number(3,0), PRIORITY number(3,0), DELIVERYMODE number(3,0), REDELIVERED number(3,0), DESTINATION varchar2(255 char), DESTINATION_TYPE number(3,0), EXPIRATION number(19,0), TIMESTAMP number(19,0), REPLYTO varchar2(255 char), REPLYTO_TYPE number(3,0), USERTYPE varchar2(255 char), MSGID varchar2(255 char), CORRELATIONID varchar2(255 char), COMPRESSED number(1,0), CLIENTFLAGS number(19,0), ENCODING varchar2(255 char), ENCODED_PROPERTIES raw(1024), ENCODED_SUPPRESS_CONSIDS raw(1024), TXNID number(19,0), ZONEID number(19,0), ROUTESRC varchar2(255 char), ROUTESEQNO number(19,0), ROUTECONSID number(19,0), PRODUCERID number(19,0), DELETED number(1,0) not null, MESSAGE_SIZE number(19,0), SMALL_MESSAGE_BODY raw(1024), LARGE_MESSAGE_BODY blob, primary key (STORE_ID))
create table EMS_SYS_RECORDS_TABLE (SYSTEM_REC_ID number(19,0) not null, VERSION varchar2(255 char), EMS_START_TIME varchar2(255 char), EMS_UPDATE_TIME varchar2(255 char), primary key (SYSTEM_REC_ID))
create table PRODUCER_RECS (STORE_ID number(19,0) not null, PRODUCER_ID number(19,0), SESSION_ID number(19,0), CONNECTION_ID number(19,0), DESTINATION varchar2(255 char), DEST_TYPE number(3,0), primary key (STORE_ID))
create table PURGE_RECS (PURGE_RECORD_ID number(19,0) not null, DEST_NAME varchar2(255 char), DEST_TYPE number(3,0), SEQ_NO number(19,0), primary key (PURGE_RECORD_ID))
create table SESSION_RECS (STORE_ID number(19,0) not null, SESSION_ID number(19,0), CONNECTION_ID number(19,0), ACKNOWLEDGE_MODE number(10,0), TRANSACTED number(1,0), IS_XA number(1,0), MSGID_PREFIX varchar2(255 char), primary key (STORE_ID))
create table TXN_RECS (TXNREC_STORE_ID number(19,0) not null, SESS_ID number(19,0), TXN_ID number(19,0), RECORD_ID number(19,0), TXN_STATE number(10,0), XID raw(1024), primary key (TXNREC_STORE_ID))
create table VALID_MSGS_RECORD (VALID_MSG_RECORD_ID number(19,0) not null, ACTIVE_HI_SEQ number(19,0), SMALL_MESSAGE_BODY raw(1024), LARGE_MESSAGE_BODY blob, primary key (VALID_MSG_RECORD_ID))
create table VALID_MSGS_RECORD_HELDMSGS (id number(19,0) not null, HELD_MSGS number(19,0), listIndex number(10,0) not null, primary key (id, listIndex))
create table VALID_MSGS_RECORD_HOLDMSGS (id number(19,0) not null, HOLD_MSGS number(19,0), listIndex number(10,0) not null, primary key (id, listIndex))
create table VALID_MSGS_RECORD_NOHOLDMSGS (id number(19,0) not null, NO_HOLD_MSGS number(19,0), listIndex number(10,0) not null, primary key (id, listIndex))
create table ZONE_RECS (STORE_ID number(19,0) not null, ZONE_ID number(10,0), ZONE_NAME varchar2(255 char), ZONE_TYPE number(5,0), primary key (STORE_ID))
alter table CONSUMER_RECS_IMPSELMAP add constraint FKD1B84D7B3B9C4484 foreign key (StoreId) references CONSUMER_RECS
create index DEL_IDX on EMS_MESSAGES (DELETED)
alter table VALID_MSGS_RECORD_HELDMSGS add constraint FK5908794BC44ABBE5 foreign key (id) references VALID_MSGS_RECORD
alter table VALID_MSGS_RECORD_HOLDMSGS add constraint FK6A06C9D5C44ABBE5 foreign key (id) references VALID_MSGS_RECORD
alter table VALID_MSGS_RECORD_NOHOLDMSGS add constraint FK610EB196C44ABBE5 foreign key (id) references VALID_MSGS_RECORD
create sequence hibernate_sequence

Note: For some reasons the required tables are not created in database, probably due to missing/incorrect configuration of hibernate ddl properties.

Anyway, I have converted the output above into a sql script and simply run them with sqlplus.

[oradb11g01 $] sqlplus emsmeta/emsmeta @meta.sql
[oradb11g01 $] sqlplus emsfs/emsfs @fs.sql

This following paragraphs describe the step to setting up a FT pair.

Make 2 copies of the tibemsd-db.conf files made in step 1. Call them tibemsd-db1.conf and tibemsd-db2.conf respectively.

In both files, make sure the stores parameter points to the same conf file.

stores                  = /home/tibadmin/ems_ft_cfg/stores-db.conf

In each files, do the following:

tibemsd-db1.conf

listen = tcp://tibems01:7222
ft_active = tcp://tibems02:7222

tibemsd-db2.conf

listen = tcp://tibems02:7222
ft_active = tcp://tibems01:7222

The config file referencing can be visualised in the following picture.


Start EMS on both servers using tibemsd-db1.conf and tibemsd-db2.conf, in any sequence. The EMS FT pair on Oracle Database 11g R2 are now operational.

tibadmin@tibems02:~/ems_ft_cfg$ startEMS-db.sh 

TIBCO Enterprise Message Service.
Copyright 2003-2010 by TIBCO Software Inc.
All rights reserved.

Version 5.1.5 V3 3/29/2010

2011-04-19 06:02:33.785 Process started from '/home/tibadmin/tibco/ems/5.1/bin/tibemsd'.
2011-04-19 06:02:33.785 Process Id: 1629
2011-04-19 06:02:33.786 Hostname: tibems02
2011-04-19 06:02:33.786 Hostname IP address: 172.16.10.11
2011-04-19 06:02:33.786 Detected IP interface: 127.0.0.1 (lo)
2011-04-19 06:02:33.786 Detected IP interface: 172.16.10.11 (eth0)
2011-04-19 06:02:33.786 Detected IP interface: 192.168.50.11 (eth1)
2011-04-19 06:02:33.786 Reading configuration from '/home/tibadmin/ems_ft_cfg/tibemsd-db2.conf'.
2011-04-19 06:02:34.891 Java Version 1.6.0_24
2011-04-19 06:02:34.904 Server name: 'EMS-SERVER'.
2011-04-19 06:02:34.904 Storage Location: '.'.
2011-04-19 06:02:34.904 Routing is disabled.
2011-04-19 06:02:34.904 Authorization is disabled.
2011-04-19 06:02:37.451 Active server 'tcp://tibems01:7222' not found.
2011-04-19 06:02:37.451 Continuing as active server.
2011-04-19 06:02:37.456 Accepting connections on tcp://tibems02:7222.
2011-04-19 06:02:37.456 Recovering state, please wait.
2011-04-19 06:04:13.561 Store '$sys.failsafe' locked by 'EMS-SERVER'
2011-04-19 06:04:13.561 Creating system record for store '$sys.failsafe' ...
2011-04-19 06:04:13.660 Store '$sys.meta' locked by 'EMS-SERVER'
2011-04-19 06:04:13.660 Creating system record for store '$sys.meta' ...
2011-04-19 06:04:14.342 Server is active.
2011-04-19 06:06:00.481 Backup server 'EMS-SERVER@tibems01' has connected.

tibadmin@tibems01:~/ems_ft_cfg$ startEMS-db.sh 

TIBCO Enterprise Message Service.
Copyright 2003-2010 by TIBCO Software Inc.
All rights reserved.

Version 5.1.5 V3 3/29/2010

2011-04-19 06:05:58.102 Process started from '/home/tibadmin/tibco/ems/5.1/bin/tibemsd'.
2011-04-19 06:05:58.103 Process Id: 10071
2011-04-19 06:05:58.103 Hostname: tibems01
2011-04-19 06:05:58.103 Hostname IP address: 172.16.10.10
2011-04-19 06:05:58.103 Detected IP interface: 127.0.0.1 (lo)
2011-04-19 06:05:58.103 Detected IP interface: 172.16.10.10 (eth0)
2011-04-19 06:05:58.103 Detected IP interface: 192.168.50.10 (eth1)
2011-04-19 06:05:58.103 Reading configuration from '/home/tibadmin/ems_ft_cfg/tibemsd-db1.conf'.
2011-04-19 06:05:58.410 Java Version 1.6.0_24
2011-04-19 06:05:58.419 Server name: 'EMS-SERVER'.
2011-04-19 06:05:58.419 Storage Location: '.'.
2011-04-19 06:05:58.419 Routing is disabled.
2011-04-19 06:05:58.420 Authorization is disabled.
2011-04-19 06:06:00.401 Accepting connections on tcp://tibems01:7222.
2011-04-19 06:06:00.401 Server is in standby mode for 'tcp://tibems02:7222'.

Lastly, I have a Gigabit connection between the EMS host and the DB host, so I will just let the shared state traffic to pass through my MSG_VLAN. For detail of the Laboratory network topology, refer to this article.

Cheers,

Wednesday, April 20, 2011

TIBCO EMS 5.1.5 Ubuntu Fault Tolerant Pair On NFS

If you'd like to jump the gun and try out the FT pair on Oracle Database, then this article is for you.

In the previous article, I briefly wrote about setting a file-based shared state TIBCO EMS FT pair on Windows platform. This article shows you how to install TIBCO EMS on linux hosts for fault tolerance configuration. The installations will use NFSv4 mounted folder for the shared states.

TIBCO EMS 5.1 is supported on the following Unix/Linux distributions under Intel Architecture.
  • Novell SUSE Linux Enterprise 10
  • Red Hat Enterprise Linux 4
  • Oracle Enterprise Linux 4
  • Mac OS X 10.4+
  • Solaris 10
Though not officially supported on Ubuntu, the installation of TIBCO EMS 5.1.5 on Ubuntu Linux is very straight forward.

This article shows you the installation made under a non-root user. If you need to use multicast you will need to start the daemon as root. I personally do not like to use setuid on the files.

There are 2 possible options when setting up FT pair using NFS storage.

1) Shared binary folder, such as /mountpoint/tibco
2) Independent binary folders on each EMS hosts, such as /home/tibadmin/tibco

In both options, the intances will use a shared NFS mountpoint for the shared states.

I have picked option 2 for my environment because I want to be able to run a single instance of TIBCO EMS if my Openfiler service is unavailable.

The installation package of TIBCO EMS 5.1.5 is TIB_ems_5.1.5_linux26gl23_x86.tar.gz

To install TIBCO EMS 5.1.5, follow the following steps:

1) Create a user account that you will use to install and run your EMS

#mkdir -p /home/tibadmin
#groupadd -g 1002 tibadmin
#useradd -u 1002 -g tibadmin tibadmin -s /bin/bash -d /home/tibadmin
#chown -R tibadmin:tibadmin /home/tibadmin

Note the uid and gid down, we will need to create a user with identical uid and gid on the Openfiler server in the absence of NIS or LDAP. This is a very important step for the NFSv4 idmap to work properly.

Login as tibadmin

#su - tibadmin
$

2) Copy the installation package into a temp folder. Make sure it is owned by tibadmin:tibadmin

$sudo chown tibadmin:tibadmin TIB_ems_5.1.5_linux26gl23_x86.tar.gz

3)Unzip the installation package

$gzip -d TIB_ems_5.1.5_linux26gl23_x86.tar.gz

4) Untar the tarball

$tar -xvf TIB_ems_5.1.5_linux26gl23_x86.tar

5) Execute the extracted installation binary

$./TIBCOUniversalInstaller-lnx-x86.bin

Follow the steps in the screens below.









6) Install the libstdc++5 package. This is required because Ubuntu 10.10 comes with libstdc++6 and not libstdc++5. Download the package (google it!) and install as follow.

dpkg -i libstdc++5_3.3.6-20_i386.deb

Repeat step 1 to 5 on another server instance, in my case it is tibems02.

Up to now, we have 2 instances of TIBCO EMS local installations. The next step is to set up the shared storage for the FT pair.

To set up a shared storage for the configuration files and message persistence, I use openFiler NAS and mount the exported directory using NFSv4 client.

1) ssh into the openfiler sever from any host with connectivity to the NFS server.

tibadmin@tibems01:~$ssh root@openfiler
root@openfiler's password: 
Last login: Wed Apr 20 12:40:18 2011 from 192.168.13.101
[root@openfiler ~]# 

2) Create tibadmin user on the openfiler server

[root@openfiler ~]# groupadd -g 1002 tibadmin
[root@openfiler ~]# useradd -u 1002 -g tibadmin -s /sbin/nologin tibadmin
[root@openfiler ~]# id tibadmin
uid=1002(tibadmin) gid=1002(tibadmin) groups=1002(tibadmin)
[root@openfiler ~]# 

3) Setup the shared filesystem exports from openfiler (or on your favourite Linux/UNIX distributions).

4) Edit the /etc/exports file on NFS server. Add fsid=0, you should see something similar to this.

openfiler1:/yoursharedfolderforemsconfig 192.168.50.10 (rw,fsid=0,no_subtree_check,no_root_squash,sync) 192.168.50.11 (rw,fsid=0,no_subtree_check,no_root_squash,sync)

Note 1: 192.168.50.10 and 192.168.50.11 are IP address NAS NIC's of my EMS hosts. Refer to this VMWare ESXi infrastructure for the full picture.

Note 2: The fsid=0 option will will be removed if you change the NFS share configurations from the webgui of openfiler. Just back up a copy of /etc/exports file, in case it is overwritten you always have a copy of it.

Note 3: While it is not the best practice to setup a NFSv4 export without a well thought through pseudo-filesystem, this configuration meets my initial requirement.

If you have time, you should look at setting up the psuedo-fs in this way.

                   /tibco (fsid=0,ro)
                            |
             -------------------------------
            |            |                  |
        ems_ft_cfg   ems_logging    ems_ft_datastore

On the clients (tibems01 and tibems02), the shared folder is mounted with the following mount options, specified in the /etc/fstab file. Note the nfs4 in particular.

$mkdir -p /home/tibadmin/ems_ft_cfg
$mount -t nfs4 -o rw openfiler1:/ home/tibadmin/ems_ft_cfg

I am not sure if NFS exported by OpenFiler meets the fault tolerance storage criteria. At least according to the TIBCO EMS User Guide, Chapter 5, the fnctl() POSIX file locking semantic on NFSv4 should work. For compliance to other criteria, a separate test is required.

struct flock fl;
 int err;
 fl.l_type = F_WRLCK;
 fl.l_whence = 0;
 fl.l_start = 0;
 fl.l_len = 0;
 err = fcntl(file, F_SETLK, &fl);

To setup for FT, the next step is to prepare the conf files on both servers (tibems01 and tibems02) and point them to the NFSv4 exported folder mouted under home/tibadmin/ems_ft_cfg.

1) Copy all the files from the sample/config directory (/home/tibadmin/tibco/ems/5.1/samples/config) into /home/tibadmin/ems_ft_cfg

tibadmin@tibems01:/$ cp /home/tibadmin/tibco/ems/5.1/samples/config/* /home/tibadmin/ems_ft_cfg/

2) Create a sub-folder under /home/tibadmin/ems_ft_cfg, name it "datastore" (it becomes "/home/tibadmin/ems_ft_config/datastore")

tibadmin@tibems01:/$ mkdir /home/tibadmin/ems_ft_cfg/datastore

3) Edit /home/tibadmin/ems_ft_cfg/tibemsd.conf, populate the following configuration parameters.

users                   = /home/tibadmin/ems_ft_cfg/users.conf
 groups                  = /home/tibadmin/ems_ft_cfg/groups.conf
 topics                  = /home/tibadmin/ems_ft_cfg/topics.conf
 queues                  = /home/tibadmin/ems_ft_cfg/queues.conf
 acl_list                = /home/tibadmin/ems_ft_cfg/acl.conf
 factories               = /home/tibadmin/ems_ft_cfg/factories.conf
 routes                  = /home/tibadmin/ems_ft_cfg/routes.conf
 bridges                 = /home/tibadmin/ems_ft_cfg/bridges.conf
 transports              = /home/tibadmin/ems_ft_cfg/transports.conf
 tibrvcm                 = /home/tibadmin/ems_ft_cfg/tibrvcm.conf
 durables                = /home/tibadmin/ems_ft_cfg/durables.conf
 channels                = /home/tibadmin/ems_ft_cfg/channels.conf
 stores                  = /home/tibadmin/ems_ft_cfg/stores.conf
   
 store                   = /home/tibadmin/ems_ft_cfg/datastore

 max_msg_memory          = 256MB

 #uncomment accordingly
  ft_active               = tcp://tibems01:7222
 #ft_active               = tcp://tibems02:7222

 ft_heartbeat            = 3
 ft_activation           = 10
 ft_reconnect_timeout    = 60

4) Make 2 copies of the tibemsd.conf files. One as tibemsd1.conf, the other as tibemsd2.conf. In tibemsd2.conf, comment out the appropriate ft_active parameter.

#ft_active               = tcp://tibems01:7222
 ft_active               = tcp://tibems02:7222

Summary:
For Primary instance

listen = tcp://tibems01:7222
ft_active = tcp://tibems02:7222

For secondary instance

listen = tcp://tibems02:7222
ft_active = tcp://tibems01:7222

To start the primary and secondary servers.

Login to tibems01 as tibadmin
tibadmin@tibems01:cd ~/tibco/ems/5.1/bin
tibadmin@tibems01:~/tibco/ems/5.1/bin$./tibemsd -config /home/tibadmin/ems_ft_cfg/tibemsd1.conf

You should see this.

TIBCO Enterprise Message Service.
Copyright 2003-2010 by TIBCO Software Inc.
All rights reserved.

Version 5.1.5 V3 3/29/2010

2011-04-16 03:27:01.550 Process started from './tibemsd'.
2011-04-16 03:27:01.550 Process Id: 1469
2011-04-16 03:27:01.550 Hostname: tibems01
2011-04-16 03:27:01.551 Hostname IP address: 172.16.10.10
2011-04-16 03:27:01.551 Detected IP interface: 127.0.0.1 (lo)
2011-04-16 03:27:01.551 Detected IP interface: 172.16.10.10 (eth0)
2011-04-16 03:27:01.551 Detected IP interface: 192.168.50.10 (eth1)
2011-04-16 03:27:01.551 Reading configuration from '/home/tibadmin/ems_ft_cfg/tibemsd1.conf'.
2011-04-16 03:27:01.582 Server name: 'EMS-SERVER'.
2011-04-16 03:27:01.582 Storage Location: '/home/tibadmin/ems_ft_cfg/datastore'.
2011-04-16 03:27:01.582 Routing is disabled.
2011-04-16 03:27:01.582 Authorization is disabled.
2011-04-16 03:27:01.681 Active server 'tcp://tibems02:7222' not found.
2011-04-16 03:27:01.681 Continuing as active server.
2011-04-16 03:27:01.686 Accepting connections on tcp://tibems01:7222.
2011-04-16 03:27:01.687 Recovering state, please wait.
2011-04-16 03:27:02.017 Server is active.


Login to tibems02 as tibadmin

tibadmin@tibems02:cd ~/tibco/ems/5.1/bin
tibadmin@tibems02:~/tibco/ems/5.1/bin$./tibemsd -config /home/tibadmin/ems_ft_cfg/tibemsd2.conf

You should see this.
TIBCO Enterprise Message Service.
Copyright 2003-2010 by TIBCO Software Inc.
All rights reserved.

Version 5.1.5 V3 3/29/2010

2011-04-16 03:31:45.008 Process started from './tibemsd'.
2011-04-16 03:31:45.009 Process Id: 15288
2011-04-16 03:31:45.009 Hostname: tibems02
2011-04-16 03:31:45.009 Hostname IP address: 172.16.10.11
2011-04-16 03:31:45.009 Detected IP interface: 127.0.0.1 (lo)
2011-04-16 03:31:45.009 Detected IP interface: 172.16.10.11 (eth0)
2011-04-16 03:31:45.009 Detected IP interface: 192.168.50.11 (eth1)
2011-04-16 03:31:45.009 Reading configuration from '/home/tibadmin/ems_ft_cfg/tibemsd2.conf'.
2011-04-16 03:31:45.020 Server name: 'EMS-SERVER'.
2011-04-16 03:31:45.020 Storage Location: '/home/tibadmin/ems_ft_cfg/datastore'.
2011-04-16 03:31:45.020 Routing is disabled.
2011-04-16 03:31:45.020 Authorization is disabled.
2011-04-16 03:31:45.076 Accepting connections on tcp://tibems02:7222.
2011-04-16 03:31:45.076 Server is in standby mode for 'tcp://tibems01:7222'.

Note: One the primary server, it has detected that the secondary server is up and connected.

2011-04-16 03:31:45.075 Backup server 'EMS-SERVER@tibems02' has connected.

Now try to shutdown the primary server, the secondary server will become active.

2011-04-16 03:41:47.082 Connection to primary server 'tcp://tibems01:7222' has been lost.
2011-04-16 03:41:47.083 Server activating on failure of 'tcp://tibems01:7222'.
2011-04-16 03:41:47.083 Server rereading configuration.
2011-04-16 03:41:47.126 Recovering state, please wait.
2011-04-16 03:41:47.135 Server is now active.

Cheers,

Thursday, April 14, 2011

Hack - Export existing volume on a physical drive via Openfiler VM

I have a piece of relatively old 150GB HDD with Linux file system, lying around for quite some time. Having installed Windows 7 64-bits as my primary OS, I hardly have a chance to make good use of that disk. To read Linux file system from Windows OS, there are a few good options out there, such as the one by Disk Internal and this opensource one.

But if you are like me, running a bunch of Linux virtual machines on a Windows 7 host, you might find this article helpful.

What we are going to do is to export the entire existing filesystem on this hard disk from the Openfiler virtual machine. But Openfiler uses Logical Volume Manager and does not give you an option to expose existing file systems without destroying the existing partition table. Well, I must say I haven't search hard enough on how to do that officially in Openfiler, but am happy with my workaround, which has been completed in 5 minutes.

This is what we will do.

1) Attach the physical volume to the Openfiler VM.

I assume you know how to figure out which physical drive is the one that you want to attach to your Openfiler VM. Be extremely careful not to attach the wrong one. Your Linux formatted disk drive will not appear in the Windows file explorer. To find out which one is your disk, try device manager, or Google is your friend.

On my Windows host, my linux harddrive is the Disk 3.


Once you have attached your physical disk to the VM, boot it up.

2) Create a share folder in Openfiler under one of your existing shares.




3) Mount the raw device to the new share folder created in step 2.

- Locate your raw device on the Openfiler server by using fdisk command.

[root@openfiler01 physical]# fdisk -l
Disk /dev/hda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000e329f

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1       19457   156288321   83  Linux

To mount,

#mount -t ext4 -o rw /dev/hda1 /mnt/home_shared/sharedvol/physical

Note: My HDD was formatted with ext4 fs.

You might need to re-export your filesystem

#exportfs -rv

Inspect the /etc/exports file in the Openfiler server.

/mnt/home_shared/sharedvol/physical 192.168.0.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)  192.168.19.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)  192.168.50.0/255.255.255.0(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)

Go to your client Windows OS (assuming it is on the same network as the Openfiler server, in my case 192.168.50.0/24), map to \\openfiler\home_shared\sharedvol\physical, you should see your existing files and should be able to copy into the drive as Samba mount.

Or from the command prompt

net use drive_letter: \\openfiler\home_shared\sharedvol\physical


Go to your NFS client Linux OS and create a mount point

#mkdir /mnt/physical

Mount with the following options

mount -o rw openfiler:/mnt/home_shared/sharedvol/physical /mnt/physical

Look at the new mounted folder, it works!

root@tibems01:/mnt/physical# df -m .
Filesystem           1M-blocks      Used Available Use% Mounted on
192.168.50.100:/mnt/home_shared/sharedvol/physical
                        150230     78025     64574  55% /mnt/physical
root@tibems01:/mnt/physical# 

Don't forget to make a backup of your openfiler /etc/exports, just in case.

Well, a hack that works and will survive Openfiler boot cycle.

Cheers,

Wednesday, April 6, 2011

svc: failed to register lockdv1 RPC service (errno 97). lockd: cannot monitor

Alright, you have searched the internet in and out, up and down, plough through the forums and read all the available man pages on earth regarding NFS and still did not find answer you longed for, because they are not directly available/described/discussed whatever it might be. Well, this might help you.

Good thing about linux is that it keeps you at your toes.

When mounting a NFS directory, you might see this error in your /var/log/messages trace.

Apr  1 09:10:45 orasoa1 kernel: [  895.012894] svc: failed to register lockdv1 RPC service (errno 97).
Apr  1 09:25:31 orasoa1 kernel: [ 1778.601154] svc: failed to register lockdv1 RPC service (errno 97).

Or when acquiring a file lock either by some applications, Oracle SOA Admin Server for example, or when simply trying to manually acquire an exclusive lock of a file on NFS.

$flock lockfile -c 'vi dosomeactivities'

You might see these errors.

Apr  1 09:25:57 orasoa1 kernel: [ 1804.313073] lockd: cannot monitor 192.168.50.100
Apr  1 09:33:59 orasoa1 kernel: [ 2285.828349] lockd: cannot monitor 192.168.50.100
Apr  1 09:36:39 orasoa1 kernel: [ 2444.869671] lockd: cannot monitor 192.168.50.100

where "192.168.50.100" is the NAS server IP address (the 'openfiler1' host).

It is probably because you don't have a DNS for your host names or simply because you haven't updated your /etc/hosts file to include the name of your NAS server.

Try this if you had similar problem, I hope it saves your precious time.

On your client host(s), stick this line into your /etc/hosts file(s)

#vi /etc/hosts

nas_server_ip_addr nas_sever_hostname #example 192.168.50.100    openfiler1
nas_client_ip_addr nas_client_hostname     #example 172.16.20.50      orasoa1

Cheers,

Friday, April 1, 2011

Oracle Service Bus (11.1.1.4.0) on Ubuntu 10.10, deployment into ESXi Virtual Infrastructure

In our previous article, we installed Oracle SOA Suite PS3 into a host on a Virtual Infrastructure.

This article discusses the installation of Oracle Service Bus 11.1.1.4.0 into the same middleware home of the SOA Suite. This is possible because Oracle has released OSB 11g as a component of SOA Suite of products. OSB 11g PS3 and SOA Suite 11g PS3 shares the same Oracle Weblogic Server version, 10.3.4.

Pre-requisite:
- Minimum swap space = 512MB. Instructions to add file-based swap space is available here.
- Java Runtime Environment. Reuse the one from previous installation of Oracle Service Bus
- NFS mount to the existing Oracle installation file system.
- oracle user account and oinstall group are created on the target host. Ensure identical uid and gid (in this case, 500 and 1001 respectively)

Required binary
- ofm_osb_generic_11.1.1.4.0_disk1_1of1.zip downloadable from Oracle website.

Installation Steps:
- unzip downloaded files to a temp folder. You will get 2 directories after unzip. Disk1 and Disk2
- cd to Disk1
- execute runInstaller. When asked for java directory, use the one (either Sun JDK or JRockit) installed in WLS.
- select the previously created middleware home of SOA Suite 11.1.1.4.0. This reduces the installation footprint because it contains the Oracle WebLogic Server 11gR1 (10.3.4) and the optional component Oracle Enterprise Pack for Eclipse (OEPE) required by the OSB 11.1.1.4.0. The other middleware home (of Weblogic Portal, wlpmw) cannot be used because it contains Oracle Weblogic Server 10.3.2 which is not compatible with SOA PS3/OSB PS3.

Oracle Server Bus OSB Installation Screens













Upon completion of OSB installation in a SOA Suite middleware home, the following elements are added to the inventory.xml file in the oraInventory/ContentsXML directory.


    
       
    

 

The following image depicts the summary of the resultant directory structure after WLS and SOA installations.
The final step is to make sure your operating environment has the right variables set up.As root,
#vi /etc/bash.bashrc
Add the following lines
export ORACLE_BASE=/u01/app/oracle
export MW_HOME=$ORACLE_BASE/product/soamw
export WLS_HOME=$MW_HOME/wls_10.3
save the file.

VoilĂ , Oracle Service Bus in the same home as SOA Suite.

Happy OSB'ing,

Wednesday, March 30, 2011

Oracle Weblogic Portal 10.3.2 on Ubuntu 10.10, deployment into ESXi Virtual Infrastructure

This article shows you how to install Oracle Weblogic Portal into a NFS folder with Oracle DB, SOA Suite and OSB installed.

The infrastructure details can be found here.

Unlike Oracle SOA Suite 11g and Oracle Service Bus 11g which use Weblogic server 10.3.4, Weblogic Portal 10.3.2 requires Weblogic Server 10.3.2. To fulfill the version compatibility, the WLP has been installed into its own middleware home, wlpmw, sits at the same level as osbmw in the file hierarchy.


To install Weblogic Portal 10.3.2, the following installation binary is used (downloadable from Oracle website).

portal103_linux32.bin

Installation steps

1) Login as oracle user
2) execute portal103_linux32.bin

$./portal103_linux32.bin

3) Follow the steps depicted in the following images







Uncheck the 'Run Quickstart' checkbox, we will create the domain later.

Happy Portaling.

Cheers,