Troubled SOA Patch Set 2

Grrrr, lost about one whole day trying to figure out a problem that is caused by 'improper' patch of SOA suite from 11.1.1.2.0 to 11.1.1.3.0.

Scouted the internet and found only one thread in OTN, but there is no answer to it other than people suggesting re-installing the whole SOA and re-apply patch from scratch.

The problem is highlighted here http://forums.oracle.com/forums/thread.jspa?threadID=1005454

How I got into this problem? Here is what I suspected.

After installing the package SOA suite 11.1.1.2.0, I created my domain with soa_server and bam_server. Played around with it, unable to start the managed servers. Googled around, realised that I had to apply the PS2. Downloaded, and patched.

After that, the problem occurs during soa_server1 startup. After a bit of digging, I suspect it is the old cache of ear and jar in some tmp folder have caused the problem. Cleaning up the cache appears to have caused the startup to throw out different sort of problem, which said 'unable to find ... in xxx ear'. After some more efforts, I finally throw in the towel and figured it would be more productive just to re-install the entire WLS, SOA suite and Jdev.

Summary of the typical sequence of steps of installation:

1) Oracle DB 11g R1
2) rcu 11.1.1.3.0
3) WLS 10.3.3
4) SOA Suite 11.1.1.2.0
5) SOA PS2 11.1.1.3.0
6) Jdeveloper 11.1.1.3.0

Everything works now.

Using Oracle Repository Creation Utility (RCU) to create Metadata Repository

The previous installation of Oracle Database 11g R2 on Ubuntu 10.04 (Lucid) was successful. This article shows you how to use oracle RCU (repository creation utility) to create metadata repository on this database instance.

Step 1 - Get the Sun JDK (if you haven't done so)

Full instructions are available here http://www.ubuntugeek.com/how-install-sun-java-runtime-environment-jre-in-ubuntu-10-04-lucid-lynx.html. Note: If you don't wish to install a jdk right now, it is alright. You can still use the jdk that comes with rcu package. It is located under the jdk folder of rcuHome.

To install sun-jdk, do the following.


kj@oradb-lucid:~$ sudo add-apt-repository 'deb http://archive.canonical.com/ lucid partner'
kj@oradb-lucid:~$ sudo apt-get update
kj@oradb-lucid:~$ sudo apt-get install sun-java6-jre sun-java6-plugin sun-java6-fonts

after the installation, voila...

kj@oradb-lucid:~$ java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing)

Step 2 - Get the RCU package from Oracle download site.

Get the package from here.

Unzip the archive to a temp folder in your $HOME. On my machine, it is /home/kj/temp.

Step 3 - Ensure environment variables are properly set
On my machine, they are

ORACLE_SID=orcl
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
JAVA_HOME=/usr/lib/jvm/java-6-sun
PATH=/u01/app/oracle/product/11.2.0/dbhome_1/bin:/usr/local/bin:/usr/bin:/bin

If your default shell is bash, you can add these entries into your .bashrc file located under your $HOME directory.

I have these in mine for both oracle and myself.

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1
export ORACLE_SID=orcl
export JAVA_HOME=/usr/lib/jvm/java-6-sun
PATH=$ORACLE_HOME/bin:$PATH

Step 4 - Start your oracle instance (if it hasn't been started)
Login as oracle user

kj@oradb-lucid:~$ su - oracle
Password:
oracle@oradb-lucid:~$

* Check the env variables. You should have all the required vars, else, ensure your have performed step 3.

Start sqlplus

oracle@oradb-lucid:~$ sqlplus / 'as sysdba'

SQL*Plus: Release 11.2.0.1.0 Production on Tue Jun 15 15:29:38 2010

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to an idle instance.

Startup the database, it will mount and open if successful.

SQL> startup
ORACLE instance started.

Total System Global Area  431038464 bytes
Fixed Size            1337016 bytes
Variable Size          281020744 bytes
Database Buffers      142606336 bytes
Redo Buffers            6074368 bytes
Database mounted.
Database opened.

Quit the sqlplus

SQL> quit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Step 5 - start the listener using lsnrctl command

oracle@oradb-lucid:~$ lsnrctl start

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 15-JUN-2010 15:35:12

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Starting /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 11.2.0.1.0 - Production
System parameter file is /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
Log messages written to /u01/app/oracle/diag/tnslsnr/oradb-lucid/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                15-JUN-2010 15:35:14
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/oradb-lucid/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
The listener supports no services
The command completed successfully

Note that the listener isn't listening to any service immediately after startup. Give your oracle instance sometime to register itself with the listener. Type lsnrctl status again to find out.

oracle@oradb-lucid:~$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 15-JUN-2010 15:37:20

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                15-JUN-2010 15:35:14
Uptime                    0 days 0 hr. 2 min. 6 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/oradb-lucid/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
Services Summary...
Service "orcl" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orclXDB" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
The command completed successfully

Step 6 - Ensure the system parameters values of the oracle instance meet the requirements.

The typical oracle 11g r2 installation should meet most of the requirements. RCU specifically requires processess to be 300 or more for WEBCENTER component. Other than this one, everything else works for me.

To alter the system parameter, connect to sqlplus as sysdba.

oracle@oradb-lucid:~$ sqlplus / as sysdba
Issue the following sql statement

SQL> ALTER SYSTEM SET processes=300 SCOPE=SPFILE;
restart the db

SQL> shutdown immediate

SQL> startup

Step 5 - Run the rcu
Go to the folder where you have untar'ed. My was $HOME/temp/. Change directory to the bin folder. The installation script is located in $HOME/temp/rcu/bin. You can run the installation script as yourself instead of as oracle because by default oracle won't have sufficient privilege on your $HOME folder.

kj@oradb-lucid:~$ ./rcu 

Follow the instruction, enter the values as depicted in the screenshots of rcu here. You shouldn't face any problem during the installation as you have adjusted your DB system parameters.

The next step is to setup Oracle Weblogic Server that serves as the container for all Oracle SOA Suite components.

Good luck.

YAIG - Yet another Oracle Database 11g R2 on Ubuntu 10.04 (Lucid) Installation Guide

Rather than re-inventing the wheel, this is my experience installing the oracle database 11gR2 on a 'virgin' ubuntu 10.04 32bit guest OS on vmware workstation, using the steps found here http://www.pythian.com/news/13291/installing-oracle-11gr2-enterprise-edition-on-ubuntu-10-04-lucid-lynx/, which in turn was based on the good work done here http://www.pythian.com/news/2329/installing-oracle-11gr1-on-ubuntu-904-jaunty-jackalope/.

I have tested the steps and found the following additional works are needed. The installation of the guest OS has on the vmware workstation been 'vanila', accepted all default settings.

VM hardware configurations:

RAM = 1048MB (1024MB doesn't work, failed the oracle installer pre-req check)

Vdisk = 20GB

CPU = 1

Core = 1

Additional step 1 - Fix the swap space

Oracle installer complained that I do not have enough swap space. The default is only around 850MB while the doco says 1.5GB of swap space is required for RAM between 1GB to 2GB (section 2.2.1 Memory Requirements of Oracle database installation guide). To setup additional swap space, follow the instruction in http://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/

How-To

# dd if=/dev/zero of=/swapfile1 bs=1024 count=1572864

note: 1.5GB = 1024MB + 512MB = 1536MB (1024 * 1536MB = 1572864 block size)

# mkswap /swapfile1

# swapon /swapfile1

to mount on startup, edit the fstab file

# vi /etc/fstab

add the following line

/swapfile1 swap swap defaults 0 0

reboot the system. Or simply run these commands in the said order.

# swapoff -a

# swapon -a

Additional step 2 - The run level

Bump up the runlevel from 2 to 3. The default run level for ubuntu is 2. According to the doco, run level 2 up to 5 in ubuntu are the same. This step is just to make the installer happy.

How-To.

Issue the following command as root. 

# init 3

Additional step 3 - Kernel Parameters

The pre-req also fails one of the recommended kernel parameters. It seems that the installer is taking the required value literally even if the configured value is larger than the recommended ones.

It says:

This is a prerequisite condition to test whether the OS kernel parameter "ip_local_port_range" is properly set. (more details)

Expected Value

 : between 9000 & 65500

Actual Value

 : between 9000 & 65535

Obviously to make the installer happy again, we will need to change the /etc/sysctl.conf again. 

Changing from

net.ipv4.ip_local_port_range = 9000 65535

to

net.ipv4.ip_local_port_range = 9000 65500

should suffice.

Additional step 4 - Limits

The same thing as in step 3 needs to happen to /etc/security/limits.conf

The pre-req statuses:

This is a prerequisite condition to test whether the hard limit for "maximum open file descriptors" is set to at least 65536. (more details)

Expected Value

 : 65536

Actual Value

 : 65535

This is a prerequisite condition to test whether the soft limit for "maximum open file descriptors" is set to at least 1024. (more details)

Expected Value

 :  1024

Actual Value

 :  1023

This is a prerequisite condition to test whether the hard limit for "maximum user processes" is set to at least 16384. (more details)

Expected Value

 :  16384

Actual Value

 :  16383

We need to fix all of that. It is weird, but the fix is easy. The root cause? I don't know.

oracle soft nproc 2047

oracle hard nproc 16384  #(was 16383)

oracle soft nofile 1024 #(was 1023)

oracle hard nofile 65536 #(was 65535)

Until you see a list of 'Failed' status with regards to missing package, you should be able to safely check the 'ignore' box and proce installation. They are ignorable as stated in the cvu_prereq.xml. (yeah, the standalone installation appears to be using the pre-req definition file which is part of the 'cluster verification utility')

My screen looked like this before I proceed with success.

Lastly, if you have elected to create a new database instance during installation and if you want to install RCU (repository creation utility) for the SOA suite on this instance, you must use AL32UTF8 as the character set encoding.

After the installation, you can try to startup the dbconsole. One of the internet forums also suggested that you need to change the emctl script (located in /$ORACLE_HOME/bin/) to use the bash shell, else you will see error like ulimit: 25: bad number if you attempt to start the dbconsole.

To start the Enterprise Manager, just type the following command as oracle user.

oracle@oradb-lucid:~$ emctl start dbconsole

For Oracle RCU installation on this database instance, please refer to this article.