Category: Server Administration Weblogic

Troubleshooting PeopleTools Version Mismatch

I got the following error message after installing the PeopleTools 8.52.10 patch:

PeopleTools release (8.52.10) for web server/ Application Designer ‘lxdevweb02.pugetsound.edu’ is not the same as Application Server PeopleTools release (8.52.02). Access denied.

CHECK APPSERVER LOGS. THE SITE BOOTED WITH INTERNAL DEFAULT SETTINGS, BECAUSE OF: bea.jolt.ApplicationException: TPESVCFAIL – application level service failure

To fix the problem, I just reconfigured the Application Server.

I found that I needed to change something to make it think that it needs to load the configuration.  So, I changed the WSL option.  After loading the configuration, I changed it back and loaded it a second time.

After booting the Application Server again, everything worked fine.  The instructions didn’t say anything about reconfiguring the Application Server, but I found that I had to do it.  I don’t know if this has anything to do with it, but I did change my PS_HOME environment variable as part of the patch installation.  The tools version  (patch number included) is in the path for my PS_HOME.

Now, on a side note, if you get just this message:

CHECK APPSERVER LOGS. THE SITE BOOTED WITH INTERNAL DEFAULT SETTINGS, BECAUSE OF: bea.jolt.ApplicationException: TPESVCFAIL – application level service failure

The problem is probably that the web server cannot talk to the application server.  Here are a few things to check:

  • Is the application server started?  Use psadmin to check the application server status.
  • Is the firewall blocking connections?  Use “sudo service iptables stop” to turn off the firewall.  If that fixes the problem, configure iptables to allow the JSL port.
  • Are the application server and web server using the same port?  Check the psappsrv.cfg file to see what the JSL port is (or use psadmin).  Check the <ps cfg home>/webserv/<domain name>/applications/peoplesoft/PORTAL.war/WEB-INF/psftdocs/<site name>/configuration.properties file for the psserver port (or use psadmin)
  • Is the app server hostname configured to localhost?  On the application server, ping the server name.  If it comes back with 127.0.0.1, you have a problem.  If your configuration is supposed to be listening on the hostname, it will be listening on localhost.  Either change /etc/hosts or put the actual IP address in the app server configuration file.
  • Is the host name on the web server the actual hostname on the app server?  It’s pretty easy to accidentally just put the web server host name as the application server.

I hope those thoughts help more than just me.

PIA needs JRockit!

The other day, I was trying to finish up an install for someone, and ran into a problem.  I couldn’t get psadmin to create a new PIA domain.  Finally, thanks to OTN, I found that I had to have JRockit JDK set as the Java environment for WebLogic.  Note that this was to create the domain not just start the domain.

Logs

If there was one thing I could change about this whole thing, it would the logs.  I spent so long on this issue just because I had no clues.  We have got to have more logs to look at for clues!  If am missing something, please comment below and point it out.  If not, Oracle, please add some logging — at least maybe print out what JVM it is using.

This is the first log file that I found: <PS_CFG_HOME>/webserv/piainstall_<domain name>.log

[sourcecode]
Install Action : CREATE_NEW_DOMAIN

Creating Domain…

Deploying Web Applications…

Deploying WebLogic Extension files…

Deploying PeopleSoft Site files…

Deploying PeopleSoft Site Doc files…

Completed.

PS_CFG_HOME: /psoft/pscfgs/fnprd

PIA_INSTALL_FAIL
[/sourcecode]

This is the second log: /tmp/piaInatallLog_<timestamp>

[sourcecode]
Install Action : CREATE_NEW_DOMAIN

Creating Domain…

Deploying Web Applications…

Deploying WebLogic Extension files…

Deploying PeopleSoft Site files…

Deploying PeopleSoft Site Doc files…

Completed.
[/sourcecode]

Also, in <PS_CFG_HOME>, I found the  wlspiainstall.properties which included all of the settings that it was using to create the domain.

Avoiding the Error

The best way to avoid this error is to make sure that you set the JAVA_HOME to your JRockit installation before installing WebLogic.  So, assuming that you have JRockit installed at /opt/jdk, you can run something like this:

[sourcecode]
export JAVA_HOME=/opt/jdk
export PATH=$JAVA_HOME/bin:$PATH
which java
java -version
[/sourcecode]

Your output should look something like this:

[sourcecode]

$export JAVA_HOME=/opt/jdk
$export PATH=$JAVA_HOME/bin:$PATH
$which java
/opt/jdk/bin/java
$java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Oracle JRockit(R) (build R28.1.0-123-138454-1.6.0_20-20101014-1350-linux-x86_64, compiled mode)
[/sourcecode]

If it looks like this, you have something wrong.  This is open source OpenJDK JVM:

[sourcecode]

$which java
/usr/bin/java
$java -version
java version "1.7.0_03"
OpenJDK Runtime Environment (IcedTea7 2.1.1pre) (7~u3-2.1.1~pre1-1ubuntu3)
OpenJDK 64-Bit Server VM (build 22.0-b10, mixed mode)
[/sourcecode]

Or, this is the Sun JDK, which caused me problems:

[sourcecode]

$java -version
java version "1.5.0_30"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_30-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_30-b03, mixed mode)
[/sourcecode]

Now, once you have that correct, you should run the Weblogic install from that command-line session.

Fixing the Problem

Now, my problem was that Weblogic had already been installed.  Apparently, it was using the Sun JDK instead of the JRockit JDK.  Normally, I would look in the <PS_CFG_HOME>/webserv/<domain>/bin/setcmd…. file to change the setting.  But, I don’t even have the domain installed at this point.

I found that I could set the JAVA_HOME and JAVA_VENDOR variables in the <weblogic home>/wlserver_10.3/common/bin/commEnv.sh file.  Note that the JAVA_VENDOR was required.  If it is not set, some of the logic later on in the script overwrites the JAVA_HOME.

So, I added these two lines near the top between the comments:

[sourcecode]

#
#*****************************************************************************

JAVA_HOME=/opt/jdk
JAVA_VENDOR=Oracle

#*****************************************************************************
# sub functions
#*****************************************************************************
[/sourcecode]

After that, my PIA built with no problem.

Resources

OTN: unable to create Web Server Domain

Step By Step: PeopleTools 8.51 Upgrade (Part 6)

Next, you need to boot the web server.  To do so, open a command window, and go to the <PS Home>/webserv/<domain name>/bin directory.  Then, type “startManagedWebLogic.cmd PIA”.

When I tried, it gave me an error when booting.  I think the error is because I am using Sun’s version of the JDK, and it trying to pass a JRocket specific command line option:

I fixed it by opening the setEnv.cmd file.  Then, I removed the first three options from the JAVA_OPTIONS_WIN variable.  So, I copied the set JAVA_OPTIONS_WIN line and commented out the original.  I deleted “-jrockit -XnoOpt -XXnoJITInline”.

I found another -jrockit location in the file, too, and I did the same thing, but I still had an issue with authentication.  So, I decided to go back to the -jrockit options, and I set the Java Home to my JRockit installation:

[sourcecode]</p>
<div id="_mcePaste"><Feb 11, 2011 9:42:07 PM EST> <Critical> <Security> <BEA-090402> <Authentication</div>
<div id="_mcePaste">denied: Boot identity not valid; The user name and/or password from the boot id</div>
<div id="_mcePaste">entity file (boot.properties) is not valid. The boot identity may have been chan</div>
<div id="_mcePaste">ged since the boot identity file was created. Please edit and update the boot id</div>
<div id="_mcePaste">entity file with the proper values of username and password. The first time the</div>
<div id="_mcePaste">updated boot identity file is used to start the server, these new values are enc</div>
<div id="_mcePaste">rypted.></div>
<div id="_mcePaste"><Feb 11, 2011 9:42:07 PM EST> <Critical> <WebLogicServer> <BEA-000386> <Server s</div>
<div id="_mcePaste">ubsystem failed. Reason: weblogic.security.SecurityInitializationException: Auth</div>
<div id="_mcePaste">entication denied: Boot identity not valid; The user name and/or password from t</div>
<div id="_mcePaste">he boot identity file (boot.properties) is not valid. The boot identity may have</div>
<div id="_mcePaste">been changed since the boot identity file was created. Please edit and update t</div>
<div id="_mcePaste">he boot identity file with the proper values of username and password. The first</div>
<div id="_mcePaste">time the updated boot identity file is used to start the server, these new valu</div>
<div id="_mcePaste">es are encrypted.</div>
<div id="_mcePaste">weblogic.security.SecurityInitializationException: Authentication denied: Boot i</div>
<div id="_mcePaste">dentity not valid; The user name and/or password from the boot identity file (bo</div>
<div id="_mcePaste">ot.properties) is not valid. The boot identity may have been changed since the b</div>
<div id="_mcePaste">oot identity file was created. Please edit and update the boot identity file wit</div>
<div id="_mcePaste">h the proper values of username and password. The first time the updated boot id</div>
<div id="_mcePaste">entity file is used to start the server, these new values are encrypted.</div>
<div id="_mcePaste">at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.do</div>
<div id="_mcePaste">BootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:959)</div>
<div id="_mcePaste">at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.in</div>
<div id="_mcePaste">itialize(CommonSecurityServiceManagerDelegateImpl.java:1050)</div>
<div id="_mcePaste">at weblogic.security.service.SecurityServiceManager.initialize(SecurityS</div>
<div id="_mcePaste">erviceManager.java:875)</div>
<div id="_mcePaste">at weblogic.security.SecurityService.start(SecurityService.java:141)</div>
<div id="_mcePaste">at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)</div>
<div id="_mcePaste">Truncated. see log file for complete stacktrace</div>
<div id="_mcePaste">Caused By: javax.security.auth.login.FailedLoginException: [Security:090303]Auth</div>
<div id="_mcePaste">entication Failed: User system weblogic.security.providers.authentication.LDAPAt</div>
<div id="_mcePaste">nDelegateException: [Security:090295]caught unexpected exception</div>
<div id="_mcePaste">at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.log</div>
<div id="_mcePaste">in(LDAPAtnLoginModuleImpl.java:244)</div>
<div id="_mcePaste">at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(Log</div>
<div id="_mcePaste">inModuleWrapper.java:110)</div>
<div id="_mcePaste">at com.bea.common.security.internal.service.LoginModuleWrapper.login(Log</div>
<div id="_mcePaste">inModuleWrapper.java:106)</div>
<div id="_mcePaste">at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)</div>
<div id="_mcePaste">at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.</div>
<div id="_mcePaste">java:39)</div>
<div id="_mcePaste">Truncated. see log file for complete stacktrace</div>
<div id="_mcePaste">></div>
<div id="_mcePaste"><Feb 11, 2011 9:42:07 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server sta</div>
<div id="_mcePaste">te changed to FAILED></div>
<div id="_mcePaste"><Feb 11, 2011 9:42:07 PM EST> <Error> <WebLogicServer> <BEA-000383> <A critical</div>
<div id="_mcePaste">service failed. The server will shut itself down></div>
<div id="_mcePaste"><Feb 11, 2011 9:42:08 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server sta</div>
<div id="_mcePaste">te changed to FORCE_SHUTTING_DOWN></div>
<div id="_mcePaste">WebLogic is no longer running.</div>
<p>[/sourcecode]

The solution ended out being to start the Admin server first.  So, from the PIA home bin directory, run “startWebLogicAdmin.cmd”:

Then, “startManagedWeblogic.cmd PIA” works:

And, it works now!

So, that wraps up our booting the servers step:

 

Please continue with Step 7.

 

Resources

Step By Step: PeopleTools 8.51 Upgrade (Part 3)

This is a continuation of my PeopleTools 8.51 posts.  In this part, we are looking at installing WebLogic and the PIA.

Sorry it has taken a while to get this post proofread — I was sick this weekend and had family visiting as well.  I will try to keep the others coming soon.

Read More

Service Administration Tip: Environment for Change Assistant

This tip helps with getting Change Assistant to run. You have to get all the pieces talking together in order to use Change Assistant.

Start with the Hub

Log into URL http://<server>:<port>/PSEMHub/hub

This page will tell you what processes are connected to the hub.

Determining the port —
check the configuration file: <weblogic home>\config.xml
Domain tag > Server tag > ListenPort attribute

or check the weblogic administration console
Click on Servers under Domain Configurations — Network Configuration
The port number should be listed in the Listen Port column.

If the hub is not running
change directory to <PIA home>/
one time startup:
startManagedWebLogic PSEMHUB
(all capitals for the PSEMHUB)
or install as a service
installNTService PSEMHUB
(make sure to also start the service)
Update: In Tools 8.49, the path is changed to <PIA home>/peoplesoft/bin

Resources
“Running the Hub on Multiple Servers”

Agent

Configure:
change the file — <ps home>/PSEMAgent/envmetadata/config/configuration.properties
Change property hubURL to the valid URL for the Hub discussed above
Change the windowsdriverstocrawl or the unixdriverstocrawl to the locations where PeopleSoft is installed

Start the agent
change directory to: <ps home>/PSEMAgent
run StartAgent
or
change directory to: <ps home>/bin/server/WinX86
run psemAgentService /install <ps home>/PSEMAgent

Resources
“Running an Agent”
“Automatically Starting an Agent in Windows”

Server Administration Tip: Logging Into the Weblogic Console

URL:
http://<hostname>:<port>/console
Note: the port may or may not be the same port number as the port you use for PeopleSoft.

Directories
web logic is generally installed in <ps home>\webserv\
generally there is only 1 folder in <ps home>\webserv and that is the domain name

Determining the host name and port
look in: <weblogic home>\setEnv.cmd
SET ADMINSERVER_PROTOCOL=HTTP
SET ADMINSERVER_PORT=<port number>
SET ADMINSERVER_HOSTNAME=<server name>
SET ADMINSERVER_SERVERNAME=WebLoginAdmin

or you can look in: <weblogic home>\config.xml
Domain tag > Server tag > ListenPort attribute

Default Login
user: system
password: password

Starting the Console
if it doesn’t connect, you may have to start it manually
startWebLogicAdmin
same directory as setEnv.cmd

Installing as a service
installNTService
with no parameters