HEUG Alliance 2011 — Here I Come!

I am very excited about the HEUG Alliance conference next week.  This is my first time attending, and I will get to present with Sean McKay from George Fox University.  So, if you are planning to attend, please catch us at 4:30 on Tuesday afternoon.  We will be talking about our Moodle interface that we worked on last year (a great project).

Also, I won’t be presenting at it, but I will be at the ACE presentation at 9:30 that morning.  Joe Rossi and Adrian Valente will be presenting the tool that we built in PeopleTools to automate enrolling and accepting payment for continuing education from self service.  So far, that tool has generated a lot of excitment, and I think that presentation will be good.

Finally, I can’t leave off Jim Marion’s presentation at 9:30 on Monday.  I am going to make it a special point to listen in on his presentation to see what tips and tricks I can pick up from him.  Ever since his announcement about his book, I have wanted to meet him, and now I will get to hear him speak!

If it wasn’t for the time conflict, I would have mentioned Grey Sparling’s presentation at 9:30 on Tuesday morning.  Their blog posts are always helpful, and I would love to hear a presentation.  I saw something about the sessions being recorded, and I will have to do some research to see if I can watch or listen to the recording.

Anyway, if you are attending and you read my blog, please stop by the MIS, Inc. booth and say hi.  I would love to meet you.

Tracing Tips and Resources

I had an issue recently where I needed to trace a process to try to determine where a bug was occurring.  So, I decided to try to document how I could setup tracing on a single process.

To enable tracing, you need to open the process definition.  Online, navigate to PeopleTools > Process Scheduler > Processes.  Then, search for the process that you want to trace.  Make sure that you change the search page so that you are searching by process name and not the process type.  Once you have the Process Definition open, go to the Override Options tab.  Then, change the parameter list to “Append” and enter the desired trace settings.

Applying the Trace Settings
Applying the Trace Settings

This is the recommended setting by Ketan on PeopleSoft Support & Tips:

-TRACE 135 -TOOLSTRACESQL 31 -TOOLSTRACEPC 1984

I think I would prefer these settings instead:

-TRACE 7 -TOOLSTRACESQL 3 -TOOLSTRACEPC 3596

My options aren’t to say that Ketan’s are wrong.  I just cut out some of the options I wouldn’t use to trim the extra info in the log file.  I also normally use traces for troubleshooting, so my options are tuned for that.

You can read below for details on what each of these does.

The -TRACE option sets the tracing for the App Engine program generally speaking.  For example, it can turn on the trace output that displays which steps run in which order.  Here are the different options (from the psappsrv.cfg file):

  • 1 = Trace STEP execution sequence to AET file
  • 2 = Trace Application SQL statements to AET file
  • 4 = Trace Dedicated Temp Table Allocation to AET file
  • 8 = not yet allocated
  • 16 = not yet allocated
  • 32 = not yet allocated
  • 64 = not yet allocated
  • 128 = Timings Report to AET file
  • 256 = Method/BuiltIn detail instead of summary in AET Timings Report
  • 512 = not yet allocated
  • 1024 = Timings Report to tables
  • 2048 = DB optimizer trace to file
  • 4096 = DB optimizer trace to tables

If you’ll notice, 135 is “Timings Report to AET file”, “Trace Dedicated Temp Table Allocation to AET file”, “Trace Application SQL statements to AET file”, and “Trace STEP execution sequence to AET file”.  My preference just leaves off the Timings Report.  I think the timings report would be good for performance tuning, but if you are troubleshooting, it is just overhead.

The -TOOLSTRACESQL option specifics the SQL tracing besides the SQL executed by steps in an App Engine program.  This would include any PeopleTools system database communication, and more importantly, it would include any database access caused by a component interface driving an online screen.

Here are the options:

  • 1 = SQL statements
  • 2 = SQL statement variables
  • 4 = SQL connect, disconnect, commit and rollback
  • 8 = Row Fetch (indicates that it occurred, not data)
  • 16 = All other API calls except ssb
  • 32 = Set Select Buffers (identifies the attributes of columns to be selected
  • 64 = Database API specific calls
  • 128 = COBOL statement timings
  • 256 = Sybase Bind information
  • 512 = Sybase Fetch information
  • 1024 = SQL Informational Trace

Again, the recommended setting is 31, which includes: “All other API calls except ssb”, “Row Fetch (indicates that it occurred, not data)”, “SQL connect, disconnect, commit and rollback”, “SQL connect, disconnect, commit and rollback”, “SQL statement variables”, and “SQL statements”.  My choice would just be to see the SQL statements and the variables because that is the most important information.  Commits and Rollbacks can be important, but you generally know when those happen and don’t need the extra information.  The fetches will tell you how many rows you will receive but it won’t give you any of the data, so it just clutters the trace file.  Those options can be helpful, but unless you need them, I would leave them off.

Finally, the -TOOLSTRACEPC option turns on tracing for PeopleCode.  This could be PeopleCode steps in an App Engine, or it could be PeopleCode in a component accessed by a Component Interface.  Here are the options:

  • 1 = Trace Evaluator instructions (not recommended)
  • 2 = List Evaluator program (not recommended)
  • 4 = Show assignments to variables
  • 8 = Show fetched values
  • 16 = Show stack
  • 64 = Trace start of programs
  • 128 = Trace external function calls
  • 256 = Trace internal function calls
  • 512 = Show parameter values
  • 1024 = Show function return value
  • 2048 = Trace each statement in program (recommended)

The recommended option is 1984, which includes: “Show function return value”, “Show parameter values”, “Trace internal function calls”, “Trace external function calls”, and “Trace start of programs”.  For me, I want to see what PeopleCode is running, so the most important option would be 2048 — Tracing each statement.  The variable assignments, fetched values, parameter values, and function returns all show the data as it flows through the PeopleCode, so I find those helpful in troubleshooting.  For me, that adds up to: 3596.

Resources

Step By Step: PeopleTools 8.51 Upgrade (Part 7)

This is a continuation of my PeopleTools 8.51 posts.  Now that we have our servers started, we can finish the Change Assistant Job.

Please see my Step by Step page for more articles like this, or you can start at the beginning of this series withPart 1.

Read More

More Memory Leak Troubleshooting

Recently, I have been doing a lot of Memory troubleshooting.

Here are some windbg commands that helped:

Load the sos module (for .Net 4.0)

.loadby sos clr

List all of my object specific to my program:

!dumpheap -stat -type <parent namespace>

List all the instances of a particular object:

!dumpheap -type <namespace>.<object class name>

List references to the object:

!gcroot <address>

Break on garbage collection:

!findroots gen any

List references to the obect (must break in garbage collection):

!findroots <address>

Show sizes in the memory:

!eeheap

List large strings in memory:

!dumpheap -type System.String -min 10000

Then, I also found that you can use the SOS module with within Visual Studio.  First, in the project settings, I had to go to the Debug tab and check “Enable unmanaged code debugging” under “Enable Debuggers”.  Then, in the immediate window, I could run the following command:

.load sos

I thought my problem seemed to be related to the Data Grid view just like this forum post.  The post lists a code fix, but my problem is finding out where I place the code.  This blog mentions putting the code in the Form Close, but my problem is during the execution of the program.

Actually, it ended up being that I was loading a hidden column into the grid that was taking a lot of memory.  I had a Select * SQL statement, and I had to list only the fields that I wanted in memory.

Resources

UserPreferenceChangedEventHandler Resources

Memory Leak Troubleshooting

This is a tad off topic, but I don’t want to loose all of these reference links for tracking down memory leaks with a VB.Net program.

Here is the article that helped out a lot.
Rico Mariani’s Performance Tidbits: Tracking down managed memory leaks (how to find a GC leak)

As far as I can tell, you download the Vadump program from here:
Windows 2000 Resource Kit Tool: Vadump.exe

Since I have Visual Studio 2010 installed, I found it here:
<Visual Studio Home>\Common7\Tools\Bin\winnt

WinDbg, I didn’t find on my system. So, I had to download it in the Windows SDK from here:
Download and Install Debugging Tools for Windows

The CLR Profiler is another tool that I tried:
CLR Profiler

I haven’t finished, so maybe, I will have to write a follow up article.

Anyone Using Virtual Machines?

Someone was asking me about using virtual machines for production PeopleSoft servers, and he pointed out this article:

Support Position for Oracle Products Running on VMWare Virtualized Environments [ID 249212.1]

From what I can tell, Oracle does not support VMWare for production servers. From the article, it looks like you can do it, but you are taking a slight risk that you would have to prove VMWare wasn’t causing the problem when you open a ticket with Oracle. I didn’t confirm this, but I think Oracle supports their own virtualization products.

Is there anyone out there using virtualization for their server environments?

Fusion and PeopleSoft

I ran across Tim’s post on the Myth of Oracle Fusion.

I don’t have any insider information, but my view from PeopleSoft was somewhat different. When Oracle purchased PeopleSoft, that’s when I began first hearing about Fusion.

What scared me at the time was that Oracle might take the PeopleTools platform that I had begun to love, and that they would drop it for something inferior but more familiar to their products. PeopleTools is somewhat unique from what I can tell, and the records, pages, and components form a very innovative platform for which one can build applications. Did Oracle really understand the benefits of this platform? Could they preserve these benefits and still combine it with their products to create Fusion?

Personally, I am glad that we haven’t seen Fusion yet. I would rather Oracle take their time and get it right than for them to mess up an awesome platform. Oracle has added some features in the recent releases of PeopleTools, and the platform continues to expand. For that, I am grateful.

That’s my take on it. Maybe I misunderstood the intentions of Oracle at the beginning, but that was my view from the PeopleSoft world.

Step By Step: PeopleTools 8.51 Upgrade (Part 6)

Next, you need to boot the web server.  To do so, open a command window, and go to the <PS Home>/webserv/<domain name>/bin directory.  Then, type “startManagedWebLogic.cmd PIA”.

When I tried, it gave me an error when booting.  I think the error is because I am using Sun’s version of the JDK, and it trying to pass a JRocket specific command line option:

I fixed it by opening the setEnv.cmd file.  Then, I removed the first three options from the JAVA_OPTIONS_WIN variable.  So, I copied the set JAVA_OPTIONS_WIN line and commented out the original.  I deleted “-jrockit -XnoOpt -XXnoJITInline”.

I found another -jrockit location in the file, too, and I did the same thing, but I still had an issue with authentication.  So, I decided to go back to the -jrockit options, and I set the Java Home to my JRockit installation:

</p>
<div id="_mcePaste"><Feb 11, 2011 9:42:07 PM EST> <Critical> <Security> <BEA-090402> <Authentication</div>
<div id="_mcePaste">denied: Boot identity not valid; The user name and/or password from the boot id</div>
<div id="_mcePaste">entity file (boot.properties) is not valid. The boot identity may have been chan</div>
<div id="_mcePaste">ged since the boot identity file was created. Please edit and update the boot id</div>
<div id="_mcePaste">entity file with the proper values of username and password. The first time the</div>
<div id="_mcePaste">updated boot identity file is used to start the server, these new values are enc</div>
<div id="_mcePaste">rypted.></div>
<div id="_mcePaste"><Feb 11, 2011 9:42:07 PM EST> <Critical> <WebLogicServer> <BEA-000386> <Server s</div>
<div id="_mcePaste">ubsystem failed. Reason: weblogic.security.SecurityInitializationException: Auth</div>
<div id="_mcePaste">entication denied: Boot identity not valid; The user name and/or password from t</div>
<div id="_mcePaste">he boot identity file (boot.properties) is not valid. The boot identity may have</div>
<div id="_mcePaste">been changed since the boot identity file was created. Please edit and update t</div>
<div id="_mcePaste">he boot identity file with the proper values of username and password. The first</div>
<div id="_mcePaste">time the updated boot identity file is used to start the server, these new valu</div>
<div id="_mcePaste">es are encrypted.</div>
<div id="_mcePaste">weblogic.security.SecurityInitializationException: Authentication denied: Boot i</div>
<div id="_mcePaste">dentity not valid; The user name and/or password from the boot identity file (bo</div>
<div id="_mcePaste">ot.properties) is not valid. The boot identity may have been changed since the b</div>
<div id="_mcePaste">oot identity file was created. Please edit and update the boot identity file wit</div>
<div id="_mcePaste">h the proper values of username and password. The first time the updated boot id</div>
<div id="_mcePaste">entity file is used to start the server, these new values are encrypted.</div>
<div id="_mcePaste">at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.do</div>
<div id="_mcePaste">BootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:959)</div>
<div id="_mcePaste">at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.in</div>
<div id="_mcePaste">itialize(CommonSecurityServiceManagerDelegateImpl.java:1050)</div>
<div id="_mcePaste">at weblogic.security.service.SecurityServiceManager.initialize(SecurityS</div>
<div id="_mcePaste">erviceManager.java:875)</div>
<div id="_mcePaste">at weblogic.security.SecurityService.start(SecurityService.java:141)</div>
<div id="_mcePaste">at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)</div>
<div id="_mcePaste">Truncated. see log file for complete stacktrace</div>
<div id="_mcePaste">Caused By: javax.security.auth.login.FailedLoginException: [Security:090303]Auth</div>
<div id="_mcePaste">entication Failed: User system weblogic.security.providers.authentication.LDAPAt</div>
<div id="_mcePaste">nDelegateException: [Security:090295]caught unexpected exception</div>
<div id="_mcePaste">at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.log</div>
<div id="_mcePaste">in(LDAPAtnLoginModuleImpl.java:244)</div>
<div id="_mcePaste">at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(Log</div>
<div id="_mcePaste">inModuleWrapper.java:110)</div>
<div id="_mcePaste">at com.bea.common.security.internal.service.LoginModuleWrapper.login(Log</div>
<div id="_mcePaste">inModuleWrapper.java:106)</div>
<div id="_mcePaste">at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)</div>
<div id="_mcePaste">at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.</div>
<div id="_mcePaste">java:39)</div>
<div id="_mcePaste">Truncated. see log file for complete stacktrace</div>
<div id="_mcePaste">></div>
<div id="_mcePaste"><Feb 11, 2011 9:42:07 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server sta</div>
<div id="_mcePaste">te changed to FAILED></div>
<div id="_mcePaste"><Feb 11, 2011 9:42:07 PM EST> <Error> <WebLogicServer> <BEA-000383> <A critical</div>
<div id="_mcePaste">service failed. The server will shut itself down></div>
<div id="_mcePaste"><Feb 11, 2011 9:42:08 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server sta</div>
<div id="_mcePaste">te changed to FORCE_SHUTTING_DOWN></div>
<div id="_mcePaste">WebLogic is no longer running.</div>
<p>

The solution ended out being to start the Admin server first.  So, from the PIA home bin directory, run “startWebLogicAdmin.cmd”:

Then, “startManagedWeblogic.cmd PIA” works:

And, it works now!

So, that wraps up our booting the servers step:

 

Please continue with Step 7.

 

Resources

Step By Step: PeopleTools 8.51 Upgrade (Part 5)

This is a continuation of my PeopleTools 8.51 posts.  This part discusses running the Change Assistant job.

Please see my Step by Step page for more articles like this, or you can start at the beginning of this series withPart 1.

Read More