Oracle Observations

July 11, 2007

ORA-00600 [15015] and all that.

Filed under: ORA-600 — bigdaveroberts @ 11:35 am

Well last Wednesday was the first fun day (at work) for a long while.

The application, database and OS were all struggling, and I suspect that the network was also experiencing problems.

After consideration, the apparent cause (based on being the earliest errors we could find evidence of) were repeated ORA-600 errors (predominantly 15015), starting within seconds of the database being restarted after the backup (which also failed).

The errors appeared to be related to the snapshot process that was dieing every 5 minutes and was then being automatically restarted by the database.

I looked up ORA-600 [15015] on Google, and got no hits and I looked up 15015 on the Metalink ORA-600 argument look up tool and received the unhelpful response:

A description for this ORA-600 error is not yet published.

I also searched in the knowledge base including the archived articles and bug database and received no hits.

So we have a stable system on a terminal release (8.1.7.4) that suddenly and for no apparent reason starts kicking out super obscure errors.

And it isn’t as if there have been any significant changes implemented.

There was one change to patch an oracle bug that reared its head when we started running the client under Citrix and one to increase the size of the SGA. Both changes were implemented more than a month ago.

Before you get excited, I have to say that I don’t know what the problem is. (Lets be frank, the only reason you are reading this is because you are experiencing the same error.)

So where do my theories lie?

There is a general tendency for Software people to blame hardware when a new problem appears in a stable system, but I wouldn’t initially blame hardware. (/var/adm/messages didn’t have anything novel in it until a disk partition filled from all the core dumps and the problem was resolved by a reboot.)

I also don’t tend towards the conspiracy theorists that assume that all problems start with an uncontrolled change made by some well meaning techie. Certainly pkginfo didn’t indicate that the system had been patched or had any new packages installed within the last 12 months.

Generally I find that unexpected problems are most often explained by the unforeseen outcomes of poorly understood changes, and while the 2 changes appear to be superficially innocuous, it is there that my suspicion starts.

The Oracle patch will probably have been installed on multiple systems for multiple customers, and while it may be possible for the interaction between multiple patches to produce unusual results, the simple fact is that the system is rarely patched and is as close to a vanilla install as possible. Thus I think that the patch is unlikely to be the cause.

Thus finally we are left with the SGA increase. This does worry me slightly, in that the size of the SGA is now close to the SHMMAX setting for the maximum shared memory segment size, to the extent that on some mornings we receive a warning:

WARNING: Not enough physical memory for SHM_SHARE_MMU segment of size 0xnnnnnnnn

in the alert log, which Metalink unhelpfully indicates may be serious on some versions and innocuous on others.

So with my suspicion that there is an issue with shared memory I have scheduled a cron job to record the results for ipcs -Am before and after each backup window, and leave the problem with a watching brief.

Obviously when Oracle comes up with a response I will post an update.

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: