The keynote by Feurstein Coding Therapy for Database Professionals actually had some good food for thought. Some of the more interesting points:

• As a developer, it can really help to understand modern ideas about how the mind works; the book On Intelligence is highly recommended.  Our brains come with lots of issues, and tons of computational power that we aren’t aware of.
• Don’t sprinkle SQL code throughout your regular PL/SQL code; try to encapsulate database calls in wrappers. This makes implementation far easier to change.  I can definitely attest to this idea, having created quite a few APIs in PL/SQL for Java applications, and it allowed a lot of flexibility in design changes over the course of time.
• Use subtypes for varchar2(x) fields — this is actually a feature I wasn’t aware of (and I’ve been coding pl/sql since, hmm, 1995 or so).
• If you get stuck, try sleeping.

He had other points, e.g. the relationships between developers & DBAs among
others (recommending the book Peopleware), but those are the ones I remember.

He’s a great speaker.

Next I saw Feuerstein’s Making the Most of PL/SQL Error Management Features. Actually knew most of the tips, but there is a new package since v10 I wasn’t aware of; DBMS_ERRLOG (since v10 even); this allows you to first call it to create an exception table for a given real table, then do DML on the original table.  Rejects will (such as duplicate primary/unique key) will go into the error-table instead of the entire transaction rolling back. He’s supplied some of his own code that adds extra features.  It should be noted that it is entirely up to the developer to handle rows in that resulting table. It would be probably really good for really big updates which take lots of time, and you don’t want the entire transaction to roll back if it finds 1 problem row.

Next I saw The Case for Manual SQL tuning by Dan Tow of Singing SQL; he obviously had some good points, being that understanding the application itself can make decisions far better than any cost-based analyzer, and that there are still parts of the cost-based analyzer that will fail miserably (ie anti-correlation between values in different tables used in the same query).  The current cost-based analyzer will usually work 90-99% of the time, but there are always, and likely will always be exceptions.

I then saw the Resolving Buffer Busy Waits by Craig Shallahamer of OraPub.  This was by far my favorite presentation, and I wasn’t expecting it to be so, having been acquainted several times with resolving buffer-busy wait problems. The presentation was actually far deeper than that, and gave me quite a bit of insights into the buffer pool itself; LRU chains, dirty-block buffers, cache-buffer chains, how block headers work, how segment headers for data/index-blevel/index-leaf/undo work. Fascinating stuff - in fact this was my favorite presentation at NOCOUG in at least 6 months.

I actually skipped the last sessions; my colleague Terry earlier had seen an Oracle Appliance session, Visual Tuning by Example was probably pretty much a sales presentation for Delphix, and I’ve done a fair amount of reading/playing on nosql (and from Oracle, was thinking it too would be mostly sales-oriented). So, I headed downstairs for the great computer history exhibit. It was different, and bigger than it was when I saw it apx 3 years ago, and totally worth it. If you haven’t seen it, I’d highly recommend it, esp if you’ve been in computers awhile. Loved the robots and very early tablet computers most, though I found myself reminiscing quite a bit about the computers I started my career with.  They even had a special video on the history of Oracle Corp.

Even for an Oracle veteran like myself, I found some really useful information in this last conference.  Kudos to Iggy and the entire NOCOUG team for delivering a great conference once again!

The commands that I used were:

#– create four 10GB files to be used for +DATA and +FRA (run as oracle)

mkdir /asmdisks

dd if=/dev/zero of=/asmdisks/data_disk1 bs=1024k count=10000
dd if=/dev/zero of=/asmdisks/data_disk2 bs=1024k count=10000
dd if=/dev/zero of=/asmdisks/fra_disk1 bs=1024k count=10000
dd if=/dev/zero of=/asmdisks/fra_disk2 bs=1024k count=10000

#– create the loop and raw devices.(run as root) This allows ASM to discover them as 4 raw devices
#– NOTE: You will also need to add the following commands to your server startup
#– scripts so that ASM can still see them after you reboot the server

losetup /dev/loop1 /asmdisks/data_disk1
losetup /dev/loop2 /asmdisks/data_disk2
losetup /dev/loop3 /asmdisks/fra_disk1
losetup /dev/loop4 /asmdisks/fra_disk2

raw /dev/raw/raw1 /dev/loop1
raw /dev/raw/raw2 /dev/loop2
raw /dev/raw/raw3 /dev/loop3
raw /dev/raw/raw4 /dev/loop4

#– change the permission to oracle:dba

chown oracle:dba /dev/raw/raw1
chown oracle:dba /dev/raw/raw2
chown oracle:dba /dev/raw/raw3
chown oracle:dba /dev/raw/raw4

chmod 660 /dev/raw/raw1
chmod 660 /dev/raw/raw2
chmod 660 /dev/raw/raw3
chmod 660 /dev/raw/raw4

After doing this, I was able to run asmca and create a +DATA and a +FRA disk group, each 10 GB in size (the raw devices are each 10GB and ASM mirrors them, so you end up with 10GB per disk group). Everything was going fine so far and I was able to create a database using the ASM disk groups that I just created.

Just when everything was working so perfectly, I decided to change things around a bit and upgrade from 11.2.0.1 Grid Infrastructure to 11.2.0.2. This release introduced Oracle Restart - billed as ‘High Availability for a non-RAC database’ and I wanted to try it out. It was also at this point that I thought to myself “I should have installed 11.2.0.2 Grid Infrastructure to begin with, rather than upgrading from 11.2.0.1″. Oh well, too late. On the other hand, this does provide a chance to do an upgrade that quite a few of our clients may actually do.

I downloaded disk3 of Patch #10098816 and followed the typical Oracle installation steps. One new thing with this version is that Oracle no longer does “In-place” upgrades, meaning each new release will have its own $ORACLE_HOME. This could cause problems for folks who hard-code the $ORACLE_HOME in scripts. Everything seemed to go without any problem until I tried to start the database I had earlier created. Upon startup, Oracle complained that it couldn’t read the spfile. Using the ASM Command Line (amscmd), I created a copy of the spfile in /tmp and tried to start with that. The file was created and readable, but this time, Oracle couldn’t read the control files. The disk groups were mounted, everything seemed fine but ASM was in an inconsistent state, sort of “stuck” between 11.2.0.1 and 11.2.0.2. After working on this to no avail for a while, I decided to take the “easy” way out. I would uninstall 11.2.0.1 and 11.2.0.2, re-install 11.2.0.2 and completely rebuild my +ASM instance. This was a test environment after all… probably not the way I would approach it in a Production situation.

With previous versions of the Oracle Installer, you could easily uninstall an ORACLE_HOME from the OUI. Now, when you click Uninstall, it directs to you to run a deinstall script from $ORACLE_HOME/deinstall. This failed for both versions and eventually I decided to just delete the $ORACLE_HOMEs and edit the oraInventory/ContentsXML/inventory.xml to remove all references to both of them. This seemed to work until running root.sh at the end the 11.2.0.2 install when I got the error and helpful suggestion:

“Improper Clusterware configuration found on this host. Deconfigure the existing clusterware by running “$CRS_HOME/install/roothas.pl -deconfig “

Of course, the roothas.pl also failed but I was able to use the –force option and it then succeeded. I re-ran root.sh and finally 11.2.0.2 Grid Infrastructure was installed.

I then ran asmca and was able to create an +ASM instance and two disk groups (+DATA and +FRA) with the files I had created earlier. The final test was whether I could use ASM to store datafiles for another instance. So I fired up the Database Configuration Assistanct (dbca) and created a database with all of its files located in the +DATA diskgroup. Success!

Now, back to Oracle Restart for a moment…like I said, Oracle Restart is new with 11.2.0.2 and is advertised as High Availability for a Non-RAC instance. It is essentially a daemon that runs (ohasd) and will restart the database/Listener/ASM if they crash unexpectedly. A shutdown abort or shutdown normal will not cause it to restart, as Oracle assumes you intend for it to be down. This could turn out to be a great thing for situations where it may take the DBA a while to login to restart a database. No need for extra downtime if the database restart can be automated!

Oracle Restart also takes care of stopping and starting the database(s) when the server reboots. This is a big change for those of us who have come to know and the love the dbora start/stop scripts. Once you start using Oracle Restart, the /etc/oratab (or /var/opt/oracle/oratab) file is no longer used by Oracle to start the database after the system reboots. After I registered my database with Oracle Restart with:
srvctl add database -d DEVDB01 -o /opt/oracle/product/11.2.0/dbhome_1
I noticed that Oracle had modified my /etc/oratab, changing the Y to an N.

A quick list of useful Oracle Restart commands are below:

srvctl stop|start|status database –d <database name>
srvctl stop|start|status asm
srvctl stop|start|status listener

In then end, I got my test environment configured exactly the way I wanted. If I could do things differently, I would have installed Grid Infrastructure 11.2.0.2 to begin with, rather than upgrading from 11.2.0.1. I also would have spent more time trying to figure out what went wrong with my initial upgrade from 11.2.0.1 to 11.2.0.2. Ideally, the upgrade would have worked and I wouldn’t have had to go through all the extra steps. This does raise a concern about doing this upgrade for our clients. I think my next step will be to setup a new environment specifically to perform the upgrade again and figure out what went wrong.

• Security
• Availability
• Performance

Some aspects of these core functions can be delegated to other groups. For example Network Engineering may manage LDAP to augment database authentication (security), or as in this case external redundancy handled disk failures in ASM (availability).

Beware: YOU MAY DELEGATE AUTHORITY, HOWEVER RESPONSIBILITY CAN BE VERY STICKY.  Do not think that because you authorize someone or something to take over part of a task you escape responsibility when things go wrong.  In this recent example I noticed warning messages in the ASM alert log after engineers completed an FRU battery maintenance on a SAN.

Note:  A recoverable backup was taken prior to the storage maintenance per standard operating procedures.

At this point we took the database out of cluster and redirected the db_recovery_file_dest to local storage, investigating why the FRA disk group mounted and then crashed the instance with an ORA-600 shortly afterwards.

Suspecting metadata corruption we attempted a repair:

SQL> alter diskgroup FRA check all repair
NOTE: starting check of diskgroup FRA
SUCCESS: check of diskgroup FRA  found no errors

A little more time goes by and then:

ORA-00600: internal error code, arguments: [kccpb_sanity_check_2]…
Shutting down instance (abort)

After contacting Oracle support and exhausting our options to mount the FRA we ended up initializing the disks with dd and rebuilding the disk group.

Summary: Backups to a Flash Recovery Area may not be as reliable as you think - especially with external redundancy on a single physical or virtual disk.  In these situations it is a good practice to maintain additional redo log members on at least two disk groups, and utilize RMAN to regularly copy archivelogs and backups to a secondary location.

Preface

Introduction

http://www.dbspecialists.com/files/presentations/changing_platforms.html

along with many other useful whitepapers about performance tuning, software installation, remote database administration, and a range of topics.

Not to downplay that capability, but given that the big wave of migrations to Linux has somewhat played out, I have found the Transportable Tablespace (TTS) feature also very useful for doing Oracle upgrades in less time than might be experienced with a traditional export/import approach, particularly in cases where the database is a fatty.  It’s still hard to beat the in-place upgrade for limiting downtime during an upgrade, but what if you’re also moving to bigger and better hardware? 

For limiting downtime during the combination of upgrading Oracle and moving to a new piece of iron, I recommend transportable tablespaces.  Trouble is there seems to be precious little information available from Oracle on how to do this.  There is absolutely no mention of using transportable tablespaces in the Upgrade Guide.  But it’s definitely doable.

For starters, you must use Enterprise Edition on your source database to generate a set of transportable tablespaces.  They can be imported into a Standard Edition database, which is also not a bad method of moving from EE to SE, by the way.  There are other limitations, too, such as with XMLTypes and opaque types, such as RAW.  BINARY_FLOAT and BINARY_DOUBLE types can only be transported using Data Pump.  In any case check the manual before proceeding to be sure that your database is a good candidate for an upgrade via TTS.

As an aside, beginning with 10gR2 there is a feature known as Transportable Database (TDB) which can be handy when the source platform is on a different OS as the target platform, but the endian format must be the same and TDB cannot be used for upgrades.

While I haven’t tried every possible upgrade, Metalink claims that beginning with Oracle 8i there should be no problem with transporting a tablespace into a higher version.  There are some differences in capability between some versions, however.  For example, directly transporting materialized views and function-based indexes is not supported until 10g.

The upgrade using TTS should proceed like this:

Preparation

1. Create an empty database on the new host using the higher Oracle version.
2. Capture the DDL for objects stored in the SYSTEM and SYSAUX tablespaces, such as PL/SQL, users, system privileges, etc.
3. Perform a full backup on the source database.

Step #2 above is one that will require testing to make sure you don’t leave anything behind.  It’s not a huge task but one that shouldn’t be taken lightly, either.

Upgrade

1. On the source database put the tablespaces to be transported into “read only” mode.  Remember that the SYSTEM and SYSAUX tablespaces cannot be transported.
2. Run your Data Pump job to export the tablespace metadata.
3. Stop or quiesce the source database.
4. Copy or otherwise make available to the target database the data files from the transported tablespaces.
5. Run you Data Pump job to import the tablespace metadata.
6. Put the tablespaces into “read write” mode on the target database.
7. Perform a full backup on the target database.

 That’s the long and short of it.  Step #4 above is the one that will likely make or break this method.  If you have to copy lots of big files across a slow connection, then you’re defeating the purpose of using this method, which is to limit downtime, although it may still be faster than export/import.  If, on the other hand, you can just swing you storage device over to a new database host, then Step #4 will take no time at all. 

In the Oracle RDBMS, every segment (table/table partition, index/index partition, queue) must be assigned to a particular tablespace. Each tablespace is composed of one or more datafiles. Datafiles do have a maximum size, depending on the blocksize for traditional smallfile tablespaces; for a tablespace with a block size of 8k, the maximum size of a datafile is just under 32Gb. So, each tablespace is typically comprised of many datafiles. Oracle has several tablespaces that are built-in; these include the SYSTEM, SYSAUX, TEMP, and UNDO/ROLLBACK tablespaces.

Let’s say that you wanted to make the work of the next DBA who manages a database you control as hard as possible. One of the easiest ways to do this is to create a database with LOTS of tablespaces; you could even create 1 tablespace per segment. This means that instead of worrying about one ‘pool’ of storage, the next DBA will need to manage dozens, or hundreds, of storage pools. Because this management is mission-critical (if ANY tablespace fills up, your application may not work), it is far more likely that one will be overlooked, and fill up. Using lots of tablespaces will also make it far more likely that a lot of space is wasted in each tablespace, because every tablespace will need to maintain a certain amt of free space to be ’safe’ from future increases in growth.  If a segment is dropped or shrunk, the space released will be only in that one tablespace, and won’t be able to be used by segments assigned to all of the other tablespaces.

If you wanted to make the the work of the next DBA who manages a database you control the easiest, you’d create just one tablespace (which could be a BIGFILE tablespace, which can use larger files than SMALLFILE tablespaces), and stick all of your segments into there. There would only be one tablespace to manage; when segments are shrunk or removed in the course of business, that space can immediately be used by any other segment. To manage extent sizes, you could simply use the EXTENT MANAGEMENT LOCAL AUTOALLOCATE feature, which automatically gives you near-to-optimimum sizes.

Prior to Oracle 10g, there were many legitimate reasons to use multiple tablespaces and there still are good reasons to segregate your segments into separate tablespaces. Some of these reasons are:

• You may need different block sizes for different segments for best performance; since each tablespace has one block size, to use different ones you need different tablespaces.
• If you aren’t using a SAN, or ASM, or similar storage solution, you may wish to segregate segments on differnet LUNs/drives for better performance. Given the size of today’s databases, and the need for high IOPS storage solutions, most non-trivial databases are using SANs/ASM/other storage solutions these days. Using these, it doesn’t give you better performance by segregating tablespaces.
• If you need to use the ‘transportable tablespaces’ feature to move large quantities of data between Oracle databases,
• If you use Oracle’s direct-path load feature, which loads data above the high-water mark of each datafile directly for faster data loading (by bypassing the internal SQL engine), then you may wish to continually create new tablespaces for this purpose.
• If you have several separate users in your database, and wish to ’split’ the database in the future so that you have one databases each for each user, then you may wish to create a dedicated tablespace for each user. This would allow you to clone half of the database using RMAN on a new machine, and OFFLINE DROP the datafiles you do not want.

In conclusion, one easy way to make your oracle database use space more efficiently, be easier to manage, and therefore be less likely to fill up, is to use fewer, larger application tablespaces, rather than more.

The Oracle Physical Standby feature is not new at all; I was using it back in 1995 when I first started working as an Oracle DBA with version 7; perhaps that’s why it’s as reliable as it is.  Although there have been cases where standby databases do encounter bugs, with proper monitoring (for example, with the monitoring done by Database Rx) this is not an issue.

So, why am I recommending Oracle Physical Standby databases over the other methods you can use to insure uptime?  What makes it so special?

• An Oracle Physical Standby database has nothing shared with the primary database; it (typically) runs on a separate server, using its own storage, it’s own memory, and it’s own network interface; machines/disks/memory/network cards all break sooner or later (it’s not “if”, it’s “when”).  Some folks say that RAC serves this purpose, but in fact it does not; all nodes in a RAC cluster read and write to the same storage subsystem, so that is a potential single point of failure.
• If you can afford the network connection necessary, you can physically place your physical standby database in a separate data center, so even if your primary data center burns up, your data is safe.
• This feature is available in all versions of Oracle, though for Standard Edition and lower, you will need to come up with scripts to move archivelogs and apply them on the standby, as we do for our Remote DBA clients.
• For Standard Edition, you can tune it so that you will lose a maximum of one archived redo log worth of data worst-case, if your primary database disappears.
• If using the Enterprise Edition, you can have Oracle itself manage it, and in fact use advanced automatic failover methods using a feature called Dataguard, that can switch primary/standby database roles very quickly and easily.
• If using the Enterprise Edition, you can set parameters to delay the application of data from the primary to the standby by a certain amount of time.  This would be useful if you suffered logical corruption on the primary database (ie someone dropping a table) and could catch it fast enough.
• If using the Enterprise Edition, you can use the Flashback Database feature to allow the standby to recover from logical errors on the primary database even easier,
• Also, if using the Enterprise Edition, you can actually make the standby stay up-to-date with the primary database in near-real-time, by making the log-writer (LGWR) process on the primary, ship redo directly to the standby, even before that redo is archived.
• Although there are restrictions, you can actually open up a standby database in read-only mode for reporting.  In versions before Oracle 11g, the standby cannot apply changes from the primary while it is open this way; however, there is a new feature in 11G called ‘Active Data Guard’ that allows this.
• In my experience, Oracle Logical standby databases are not as reliable as compared with Physical Standbys.
• Another advantage; you can use your physical standby database to do RMAN backups from, which can dramatically reduce the load on your primary database.

The main disadvantage to setting up a standby database, is that you will double your hardware cost; you will need a server/disk able to handle your peak production loads.  It will also cost you an extra Oracle license for that instance.  This will double the amount of space in your datacenter used, and also double the amount of electricity/heat produced.  For many businesses though, this cost is far less then the cost incurred by a lengthy database recovery.

There is nothing more relieving than when a primary database malfunctions, and you are able to continue production with failover times usually measured in under 10 minutes, rather than potentially waiting days/weeks for new hardware and a full restore from backup.  There is no other feature that as dramatically improves the potential uptime of an Oracle Database as much, as using a Physical Standby database.