25 Jan

Too Many Flashbacks

With apologies to Dr. Seuss.

Did I ever tell you the makers of RAC
had seven features and named each flashback?

Well they did, and it wasn’t a smart thing to do.
You see, when the customers wanted a clue
as to how to keep data from getting deleted
the RAC folks said “flashback” and customers heeded.

They turned on all seven of those flashback features
Each one was a slightly dissimilar creature.
Some used the UNDO, some used flashback files
Some just renamed tables to bin$ styles.

One was a place you keep things for recovering
Another was just for forensic discovering
With so many features called by the same name
when thinks broke no one knew just which one they should blame.

On a Friday three minutes past seventeen hundred,
users ask, “What’s the deal? Our data’s been plundered.”
“It looks like all names in the customer table,
are now ‘John Q Public,’ a certain mislabel.”

It was Nimrod, an intern fresh from his instruction.
What he thought was just test was really production.
“No Problem,” says Morton, the wise DBA,
“Flashback is on. I’ll restore it today.”

First Morton asked Nimrod what time he committed
his update that had any where clause omitted.
“I ran it at just past noon yesterday lunch,
It ran for so long that I went for a munch.”

By now it had been almost thirty one hours
but Morton knew that he could call on his powers,
and on the mere fact that that the undo_retention
was set to two days, a quite lengthy extension.

But woe! When he tried to engage Flashback Query,
he got “snapshot too old” and it ruined his theory
that undo_retention makes Oracle hold
all undo data no matter how old.

It turns out that there was no undo_guarantee
or autoextend which would also be key
to use all Flashback features reliant on Undo
A realization he would slowly come to.

Now Flashback Transaction and Versions Between
were out of the picture, although unforseen.
But he still had four more kinds of Flashback to try
So he thought them each through, to see if they might fly.

Undropping the table would use flashback syntax,
But that wouldn’t help, since the table was intact.
With database flashback all could be reverted
But subsequent changes would then be subverted

Do you know who was governor of California
When Flashback Data Archive came out and I’ll warn ya,
it will not help poor Morton, though it’s called Total Recall.
He’d have had to enable it for tables to see all.

The last of the flashbacks was the old FRA,
the Flashback Recovery Area they say.
And it’s just a directory where things are kept
Like logs and old backups made while Morton slept.

So after all that there was nary a way
to use Flashback of any sort to save the day.
Old-fashioned LogMiner was what Morton used
To restore all the rows that poor Nimrod abused.

With so many flashbacks and so much confusion
I bet that Oracle regrets the profusion
of so many things that they call the same name.
But now its too late and there’s no one to blame.

19 Jan

The strangest Oracle problem I ever encountered – can you guess the cause?

Before I joined Blue Gecko, I did independent remote DBA work, and called myself ORA-600 Consulting. Stemming from my hair-raising experiences in the trenches at Amazon in the late ’90s / early 2000s, I decided to specialize in emergency DBA work for companies in the midst of crises (I know, great idea for someone who wanted to get away from the Amazon craziness, right?).

One day in 2009, a company in Florida called my cell phone at 2AM. They described their problem as follows:

We have a 32-bit Intel server running Red Hat Enterprise Linux 4 and Oracle Database Enterprise Edition 9.2.0.1. There are four databases ranging in size from 20G to 100G. The storage is EXT3 filesystems on partitions of an Apple Xserv RAID5 array.

We had a power outage yesterday, and the database server powered down and booted back up. Prior to yesterday, it has not rebooted for about one year. We have been running trouble-free for the previous year. Upon reboot, Oracle started automatically, but all of the databases appeared as they did about one year ago. It is like the database hasn’t been saving the changes we have been making for the past year. None of the inserts, updates or deletes made in the past year are present in the databases. We are absolutely flummoxed. Please help!

I logged into the server and it was just as they described. Even the alert log and messages files ended suddenly about one year prior, and picked up again on the day of the most recent reboot. There was no trace of the intervening 12 months of work. The customer was ready to resort to their backups, but wanted to understand the problem before they proceeded. In addition, restoring backups would mean losing the last 24 hours of transactions, since archivelogs had not gone to tape for that long, and they were missing just like everything else from before the most recent reboot.

They weren’t the only ones who were flummoxed. I just sat there thinking, “where do I start?” After some poking around, though, I solved the problem. Any guesses what went wrong here? I’ll post the solution in about a week. No fair posting the solution if I’ve told you this story before!

13 Jan

OurSQL Episode 74: Off the Charts, part 1

This week we kick off a multi-part series on monitoring. In this episode we discuss the MySQL Enterprise Monitor offering from Oracle and the MONyog offering from Webyog.

News
The Independent Oracle User Group is looking for nominations to the MySQL Council.

The next OTN Developer Day for MySQL will be Thursday, February 9th, 2012 in Frankfurt, Germany. Registration is free.

read more

6 Jan

OurSQL Episode 73: What happened?

This week we present a year in review for the MySQL Ecosystem, including updates from Oracle's MySQL, SkySQL, Percona and MariaDB.

News:
The MySQL developer’s room at FOSDEM has almost 40 submissions, and only about a dozen slots, so they need your vote to figure out what sessions will be presented. Send in your votes via twitter or e-mail, see Giuseppe's blog post and session descriptions.

read more

31 Dec

OurSQL Episode 72: Blooper Retrospective

This week we play a bunch of bloopers, some you have heard, some you haven't, as our year-end gift to you. We hope these make you laugh!

Sugus candy

26 Dec

OurSQL Episode 71: Table Manners, part 2

This week, we continue our discussion about MySQL and its forks. We discuss the Percona server and MariaDB.

Percona Live comes to Washington, DC on Wednesday, January 11th, 2012 at the Walter E. Washington Convention Center. There is a 50% discount for students, faculty and staff of educational organizations, and a 35% discount for government employees. http://www.percona.com/live/dc-2012/

Percona Server
Percona software

read more

16 Dec

OurSQL Episode 70: Table Manners, part 1

We called this podcast table manners because we are often asked what MySQL fork to use. This is part 1 of a 2-part series. This week we delve into Oracle's MySQL and Drizzle.

News:
There will be a "MySQL and Friends" devroom at FOSDEM 2012 in Brussels, Belgium on Sunday, February 5th, 2012. If you are interested in giving a talk, please submit it before December 26th through the submission form. There will also be a dinner on Saturday February 4th, for anyone making travel plans.

Last week we talked about the debian packages available for MySQL 5.5. We went looking for the packages and could not find the Debian drop-down for downloading MySQL 5.5, but we’re happy to report that the drop-down menu item is there now. http://dev.mysql.com/downloads/mysql/

read more

9 Dec

OurSQL Episode 69: Replication Secret Agent

In this episode we talk about the MySQL Master High Availability Agent, aka MHA.

News

There will be a "MySQL and Friends" devroom at FOSDEM 2012. The dev room is scheduled for Sunday 5th February 2012, whole day. If you are interested to give a talk, please submit it before December 26th through the submission form. You can read the original announcement by Frédéric Descamps.

Meet the MySQL Experts Podcast

read more

2 Dec

OurSQL Episode 68: Scaling, or Sailing?

This week Sheeri visits the Schooner offices and inteviews Dr John Busch, the CTO, Founder, and Chairman of the Board for Schooner. Schooner is able to fully use multi-core processors and do fully synchronous replication without losing performance. The new 5.1 release offers parallel synchronous replication with automated failover between remote data centers.

News
Both Sheeri and Gerry have new jobs - Sheeri wrote about her change at http://palominodb.com/blog/2011/11/23/last-day-palominodb

read more

1 Dec

ORA-03113: End-of-file on communication channel

Applications typically connect to an Oracle service over a network using TCP/IP sockets. When that connection is lost, the application-side Oracle client software raises ORA-03113: End-of-file on communication channel. Essentially, this error indicates that the connection to the Oracle service has been lost.

The most common reason for losing the server connection is that the Oracle server process that was servicing the application has died. In Oracle, individual server processes can (and often do) die without the whole service (instance) failing.

The best way to troubleshoot ORA-03113 is to examine the Oracle instance for errors. The DBA should look for errors in the alert log indicating server process death. By addressing the errors that are crashing individual server processes, the overall problem can be solved.

If an application continues to try to make requests against a dead connection that has failed with ORA-03113, it will raise ORA-01041: Internal error. Hostdef extension doesn’t exist. or ORA-03114: not connected to ORACLE.

Example:

SQL> select * from emp;
select * from emp
                *
ERROR at line 1:
ORA-03113: end-of-file on communication channel

SQL> select * from emp;
ERROR:
ORA-03114: not connected to ORACLE

To view the last 100 lines of the alert log on the server, use ADR:

$ adrci

ADRCI: Release 11.2.0.2.0 - Production on Thu Dec 1 15:38:15 2011
ADR base = "/u01/app"

adrci> show homes
diag/tnslsnr/ip-10-100-255-165/listener
diag/rdbms/uw01/uw01

adrci> set home diag/rdbms/uw01/uw01
adrci> show alert -tail 100

Alternately, do it the old-fashioned way (on Linux or Unix):

$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.2.0 Production on Thu Dec 1 15:43:43 2011

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> show parameter background_dump

NAME				     TYPE	 VALUE
------------------------------------ ----------- -----------------------------------
background_dump_dest		     string	 /u01/app/diag/rdbms/uw01/uw01/trace

SQL> exit
Disconnected from Oracle Database 11g...

$ tail -100 /u01/app/diag/rdbms/uw01/uw01/trace/alert_uw01.log