Coldfusion 7 MX cannot understand oracle sometimes

Like every Monday,  I came into work today and got a complaint about our portal’s coldfusion business application not working.  The coldfusion applications were built on Coldfusion 7 MX and the backend was Oracle 11g. The exception on the portal look as below:

Portal Exception
The Coldfusion Exception

After 30 minutes of troubleshooting, I went into the coldfusion admin page and noticed that when verifying the oracle connection it gave an “Internal error: Net8 protocol error” message. Later on googling a bit, I understood from this forum post that the exception was because coldfusion 7 MX was unable to understand the “password about to expire” signals from oracle for the user that coldfusion uses to connect to oracle. So we reset the password for that user in the oracle database using the alter command and the problem was fixed.

Thanks to Google and the posted solution this production issue was solved within an hour.

Fix to show the hidden content behind the overflow scrollbar in Internet Explorer

One of my web applications at work was having an issue of displaying the content completely in the internet explorer.

We had our coldfusion fusebox applications embeded into a PHP template and I had the style set to display only the horizonal overflow scrollbar i.e. “overflow:auto; overflow-y:hidden” . This works fine with Firefox but when we open the same in Internet explorer we have an issue. The scrollbar hides some content behind the itself.

The solution to this problem in IE, as mentioned in the referenced link, is to add around 20 pixels of padding at the bottom of the page.

This fixed my problem and works like a charm. Hope this helps others too. Thank you to the guy who wrote the referred link.

sendmail service queue clearing and ORA-24247 error fix using instructions to add ACL info.

The other day at work we had a request from the client to send an e-blast (mass e-mail) to all the email addresses listed in our database (~55k emails).

In the past this was done using a procedure in the oracle database which used one of our smtp mail servers and oracle UTL_SMTP package. But due to a missing smtp mail server we could not run the procedure. We tried changing the mail server ip from a non-working one to a working one, but this didnt work. We kept getting an error that said

ORA-24247 network access denied by access control list (ACL)

As this was a time critical task and we did not have time to figure out where the missing server went, I created a PHP script that retrieves the email addresses from the oracle database and sends the message to all of them one by one using the mail() method in PHP. This when tested worked great. I estimated the script runtime to be 4 hours. I started the script at 630 PM and came back next morning and it was still running!

The script had already sent 50 – 300 messages to each recipient by then. To stop the php script I stopped the PHP page which was running all night. This did NOT stop the script in the backend. To stop the mailing process I went through google and figured out that the mail() process used the sendmail service on the redhat server. So I manually stopped the sendmail service which stopped the mails from sending.

I then asked the network administrator to check if there were any mails in the SMTP server queue. There were 1000s of messages in queue. I requested him to stop all the messages in the queue but the messages with RETRY status could not be stopped. There was a unapplied patch that needed to be run to make this work. At least the mails was not sending the mail but stuck at the queue.

After 3 days when I restarted the sendmail service on the server, it resumed sending the messages to all the recipients in the queue. This increased concerns. I then researched a little bit more and came to know that the sendmail service in redhat has its own mail queue which can be viewed using the mailq command in linux or using the sendmail –v -q command. There were 181k messages in queue waiting to be sent. All the queueud messages were stored in the folder /var/spool/mqueue. (Reference)

To delete all the messages in queue I ran the command rm /var/spool/mqueue but this didn’t work and gave me an error. “/bin/rm: Argument list too long” . This was probably due to the limitations of rm command to have a length of arguments as a max of 1024. The alternate command (reference) to delete all the 181k files is “find . -name ‘*’ | xargs rm”. This deletes all the files in the current directory regardless of the number of files.

This way the spooled messages were cleared from the queue but the problem the database procedure not able to access the mail server still existed. So I went through various google articles referring to the ORA-24247 error. I came to know that this error was due to an extra security layer in oracle 11g. There is an XML table in  the oracle 11g database that tells the packages about particular accessible server ips. This article helped me out in understanding the extra security ACL list information and updating it to serve my purpose.

This way I fixed the oracle script access to the smtp server and solved the mailq problem with the e-blast messaging project. Just shared it to help others fix this problem if the come across it.

CA Arc serv backup agent for oracle

The other day I was helping out the network engineers install an Arc Serv Backup agent on one of our servers for them to manage backups and restores for that server. I installed the Arc Serv agent for the server on the Red Hat Linux server without any problems but when it came to installing the Oracle backup client, it overwrote all the oracle environmental variables. thus ending up in crashing the whole production database.

It initially didn’t give any problem. the problem became prominent only after a few minutes when our customers were having trouble. I and the team then worked on the troubleshooting of it. And in the end we figured it out and manually setup the Oracle_home variable to fix everything on the oracle production server.

Thank god we were able to figure out the source of the problem in a short time, it took only 1 hour to startup the oracle server back. I got a chance to explore Oracle setup to fix this issue.

“We learn from our mistakes, so we have to make some mistakes” 🙂

Apache start error – “DocumentRoot must be a directory”

Fix for the “DocumentRoot must be a directory” error when changing the document root from default in the apache configuration file (httpd.conf)

I was going through this error for around 10 days now. I was setting up a new linux server for test on a Red Hat Linux Enterprise Edition 5. I changed the documentroot from “/etc/httpd” to “/web/portal” and when I restarted the apache server it gave me the error saying “DocumentRoot must be a directory”. I made sure I had the permissions right on the folder and the folder existed but the error seemed to persist.

I went through many forums and then came across this post. The solution was to make sure the Security settings for the linux system were setup properly. So what was needed was to adjust the setting in the security level settings using the command system-config-securitylevel (or redhat-config-securitylevel) and then the server started without any problem.

Learning a little bit of system administration

I am learning usage of VMware to setup multiple servers on a single physical system.

After working on web development on a localhost and pushing changes to a staging then production server for a long time, now I am going to learn how to setup a developer server, staging server,  production server, database servers and other required servers on a given number of resources optimally using Virtual Machines.

Currently at work we use VMware on host systems  and run multiple servers on the same system making optimal usage of the resources. First thing I am doing is reading up on technical information about VMware here. Then I will be learning how to set ip addresses within a network and assign DNS to them. Then I will install a linux server on  one of the VM clients and setup a server etc..

I will be doing all this in my laptop and from now on instead of installing linux on a seperate partition as I have been doing for years, I will be installing it on a VM host on my laptop and will operate them from my windows like I always wanted to do (without rebooting the system).

I have played around with these virtual pc stuff in my school days but never went too deep into it. This is the time when I need to go into it deep. I am very excited to experiment with all this. I should have learned all these earlier but never mind. Its never too late to learn 🙂