It is bit complicated and boring but in the end I was successful. I had the old version right from December v2.8 and I was avoid the upgrade as the Automatic Upgrade would hang at “Unpacking updates…” So i did a manual upgrade. These are the brief steps how i did it:

1. Backup your database to be on safe side. Though I never had to recover from it. Things went right.
2. Backup ALL your WordPress files in your WordPress directory. Don’t forget your .htaccess file as well.
3. Deactivate ALL your Plugins. This is important.
4. Ensure first three steps are completed. Do not attempt the upgrade unless you have completed the first three steps.
5. Download and extract the WordPress package from http://wordpress.org/download/.
6. Delete the old WordPress files on your site, but DO NOT DELETE

• wp-config.php file;
• wp-content folder; Special Exception: the wp-content/cache and the wp-content/plugins/widgets folders should be deleted.
• wp-images folder;
• wp-includes/languages/ folder–if you are using a language file do not delete that folder;
• .htaccess file–if you have added custom rules to your .htaccess, do not delete it;
• robots.txt file–if your blog lives in the root of your site (ie. the blog is the site) and you have created such a file, do not delete it.

7. Upload the new files from your computer’s hard drive to the appropriate WordPress folder on your site.
8. Run the WordPress upgrade program and follow the instructions on the screen.
9. Update Permalinks and .htaccess. Update your Permalink Structure and merge the custom rules, if necessary, into your .htaccess file.
10. Install updated Plugins and Themes. Please review the list of Plugins that work in Version 3.2.1. Check for Theme Compatibility with 3.2.1 and ask your Theme author for any new version.
11. Reactivate Plugins
12. Replace the wp-config.php file
13. Review what has changed in WordPress. Open the Admin Page.

In case if something goes wrong, replace the old files that you had backed up.
All the best!