OK. If you've arrived here from somewhere on the web you will already know that this issue is widespread and isn't only a WordPress issue. I have spent several days on this issue and received prompt, courteous and knowledgeable help from my hoster GoDaddy.
I think the way to troubleshoot this problem (and resolve it - because I have) is as follows - I would suggest doing it in this order.
First, this isn't necessarily a memory issue. Spending time on this right from the start won't help. You will doubtless have tried this already. Fine, something to do in the future.
Next: BACKUP your database.
If you're using WordPress I would install a plugin WPPhpMyAdmin - I use it all the time and it saves logging into to another screen - you can always open up other tabs to your site in the your browser.
1) From within phpMyAdmin browse your database. You should automatically be in the 'Structure' screen - just look down at the various 'Records' - nothing should seem too large. I had a huge number of items in my 'postmeta' records - this was the source of my problem.
2) Browse the 'postmeta' records. You are likely to see the problem - I had a huge number of entries under some kind of hash reference, might be a ping cache, lots of things. One way to identify this is by clicking on the column 'metakey' which will run everything in alphabetic order.
3) If this is the case then run a script from within the 'SQL' tab : DELETE FROM wp_postmeta WHERE meta_key = 'YourMetaKey';
where 'YourMetaKey' refers to the 'metakey' which you've identified as holding all these pointless records.
4) The problem should go away.
5) Keep your database nice and clean. These are some of the best tools for this:
WP Cleanup - if you only want one plugin - this is the one.
WP Optimize - you don't really need this if you've the one above.
WPDBSpringClean - keeps those plugins at bay which you've uninstalled.
WPMassDelete - helpful for getting rid of old posts.
And, I'm sure I don't need to say this: backup, backup, backup. And, oh yes, backup!
If this helps you as much as it might have helped me a week ago then I will have paid back something to a community which has helped me along all these years.