The waitlist feature applies to the whole event when "Max Participants" is set on the event info & configuration page, not to limits set for individual options in a price field.
The usual behaviour when an price field option is "fully subscribed" is that the option can no longer be selected during the registration process.
So the behaviour you describe is not usual and the behaviour you want is not 'out-of -the box' CiviCRM.
Can you reproduce your resultsSomeone else will have to advise on how to get the behaviour you want.
But a first step would be to upgrade your site to 4.5 demo site. The8 which is the latest version in the 4.5 series so that the behaviour when an price field option is 4.5"fully subscribed" is operating correctly.8 perhaps It would not be sensible to create custom code on a site with a bug in 4.5.5 was fixed by 4.5.8.