The waitlist feature applies to the whole event when "Max Participants" is set on the event info & configuration page, not to limits set for individual options in a price field.
The usual behaviour when an price field option is "fully subscribed" is that the option can no longer be selected during the registration process.
So the behaviour you describe is not usual and the behaviour you want is not 'out-of -the box' CiviCRM.
Can you reproduce your results on the 4.5 demo site. The latest version in the 4.5 series is 4.5.8 perhaps a bug in 4.5.5 was fixed by 4.5.8.