It has been confirmed by @peternz@petednz and by testing without the 'Separate membership payment' option, that unsetting this option resolves the 'transaction is a duplicate' error.
In our case the membership types were moved out of the form into a price set.
