Using the contact - getoptions method you can read the list of options for a custom field with select type.
/civicrm/civicrm/extern/rest.php?entity=Contact&action=getoptions&api_key=userkey&key=sitekey&json={"sequential":1,"field":"custom_5"}
But how can I create new options using the API?