I don't know if this is the case here, and it seems more regular than my experience, but my coworkers and I have seen cron jobs walk fairly regularly. (We're more commonly using Ubuntu 14.04 or 12.04, however.)
The usual issue I see is the following with a daily scheduled job:
- cron run starts at 1:00:00
- there are a bunch of things to do, so it gets to the daily job at 1:00:20
- the next day at 1:00:00, cron runs again
- there's less to do, so it gets to the daily scheduled job at 1:00:10
- being "daily", the scheduled job checks to make sure its previous run was more than one day ago, and that check fails since it was 23 hours, 59 minutes, 50 seconds ago
- the scheduled job executes the next time cron runs
In my experience, that would normally make the scheduled job walk one cron interval (usually 5, 6, 10, or 15 minutes) rather than an entire hour. However, if you only have the cron job running hourly, that would explain your phenomenon exactly.
Unfortunately, there's no way to guarantee that a scheduled job will get run at a precise time each day unless you have a separate cron job to run the single scheduled job. However, reducing the cron interval will mean that when the schedule job does walk, it'll take smaller steps.
(Side note: this situation is why I always have the scheduled reminders job run hourly rather than daily. Otherwise, the scheduled job might walk all the way to 11:00 pm, potentially reminding people about the next day's event after the recipients have gone to bed.)
UPDATE: As Karin describes below, this was addressed in 4.6.3, so the later 4.6.x releases all fix this. However, some changes introduced in 4.7 had the result of undoing this. The relevant issue is CRM-18671 for describing what's going on.
