Resume scheduled task after server went down

Question

Experimentation has shown that if the Adaxes service is not up when a Scheduled Task is supposed to run, the ST obviously does not happen. After the service comes back up, such missed STs do not "catch up", either. It seems that the Adaxes scheduler looks at each ST's Next Run datetime and uses the "=" operator rather than the ">=" operator to decide whether or not to run it.

This means that if the Adaxes server is down for some reason, any STs that ought to have run during that time will be stuck in limbo once the server comes back up: their Next Run datetime is in the past, so the scheduler just ignores them.

I have found that disabling and re-enabling such a stuck ST will recalculate the Next Run datetime, and so that ST is back in business. However, this requires manual intervention, which I would like to avoid. Is there some way the Adaxes system can sort out this situation automatically?

Note that at our site we have a single Adaxes server, not a multi-server deployment.

Answer 1

Hello Alex,

As per our check, such situations do not take place. When a server comes back from down and Adaxes service is started and initiated (you just need to connect to it via the Administration console), the scheduled tasks that have their next run time in past will be executed automatically. Sometimes, there can be a delay, but they will still run without any manual interaction.

Comments

I did some more experimentation, and it does seem that after a service restart (and a re-connect in the Admin Console) that the system attempts to bring things up to date. It did not work 100%, however, I still ended up in a situation where the enabled Scheduled Task had a Next Run datetime that was in the past, and even allowing some time to pass did not resolve that.

I believe my experimental setup may have had something to do with this. I created a ST that runs once per minute; all it does is use LogMessage() to write the current date and time to the log. Perhaps because of the frequency, things never really got a good chance to catch up.

Here's what I did:

1. Let the ST occur a few times so that there were several log entries.
2. Closed the Admin Console.
3. Stopped the Softerra Adaxes Service.
4. Waited for two or three minutes (i.e. a couple of times were missed).
5. Started the Softerra Adaxes Service.
6. Opened the Admin Console, and let it reconnect with the service.
7. Examined the Logging section, and also looked at the ST's Properties, specifically its "Next run" datetime.

There were one or two new entries in the Logging section, which occurred after I restarted the service. This shows an attempt to catch up on the missed runs that ought to have occurred during the down time. However, the ST's Properties pane showed that its Next run datetime was already in the past (although it was scheduled for after the restart of the service). Therefore, the ST did not run again, even after waiting for some time. Disabling and re-enabling the ST straightened that out because that recalculated the Next run datetime to a point in the future.

So it seems that Adaxes tries its best to catch up on missing STs, but if there's another missed time while it is doing that, the problem I originally mentioned re-occurs.

I don't think this is a serious problem, because we have no intention of scheduling once-per-minute tasks; that was done only for experimental purposes. When the scheduled tasks occur at a more reasonable interval, like once per hour or once per day, it seems likely that things will work as expected.

The big takeaway for me is that Adaxes does indeed attempt to "catch up" on missed Scheduled Tasks.  That's an important thing to realize.

---

Hello Alex,

Thank you for the provided details, Could you, please, post here or send us (support@adaxes.com) screenshots of the entries you see in the logs? Screenshots providing more insight on the behaviour will also be much appreciated.

---

Please note that I'm content with the behavior of the product, and I'm not requesting any kind of fix/change.  However, because you are interested, here is a what I had set up:

I created a new Scheduled Task that ran the following PowerShell script once every minute:

$now = (Get-Date -Format G)
$Context.LogMessage("Current datetime is $now", "Information")

As described in my previous comment, I let the ST run a few times (A in the screen snapshot). Then I closed the Admin Console and stopped the service for a few minutes.  After that I restarted the service and opened the Admin Console again.  After the reconnection I examined the Logging section, and saw that there were some "catching up" STs (B in the screen snapshot).

However, looking at the Properties of the ST showed that the Next run on that ST was at 9:01 -- that is, before the most recent logged item at 9:02:40.  Therefore, no further items appeared in the Logging section.  I do not have a screen snapshot of that, because eventually I disabled and re-enabled the ST.  This resulted in the top-most line in the above image, at 9:13:07.

I hope you find this information helpful.  But to re-iterate, I'm not looking for any changes to the product.  I do not plan to schedule intervals as short as once per minute.