Server Side Issue Turned Out To Be Design Issue
Thanks to Arthur, the poor guy who has to fix the daily report update whenever it is not working, took the time to track down the problem. He finally figured out the obscure reason that caused the random problem of reports not uploaded. It turned out it is a design issue, meaning that it is all my fault.
In the beginning, the report generating server was never meant to do more than a few simple things. Hence the process of report generation was created with an ad-hoc solution. Over time, however, as more and more responsibility is placed onto the server, the scheduling become a mess and conflicts among the scheduled tasks started to happen from time to time when they do not complete on time.
Yesterday, we tried out an alternative way to schedule the tasks to see if it can work more efficiently.
Looks like we have a good solution on hand to take care of the problem for good.
Another lesson learned – taking a dirty shortcut to get something done in the beginning is never a good idea in the long run.