We won't sugar-coat it, our servers were offline for a 48 hour period recently, which was a first in our five years in service. Like all disasters, it wasn't one item in particular but a many factors that lead to the outage.
Regional IDX is committed to providing the best service in real estate web sites and real estate web hosting. We thank you for your patience and understanding while we worked to get this resolved.
The root cause of the problem was incorrect power supplied to the data center that houses the main Regional IDX servers and the fiber optic connections. We were not being supplied with the 480V of power from our two redundant power feeds.
Instead, we were being provided a fluctuating source of power, which is not unheard of. However, there was more than a 60V variance degree that caused our battery systems (UPS systems) to kick on continually to cover what it sees as a brown out.
This has burned up batteries in several strings that are tied to both primary and redundant UPS systems leading to a power failure.
This interruption subsequently lead to an improper shutdown of the servers which normally wouldn't be too terrible.
However the servers were going through routine nightly maintenance processes and, in return, resulted in a corrupt filesystem.
Since the Regional IDX servers are very large, it took more than 20 hours to diagnose, correct the filesystem and to restore the more than 2,000,000 photos.
We have replaced the bad batteries and now are fully aware of the actual cause for our interruption.
It's interesting note that during the scenario, there was the enormous number of phone calls and emails we received in such a short amount of time. This means that our clients and their customers are VERY in tune to their Regional IDX web sites.
As stated earlier, Regional IDX is committed to providing the best service in real estate web sites and real estate web hosting. We thank you for your patience and understanding while we worked to get this resolved.
Sincerely,
Regional IDX