Hello, We have planned a maintenance operation during the night of Thursday to Friday, which will correct some of the problems encountered, and which will permit us to advance towards implementing a permanent solution to these issues. With this kind of incident, Gandi prefers to offer transparency regarding an appropriate technical resolution, and we have therefore made best efforts to communicate all the details we have concerning this problem, as well as our action plan to fix the issue. At the same time, we have taken a certain amount of time before implementing the first corrective measures, since the technical analysis required was considerably more complicated than usual. For the time being, here are the first results of our analysis: - A new make and model of hard disks that we are using exhibits difficulty when processing write operations for certain block sizes. This can, on occasion, cause a complete block of write operations for a certain non-negligible period of time. We are currently working with our supplier to resolve this problem. - At the heart of our storage technology, STMF (SCSI Target Mode Framework in the Solaris operating system) was subject to a regression which severely handicapped its ability deal with higher loads, and most likely also provoked other peripheral issues. We have temporarily corrected this problem while we await an official patch. - We are performing detailed analysis of a problem with write cache synchronisation on the disks, which occasionally take an abnormal amount of time to complete. This problem is not easily reproduced, and we must perform debug traces on the production systems in order to obtain more detailed information. In order to further advance on these points, and to correct the initially identified problems, we need to restart the affected storage systems. This operation will be undertaken during the night of Thursday 23 to Friday 24 February, between 22:30 and 02:30 GMT, and will entail a total suspension of disk activity on your server(s) for approximately 15 to 20 minutes. During this maintenance period, please refrain from effecting any operations on your virtual servers, such as stop/start or reboot. The servers will resume their normal read/write operations once the maintenance operations has been completed. We will keep you informed on the progress of this operation via our website: https://www.gandi.net/news/ In the evening of Friday 3 February, we erroneously restarted our storage platform for the same kind of operations. Although this particular incident was resolved rapidly for the majority of our customers, we have nevertheless decided to reimburse the whole day of hosting services to the affected customers. All of your hosting products will have therefore had their expiry date extended as a result.
Nicolas Chipaux GANDI - http://www.gandi.net