Gandi Groups


Hosting General: Planned maintenance on hosting filers.

You should be logged in to post new messages. Create an account.

Hello,

We have planned a maintenance operation during the night of Thursday to
Friday, which will correct some of the problems encountered, and which will
permit us to advance towards implementing a permanent solution to these
issues.

With this kind of incident, Gandi prefers to offer transparency regarding an
appropriate technical resolution, and we have therefore made best efforts to
communicate all the details we have concerning this problem, as well as our
action plan to fix the issue.

At the same time, we have taken a certain amount of time before implementing
the first corrective measures, since the technical analysis required was
considerably more complicated than usual.  For the time being, here are the
first results of our analysis:

- A new make and model of hard disks that we are using exhibits difficulty
when processing write operations for certain block sizes.  This can, on
occasion, cause a complete block of write operations for a certain
non-negligible period of time.  We are currently working with our supplier
to resolve this problem.

- At the heart of our storage technology, STMF (SCSI Target Mode Framework
in the Solaris operating system) was subject to a regression which severely
handicapped its ability deal with higher loads, and most likely also
provoked other peripheral issues.  We have temporarily corrected this
problem while we await an official patch.

- We are performing detailed analysis of a problem with write cache
synchronisation on the disks, which occasionally take an abnormal amount of
time to complete.  This problem is not easily reproduced, and we must
perform debug traces on the production systems in order to obtain more
detailed information.

In order to further advance on these points, and to correct the initially
identified problems, we need to restart the affected storage systems.

This operation will be undertaken during the night of Thursday 23 to Friday
24 February, between 22:30 and 02:30 GMT, and will entail a total suspension
of disk activity on your server(s) for approximately 15 to 20 minutes.

During this maintenance period, please refrain from effecting any operations
on your virtual servers, such as stop/start or reboot.  The servers will
resume their normal read/write operations once the maintenance operations
has been completed.

We will keep you informed on the progress of this operation via our
website:  https://www.gandi.net/news/

In the evening of Friday 3 February, we erroneously restarted our storage
platform for the same kind of operations.  Although this particular
incident was resolved rapidly for the majority of our customers, we have
nevertheless decided to reimburse the whole day of hosting services to the
affected customers.  All of your hosting products will have therefore had
their expiry date extended as a result.
Nicolas Chipaux
GANDI - http://www.gandi.net