Object Storage Internal error 500. Bucket unavailable for 3 days+

#1

The promised 99.9% Object Storage Availability on the front page is fake… another fake promise from Scaleway.

At the moment my object storage bucket is not accessible. I can’t use it as any action triggers internal error:
2019/07/09 17:34:17 ERROR : /: Dir.Stat error: InternalError: We encountered an internal error. Please try again.
status code: 500, request id: tx795027a68d2d416f8d704-005d24b3f8, host id: tx795027a68d2d416f8d704-005d24b3f8

This server was used for production so every minute counts, but still no detailed update from Support team for 20 hours and counting. Ticket was opened yesterday at 5pm (#MHRO-3422-QMKE).

I use Scaleway VPS for over a year now and already had quite a few problems with VPS’s which lasted for days. Have countless closed tickets and forum posts. Now the Object storage.

I believe that this amount of failed deliveries could be enough to sue Scaleway on legal bases.

No comments…

#2

Hello @Max_Millow,

Your issue is currently being investigated by the concerned team. Your data was accessible at all time, so there was no issue concerning the SLA. My colleagues will reply you directly in the ticket with more information.

#3

Thank you bene for prompt reply.

However my data was not accessible at all times. Didn’t I made this clear in the first post?
At this very moment I can not access my data via the Scaleway dashboard nor via the API.
see screenshots below…

The error response also has code 500 which means your internal error. Any actions to list, copy, move files are triggering the same error.

Rclone logs:

2019/07/09 18:00:41 ERROR : /: Dir.Stat error: InternalError: We encountered an internal error. Please try again.
status code: 500, request id: txc96af45037044910af226-005d24ba28, host id: txc96af45037044910af226-005d24ba28
2019/07/09 18:01:29 ERROR : /: Dir.Stat error: InternalError: We encountered an internal error. Please try again.
status code: 500, request id: txe4b2c3a404c743aab047a-005d24ba58, host id: txe4b2c3a404c743aab047a-005d24ba58
2019/07/09 18:02:27 ERROR : /: Dir.Stat error: InternalError: We encountered an internal error. Please try again.
status code: 500, request id: tx0a588acdb9cc41a19aa7a-005d24ba92, host id: tx0a588acdb9cc41a19aa7a-005d24ba92

#4

Hello,

my colleagues have informed me that the root cause has been identified and corrected.
Don’t hesitate to come back to us directly in the ticket if you have any further question.

Benedikt

#5

Hi.

The folder data shows as empty when mounted using rclone.
Using the same config and connections string as before.
The Scaleway Dashboard to browse files in the bucket also throws the same error as before, when trying to browse files in folder data.

Logs:

2019/07/11 19:55:11 ERROR : /: Dir.Stat error: SerializationError: failed to decode REST XML response
status code: 200, request id: txc087ea70d1614e1fab821-005d2777fe
caused by: XML syntax error on line 2: illegal character code U+0001

#6

Too much JavaScript to make everything look good. ITIL studied 40/40 for sure… or not. Real provider is Google and Amazon, but Scaleway… a what? Scaleway… hate that. It’s damn Online.net for me and I love it. I know it as Online.net, it will be Online.net for me and they provide best servers for hobbyists and enthusiasts. What’s more fun than be a big boy and play like a kid at sandbox with Windows Server and attach machines to it, because bandwidth is unmetered. Sync Center works nicely, because this company does not bill for transfers.

#7

The ticket is closed, and problem is kinda resolved.
The rclone uploaded a bad filename which caused bucket crash on listing. the data was unharmed but it took support 3 days to deliver information. SLA badly breached.

However Scaleway applied a hotfix so the buckets wont crash next time this happens.

Here is the Github bug report: https://github.com/ncw/rclone/issues/3345