C14 usage questions


#1

Hi,
I’m interested in C14, but very confused on how it works, from the descriptions given in the information page. Note that I’m refering to the standard service, not intensive.

From what I can tell:

  • you interact with one or more safe-deposit boxes by uploading to it
  • archives are like snapshots of this deposit box; you do not directly interact with them
  • but the C14 CLI tool only supplies an upload to archive function, so maybe you can upload to both archives and safe-deposits??
  • I notice that it also offers a freeze function - does that mean that the upload doesn’t really go to an archive, but instead, the safe-deposit?
  • all uploads (and downloads?) to this safe deposit box are free
  • what about uploads directly to an archive?
  • only archive freeze/unfreeze and delete are charged as operation costs

So my questions:

  • can data in the safe-deposit box be downloaded?
  • can data in this deposit be deleted?
  • can one file in an archive be updated/deleted, or does this require unfreezing the whole archive, changing the file, then re-freezing everything, incurring 2x operation costs over the whole size of the archive? If so, does re-freezing overwrite the old archive, or does that also need to be removed, incurring another operation cost? Also, from this response, it seems like the storage cost for a month is incurred immediately, so if re-freezing creates a new archive, does that mean you also get hit with an additional 1 month’s worth of storage costs as well?
  • data in the deposit is frozen automatically after 7 days - is this 7 days since the data was uploaded, or from when the deposit was created?
  • once data from the deposit is archived, I assume the data is no longer visible in the deposit? If so, how do sync protocols, eg rsync, work, since it’d just re-upload everything after the archival process?
  • does unfreezing essentially extract the contents of the archive into the deposit?
  • Under “Simulate the cost of storage of your data” on the C14 info page, there’s a “Time” option - what does this mean?
  • I assume the “standard” and “intensive” service levels are a property of the safe-deposit? Is it possible to change the service level of the deposit, or does this require re-uploading?
  • It seems like there’s a benefit to having many small archives instead of just one big archive, if you anticipate that the data may change. Is there any downside to doing this other than making management more difficult?

Say that I have a 100GB set of files that I want to back up to C14. The data set rarely ever changes, but 1GB of data is added to it every day.
So firstly I need to set up a safe-deposit box, then transfer up the initial 100GB set. I set it up so that rsync executes every day, syncing a local folder to the C14 deposit box.
This means that on day 1, 100GB is uploaded, then on day 2, 1GB is uploaded (or added), day 3, another 1GB uploaded. On day 7, the deposit now contains 106GB of data, which then automatically gets archived (€1.06 charged + €0.212 for 1 month of storage).
Assuming that all the data in the deposit is now gone, on day 8, the rsync job would upload 107GB of data? This continues to day 15, where the 114GB of data in the deposit is archived (€1.14 charged + €0.228 for a month). However, now I’ve got two sets of data, so I need to remove the old 106GB archive (another €1.06 charged) if I don’t want to be paying for it into eternity.

Is my understanding above correct?

If I then, later, wish to restore data from C14, I need to unfreeze the latest archive into the deposit (€1.14 charged; does this conflict with existing data in the deposit?), download the data, then delete data from the deposit to prevent it automatically being archived again?

Thank you for reading!


#2

+1

Great thread - please clarify online.net, for me, especially the rsync informations would be very interesting.


#3

I beleive C14 is for archive, not for backup. Maybe it will change with “intensive”, but for now it’s not very easy to use it for incremental backup like you want.


#4

Safe deposit box is an ftp/scp temporary storage area that looks like a normal file system that you can upload to and download from. Archive (C14) is like an iron box sealed underground. You cannot communicate directly with an archive. Rather, there is an automated process that transfers safe deposit boxes to the archive and back.

C14 archives split the user data across a lot (dozens I think) of servers using software erasure codes, so transferring to and from safe deposit boxes can take hours or in my experience sometimes overnight. During the transfer, the safe deposit box is inaccessible. After transferring the SDB to archive, the SDB is emptied. The erasure code means many hard drives would have to fail simultaneously for data to be lost.

In principle you could use the safe deposit box as a temporary ftp area and store files for a few days for free. The TOS doesn’t allow that but some people do it anyway and haven’t had trouble so far. If you leave data in the SDB for 1 week, it gets archived automatically and you are charged for the operations and storage.

You can’t conveniently incrementally change the contents of an archive, so if you want something like that, you’re better off with multiple small archives representing the deltas.

Your idea of uploading 1GB a day to C14 doesn’t sound like fun, though maybe it could be ok if automated. I’ve been using C14 to back up larger datasets, like 100GB. C14 is not an object store like S3. You might be better off with a weekly incremental upload/archive than a daily one. Or you could upload to the ftp (safe deposit) area daily, but only archive once a week.

I did the math and found that intensive is cheaper than standard unless you leave the data at rest for more than 6 months, then delete it without reading it. That’s because de-archiving and deletion each get an operation charge. So far I’m only using intensive. Even intensive, though, is more for long term storage/backup than for daily operations.


#5

Hi @o_o

Thanks for your feedback, I published a blog post a few months ago on how C14 works with examples etc. you should check it out https://blog.online.net/2016/07/04/c14-the-secure-cold-storage-platform-for-free-during-the-summer/ it may help.

When you create a new archive, a “temporary space” aka “safe-deposit box” is created. You access this area via different protocol: FTP, SFTP, Rsync, SCP, etc. and can upload / remove files while the safe-deposit box is opened.

Note: [quote=“o_o, post:1, topic:4675”]
can one file in an archive be updated/deleted, or does this require unfreezing the whole archive, changing the file, then re-freezing everything, incurring 2x operation costs over the whole size of the archive? If so, does re-freezing overwrite the old archive, or does that also need to be removed, incurring another operation cost? Also, from this response, it seems like the storage cost for a month is incurred immediately, so if re-freezing creates a new archive, does that mean you also get hit with an additional 1 month’s worth of storage costs as well?
[/quote]

An archive is immutable, that means you can’t update its content. So no, you can’t update or delete a file in an archive. To do that you have to

  1. Unfreeze your archive
  2. Access the safe-deposit box to add / remove / update files
  3. Freeze your archive
  4. Remove the previous one if you do not need it anymore.

Remember that the main goal of C14 is to a offer an affordable and secure cold storage and archiving platform so generally you do not have to update an archive once frozen. For intensive archive accesses, we provide the **intensive service level ** which is without operation fees.

It’s frozen 7 days after the creation date. If this delay is too short, we also provide an option for large bucket.

Unfortunately it’s not possible sync data with an archives are they are immutable.[quote=“o_o, post:1, topic:4675”]
does unfreezing essentially extract the contents of the archive into the deposit?
[/quote]

Unfreezing extract the content of your archive in the safe deposit box.[quote=“o_o, post:1, topic:4675”]
Under “Simulate the cost of storage of your data” on the C14 info page, there’s a “Time” option - what does this mean?
[/quote]

The Time option lets you see the storage cost over time.

The service level: standard, enterprise and intensive is a specification of the archive that have direct impact on the pricing, SLA and sustainability. It’s not possible to change the service level of an existing archive.

Hope I answer your questions, let me know if you need additional details.

Edouard


#6

Sorry for the late reply, and thanks a lot for the responses! C14 seems to make more sense to me now.

I find the idea of charging for deletion a little odd. Like, what happens if one just simply stops paying? My assumption would be that the account is lost and data is removed, but this would save paying for deletion if nothing else on the account was ever needed. One could, in theory, set up multiple accounts with one archive each. I suppose this would be a violation of some ToS, but the whole thing seems a little odd to me.

In theory, it should be possible to support an incremental backup if each archive represents a diff, but I suspect the tooling support for this isn’t really there? For one, restoration sounds like it could be a pain if you needed to merge a bunch of archives to get all your data.

Anyway, thanks again!