This project has moved. For the latest updates, please go here.

New Backup feature is ready for testing

Oct 21, 2010 at 8:57 AM

As a preparation for my .mirror feature, which is more complicated than I first expected, I decided to commit a little additional feature. You will find this in the next days in the Liquesce repository. Like the .mirror feature it is an optional configureable thing. So no one has to use this... but if you like it let me know ;)

My first thought of a backup drive was to have two Liquesce drives with different physical disks. So thats why I first requested to have multiple Liquesce disks. Nice thing, but we would have 2 pools of free disk spaces (for virtual data disk and virtual backup disk). I don't like this.

Since I have implemented the tools I need right now from the .mirror preparation I am implementing the following thing:
Imagine you have only one Liquesce disk and a .backup directory in it. If you copy data from your root dir to the .backup , Liquesce should automatically detect these copys as backups and places the files/folders on a different physical disk than the original file. So if one disk crashes you don't loos data (at least since the last backup you did in .backup). It isn't possible to put other files/folders in the .backup dir if they are not found in the root. This is to prevent a wrong usage of the .backup folder. And you can choose which files you want to back up by not copying the others.

So to make it more clear:
physical Disks:
D:\
E:\
Liquesce Disk:
T:\

You have T:\.backup .
First you create a data file in T:\folder\newfile.txt. It will be placed for example on D:\folder\newfile.txt .
Then you copy T:\folder\newfile.txt to T:\.backup\folder\newfile.txt . The new .backup thing will make sure that this file will be placed on E:\.backup\folder\newfile.txt .

If D:\ crashes then you have the backup. If E:\ crashes then you have the original file.
Easy feature but a powerful result :-D

There is one important limitation for this: If you move files from one physical drive to an other then it is possible that the original and the backup is on the same physical disk. So you shouldn't write on the physical disks if you use .backup

Like discussed in the .mirror thread we can think about presenting the .backup folder as an own Liquesce drive and filter it in the data drive.

It should be also possible to use .backup with file sync programms. Just sync you data in the .backup folder. So we would also be able to have a kind of non realtime mirroring feature in a few days, like in the WHSv1 but better and more transparent :-).
Right now I am not thinking of dropping the .mirror feature which should write the mirrored files live in the end of the implementation, like in WHSv2 but better and also more transparent :-D.

Hope you will like it!

Coordinator
Oct 21, 2010 at 5:50 PM

So will this feature also deal with further updates in T:\.backup\folder\ so that once it is in the .backup all modifications will also go into the .backup ?

It's not quite like a the .mirror which is doing all dirs / subdirs of all things marked, just the subset that the user has decided to start..

Also how a re you going to notify the user that once they have used all the space on the other lcations of .backup that they ar now actually copying onto the same drive ?

Oct 22, 2010 at 10:41 AM

Just to make it clear:

1.) T:\.backup is the backup folder with all backed up data and it isn't the backup source. The backup source is in T:\ (excluding the .backup of course).

2.) Only the things which the user copys to the .backup will be backed up. (So, Yes)

3.) Insufficient space notification is the same like in the normal drive since it is on the same virtual liquesce drive. Only one physical disk is excluded for storage (that one with the original file).

 

Ok from the logical view it should do the following in pseudo pseudo code:

If something is changed in .backup and .backup feature activated

    If there is a file/folder in the original data dir with the same name

       make sure that the backup file/folder will be placed on a different disk

    Else

       throw a access denied error // to prevent wrong usage of the .backup feature

    EndIf

EndIf

 

So you have to use a backup/sync program of your choice to bring your data in the .backup folder or you copy it manually with explorer or script. The point is that you will get an automatically redundancy if you copy your files in .backup. Everthing with automated copy will be in the other .mirror feature.

 

All folder operations are already working in my working copy.

I have troubles with the file write because there is already so much manual code and the Roots class isn't used for all path requests completely. I think I have to change the Roots.GetPath() and then the CreateFile Method. First tries were not working... :-( I'd like to have a cleaner abstraction of the whole root/path/relative directory operations. I think we should not be able to see anything related with path lookup tables or root lists from the LiquesceOps.cs and just implement the file operations in there. All path manipulations should go to Roots class.

The sharing things in Roots.GetPath() makes me also troubles because I'm not really seeing the sense of it. I don't like this recursion thing in the GetPath also...

But I keep on testing and implementing...

 

PS:

What about a naming convention?

eg:

path = C:\drive1\folder\file.txt

root = C:\drive1

relative = \folder\file.txt

just to get the code more clear...

Oct 23, 2010 at 11:40 AM

First .backup finished

 

Configuration:

Go to the "Global Advanced Settings" and switch the "Allocation Mode" to "backup".

Create a new ".backup" folder in the root of your virtual liquesce drive.

 

Usage:

Just copy your files or folders from the root directory to the .backup directory.

copy

Liquesce .backup won't copy anything automatically. There are many third party apps which can automatically copy or sync folders. Tell us which are the best!

Liquesce .backup just takes care that the original file and the backup isn't placed on the same disk.

allocate

If one disk crashes you already have the original file or the backup version of this file.

 

Limitations:

You are not allowed to change the file's/folder's location from one physical disk to an other. Liquesce .backup isn't recognizing these changes and it's possible that the moved file is on the same physical disk as the .backup file. Just copy over your virtual liquesce drive if you are using .backup

 

Current Status:

The base feature set should be implemented right now. There are no known bugs with .backup. If you find one please report how to reproduce and a description of the issue.

There is room for improvement with the copying performance of many small files. Or better: the allocation search takes some time so if you copy many files it will be very slow. We will see when I have time to make it faster. The copying process itself should be as fast as in the normal liquesce drive. So for large files you don't even recognize a difference.

 

I hope there will come some fans for this feature... I'm looking forward to implement other backup and mirroring strategies for Liquesce so I want to know your opinion.

Oct 24, 2010 at 10:01 AM

Update:

Current version isn't stable any more. Seems that I've destroyed something since the all modes are making troubles.

Use the latest release if don't want to run in troubles.

sorry, I'm working on it.

Coordinator
Oct 24, 2010 at 10:48 AM

I just about to get latest and make a relase for this [url:http://liquesce.codeplex.com/releases/view/54504] when i saw the above..

So have left it as planning !

Oct 24, 2010 at 11:36 AM
Edited Oct 24, 2010 at 12:00 PM

Ok thx...

But I have problems with this right now... Sometimes I get not enough resources errors... No idea where are the comming... I hope I don't damaged something.

Did you tested the new build before you put it on the site?

Stefan Lechner MSc


2010/10/24 smurfiv <notifications@codeplex.com>

From: smurfiv

I just about to get latest and make a relase for this [url:http://liquesce.codeplex.com/releases/view/54504] when i saw the above..

So have left it as planning !

Read the full discussion online.

To add a post to this discussion, reply to this email (Liquesce@discussions.codeplex.com)

To start a new discussion for this project, email Liquesce@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com

 

Oct 24, 2010 at 11:38 AM
Edited Oct 24, 2010 at 12:00 PM

And I also had issues with the logging thing...

On trace level it got very unstabel :-/


Stefan Lechner MSc


2010/10/24 Stefan Lechner <stefanlechner@gmx.at>
Ok thx...

But I have problems with this right now... Sometimes I get not enough resources errors... No idea where are the comming... I hope I don't damaged something.

Did you tested the new build before you put it on the site?

Stefan Lechner MSc



2010/10/24 smurfiv <notifications@codeplex.com>

From: smurfiv

I just about to get latest and make a relase for this [url:http://liquesce.codeplex.com/releases/view/54504] when i saw the above..

So have left it as planning !

Read the full discussion online.

To add a post to this discussion, reply to this email (Liquesce@discussions.codeplex.com)

To start a new discussion for this project, email Liquesce@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com


 

Coordinator
Oct 25, 2010 at 5:50 PM
fpdragon wrote:

And I also had issues with the logging thing...

On trace level it got very unstabel :-/

Drop the threads down to 1, and anything that is requireing a timeout will probably start to be hit as well.

Instability is showing that your implemenations do not like being stressed, this is bad, and probably has something to do with access variables via multiple threads

Oct 28, 2010 at 11:47 AM

I've noticed one problem with the structure we have right now:

Right now it is only possible to copy files which are existing in the original directory. I've implemented that to prevent to use .backup as a normal folder for storing any files.

The thing is that many sync programs are making temp files for indexing and they store them to the destination location. In our case this destination would be the .backup folder but the access is denied for these temp files.

 

I'm thinking of removing the write restrictions on the .backup folder to make it possible to use every sync program you like but you have to keep in mind to use the backup folder in the correct way.

An additional tool for checking the backup consistency is planed and I already started with the implementation. So you should be able to see if your backup is secure with this tool.

Coordinator
Nov 12, 2010 at 7:08 PM

Could this be extended to use the MS http://msdn.microsoft.com/en-us/sync/bb887623.aspx

Microsoft Sync Framework File Synchronization Provider services.

It would mean that the whole backup regime is then controlled by Liquesce or a sub-service of it.