This project has moved and is read-only. For the latest updates, please go here.

Folder distribution - seperate thread

Oct 5, 2010 at 5:33 PM

 

The question is ....

How did you plan to handle the redistribution of FIles Folders ?

 

The simples solution is ... the sight of the file which is currently written ... if out of space then seach the next writable SourcePath and try it there ......

If i leave the problems how to handle this (Buffer, unknown End-Filesize) behind, there is another problem ....

Mostly we store Media Files, which consist of more than one FIle , Songs of an Album+Cover+Lyric, The movie and a lot of extrafiles if you use Software like XBMC, ....

So its a bad idea to seperate them from the Moviefile ... it work but its not nice ....

 

So we say OK .... then another Rule ... if we move one file of a folder ... then the whole folder has to move .....

That works fine .... until the point ... you hav a subfolder .... as Example with Coverart or whatever and the space run out during writing the Cover ....

Now ... following the Rule we made ... we move the whole Folder .... resulting in seperating the CoverSubFolder from the Movie ..... not what we want ...

 

A better way is to define a depth from which wee keep folders together -> KeepTogeterDeept

0 ... means at Root level ... this is the simplest and most common kind ... most moviecollections work with this scheme ....

If during writing of a file, the file cant be written, use the normal scheme to find another place to store the file. (Creating folder structure, writing old file to the new location, continua wrting file ....)

But insert the Path in a reallocation List . (i dont think that moving a lot of files around in the same thread as writing a file (that which is the reason of out of space) is a good idea)

Somwehere in the near future at a magical moment the Reallocation Thread start, build a list of files that have to be moved and start his work of moving files.

Why build a list ? Cous there is the possibility that he cant move one ore more files couse they are used,

But thats ok ... for the user ... thanks joining there is no difference ... the Reallocation Thread can go sleep .... and he can try it later if the file is not in use .....

 

Another question is the UserApp write access to a file .... during reallocation ....

Reading is no problem ... we can copy it to the new destination and delete the old one after nobody has a lock on it ....

 

For writing there are 2 possiblitys ... the simplest ... the file (source) is during reallocation read only .... easy to do but not so nice ....

Or we can buffer the write access to temp file .....

Now there are even 2 solutions...  using the temp file until the handle is closed, and then do the changes to the new destination ... more simple to implement ...

Or after moving is done we do the changes on the new destination and do all future writes to the new file until the handle is closed .... a bit more complex but the space on the source can be freed by deleting the old file ...

 

Ok if we look a little bit closer the thing about the tampfile is more complex as at first sight .... UserApp can make random ReadWrite so we cant make simle a tempfile in the size of the sourcefile and write in it.

The tempfile has to be like a list of data and offsets, every writeAcces is loged by Data and Offset, every Read access is done to the Sourcefile.

There is only one little disadvatage ... after closing the filehandle of a example 10GB File which is heavily modified, there is a delay until the changes are written to the new Destination.

So if UserApp do something like heavily changing a file, closing it and open it to verify modifications, the merging of tempfile and destination file arent completed yet, so we can only open the source file (that whitout the changes).

But we can do better ........

We can read from the tempfile for the changed offsets ..... so ... if UserApp want to read the file BEFORE tempfile is merged to destination, we use the data in tempfile.

 

So thats enouge to read .... what do you think .... is this nonsens ... ?