This project has moved. For the latest updates, please go here.

An alternative to Dokan

Coordinator
Oct 11, 2010 at 8:41 PM

Seems like the Dokan thing and getting it to work on x64 is an art that is not being shared via GoogleCode.. The Issues are open but the users of Doakn that might have solutions or ideas are not sharing.

So,

I'm looking for an alternative that allows a service to attach to a "signed" driver that is actually compiled in 32 and 64 bit..

I have 1 at the moment but do not want to tell what it is whithout having some input to see if others have come across it.

The idea is to replace the underlying Dokan calls closely, or completely, this will depndent on ease of P/Invoke calls and whether I should resort to C/C++ interface DLL.

 

Commetns please.

Oct 11, 2010 at 9:24 PM

I agree tottaly.


Pity since Dokan probably has the potential, but really an Open Source IS an Open Source.

(unlike what Oracle and some people think)

Which one?

This doesn't know anything radically different:

http://sourceforge.net/apps/mediawiki/fuse/index.php?title=OperatingSystems#Windows

This is commercial:

http://www.eldos.com/cbfs/

This is dead:

http://hg.sharesource.org/fuse4win/summary

 

...seems Dokan has the most wide recognition but...

 

 

Coordinator
Oct 11, 2010 at 9:55 PM

I started looking at adapting WinMount.sys (The new WIM thing) but got diverted onto IMDisk. Trouble is IMDisk is for images and not really for existing HDD;s, but when creating a share and then opening with notepad. It just worked. Also it has the potential to load the drive before the OS starts, thus allowing co,plex shares to be reset (Not sure how that would play out for the initial creation tho')

Still looking as the documentation on both of the above is mostly non-existent, so relying on low level code trace without resorting to compliing code.... This may take a while !

Oct 11, 2010 at 10:29 PM

Maybe you can/should work WITH the guy?


(I see he is still active)

 

Coordinator
Oct 11, 2010 at 10:35 PM

I seem to have misled myself on both of those.. They are both targetting a Blob of space, Img files, a large area of Ram or like WHS a different loading space, that is not visible by any other OS component.

So back to the drawing board for a while.

Oct 11, 2010 at 11:08 PM

(thinking aloud)

That guy that made CallBackFS must be nuts... Did you see the prices and the freshman-in-college explanation? Duuuuuh!

 

Coordinator
Oct 12, 2010 at 2:23 PM
Edited Oct 12, 2010 at 2:35 PM

OK.. Disk I/O Drivers are probably no the way to go..

So I will investigate the following to see if it can be adapted to the IFS that's needed.

http://www.pismotechnic.com/pfm/doc/

 

I also toyed with using a WebDav for Win7 onwards, but that just kills all throughput.. http://groups.google.com/group/dokan/browse_thread/thread/7a42a652a2826164?pli=1

Oct 12, 2010 at 9:35 PM

I found the following things which might be useful:

 

http://www.acc.umu.se/~bosse/

http://en.wikipedia.org/wiki/Filesystem_in_Userspace 

http://www.eldos.com/cbfs/ < Shame about the price!

Coordinator
Oct 13, 2010 at 9:11 PM
kayomani wrote:
http://www.acc.umu.se/~bosse/

Tried them, FileDisk-15 is the only one signed, and then it needed improvements to get to #17 for win 7 x64  which is not signed.

 http://en.wikipedia.org/wiki/Filesystem_in_Userspace

Yes, Been there a few times.

the majority of them are Fuse related backend, which mean they are *Nix implementations..

 

 http://www.eldos.com/cbfs/ < Shame about the price!

And the wonderful descriptions given ;-)

Coordinator
Oct 13, 2010 at 9:15 PM
Edited Oct 13, 2010 at 9:17 PM

This is a quote I got from Olof from the IMDisk project

If you are looking for handling requests to the directory structure, then you are not looking for a virtual disk driver, but a virtual filesystem driver or a filesystem redirector driver. That is a completely different task. A virtual disk driver just, as you say, handle I/O requests to a "blob" that can contain anything, the disk driver does not care about structure of that data. The filesystem driver on top of the disk driver handles the filesystem structures and that is where you need to put your driver. A filesystem redirector driver provides a filesystem structure without underlying disk device, for example when mounting a network file share.

Unfortunately, filesystem drivers (with variations such as filesystem filter drivers and filesystem redirector drivers) are the most complex and difficult drivers to develop. I have no experience from that myself,

<...>

As I said, handling filesystem I/O requests is a completely different task from handling disk I/O requests. I would think that you could use "devio" to start a project with, but you you need to change the "read 16 bytes at position 32" style calls to "read 16 bytes from file \abc.txt" style calls. But the heavy work here lies in the kernel mode filesystem driver that you need.

 

It also seems that the Pismo project is also like the IMDisk project

Coordinator
Oct 13, 2010 at 10:17 PM

Here's a thought..

Dokan works on 64 Bit. (Just doesn't like shares !)

1) How about writing a dokan network client that needs to be installed on (well) x64 Clients that use Dokan to create a drive letter that is then re-routed to the Dokan Service ? This is bit like the "Wuala Setup".

Or

2) implement WebDav and then see if the Explorer share thing allows applications (Like media player) to actually talk to the WebDav (IIS) re-routed to Dokan Service mount.

Comments ?

Oct 14, 2010 at 9:01 AM

1) If there are no better options it's a solution. It messes of "just working" with shares though (over VPN for example).

 

2) We have to be sure that the host (a) has certain (and unfortunately rather high end) prerequisites, (b) our setup won't conflict an existing setup (an existing IIS with services etc.)...

 

 

Coordinator
Oct 14, 2010 at 9:26 AM

2) High End? Why? The machine is already running x64 (That's part of the problem) This would not be needed on a 32 bit host, so that's not a problem.

IIS when idle is failry quiet as far as CPU cycles are concerned, And it mostly just forwards the requests through (Although IIS below 6.5 used DCom mostly !!)

Anyway, will have to see what can be done.

Both solution require exposing an interface into the Liquesce service that will perform similiar actions, but that is once I've found which is best / works / etc..

That interface could then be used by the plug-ins to perform their actions without actually ruining in the service space (Much better to apportion the blame as to what went wrong !)

Coordinator
Oct 15, 2010 at 7:00 PM

While investigating the setup of IIS 7, I found thatyou can setup WebDav folders to work with FTP sites.

Step 5 in this page http://maximumpcguides.com/windows-vista/how-to-create-a-webdav-share-in-windows-vista/

So All that is needed now is for the setup of IIS to perform FTP - Hold on, that means I do not have to write anything and Explorer will take care of things for me - cool

Anyone want to give the following a try ?

http://technet.microsoft.com/en-us/library/cc771012%28WS.10%29.aspx

 

Oct 15, 2010 at 9:38 PM
Edited Oct 15, 2010 at 9:40 PM

I will try to do this during the weekend.

I know how to set FTP on IIS (nothing much to it - yet it sucks compared to 3rd party but that is another story).

What happens after FTP is setup? You want webdav set up directly on the disks? On the view? Where?

(ok it's late and I have a headache - you mean on the view)

 

Coordinator
Oct 15, 2010 at 10:33 PM

Cool I do not and due to the recent updates my VPC's have decided to become unstable.. Thanks MS !

Anyway, This looks like a better set of instructions for IIS 7.5

http://learn.iis.net/page.aspx/263/installing-and-configuring-ftp-on-iis-7/

And this sequence looks like the instructions to get a directry hosted as FTP

http://www.windowsnetworking.com/articles_tutorials/Configuring-IIS-Host-FTP-Site-Part1.html

Then follow the http://maximumpcguides.com/windows-vista/how-to-create-a-webdav-share-in-windows-vista/

link to get Explorer to see the ftp as a normal share thus allowing the client to treatt he shared folder via the FTP protocol as a normal mapped drive / directory.

All Sounds good in theory.. Will have to have a go, but will not have musch time this weekend.

Oct 16, 2010 at 12:47 AM
Edited Oct 16, 2010 at 12:48 AM

I have just tried the following:

  • IIS 7.5 ftp via ftp client and via mapped drive

The mapped drive worked but was a bit laggy especially when transferring a file however the problem lies in the transfer rates.  I could pull my test file over the network over a normal windows share at 105mb/s and similar via direct ftp.  However if I redirect the ftp to point at dokan which is in turn points to the same place I get about 6.5mb/s.  I tried increasing the buffers, 8 threads and 256k instead of the default to no benefit.

 

  • IIS 7.5 webdav

Performance was a bit better with this, I achieved about 65mb/s via dokan which is acceptable, again about 105 via standard disk access.  However both windows and the commercial webdav mounter I tried both limit files to 4gb (see http://social.answers.microsoft.com/Forums/en-US/xphardware/thread/d208bba6-920c-4639-bd45-f345f462934f) which makes this unusable.  Webdav also didn't like being pointed at the dokan root dir but a sub folder worked fine o.O.

 

If I copy my test file locally from dokan to a local hard disk, again I get about 110mb/s.  In short unless I set something up wrong I don't think either is an acceptable work around.

Coordinator
Oct 16, 2010 at 8:40 AM

Thanks for testing these.. At least a workaround is better than not working at all.

Anyway.. The 4GB limit should not be a problem if the WebDav server supports resumable modules (Level 2 I think it called), and the client should be able to perform that as well.

This site states that his implementation also gets round the rather poor performance of the default WebDav that is serviced by IIS  http://www.webdavsystem.com/server/documentation/upload

ftp.. How easy was it to set this up ?

I see that you did not state that you needed any third party bit's. Have you used alternative FTP servers ?

Also, Did you have Office installed on the clients, As this changes the network (WebClient) provider to another type that should be capable of Level 2 calls (Like locking  etc.)

Oct 16, 2010 at 12:24 PM

Haven't tried any of the two yet (too busy), maybe later in the day, but I think this method is a long way round the straight line (that normal sharing is). I doubt a 3rd party FTP server will help because if something fits better with a Microsoft technology, it should be Microsoft components.

I think Dokan itself is the limitation. A "lower-level" filesystem driver that is in fact implementing FUSE is the way to go. Dokan is based on very high-level making it come on-line too late to allow for any other "trick" to work behind OS.

Anyway, we'll keep looking.

 

Oct 16, 2010 at 5:39 PM
Edited Oct 16, 2010 at 5:41 PM

What kind of authentication do you guys use?

I tried to use SSL since SBS7 has a self-signed certificate, but doesn't work.

Also I don't really see much usability to webdav as I cannot map a drive letter to it anyway.

 

Oct 17, 2010 at 1:51 PM
Edited Oct 17, 2010 at 1:53 PM

The 4gb limit isn't server side but on the client, I tried using 'webdrive' also but that also had the limit..  I have office 2010 installed.

IIS ftp is very easy to setup and I am not sure what is the cause of performance drop I am seeing from it, I think it might just be the way the client is designed tbh.  

 

@NLS I didn't bother with SSL as it might decrease performance and with the right setup you can map it to a network drive although i'll admit it took me about an hour or fiddling to get it working!

I am inclined to say that these work around's don't look very promising to me and it might be better to look at serving up remote files via the service on the remote machine as I feel performance may end up being better.   I'll have a bit of a play at some point soon and see what kind of performance I can get over my gig network.

Coordinator
Oct 17, 2010 at 5:14 PM
NLS wrote:
I haven't followed the development of Dokan, but IS development active? I mean is there any point waiting for some breakthrough concerning our issues or we are really knocking an empty house's door?

(in that case replacing Dokan altogether maybe is what must happen ANYWAY)

 Does not seem to be any answers going into the Issue list, and no contact with the author to or from his website.

So Still hunting for something that allows an IFS User Mode signed driver.

The other alternative is to recompile / fix / re-issue and find how to create a signed driver fot x64, I'm sure a few other Dokan users would appreciate that as well !

Oct 17, 2010 at 5:36 PM

Exactly (I didn't want to say it - someone had to say this himself ;)).

 

Coordinator
Feb 28, 2011 at 9:14 PM
kayomani wrote:
The 4gb limit isn't server side but on the client, I tried using 'webdrive' also but that also had the limit..  I have office 2010 installed.

From a hunt on the web I found this:
The maximum size you can set in ASP.NET is 2097151Kb = 2Gb. If you need to upload files larger than 2Gb you must implement resumable upload interfaces and upload files with segments. Note that you will need the WebDAV client application that supports resumable upload in this case, such as based on IT Hit WebDAV Client API for .NET.

So, It seems having Office and IIS allows 4GB, but in order to get anything bigger requires a better client !
Any new information be added from anyone ?

Coordinator
Feb 28, 2011 at 9:17 PM
NLS wrote:
Haven't tried any of the two yet (too busy), maybe later in the day, but I think this method is a long way round the straight line (that normal sharing is). I doubt a 3rd party FTP server will help because if something fits better with a Microsoft technology, it should be Microsoft components.

Agreed

NLS wrote:
I think Dokan itself is the limitation. A "lower-level" filesystem driver that is in fact implementing FUSE is the way to go. Dokan is based on very high-level making it come on-line too late to allow for any other "trick" to work behind OS.

Dokan does seem to be getting "In my Way"

NLS wrote:
Anyway, we'll keep looking.

Any news / ideas on this ?

Mar 2, 2011 at 12:03 AM
Edited Mar 2, 2011 at 3:52 AM

Just thinking 'aloud':

For so many reasons, this isn't a good idea (you may need to turn off the native windows sharing service, you need cygwin, security is stuffed, performance could be bad, the list probably goes on and on...)

BUT...

http://smithii.com/samba

I had a thought about running a SMB server on Windows, so did a search and found that link.

(Possible solutions may be, which again may make this a bad idea, run in a virtual machine to get a virtual network card where you can use the ports required and still have real Windows shares, Hmmm or *maybe* [I don't know] you can set up a virtual network card without VM, recompile for native execution to remove cygwin, add a Dokan style interface to it to remove Dokan, or if Dokan works for everything except shares [which may be the case, all my issues are with shares] then just point the Samba server to the Dokan drive where it is only accessed locally by the daemon?)

What I do like about it: this solution has the potential to use no additional/buggy drivers.  In the discard Dokan case, the local machine maps a shared drive to itself and it all just works.

(EDIT: WinPCap may help getting a virtual network card working, if another way can't be found.)

(EDIT2: http://www.blisstonia.com/eolson/notes/smboverssh.php <- this guy's done the work in keeping native windows sharing, from what I understand.)

Coordinator
Mar 2, 2011 at 7:01 AM

Thinking is good.. It makes things easy for others :-)
Anyway, I can see that Samba has it's merits but, you have to perform a lot of jumps to get it visible, and this makes it a very techy solution, And I already have something that works a lot simpler than that via normal "Junctions" - An idea that started the Diskbuncher work (I wonder how far that has got!)

So would not like to go down the Samba route on my own !

Mar 2, 2011 at 7:38 AM

Yeah, I'm using junctions too for the moment, until something else comes along which matches its reliability and speed.  Not being able to merge folders with it though is what's making it much less ideal than it could be.

I just looked at DiskBuncher, interesting! and I've not come across it before, but I didn't see any documentation related to the "merge" concept...  Still only on their first release too, by the look of it.

I may download and patch samba (so I've got source) to try it all out - it should work with Liquesce right away.  I'll be modifying the source to bind directly to the new virtual network adaptor too, rather than doing all that port forwarding stuff, so that will be much simpler.

Will be interesting to see if it all hangs together.  Are you in a position to have a look at that "attribute-change effecting modified date" issue and perhaps release a patch for testing if there is a problem?  That's my only real issue which *stops* me using Liquesce at the moment.

Coordinator
Mar 2, 2011 at 9:19 AM
Jaso wrote: Are you in a position to have a look at that "attribute-change effecting modified date" issue and perhaps release a patch for testing if there is a problem?  That's my only real issue which *stops* me using Liquesce at the moment.

Give me a few days - I need to get back into TFS on codeplex and see if the Merge / Branch thing "really works"

Mar 2, 2011 at 10:08 AM
Edited Mar 2, 2011 at 10:12 AM

That's Ok...  I'm stuck getting packet forwarding working between the hardware network adaptor and the virtual loopback adaptor - so while I could get Samba working on the machine with the Dokan drive - I still can't share it to another PC without turning off native sharing which I don't want to do - as it breaks my working junctions!

I also just tried adding (via the advanced ipv4 dialog, additional IP addresses) a 2nd IP address to the real network, but it's just an alias, so is already using port 445 & 139.

It shouldn't be this hard!

Mar 4, 2011 at 12:44 PM
Edited Mar 4, 2011 at 10:38 PM

After quite a few hours [could have been 16 or more] playing with virtual networks, I gave up.  But at least I've had a small success of my own:

I looked at the source code to the backup software I have and I decided it shouldn't be too hard to make it "see" a merged drive by playing with the directory walker function...  I edited the recursive directory scan - to scan each directory level "through" each drive in the "merge" set - and ignore directories already seen (as they are already scanned, by definition of "through" previously) and if "Bob" isn't my uncle, this thing worked...  Then I modified the write file function (I used "?" as a drive letter to specify a merge set) so it looks for enough space in an existing directory/drive...  changed "?" to that drive and...  It turns out "Bob" is my uncle.  If nowhere has a/ an existing path, and b/ enough space - the file isn't copied and I know to move things around to make space - or create a new directory somewhere else with space...  Finally, it all works!  (I still use Junctions for sharing everything though.)  I did add commented out code to find a space on a drive where a directory could be created, which is what the old backup code did, but I decided I still wanted to keep control over where things went.

(Oh, and I looked at the oem.inf file in the inf directory for my new USB wireless lan - it was installed a week after the BSODs first started, so I'm sure Dokan was causing the BSOD's I was seeing when waking from Hibernate.  Still not seen another one either.)

Coordinator
May 25, 2011 at 9:12 PM

I have banged my head on this for long enough and hence my current direction into FTP or another server interface, with a forced client that does the Dokan mapping.
See the http://amalgam.codeplex.com/ project that I have just started as a possible addition to the FTP service.
This may well be my development for the fullblown Liquesce solution.