Please, first refer to the
tasks topic.
We need a Node.js module to get a sort of stream for folder archives.
This is a possible solution to the
multiple files download problem.
The stream should take as input a bunch of files.
The output would be a ZIP compatible archive containing those files.
HFS 2.2 and 2.3 have a solution like this, but TAR compatible instead of ZIP. It's a possible starting point if you want to get an idea of an algorithmic solution.
The reasons why TAR was chosen are
1) it's easier to implement
2) it permits random access in a reasonable time. In other words: the stream.seek() method.
Point 2 is very important, because people may stop and resume a very big download.
Any compression won't allow this, because the output size is not predictable without compressing the whole data.
ZIP format allows zero compression. I guess (and hope) this will permit us to implement a very quick stream.seek().
Weeks ago i've found a zip stream module, but it was not implementing any kind of seek(), thus not allowing resume.
In the worst case we can drop the resume support and use something like that.
Anyway, home internet connections are usually slow and not reliable. It's not good when your 1gig download stops at 90% and you have to restart again.
We may even give both options, tar and zip. One resumable but not the other.
The most important point is that the ZIP must be "virtual". No intermediary file must be created. The load on the server must be very close as the load of transfering the single files.
The expected extra load of having the archive is near 0% for both CPU drive.