rejetto forum

Software => HFS ~ HTTP File Server => HTML & templates => Topic started by: LeoNeeson on May 13, 2018, 07:32:01 AM

Title: [Question] How to download and save an external file (Macro)
Post by: LeoNeeson on May 13, 2018, 07:32:01 AM
» Update: February 10, 2020
Latest version/revision: HERE (https://rejetto.com/forum/index.php?topic=12059.msg1065252#msg1065252).

» Change log:
May 27, 2018 (https://rejetto.com/forum/index.php?topic=12059.msg1062848#msg1062848) > Fixed a serious security hole found on this script (by Mars)
August 02, 2019 (https://rejetto.com/forum/index.php?topic=12059.msg1065252#msg1065252) > Using %encoded-folder% as the destination (by danny).

» To-do list:
- Progress bar (when uploading)
- Show a link (to the uploaded file)



» ATTENTION: The text below, is left as-is (only as a reference), so, it's not recommended for public use, because in those previous/initials stages of this script, a serious security hole was found (that allowed access to any file on the entire hard disk hosting HFS), which was solved HERE (https://rejetto.com/forum/index.php?topic=12059.msg1062848#msg1062848) (thanks to Mars). There is no need to panic, but if you use an old version, just be sure to restrict the access of this script only to yourself. It is recommended to always use the latest version.



Hi there! I was reading these wiki pages (Template macros (http://www.rejetto.com/wiki/index.php?title=HFS:_Template_macros) & scripting commands (http://www.rejetto.com/wiki/index.php?title=HFS:_scripting_commands#File_manipulation)), trying to being able to download and save an external file to the current folder (Remote upload), but I can't get it working.

Please check my code (in my example, there is an URL already loaded, of a Tiny Core Linux (http://distro.ibiblio.org/tinycorelinux/downloads.html) ISO file). The following code works fine, but it's NOT recommended as it could overload the RAM if you are downloading a big file (I only leave this as reference):

THIS CODE WORKS:
Code: [Select]
<form method='post'>
URL: <input name='url' value="http://distro.ibiblio.org/tinycorelinux/9.x/x86/release/Core-9.0.iso">
<br>Filename: <input name='dest' value="%folder%Core-9.0-GOOD.iso">
<br><input type='submit'>
</form>
{.set|url|{.postvar|url.}.}
{.break|if={.not|{.^url.}.}.}
{.set|dest| {.or|{.filename|{.postvar|dest.}.}|{.filename|{.^url.}.}|downloaded.} .}

{.load|{.^url.}|var=data.}
{.and
| {.length|{.save| ./{.^dest.} |var=data.}.}
| Downloaded {.^dest.} for {.length|var=data.} bytes
.}

The following is the recommended code by rejetto here (http://rejetto.com/forum/index.php?topic=7699.0) (chinking the file every 1MB), but it doesn't work as expected (the download has a never ending loop). This code not only doesn't work, but besides that, it also have missing the text "Downloaded Core-9.0.iso for 13256704 bytes" and the event doesn't get logged on HFS's log.

THIS CODE DOESN'T WORK: (why?)
Quote
<form method='post'>
URL: <input name='url' value="http://distro.ibiblio.org/tinycorelinux/9.x/x86/release/Core-9.0.iso">
<br>Filename: <input name='dest' value="%folder%Core-9.0-BAD.iso">
<br><input type='submit'>
</form>
{.set|url|{.postvar|url.}.}
{.break|if={.not|{.^url.}.}.}
{.set|dest| {.or|{.filename|{.postvar|dest.}.}|{.filename|{.^url.}.}|downloaded.} .}

{.set|from|0.}
{.set|chunk|1000000.}
{.save|{.^dest.}|.}
{.while|chunk|{:
   {.load|{.^url.}|var=data|size={.^chunk.}|from={.^from.}.}
   {.if|{.length|var=data.}
      | {:{.append|{.^dest.}|var=data.}
         {.inc|from|{.length|var=data.}.}
        :}
      | {:{.set|chunk|0.}:}.}
:}|timeout=0.}

Could someone point me where is the fault?... :-\
Title: Re: [Question] How to download and save an external file (Macro)
Post by: LeoNeeson on May 16, 2018, 08:38:41 AM
Thank you! I did a deep search on the forum, and I'm ashamed that I did this very same question only FOUR years ago, here (http://rejetto.com/forum/index.php?topic=11430.0)... (I've totally forgot that thread. LOL!, I'm getting older). I've also found those rejetto's posts about this, from back 2009: here (http://rejetto.com/forum/index.php?topic=7699.0) and here (http://rejetto.com/forum/index.php?topic=7668.msg1046929#msg1046929).

I've edited the message above with an updated code.
Title: Re: [Question] How to download and save an external file (Macro)
Post by: LeoNeeson on May 16, 2018, 11:48:33 AM
The example has a variable of a larger size than possible.

So, did you try this:  {.set|chunk|4000.}

. . . maybe not optimized, but at least it is possible to have a variable that size.
I've tried and it has the same effect (it doesn't work properly). I'm curious to know why the first code works (limited by the RAM size) but not the second one... :-\
Title: Re: [Question] How to download and save an external file (Macro)
Post by: LeoNeeson on May 19, 2018, 11:45:48 AM
@Rejetto/Mars: It would be great if you could check this (just to be sure there is no bug in HFS). After doing several tests, and I've found out that if you use a local file instead of a remote external file (as remote upload url), the first code works fine (loading the file into the RAM), but the second code (based on this (http://rejetto.com/forum/index.php?topic=7699.0) post), creates a infinite endless loop archive (which you can't stop unless you kill the hfs.exe process). I guess this is not a normal behavior (perhaps a bug?).

Try yourself, testing the files included the in the attached zip file.
Cheers,
Leo.-
Title: Remote upload: Endless loop archive (bug?)
Post by: LeoNeeson on May 19, 2018, 11:48:07 AM
@Rejetto/Mars: It would be great if you could check this (just to be sure there is no bug in HFS). After doing several tests, and I've found out that if you use a local file instead of a remote external file (as remote upload url), the first code works fine (loading the file into the RAM), but the second code (based on this (http://rejetto.com/forum/index.php?topic=7699.0) post), creates a infinite endless loop archive (which you can't stop unless you kill the hfs.exe process). I guess this is not a normal behavior (perhaps a bug?).

Try yourself, testing the files included the in the attached zip file.
Cheers,
Leo.-

EDIT: I know there is no point doing a 'Remote upload' of a local file. It was just a way of testing the script code without having to download a file (if it works with a local file, it should work with a remote external file, right?)
Title: Re: Remote upload: Endless loop archive (bug?)
Post by: Mars on May 19, 2018, 09:55:03 PM
I ended up understanding the reason for the infinite loop, becauseI almost made the same error as rejetto there is little in my previous message on another subject  ::)

I added at the same time a check of the steps in the log of hfs, which can be deleted if upload is correct

the url source is added to the log to check its format

also do an analysis to see the different formats of the url using
<input name='url' value="%encoded-folder%100MB.zip">
<input name='url' value="%url%100MB.zip">


Quote
<form method='post'>
URL: <input name='url' value="folder%100MB.zip">
<br>Filename: <input name='dest' value="%folder%CopyOf100MB-BAD.zip">
<br><input type='submit'>
</form>
{.set|url|{.postvar|url.}.}
{.break|if={.not|{.^url.}.}.}
{.set|dest| {.or|{.filename|{.postvar|dest.}.}|{.filename|{.^url.}.}|downloaded.} .}
{.set|from|0.}
{.set|chunk|1000000.}
{.save|{.^dest.}|.}
{.add to log| loading from '{.^url.}'.}
{.while|chunk|{:
   {.load|{.^url.}|from={.^from.}|var=data|size={.^chunk.}.}
   {.add to log| chunk={.^chunk.} from={.^from.} size= {.length|var=data.}.}
   {.if|{.length|var=data.}
      | {:{.append|{.^dest.}|var=data.}{.inc|from|{.length|var=data.}.} :}
      | {:{.set|chunk|0.}:}.}
:}|timeout=0.}
{.add to log| saved file contain {.^from.} bytes.}

I've tried and it has the same effect (it doesn't work properly). I'm curious to know why the first code works (limited by the RAM size) but not the second one... :-\

{.while|chunk| ... .}   the name of the variable is always TRUE --> infinite loop  :'(

 the loop must be tested on the value of the variable {.while|{.^chunk.}| ... .}

all messages have been fixed to avoid future worries  ;)


big mistake on my part   :o , the rejetto code is correct chunk must be used as a variable inside a while macro
Title: Remote upload: now it writes an empty file
Post by: LeoNeeson on May 20, 2018, 07:30:41 AM
I was almost sure your changes would work, until I've tested it (and it doesn't work). The endless loop was solved, but now it writes an empty file (in the log shows "saved file contain 0 bytes"). The same goes using a remote file.

I've tried using any of these url formats (one at time), but with same result.
Code: [Select]
<input name='url' value="%encoded-folder%100MB.zip">
<input name='url' value="%url%100MB.zip">
<input name='url' value="%folder%100MB.zip">

...and I've tried even using:
Code: [Select]
<input name='url' value="100MB.zip">
<input name='url' value="http://127.0.0.1/HFS/100MB.zip">

But the log always show...
Code: [Select]
Requested GET /HFS/
loading from '/HFS/100MB.zip'
saved file contain 0 bytes
Requested POST /HFS/

...and it writes an empty file (CopyOf100MB.zip or whatever filename you use). I've tried changing folder settings permissions (allowing everything), and I double checked everything (even starting with a new configuration file on 2 different machines), but always with the same results.

I also tried using #global variables, running it in the "HFS > Debug > Run script..." window (as described here (http://rejetto.com/forum/index.php?topic=7699.0)), with the same result (an empty file).

Rejetto said (http://rejetto.com/forum/index.php?topic=7699.0) (in 2009): "this requires some bugfixes available in #248 that i'm going to publish now.". Could be possible he forgot to apply those bugfixes on that release?... ???

...or what else could be?... :-\
Title: Re: [Question] How to download and save an external file (Macro)
Post by: Mars on May 20, 2018, 09:21:47 PM
there is actually a malfunction at the level of hfs, but I can not do anything without the help of rejetto

here is what it is necessary to use in the file utilLib.pas

Quote
function loadFile(fn:string; from:int64=0; size:int64=-1):ansistring;
var
  f:file;
  bak: byte;
begin
result:='';
IOresult;
if not validFilepath(fn) then exit;
if not isAbsolutePath(fn) then chDir(exePath);
assignFile(f, fn);
bak:=fileMode;
fileMode:=0;
try
  reset(f,1);
  if IOresult <> 0 then exit;
  if (size < 0) or (size > filesize(f)-from) then
    size:=filesize(f)-from;
  setLength(result, size);
  seek(f, from);
  blockRead(f, result[1], size);
  finally
  closeFile(f);
  filemode:=bak;
  end;
end; // loadFile


I took the opportunity to improve the security of the script, which is in green can be deleted later

Quote

<form method='post'>
URL: <input name='url' value="folder%100MB.zip">
<br>Filename: <input name='dest' value="%folder%CopyOf100MB-BAD.zip">
<br><input type='submit'>
</form>
{.set|url|{.postvar|url.}.}

{.set|url|/100MB.zip.}

{.break|if={.not|{.^url.}.}.}
{.set|dest|/download/{.or|{.filename|{.postvar|dest.}.}|{.filename|{.^url.}.}|downloaded.}.}
{.delete|{.^dest.}.}
{.add to log| loading from '{.^url.}' {.filesize|{.^url.}.} bytes .}
{.break|if={.not|{.filesize|{.^url.}.}.}|result=Source file can't be downloaded: server return nul size.}
{.set|from|0.}
{.save|{.^dest.}|.}
{.comment| define CHUNK with the min size, if nul then WHILE is never executed.}
{.set|chunk|{.min|{.filesize|{.^url.}.}|10000000.}.}
{.while|chunk|{:
   {.load|{.^url.}|from={.^from.}|var=data|size={.^chunk.}.}
   {.add to log| chunk={.^chunk.} from={.^from.} size={.length|var=data.}.}
   {.if|{.length|var=data.}
      |   {:
         {.append|{.^dest.}|var=data.}{.inc|from|{.length|var=data.}.}
         {.comment| redefine CHUNK with the min size, if nul then WHILE is stoped.}
         {.set|chunk|{.min|{.^chunk.}|{.sub|{.filesize|{.^url.}.}|{.^from.}/sub.}.}.}
         :}
      | {:{.set|chunk|0.}:}
   /if.}:}
   |timeout=0
   |else={:{.add to log|Error during WHILE.}
   :}/while.}
{.add to log| saved file contain {.^from.} bytes.}

for ease of testing, I created a real folder named 'download' and added the file '100MB.zip', both in the root of the VFS, So with the script the zip is copied in pieces in 'download'

{.set|url|/100MB.zip.}
{.set|dest|/download/{.or|{.filename|{.postvar|dest.}.}|{.filename|{.^url.}.}|downloaded.}.}
Title: Re: [Question] How to download and save an external file (Macro)
Post by: rejetto on May 20, 2018, 10:29:20 PM
hi guys, i started reading this thread but only half of it as it is a bit long. I'll try to finish tomorrow as i have to go to bed.
Title: Re: [Question] How to download and save an external file (Macro)
Post by: LeoNeeson on May 22, 2018, 12:18:38 AM
@Rejetto: Mars's bugfix is a nice opportunity to officially release (along with the source code) the long-awaited 'stable' release v2.3l Build #300, which will also be the first stable release featuring the multi-language support (that you have published as beta test build, five months ago here (http://www.rejetto.com/forum/index.php?/topic,11898.0.html;msg=1062502), along with another bugfixes here (http://www.rejetto.com/forum/index.php?/topic,11954.0.html;msg=1062521)).

(there is no hurry, it's only a suggestion.. ;) )
Title: Re: [Question] How to download and save an external file (Macro)
Post by: Mars on May 22, 2018, 02:02:05 PM
@Rejetto

I ask myself an essential question: 'httpGet' is of type string and 'reply' of type Tstringstream, I imagine that the macro 'load' is originally planned to treat only text files, that does not mean it does not pose a problem whereas with binary files it would be more appropriate to use 'httpGetFile' with 'reply' as Tfilestream

I am also studying the implementation for the macro 'load' of urls based on TFtpClient


Title: Re: [Question] How to download and save an external file (Macro)
Post by: rejetto on May 22, 2018, 09:03:12 PM
ok, mars' fix is correct. Thanks mars!
i included it for the next build, but if you don't want to wait, you can download this preview
https://drive.google.com/open?id=12GRINYSKe8H-VZtpsbvJkabvdTrB48F3
Title: Re: [Question] How to download and save an external file (Macro)
Post by: Mars on May 22, 2018, 09:11:33 PM
Thank's Boss

it is leoneeson who needs it most for the moment, I suggest you wait before publishing a new version, this is related to my idea about download by ftp links (I'm halfway because I 'advance by student', you would have to look on your side to check that a download is feasible from an url https, I failed to validate it, despite the design of the procedure in scriptlib

if reMatch(fn, '^https?://', 'i!') > 0 then
    try result:=httpGet(fn, from, size)
Title: Re: [Question] How to download and save an external file (Macro)
Post by: rejetto on May 22, 2018, 09:12:26 PM
I ask myself an essential question: 'httpGet' is of type string and 'reply' of type Tstringstream, I imagine that the macro 'load' is originally planned to treat only text files, that does not mean it does not pose a problem whereas with binary files it would be more appropriate to use 'httpGetFile' with 'reply' as Tfilestream

i'm not sure what it's good for, but i normally use string-s for generic/binary buffer handling in Delphi, as i find it to be very handy. Binaries are handled correctly.
I wonder if the code above is actually working good with URLs, i didn't try, i just used Leo's test.
Even if it works, i wonder what's the time overhead given by making a new request for each megabyte. Maybe it's acceptable. If not you can increase it to 10MB and should be fine.
Title: Re: [Question] How to download and save an external file (Macro)
Post by: Mars on May 22, 2018, 09:34:18 PM
Hi I will have a little hesitation to use a string to store binary values but it is true that delphi behaves differently from C which uses a 0 end of chain, but hey as it works as it is, I do not there is not much to say


in another vein, the https hangs on the http protocol, you would have to look into the issue so that the macro 'load' is 100% functional as for a simple http link


on my side I am worried about the problem considering the links ftp because it is more within range of my competences

at this moment, we will link our results to release a new build ;)
Title: Re: [Question] How to download and save an external file (Macro)
Post by: LeoNeeson on May 23, 2018, 11:04:48 AM
@Rejetto: There is no hurry for me, take all the time you need before publishing a new stable version (I also confirm that the 'remote upload' feature works perfect now on this (https://rejetto.com/forum/index.php?topic=12059.msg1062781#msg1062781) new build). :)

@Mars: Your code posted here works fine with local files (and with small changes on the code, also with external files).

» EDIT: I'm currently doing several tests...
(I'll post the results on the next few days). ;)
Title: [Testing] Remote upload (Macro)
Post by: LeoNeeson on May 27, 2018, 04:06:25 AM
@Mars: After doing several tests, everything seems to be working OK, but somehow, I've found a small rare bug/issue: if connection gets closed (or dropped) by the remote server (or by bad connection quality) while downloading a chunk, then the result was getting a corrupted file (I've ended up with a corrupted 15,7MB file, instead of being 12,6MB). It only happened once, and then I couldn't recreate the error. I don't know if we can get a workaround or modify the code to avoid bugs like this happen again, but I will leave you the logs of my tests:

GOOD download (normal log):
Quote
10:25:05 127.0.0.1:1281 Requested GET /
10:25:13 127.0.0.1:1281 loading from 'http://distro.ibiblio.org/tinycorelinux/9.x/x86/release/Core-9.0.iso' 13256704 bytes
10:25:34 127.0.0.1:1281 chunk=10000000 from=0 size=10000000
10:26:08 127.0.0.1:1281 chunk=3256704 from=10000000 size=3256704
10:26:09 127.0.0.1:1281 saved file contain 13256704 bytes
10:26:09 127.0.0.1:1281 Requested POST /

BAD download (issue log):
(Marked in bold = are the abnormal execution of code)
(Marked in red = comments added by me to describe the situation)
Quote
10:36:05 127.0.0.1:1257 Requested GET /
10:36:17 127.0.0.1:1257 loading from 'http://distro.ibiblio.org/tinycorelinux/9.x/x86/release/Core-9.0.iso' 13256704 bytes
10:36:53 127.0.0.1:1257 chunk=10000000 from=0 size=10000000 > Connection dropped!
10:37:16 127.0.0.1:1269 loading from 'http://distro.ibiblio.org/tinycorelinux/9.x/x86/release/Core-9.0.iso' 13256704 bytes
10:37:38 127.0.0.1:1269 chunk=10000000 from=0 size=10000000
10:37:47 127.0.0.1:1269 chunk=3256704 from=10000000 size=3256704
10:37:48 127.0.0.1:1269 saved file contain 13256704 bytes
10:37:48 127.0.0.1:1269 Requested POST /
10:37:48 127.0.0.1:1257 chunk=3256704 from=10000000 size=3256704 > WTF!?
10:37:49 127.0.0.1:1257 saved file contain 13256704 bytes > Not true, file was bigger, (exactly 16513408 bytes)!
10:37:49 127.0.0.1:1257 Requested POST /

Code used (RemoteUpload3.tpl):
Quote
<form method='post'>
URL: <input name='url' value="http://distro.ibiblio.org/tinycorelinux/9.x/x86/release/Core-9.0.iso">
<br>Filename: <input name='dest' value="%folder%Core-9.0.iso">
<br><input type='submit'>
</form>
{.set|url|{.postvar|url.}.}
{.break|if={.not|{.^url.}.}.}
{.set|dest|{.or|{.filename|{.postvar|dest.}.}|{.filename|{.^url.}.}|downloaded.}.}
{.delete|{.^dest.}.}
{.add to log| loading from '{.^url.}' {.filesize|{.^url.}.} bytes .}
{.break|if={.not|{.filesize|{.^url.}.}.}|result=Source file can't be downloaded: server return nul size.}
{.set|from|0.}
{.save|{.^dest.}|.}
{.comment| define CHUNK with the min size, if nul then WHILE is never executed.}
{.set|chunk|{.min|{.filesize|{.^url.}.}|10000000.}.}
{.while|chunk|{:
   {.load|{.^url.}|from={.^from.}|var=data|size={.^chunk.}.}
   {.add to log| chunk={.^chunk.} from={.^from.} size={.length|var=data.}.}
   {.if|{.length|var=data.}
      |   {:
         {.append|{.^dest.}|var=data.}{.inc|from|{.length|var=data.}.}
         {.comment| redefine CHUNK with the min size, if nul then WHILE is stoped.}
         {.set|chunk|{.min|{.^chunk.}|{.sub|{.filesize|{.^url.}.}|{.^from.}/sub.}.}.}
         :}
      | {:{.set|chunk|0.}:}
   /if.}:}
   |timeout=0
   |else={:{.add to log|Error during WHILE.}
   :}/while.}
{.add to log| saved file contain {.^from.} bytes.}

» Ideas for solutions/enhancements: I don't know if there is a way to 'detect' if a connection was closed (or dropped) while downloading a chunk, but I think a simple verification of file size after finish the download, could be of help. Also, perhaps a 'progress' bar could be of help, meanwhile the file is downloading (if the download on the server takes too long).

I was trying to add a 'progress' bar, like HFS has by default for uploads (useful if the download on the server takes too long), but using %progress-files%, %total% and %speed-kb% doesn't seem to have effect.
Title: Re: [Question] How to download and save an external file (Macro)
Post by: Mars on May 27, 2018, 02:10:04 PM
the solution is to memorize the size of the initial file in a temporary variable, and when the complete download is complete, to check that it matches the saved file. If this is not the case, we force the physical deletion of the loaded file, except if source file is on a FTP server, in this case interrupted download can be restarted later

at the beginning of script, we test used protocol, if FTP  is used then we start download from the last know position, else destination is simply deleted

The FTP protocol is functional, but the download can only be done in full or be taken at the position of its interruption. Due to my limited knowledge about using sockets, I could not set up partial loads like with the http

* bug corrected: the file was tranferred to the Recycle Bin instead of being physically destroyed, which could have led to disk saturation

there is a serious security hole in the use of this script, this one in the state allows access to the entire hard disk hosting HFS

the macro load is intended to work with windows-type paths as c:\windows\explorer.exe

so if it is offered the opportunity to a user of your hfs to use the form, it will be able to recover any file of your hard disk by transferring it in hfs

moreover the url can be encrypted using % followed by a hex code representing a character of the ascii table as c%3A%5Fwindows%5Fexplorer.exe

it is therefore absolutely necessary to detect and prevent any continuation of the script if the url contains the \

Quote
<form method='post'>
URL: <input name='url' value="http://distro.ibiblio.org/tinycorelinux/9.x/x86/release/Core-9.0.iso">
<br>Filename: <input name='dest' value="%folder%Core-9.0.iso">
<br><input type='submit'>
</form>

{.set|url|{.postvar|url.}.}

{.break|if={.not|{.^url.}.}.}

{.break|if={.count substring|\|{.decodeuri|{.^url.}.}.}|result=Direct acces on hard disk not allowed.}

{.set|dest|{.or|{.filename|{.postvar|dest.}.}|{.filename|{.^url.}.}|downloaded.}.}

{.set|filesize|{.filesize|{.^url.}.}.}

{.add to log|Start loading from '{.^url.}' {.^filesize.} bytes.}

{.if|{.match|^ftp?://|{.^url.}.}| {:{.set|from|{.^filesize.}.}:} | {:{.delete|{.^dest.}|bin=0.}{.set|from|0.}:}.}

{.break|if={.not|{.^filesize.}.}{.=|{.^filesize.}|{.filesize|{.^dest.}.}.}|result=Source file can't be downloaded: server return nul size or destination match source size.}

{.save|{.^dest.}|.}
{.^from.}
{.comment| define CHUNK with the min size, if nul then WHILE is never executed.}
{.set|chunk|{.min|{.^filesize.}|10000000.}.}
{.^chunk.}
{.while|chunk|{:
 {.add to log|loading {.^chunk.} bytes.}
   {.load|{.^url.}|from={.^from.}|var=data|size={.^chunk.}.}
   {.length|var=data.}
   {.add to log|Download: from={.^from.} request={.^chunk.}  loaded= {.length|var=data.}.}
   {.if|{.length|var=data.}
      |   {:
         {.add to log|TPL sauvegarde en cours.}
         {.append|{.^dest.}|var=data.}{.inc|from|{.length|var=data.}.}
         {.comment| redefine CHUNK with the min size, if nul then WHILE is stoped.}
         {.set|chunk|{.min|{.^chunk.}|{.sub|{.^filesize.}|{.^from.}/sub.}.}.}
         :}
      |   {:
         {.set|chunk|0.}
         {.if|{.and| {.not|{.match|^ftp?://|{.^url.}.}/not.} | {.not|{.^filesize.} = {.filesize|{.^dest.}.}/not.}   /and.}
            |{:{.delete|{.^dest.}|bin=0.}:}.}
         {.add to log|End of download.}
         :}
   /if.}:}
   |timeout=0
   |else={:{.add to log|Error during WHILE.}:}
/while.}
{.add to log|Saved file contain {.^from.} bytes.}

I am studying the case taking into account the eventuality where the source file would be also the file of destination and would be destroyed irremediably during the transfer and its size becomes zero
Title: Re: [Question] How to download and save an external file (Macro)
Post by: LeoNeeson on May 28, 2018, 04:58:44 AM
there is a serious security hole in the use of this script, this one in the state allows access to the entire hard disk hosting HFS
Thanks for reporting the security issue. I've updated the first post to warn users about this.

I am studying the case taking into account the eventuality where the source file would be also the file of destination and would be destroyed irremediably during the transfer and its size becomes zero
Do you mean: if the same destination file exist, then delete the destination file? I think it's better to automatically rename the destination file, not to delete/overwrite it. So, if destination file ("filename.zzz") exists, then save new file destination as "filename_001.zzz". If "filename_001.zzz" exists, then save as "filename_002.zzz", or "filename_003.zzz", "filename_004.zzz", and so on...
Title: Re: [Question] How to download and save an external file (Macro)
Post by: Mars on May 30, 2018, 07:13:27 PM
Do you mean: if the same destination file exist, then delete the destination file? I think it's better to automatically rename the destination file, not to delete/overwrite it. So, if destination file ("filename.zzz") exists, then save new file destination as "filename_001.zzz". If "filename_001.zzz" exists, then save as "filename_002.zzz", or "filename_003.zzz", "filename_004.zzz", and so on...

I advance slowly but I found a solution for that, what posed me the most problem was to choose to integrate it in the macro save or to create a new one, the second chose turned out much wiser, and the name of the macro will be left to the complacency of rejetto if a better name is dear to him.

I made the temporary choice as we could find it in the wiki:  ;D

 {.first free|A.} 
 You can specify A as a file name, or as a URL. It will load and expand to it. The file or URL you specify must be accessible from the server machine. A can be C:\windows\win.ini or also absolute /another_file_in_VFS/the_file_i_want.txt or also relative from current folder hello/the_file_i_want.txt
 in case the path is not provided, the macro generates an error of unsatisfactory parameters
 Optional parameter limit  to specify a max seaching index. if ommited default value is 10.
 Example  {.first free|c:\temp\hfs.tmp|limit=5.}

the principle is to provide  to the macro a path in the form of url or absolute, an increment is then used to find a new free file name, without exceeding the number of 10, unless we specify a parameter named 'max'

the function will return an empty string in case of invalid path, or of limit exceded

Code: [Select]
  function firstfree():string;
    var
      dest, ext: string;
      i, limit : integer;
    begin
      result:='';
      try
      limit:=parI('limit', 10);
      pars.Delete(pars.IndexOfName('limit')); // to be sure there is a valid parameter for the path in pars[1]
      p:=trim(pars[1]);
      // we make sure that the file can be put on a physical support
      if not DirectoryExists(uri2diskMaybeFolder(p)) then exit;
      ext:=extractFileExt(p);
      dest:= copy(p,1,length(p)-length(ext));
      i:=0;
      while fileExists(uri2diskMaybe(p)) do
        begin
        inc(i);
        // test limit here to avoid prolonging the loop unnecessarily
        if (i > limit) then exit; // function must return an empty string
        p:=format('%s (%d)%s', [dest, i, ext]);
        end;
      except exit; end;
      result:=p;
    end;
Title: Re: [Question] How to download and save an external file (Macro)
Post by: LeoNeeson on February 09, 2020, 01:02:22 AM
Howabout this way, with %encoded-folder% as the destination?  Better security.
Code: [Select]
<center><form method='post'>Paste a URL: <input name='url' value=""><input type='hidden' name='dest' value=""><br><input type='submit' value='Transfer'></form>
{.set|url|{.postvar|url.}.}
{.break|if={.count substring|\|{.decodeuri|{.^url.}.}.}|result=Direct access on hard disk not allowed.}
{.set|dest|%encoded-folder%{.or|{.filename|{.postvar|dest.}.}|{.filename|{.^url.}.}.}.}
{.set|filesize|{.filesize|{.^url.}.}.}
{.if|{.match|^ftp?://|{.^url.}.}| {:{.set|from|{.^filesize.}.}:} | {:{.delete|{.^dest.}|bin=0.}{.set|from|0.}:}.}
{.break|if={.not|{.^filesize.}.}{.=|{.^filesize.}|{.filesize|{.^dest.}.}.}.}
{.save|{.^dest.}|.}
{.^from.}
{.comment| define CHUNK with the min size, if nul then WHILE is never executed.}
{.set|chunk|{.min|{.^filesize.}|10000000.}.}
{.^chunk.}
{.while|chunk|{:
 {.add to log|loading {.^chunk.} bytes.}
   {.load|{.^url.}|from={.^from.}|var=data|size={.^chunk.}.}
   {.length|var=data.}
   {.add to log|Download: from={.^from.} request={.^chunk.}  loaded= {.length|var=data.}.}
   {.if|{.length|var=data.}
      |   {:
         {.add to log|saving.}
         {.append|{.^dest.}|var=data.}{.inc|from|{.length|var=data.}.}
         {.comment| redefine CHUNK with the min size, if nul then WHILE is stoped.}
         {.set|chunk|{.min|{.^chunk.}|{.sub|{.^filesize.}|{.^from.}/sub.}.}.}
         :}
      |   {:
         {.set|chunk|0.}
         {.if|{.and| {.not|{.match|^ftp?://|{.^url.}.}/not.} | {.not|{.^filesize.} = {.filesize|{.^dest.}.}/not.}   /and.}
            |{:{.delete|{.^dest.}|bin=0.}:}.}
         {.add to log|End of download.}:}/if.}:}|timeout=0.}</center>
Thanks! I will check it out :)

if reMatch(fn, '^https?://', 'i!') > 0 then
    try result:=httpGet(fn, from, size)
Where does that part go?
It's already part of the source code (in the file 'scriptLib.pas'). I think it was pointed (cited) by Mars to ask to Rejetto why HTTPS doesn't work. I hope Rejetto could bring us some information about this...
Title: Re: [Question] How to download and save an external file (Macro)
Post by: rejetto on February 09, 2020, 12:07:49 PM
i'm sorry but the original post from danny was deleted and i don't understand what information you are asking to me
Title: Re: [Question] How to download and save an external file (Macro)
Post by: LeoNeeson on February 09, 2020, 12:22:06 PM
i'm sorry but the original post from danny was deleted and i don't understand what information you are asking to me
Danny said this: "The transload feature could be ready for prime-time, if it were possible to load files from https:// url source.". But it seems HFS can't handle any kind of HTTPS, right?...
Title: Re: [Question] How to download and save an external file (Macro)
Post by: rejetto on February 09, 2020, 11:03:13 PM
yes, for what i've seen, HFS is not able to load https URLs at the moment