Author Topic: [Question] How to download and save an external file (Macro)  (Read 2590 times)

0 Members and 1 Guest are viewing this topic.

Offline LeoNeeson

  • Insane poster
  • *****
  • Posts: 465
  • Solitario...
    • View Profile
    • twitter.com/LeoNeeson
WARNING: there is a serious security hole in the use of this script (which was solved here), that allows access to any file on the entire hard disk hosting HFS. Be sure to restrict the access of this script only to yourself.


Hi there! I was reading these wiki pages (Template macros & scripting commands), trying to being able to download and save an external file to the current folder (Remote upload), but I can't get it working.

Please check my code (in my example, there is an URL already loaded, of a Tiny Core Linux ISO file). The following code works fine, but it's NOT recommended as it could overload the RAM if you are downloading a big file (I only leave this as reference):

THIS CODE WORKS:
Code: [Select]
<form method='post'>
URL: <input name='url' value="http://distro.ibiblio.org/tinycorelinux/9.x/x86/release/Core-9.0.iso">
<br>Filename: <input name='dest' value="%folder%Core-9.0-GOOD.iso">
<br><input type='submit'>
</form>
{.set|url|{.postvar|url.}.}
{.break|if={.not|{.^url.}.}.}
{.set|dest| {.or|{.filename|{.postvar|dest.}.}|{.filename|{.^url.}.}|downloaded.} .}

{.load|{.^url.}|var=data.}
{.and
| {.length|{.save| ./{.^dest.} |var=data.}.}
| Downloaded {.^dest.} for {.length|var=data.} bytes
.}

The following is the recommended code by rejetto here (chinking the file every 1MB), but it doesn't work as expected (the download has a never ending loop). This code not only doesn't work, but besides that, it also have missing the text "Downloaded Core-9.0.iso for 13256704 bytes" and the event doesn't get logged on HFS's log.

THIS CODE DOESN'T WORK: (why?)
Quote
<form method='post'>
URL: <input name='url' value="http://distro.ibiblio.org/tinycorelinux/9.x/x86/release/Core-9.0.iso">
<br>Filename: <input name='dest' value="%folder%Core-9.0-BAD.iso">
<br><input type='submit'>
</form>
{.set|url|{.postvar|url.}.}
{.break|if={.not|{.^url.}.}.}
{.set|dest| {.or|{.filename|{.postvar|dest.}.}|{.filename|{.^url.}.}|downloaded.} .}

{.set|from|0.}
{.set|chunk|1000000.}
{.save|{.^dest.}|.}
{.while|chunk|{:
   {.load|{.^url.}|var=data|size={.^chunk.}|from={.^from.}.}
   {.if|{.length|var=data.}
      | {:{.append|{.^dest.}|var=data.}
         {.inc|from|{.length|var=data.}.}
        :}
      | {:{.set|chunk|0.}:}.}
:}|timeout=0.}

Could someone point me where is the fault?... :-\
« Last Edit: May 28, 2018, 04:55:00 AM by LeoNeeson »
• HFS ahora también disponible en Español! (Clic aqui) :)
• HFS is now also available in Spanish! (Click here)

Offline danny

  • Experienced poster
  • ***
  • Posts: 42
    • View Profile
Re: [Question] How to download and save an external file (Macro)
« Reply #1 on: May 13, 2018, 01:33:22 PM »
{.load|http://distro.ibiblio.org/tinycorelinux/9.x/x86/release/Core-9.0.iso|/Upload.}
That loads 5596 bytes of the iso into ram.  Needs the {.save


However, from rejetto's posts, I found an example of saving big files 
Except that the chunk size maximum is bigger than the buffer, which may fill and exit.
{.set|#url|http://distro.ibiblio.org/tinycorelinux/9.x/x86/release/Core-9.0.iso.}
{.set|#out|test.}
{.set|#from|0.}
{.set|#chunk|1000000.}
{.save|{.^#out.}|.}
{.while|chunk|{:
   {.load|{.^#url.}|var=#x|size={.^#chunk.}|from={.^#from.}.}
   {.if|{.length|var=#x.}
      | {:{.append|{.^#out.}|var=#x.}
         {.inc|#from|{.length|var=#x.}.}
        :}
      | {:{.set|#chunk|0.}:}.}
:}|timeout=0.}

I guess you could try the chunk size at 4096 if that is bytes or 32768 if bits?  Hard drives write 4k chunks.  Even 5k might work; but, larger would probably exit the script before it could append more.
Might want to replace "test" with the destination?  Folder or filename?
Seems that "Remote Upload" is in alpha status. 

P.S.
Maybe Openmediavault has Remote Upload for your Raspberry Pi? 

« Last Edit: May 20, 2018, 01:11:01 PM by Mars »

Follow members gave a thank to your post:


Offline LeoNeeson

  • Insane poster
  • *****
  • Posts: 465
  • Solitario...
    • View Profile
    • twitter.com/LeoNeeson
Re: [Question] How to download and save an external file (Macro)
« Reply #2 on: May 16, 2018, 08:38:41 AM »
Thank you! I did a deep search on the forum, and I'm ashamed that I did this very same question only FOUR years ago, here... (I've totally forgot that thread. LOL!, I'm getting older). I've also found those rejetto's posts about this, from back 2009: here and here.

I've edited the message above with an updated code.
« Last Edit: May 16, 2018, 08:56:58 AM by LeoNeeson »
• HFS ahora también disponible en Español! (Clic aqui) :)
• HFS is now also available in Spanish! (Click here)

Offline danny

  • Experienced poster
  • ***
  • Posts: 42
    • View Profile
Re: [Question] How to download and save an external file (Macro)
« Reply #3 on: May 16, 2018, 09:17:04 AM »
The example has a variable of a larger size than possible.

So, did you try this:  {.set|chunk|4000.}

. . . maybe not optimized, but at least it is possible to have a variable that size.

Offline LeoNeeson

  • Insane poster
  • *****
  • Posts: 465
  • Solitario...
    • View Profile
    • twitter.com/LeoNeeson
Re: [Question] How to download and save an external file (Macro)
« Reply #4 on: May 16, 2018, 11:48:33 AM »
The example has a variable of a larger size than possible.

So, did you try this:  {.set|chunk|4000.}

. . . maybe not optimized, but at least it is possible to have a variable that size.
I've tried and it has the same effect (it doesn't work properly). I'm curious to know why the first code works (limited by the RAM size) but not the second one... :-\
• HFS ahora también disponible en Español! (Clic aqui) :)
• HFS is now also available in Spanish! (Click here)

Offline LeoNeeson

  • Insane poster
  • *****
  • Posts: 465
  • Solitario...
    • View Profile
    • twitter.com/LeoNeeson
Remote upload: Endless loop archive (bug?)
« Reply #5 on: May 19, 2018, 11:48:07 AM »
@Rejetto/Mars: It would be great if you could check this (just to be sure there is no bug in HFS). After doing several tests, and I've found out that if you use a local file instead of a remote external file (as remote upload url), the first code works fine (loading the file into the RAM), but the second code (based on this post), creates a infinite endless loop archive (which you can't stop unless you kill the hfs.exe process). I guess this is not a normal behavior (perhaps a bug?).

Try yourself, testing the files included the in the attached zip file.
Cheers,
Leo.-

EDIT: I know there is no point doing a 'Remote upload' of a local file. It was just a way of testing the script code without having to download a file (if it works with a local file, it should work with a remote external file, right?)
« Last Edit: May 19, 2018, 12:30:38 PM by LeoNeeson »
• HFS ahora también disponible en Español! (Clic aqui) :)
• HFS is now also available in Spanish! (Click here)

Offline Mars

  • Operator
  • Insane poster
  • *****
  • Posts: 1828
    • View Profile
Re: Remote upload: Endless loop archive (bug?)
« Reply #6 on: May 19, 2018, 09:55:03 PM »
I ended up understanding the reason for the infinite loop, becauseI almost made the same error as rejetto there is little in my previous message on another subject  ::)

I added at the same time a check of the steps in the log of hfs, which can be deleted if upload is correct

the url source is added to the log to check its format

also do an analysis to see the different formats of the url using
<input name='url' value="%encoded-folder%100MB.zip">
<input name='url' value="%url%100MB.zip">


Quote
<form method='post'>
URL: <input name='url' value="folder%100MB.zip">
<br>Filename: <input name='dest' value="%folder%CopyOf100MB-BAD.zip">
<br><input type='submit'>
</form>
{.set|url|{.postvar|url.}.}
{.break|if={.not|{.^url.}.}.}
{.set|dest| {.or|{.filename|{.postvar|dest.}.}|{.filename|{.^url.}.}|downloaded.} .}
{.set|from|0.}
{.set|chunk|1000000.}
{.save|{.^dest.}|.}
{.add to log| loading from '{.^url.}'.}
{.while|chunk|{:
   {.load|{.^url.}|from={.^from.}|var=data|size={.^chunk.}.}
   {.add to log| chunk={.^chunk.} from={.^from.} size= {.length|var=data.}.}
   {.if|{.length|var=data.}
      | {:{.append|{.^dest.}|var=data.}{.inc|from|{.length|var=data.}.} :}
      | {:{.set|chunk|0.}:}.}
:}|timeout=0.}
{.add to log| saved file contain {.^from.} bytes.}

I've tried and it has the same effect (it doesn't work properly). I'm curious to know why the first code works (limited by the RAM size) but not the second one... :-\

{.while|chunk| ... .}   the name of the variable is always TRUE --> infinite loop  :'(

 the loop must be tested on the value of the variable {.while|{.^chunk.}| ... .}

all messages have been fixed to avoid future worries  ;)


big mistake on my part   :o , the rejetto code is correct chunk must be used as a variable inside a while macro
« Last Edit: May 20, 2018, 01:10:17 PM by Mars »
FRENCH MEMBER : Si vous comprenez la langue française,  n'hésitez pas à l'utiliser pour une meilleure aide de ma part

Offline LeoNeeson

  • Insane poster
  • *****
  • Posts: 465
  • Solitario...
    • View Profile
    • twitter.com/LeoNeeson
Remote upload: now it writes an empty file
« Reply #7 on: May 20, 2018, 07:30:41 AM »
I was almost sure your changes would work, until I've tested it (and it doesn't work). The endless loop was solved, but now it writes an empty file (in the log shows "saved file contain 0 bytes"). The same goes using a remote file.

I've tried using any of these url formats (one at time), but with same result.
Code: [Select]
<input name='url' value="%encoded-folder%100MB.zip">
<input name='url' value="%url%100MB.zip">
<input name='url' value="%folder%100MB.zip">

...and I've tried even using:
Code: [Select]
<input name='url' value="100MB.zip">
<input name='url' value="http://127.0.0.1/HFS/100MB.zip">

But the log always show...
Code: [Select]
Requested GET /HFS/
loading from '/HFS/100MB.zip'
saved file contain 0 bytes
Requested POST /HFS/

...and it writes an empty file (CopyOf100MB.zip or whatever filename you use). I've tried changing folder settings permissions (allowing everything), and I double checked everything (even starting with a new configuration file on 2 different machines), but always with the same results.

I also tried using #global variables, running it in the "HFS > Debug > Run script..." window (as described here), with the same result (an empty file).

Rejetto said (in 2009): "this requires some bugfixes available in #248 that i'm going to publish now.". Could be possible he forgot to apply those bugfixes on that release?... ???

...or what else could be?... :-\
• HFS ahora también disponible en Español! (Clic aqui) :)
• HFS is now also available in Spanish! (Click here)

Offline Mars

  • Operator
  • Insane poster
  • *****
  • Posts: 1828
    • View Profile
Re: [Question] How to download and save an external file (Macro)
« Reply #8 on: May 20, 2018, 09:21:47 PM »
there is actually a malfunction at the level of hfs, but I can not do anything without the help of rejetto

here is what it is necessary to use in the file utilLib.pas

Quote
function loadFile(fn:string; from:int64=0; size:int64=-1):ansistring;
var
  f:file;
  bak: byte;
begin
result:='';
IOresult;
if not validFilepath(fn) then exit;
if not isAbsolutePath(fn) then chDir(exePath);
assignFile(f, fn);
bak:=fileMode;
fileMode:=0;
try
  reset(f,1);
  if IOresult <> 0 then exit;
  if (size < 0) or (size > filesize(f)-from) then
    size:=filesize(f)-from;
  setLength(result, size);
  seek(f, from);
  blockRead(f, result[1], size);
  finally
  closeFile(f);
  filemode:=bak;
  end;
end; // loadFile


I took the opportunity to improve the security of the script, which is in green can be deleted later

Quote

<form method='post'>
URL: <input name='url' value="folder%100MB.zip">
<br>Filename: <input name='dest' value="%folder%CopyOf100MB-BAD.zip">
<br><input type='submit'>
</form>
{.set|url|{.postvar|url.}.}

{.set|url|/100MB.zip.}

{.break|if={.not|{.^url.}.}.}
{.set|dest|/download/{.or|{.filename|{.postvar|dest.}.}|{.filename|{.^url.}.}|downloaded.}.}
{.delete|{.^dest.}.}
{.add to log| loading from '{.^url.}' {.filesize|{.^url.}.} bytes .}
{.break|if={.not|{.filesize|{.^url.}.}.}|result=Source file can't be downloaded: server return nul size.}
{.set|from|0.}
{.save|{.^dest.}|.}
{.comment| define CHUNK with the min size, if nul then WHILE is never executed.}
{.set|chunk|{.min|{.filesize|{.^url.}.}|10000000.}.}
{.while|chunk|{:
   {.load|{.^url.}|from={.^from.}|var=data|size={.^chunk.}.}
   {.add to log| chunk={.^chunk.} from={.^from.} size={.length|var=data.}.}
   {.if|{.length|var=data.}
      |   {:
         {.append|{.^dest.}|var=data.}{.inc|from|{.length|var=data.}.}
         {.comment| redefine CHUNK with the min size, if nul then WHILE is stoped.}
         {.set|chunk|{.min|{.^chunk.}|{.sub|{.filesize|{.^url.}.}|{.^from.}/sub.}.}.}
         :}
      | {:{.set|chunk|0.}:}
   /if.}:}
   |timeout=0
   |else={:{.add to log|Error during WHILE.}
   :}/while.}
{.add to log| saved file contain {.^from.} bytes.}

for ease of testing, I created a real folder named 'download' and added the file '100MB.zip', both in the root of the VFS, So with the script the zip is copied in pieces in 'download'

{.set|url|/100MB.zip.}
{.set|dest|/download/{.or|{.filename|{.postvar|dest.}.}|{.filename|{.^url.}.}|downloaded.}.}
FRENCH MEMBER : Si vous comprenez la langue française,  n'hésitez pas à l'utiliser pour une meilleure aide de ma part

Offline rejetto

  • Administrator
  • Insane programmer
  • *
  • Posts: 12856
    • View Profile
Re: [Question] How to download and save an external file (Macro)
« Reply #9 on: May 20, 2018, 10:29:20 PM »
hi guys, i started reading this thread but only half of it as it is a bit long. I'll try to finish tomorrow as i have to go to bed.

Offline LeoNeeson

  • Insane poster
  • *****
  • Posts: 465
  • Solitario...
    • View Profile
    • twitter.com/LeoNeeson
Re: [Question] How to download and save an external file (Macro)
« Reply #10 on: May 22, 2018, 12:18:38 AM »
@Rejetto: Mars's bugfix is a nice opportunity to officially release (along with the source code) the long-awaited 'stable' release v2.3l Build #300, which will also be the first stable release featuring the multi-language support (that you have published as beta test build, five months ago here, along with another bugfixes here).

(there is no hurry, it's only a suggestion.. ;) )
• HFS ahora también disponible en Español! (Clic aqui) :)
• HFS is now also available in Spanish! (Click here)

Offline Mars

  • Operator
  • Insane poster
  • *****
  • Posts: 1828
    • View Profile
Re: [Question] How to download and save an external file (Macro)
« Reply #11 on: May 22, 2018, 02:02:05 PM »
@Rejetto

I ask myself an essential question: 'httpGet' is of type string and 'reply' of type Tstringstream, I imagine that the macro 'load' is originally planned to treat only text files, that does not mean it does not pose a problem whereas with binary files it would be more appropriate to use 'httpGetFile' with 'reply' as Tfilestream

I am also studying the implementation for the macro 'load' of urls based on TFtpClient


« Last Edit: May 22, 2018, 02:07:11 PM by Mars »
FRENCH MEMBER : Si vous comprenez la langue française,  n'hésitez pas à l'utiliser pour une meilleure aide de ma part

Offline rejetto

  • Administrator
  • Insane programmer
  • *
  • Posts: 12856
    • View Profile
Re: [Question] How to download and save an external file (Macro)
« Reply #12 on: May 22, 2018, 09:03:12 PM »
ok, mars' fix is correct. Thanks mars!
i included it for the next build, but if you don't want to wait, you can download this preview
https://drive.google.com/open?id=12GRINYSKe8H-VZtpsbvJkabvdTrB48F3

Offline Mars

  • Operator
  • Insane poster
  • *****
  • Posts: 1828
    • View Profile
Re: [Question] How to download and save an external file (Macro)
« Reply #13 on: May 22, 2018, 09:11:33 PM »
Thank's Boss

it is leoneeson who needs it most for the moment, I suggest you wait before publishing a new version, this is related to my idea about download by ftp links (I'm halfway because I 'advance by student', you would have to look on your side to check that a download is feasible from an url https, I failed to validate it, despite the design of the procedure in scriptlib

if reMatch(fn, '^https?://', 'i!') > 0 then
    try result:=httpGet(fn, from, size)
FRENCH MEMBER : Si vous comprenez la langue française,  n'hésitez pas à l'utiliser pour une meilleure aide de ma part

Offline rejetto

  • Administrator
  • Insane programmer
  • *
  • Posts: 12856
    • View Profile
Re: [Question] How to download and save an external file (Macro)
« Reply #14 on: May 22, 2018, 09:12:26 PM »
I ask myself an essential question: 'httpGet' is of type string and 'reply' of type Tstringstream, I imagine that the macro 'load' is originally planned to treat only text files, that does not mean it does not pose a problem whereas with binary files it would be more appropriate to use 'httpGetFile' with 'reply' as Tfilestream

i'm not sure what it's good for, but i normally use string-s for generic/binary buffer handling in Delphi, as i find it to be very handy. Binaries are handled correctly.
I wonder if the code above is actually working good with URLs, i didn't try, i just used Leo's test.
Even if it works, i wonder what's the time overhead given by making a new request for each megabyte. Maybe it's acceptable. If not you can increase it to 10MB and should be fine.