rejetto forum

Software => HFS ~ HTTP File Server => Topic started by: raybob on August 17, 2011, 01:43:21 AM

Title: Macros to serve HTTP status codes
Post by: raybob on August 17, 2011, 01:43:21 AM
Can you create a macro command to directly serve different HTTP codes to the user, for example something like {.if| I want to give this user a deny-403 code |{:{.serve|403.}:}.}

Reason I ask is because in certain cases I want to direct a user to the deny page but using {.redirect|/~deny.} will confuse search engines and browsers into thinking that the page exists and is accessible but got temporarily moved....  This would also be useful for serving a 307 code since using {.redirect.} only constitutes a temporary redirect, not a permanent one.

Thanks!
Title: Re: Macros to serve certain HTTP codes
Post by: FRENCH CAN CAN on August 18, 2011, 11:38:07 AM
At the head section of your template:

<meta name="robots" content="noindex,nofollow,noarchive,nosnippet">

you should get the desired effect.
Title: Re: Macros to serve certain HTTP codes
Post by: raybob on August 19, 2011, 08:51:34 PM
No, I'm looking for a way to actually use a macro to serve HTTP status codes of my choosing in the response headers; for users not just search engines, and for a variety of applications.
Title: Re: Macros to serve HTTP status codes
Post by: raybob on September 02, 2011, 04:33:44 PM
Like {.if| I want to hide this page |{:{.http status|404.}:}.} or {.if| I want to deny this user |{:{.http status|403.}:}.}  or  {.if| I permanently moved this file |{:{.http status|301.}{.add header|Location: http://website.tld/.}:}.}

Something to that effect would be very useful.

This can very easily be done in PHP...  <?php header("HTTP/1.1 404 Not Found"); ?>
Title: Re: Macros to serve HTTP status codes
Post by: rejetto on September 17, 2011, 05:48:12 PM
yes i mean to add something like that.
that was already in the to-do list, in this line

+ macros missing to cache a folder: {.reply|content=|var=|code=|filename=|mime=.}


i meant it as a way to provide cached content, but it would work for your purpose too.
keep an eye on future builds for new commands.
unfortunately i didn't solve my problems stopping me from developing hfs yet. :(
Title: Re: Macros to serve HTTP status codes
Post by: raybob on September 18, 2011, 06:02:19 PM
Rather than creating a new topic, I'll just ask here....

Have you ever thought about creating a command that functions similar to javascript setTimeout() or PHP sleep()
Title: Re: Macros to serve HTTP status codes
Post by: rejetto on October 04, 2011, 09:40:13 AM
mmm, nope.
But sleep is out of question because HFS is not multi-threaded.
Would you need a setTimeout ?
Title: Re: Macros to serve HTTP status codes
Post by: raybob on October 07, 2011, 03:24:55 PM
But sleep is out of question because HFS is not multi-threaded.

Is that something you plan on doing in the future is making HFS multithreaded?  Perhaps one thread per connection would be effective :)

HFS frequently maxes out my CPU on its allotted core (and this is a Core i7 here) and then the GUI freezes up....

Generally if I have >8 active downloads HFS lags to the point that it's basically knocked offline because I can't even access it from within my LAN in under a minute.  That's on a VM server with an i7 and 3GB RAM.

EDIT:

Also I ask about setTimeout or something like that because capturing the output of a batch file only works if the batch takes under a certain amount of time to complete, like HFS doesn't wait for it.
Title: Re: Macros to serve HTTP status codes
Post by: rejetto on October 23, 2011, 06:32:03 PM
Is that something you plan on doing in the future is making HFS multithreaded?

that won't come until a full rewrite of HFS.
would i have the time, i would go for a multiplatform version based on Python.

Quote
Generally if I have >8 active downloads HFS lags to the point that it's basically knocked offline because I can't even access it from within my LAN in under a minute.  That's on a VM server with an i7 and 3GB RAM.

wow. What's the total upload speed at that moment?

Quote
Also I ask about setTimeout or something like that because capturing the output of a batch file only works if the batch takes under a certain amount of time to complete, like HFS doesn't wait for it.

but you don't know how long will the batch take, right?
you may wait too long, or arrive too early.
it's better if the "fetcher" of the result is in a timed event, isn't it?
Title: Re: Macros to serve HTTP status codes
Post by: rejetto on October 23, 2011, 06:33:32 PM
Bandalit moved to http://www.rejetto.com/forum/index.php/topic,9941.0.html
Title: Re: Macros to serve HTTP status codes
Post by: raybob on October 30, 2011, 05:35:25 PM
wow. What's the total upload speed at that moment?

My internet doesn't go above 120KB/s (which is slightly problematic).  HFS doesn't lag based on total upload speed but rather number of concurrent connections.  I can easily get over 8MB/s with no lag on my local LAN if no one else is connected.

That being said there is a total of ~200 macros that get executed under [request] according to the macro-log and 542 items in my VFS   ;D

Do you think the fact that it's under a VM has much effect?  Every other application seems to run the same speed...

BTW what I said before wasn't totally accurate.... here's what I measured: >4 downloads makes the GUI slower, >10 downloads makes the GUI unusable by taking >10 seconds to show anything, >12 downloads makes a page take longer than 10 seconds to load and >15-20 downloads will make Firefox time out when loading a page.  HFS has only been knocked offline once for me (completely unresponsive in GUI and over HTTP) and it had around 30 (concurrent) downloads.

I also tried disabling browsing compression but it didn't seem to have an effect.
Title: Re: Macros to serve HTTP status codes
Post by: rejetto on October 31, 2011, 06:39:18 PM
your [request] script is executed only at the beginning of the connection.
in case the gui is slow also when no connections are established, it must be something else.
a quick test could be done by disabling your scripts for a while: you can just rename the hfs.events file
Title: Re: Macros to serve HTTP status codes
Post by: raybob on October 31, 2011, 09:00:47 PM
I'll try that next time I get heavy traffic.
Title: Re: Macros to serve HTTP status codes
Post by: raybob on November 11, 2011, 05:39:50 AM
Disabling events actually didn't seem to have an effect.

Today during some heavy traffic I tried temporarily disabling options and I managed to hit a record of 8 HTTP requests/second (something like 1000 HTTP packets per second) with literally zero lag.  It was pretty cool  :D  Restoring the options pretty much immediately locked up the interface.

The following things I tried all had no effect:

Disabling Apache Log
Disabling Compressed Browsing
Disabling preventing download accelerators
clearing ban list
turning off high speed handling
Title: Re: Macros to serve HTTP status codes
Post by: rejetto on November 13, 2011, 10:57:30 PM
i didn't understand, what's the result you just told, was it something different? and how did you get it?
Title: Re: Macros to serve HTTP status codes
Post by: raybob on November 14, 2011, 03:26:23 AM
No, I'm saying that going with my options, nothing I change seems to have an effect, yet clicking temporarily reset options fixes it immediately.
Title: Re: Macros to serve HTTP status codes
Post by: rejetto on November 14, 2011, 10:09:26 AM
ah ok.
So, it is an option OR the template.
try to backup your template then "html template > restore default"
Title: Re: Macros to serve HTTP status codes
Post by: raybob on November 15, 2011, 03:47:11 AM
Next time I'm getting traffic I'll try replacing the template.  That very well could be the problem since it's over 300KB in size ;)

Though how could the template be the problem if HFS also lags downloading files, not just pages?

I'll let you know the results.

---------------------------------------------------------

OK, without traffic, I tested how long it took HFS to generate a page with a bunch of {.exec.} commands in it..... usually it would take ~770 milliseconds but as high as 2 seconds to generate it, with my standard template.  Changing the template to a nearly blank one had no effect.

Next I'll test with heavy concurrent traffic.

---------------------------------------------------------

Two more things.... Do you think it has an effect that my VFS has 634 items in it?  I feel like this is the most likely cause.

Anyway, here's my customized options just as another thought:

Code: [Select]
# 2.3 beta (build 279)
window-max=yes
# default: no

easy=no
# default: yes

port=80
# default:

files-box-ratio=0.1609375
# default: 0

log-max-lines=0
# default: 2000

log-file-name=C:\Documents and Settings\admin\Desktop\Logs\HFS Main Log.log
# default:

log-font-name=Arial
# default:

log-font-size=8
# default: 0

log-date=yes
# default: no

log-only-served=no
# default: yes

log-server-start=yes
# default: no

log-server-stop=yes
# default: no

log-replies=yes
# default: no

log-file-tabbed=yes
# default: no

log-apache-format=%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"
# default:

tpl-file=C:\Documents and Settings\admin\Desktop\Global Template.tpl
# default:

confirm-exit=yes
# default: no

keep-bak-updating=no
# default: yes

custom-ip=www.filesplat.com
# default:

search-better-ip=no
# default: yes

start-minimized=yes
# default: no

auto-save-vfs=yes
# default: no

use-comment-as-realm=no
# default: yes

getright-template=no
# default: yes

dont-include-port-in-url=yes
# default: no

delete-partial-uploads=yes
# default: no

rename-partial-uploads=%name%.partialupload
# default:

tray-icon-for-each-download=no
# default: yes

enable-fingerprints=no
# default: yes

send-hfs-identifier=no
# default: yes

list-protected-items=yes
# default: no

browse-localhost=no
# default: yes

default-sorting=ext
# default: name

auto-save-vfs-every=60
# default: 0

last-update-check=40575.7608544907
# default: 0

tray-message=Uptime: %uptime%\nDownloads: %downloads%\nTotal Hits: %hits%
# default: %ip%\nUptime: %uptime%\nDownloads: %downloads%

max-ips=100
# default: 0

max-connections=400
# default: 0

max-contemporary-dls=25
# default: 0

max-contemporary-dls-ip=5
# default: 0

flash-on=
# default: download

graph-rate=1
# default: 10

no-reply-ban=yes
# default: no

(long ban list removed for privacy)

add-to-folder=/
# default:

last-file-open=C:\Documents and Settings\admin\Desktop\Server File System.vfs
# default:

encode-spaces=no
# default: yes

mime-types=*.js|text/javascript|*.htm;*.html|text/html|*.jpg;*.jpeg;*.jpe|image/jpeg|*.gif|image/gif|*.png|image/png|*.bmp|image/bmp|*.ico|image/x-icon|*.css|text/css|*.txt|text/plain|*.mpeg;*.mpg;*.mpe|video/mpeg|*.avi|video/x-msvideo|*.tpl|text/plain|*.swf|application/x-shockwave-flash
# default: *.htm;*.html|text/html|*.jpg;*.jpeg;*.jpe|image/jpeg|*.gif|image/gif|*.png|image/png|*.bmp|image/bmp|*.ico|image/x-icon|*.mpeg;*.mpg;*.mpe|video/mpeg|*.avi|video/x-msvideo|*.txt|text/plain|*.css|text/css|*.js|text/javascript

in-browser-if-mime=yes
# default: no

icon-masks=.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||.jpg|10001||.mp3|10001||.pdf|10002||
# default:

(address2name removed for privacy)

recent-files=C:\Documents and Settings\admin\Desktop\Server File System.vfs|C:\Documents and Settings\admin\Desktop\Server File System.vfs|C:\Documents and Settings\admin\Desktop\test.vfs
# default:

trusted-files=C:\Documents and Settings\admin\Desktop\Server File System.vfs|C:\Documents and Settings\admin\Desktop\Server File System.vfs
# default:

tray-instead-of-quit=yes
# default: no

compressed-browsing=no
# default: yes

hints4newcomers=no
# default: yes

log-toolbar-expanded=yes
# default: no

do-not-log-address=192.168.1.1-192.168.1.80
# default:

min-disk-space=10
# default: 0

out-total=229392697625
# default: 0

in-total=27657785166
# default: 0

hits-total=404436
# default: 0

downloads-total=85054
# default: 0

upload-total=1922
# default: 0

connections-columns=IP address;174|File;417|Status;196|Speed;72|Time left;77|Progress;980|
# default: IP address;206|File;206|Status;206|Speed;50|Time left;50|Progress;50|

update-daily=no
# default: yes

ip-services=http://www.whatismyip.org/|;http://www.melauto.it/public/rejetto/ip.php|;http://checkip.dyndns.org|:;http://www.canyouseeme.org|td><b>
# default:

ip-services-time=40662.6237840394
# default: 0


-------------------------------------------------------------------

EDIT

I took my VFS, cleared it down to 2 items, and my personal page which typically took 0.7-0.8 seconds to load now took around 0.6 seconds to load.  Not really solved yet.

I'll have to test that with high traffic though.
Title: Re: Macros to serve HTTP status codes
Post by: rejetto on November 15, 2011, 02:42:21 PM
i didn't see anything strange in your options.
it must be the tpl.
Title: Re: Macros to serve HTTP status codes
Post by: raybob on December 03, 2011, 01:26:42 AM
After some testing, I can conclusively say that it IS my VFS that is causing the lag.  My VFS is 168KB in size and has ~750 items.  After renaming the VFS and restarting HFS, it was able to respond to all requests almost instantly, and without a frozen GUI.  The problem has been getting worse lately and I guess my large VFS explains it.  This also explains why the problem doesn't occur in FHFS :)

At one point today I had 11 simultaneous downloads and it took ~5 minutes to generate a single user page with ~20 files.  Also it seems as if HFS puts files in priority over pages (good for HFS' original purpose but not for me lol) as starting a file download was almost instant, while generating a page was extraordinarily slow.

ALSO I'm unable to test this with old versions because apparently there's something in my events script that doesn't work before 279 ( {.cookie.}? ) and so no URLs load at all if using 274  :(

Another thought:  I was wondering why the VFS in FHFS has paths stored as clear text, while the VFS in my Filesplat server seems to be completely encoded.... it was started with build 274 (or maybe even it was 2.2f).  Was it changed since then, and would I have a performance benefit if I somehow converted to the new version?  

This has gotten so bad that with 8 downloads it takes several minutes to load a page with macros.

Two more things I thought of:  1) Would HFS would better outside a VM?  2) Would HFS run better with a professional or server version of windows?

And yet another find.... with high traffic loading pages with few macros (preferences page etc.) takes only a few seconds, but loading macro-heavy pages (like user files pages) can take up to several MINUTES.

LASTLY:  If you want to test the site yourself Rejetto go to www.filesplat.com and see how long it takes to load the main page... see the source at the top for the %build-time% and then go to /~currentdls to see the number of downloads at that moment.
Title: Re: Macros to serve HTTP status codes
Post by: rejetto on January 15, 2012, 09:17:00 PM
it's strange that 750 items are "too much".
consider that the VFS can also contain scripts in it. That may be the problem.

HFS gives no priority to downloads over pages. That must be an apparent side effect of your speed problem.

the reason for encoded VFS is that above 10KB the file gets compressed.

1) VMs always introduce an overhead, i don't know if it's appreciable in your case

2) i have no idea, but i don't expect that

the build time for me was 0.024s
Title: Re: Macros to serve HTTP status codes
Post by: raybob on January 15, 2012, 11:23:40 PM
Hmmm.... as a result of me upgrading to Server 2008 the loading speed for pages is literally 10-20x faster....

The front page typically takes .016 seconds now whereas a file page with 20 files, 2 folders, 2 hash calculations that used to take 12 seconds now takes 0.5-2 seconds to load.

I haven't had any serious load lately but when I do I'll let you know if it's any faster.

EDIT:  My problem seems to be solved... while 4 concurrent downloads would mostly lock it up before now it causes pretty much zero increase in page loading time....

I'm thinking it might be due to increased IO performance.... VirtualBox seems to like VDI files much more than VMDK, that might be one of the causes.
Title: Re: Macros to serve HTTP status codes
Post by: rejetto on January 20, 2012, 11:20:54 AM
fucking strange
Title: Re: Macros to serve HTTP status codes
Post by: raybob on September 06, 2012, 07:42:49 PM
yes i mean to add something like that.
that was already in the to-do list, in this line

+ macros missing to cache a folder: {.reply|content=|var=|code=|filename=|mime=.}


i meant it as a way to provide cached content, but it would work for your purpose too.
keep an eye on future builds for new commands.
unfortunately i didn't solve my problems stopping me from developing hfs yet. :(

Has this yet come to reality?  :)
Title: Re: Macros to serve HTTP status codes
Post by: rejetto on September 07, 2012, 02:10:46 PM
nope :)