rejetto forum
Software => HFS ~ HTTP File Server => Programmers corner => Topic started by: danny on March 23, 2021, 05:57:47 AM
-
This won't be complete because full equal-opposite metrics/status/measures aren't provisioned.
But, this is what I've 'cooked-up' so far:
{.if|{.{.calc|%number-addresses-downloading%*%speed-out% .}> 7500.}|{: busy :}|{: archive :}.}
can also be used for style display none or style display initial (to hide/show buttons)
or
{.if|{.{.calc|%number-addresses-downloading%*%speed-out% .} < 7500.}|{:
archive script
upload script
:}.}
If reasonable load, do script (else omit)
or
soft version {.if|{.%connections% > 39.}|{: busy excuse + javascript recheck timer :}|{: do file list :}.}
connection count management without high cpu load cost
hard version {.if|{.%connections% > 49.}|{:{.redirect|/~busyexcuse.}:}.}
new [busyexcuse|public] section is page for javascript recheck timer wait seconds, if less than x connections then back..
connection count management without high cpu load cost + connection desist (disconnect/redirect) kills downloader/scanner.
P.S.
Multi-user scalable: These above examples run condition detect on the server but the stalling waiting part runs on the Client cpu.
To make stronger logic, the scope was purposefully confined to doable. Just sayin' that hard road North got 2 more signposts, like Snacks > and < Else.
-
whether it is possible to perform a comparison test without going through a macro as it
{.if|{.8000 > 7500.}|{: busy :}|{: archive :}.}
it is strongly preferable to place the equivalent macro as a precautionary measure
{.if|{.>|8000|7500.}|{: busy :}|{: archive :}.}
this allows you to acquire good habits and write functional code, HFS does not support direct literal numerical calculations as javascript
{.%number-addresses-downloading%*%speed-out% > 7500.} is bad
{.>|{.calc|%number-addresses-downloading%*%speed-out%.}|7500.} is correct
{.256*3 < 10 .} is similar to "256*3" < "10" as strings comparaison
you can test this in menu>>debug>run script
{.if|{. 25.6*2 < 100.}|true|false.} ->> "25.6*2" > "100" >:(
{.if|{. {.calc|25.6*2.} < 100.}|true|false.} ->> 56.2 < 100 ;D
-
Thank you for doing the edits and the briefing. Awesome!!!
I wonder how to get upload speed/#/status when not on the upload page?
-
@Mars
Would also like to streamline error handler for sections [overload] and [max contemp downloads].
Currently, I have: [overload]
{.if not|%user%|{:{.if|{.%url% = /.}|{:{.disconnect.}:}.}:}.}{.add header|Cache-Control: no-cache, max-age=0.}<!DOCTYPE html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1"><META HTTP-EQUIV="Refresh" CONTENT="3;URL=./"><TITLE>Overload</TITLE><link rel="icon" href="data:,"></head><body bgcolor="black" text="white" alink="white" link="white" vlink="white"><center><h2><br>High traffic mode engaged.</h2>Returning to previous page after overload has cleared.</center></body>{.disconnect|{.current downloads|ip|file=this.}.}{.if|{.{.current downloads|ip=%ip%|file=this.} > 1.} |{: {.disconnection reason|knackered.} :}/if.}
[max contemp downloads]
{.add header|Cache-Control: no-cache, max-age=0.}<!DOCTYPE html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1"><META HTTP-EQUIV="Refresh" CONTENT="3;URL=./"><TITLE>Downloads</TITLE><link rel="icon" href="data:,"></head><body bgcolor="black" text="white" alink="white" link="white" vlink="white"><center><h2><br>There are ongoing downloads.</h2>More available <i>after</i> current downloads finish.</center></body>{.disconnect|{.current downloads|ip|file=this.}.}
This is bulky/verbose and partially successful. Main trouble is when in overdo condition, file downloaded consists of error page internally. It would be nice to Stop the download without filling the requested file internals with contents of error page.
Causing the client to stop a download requires either server-side redirect or disconnect, macros. So, these pages need to start with if download then disconnect; but, how to?
-
under normal conditions if a download is interrupted the server sends the client a response with Not served: 503 - Overload, this is what should be reproduced instead of performing a sudden disconnection, perhaps a special section with a macro add header, I don't know more, on the other hand there is an event [download] but I don't know how to manage it
you have to look at the list of working variables even if it means creating variables {.set | # var1.} in the sections of the template to reuse them in the event as {. ^ var1.}
-
under normal conditions if a download is interrupted the server sends the client a response with Not served: 503 - Overload, this is what should be reproduced instead of performing a sudden disconnection, perhaps a special section with a macro add header...
Some unexpected results:
Attempts to send those types of headers goes to Status=thinking, unresponsive console ui, stuck.
Problem: header does not reach client
Exception: sending a header that hfs and client don't recognize, reaches the client (but a meaningful header does not)
P.S. For preventing the "corrupt file generator" effect, (right-click+save-as with download limit set), it looks like the redirect macro is a working solution, because it changes the saved filename to section name with .htm extension (a proper file albeit not the one you clicked). This could provide better opportunity for downloading the correct file (instead of a corrupt file). Also new custom section name can be a message [server is busy], which would then look like user tried to download "server is busy.htm". Very clanky, but workable.
...creating variables {.set | # var1.} in the sections of the template to reuse them in the event as {. ^ var1.}
For the error redirect macro, I need to set global-variable to %encoded-folder% (or url) Before to redirection to [server is busy]. I need it to remember the original location.
is this correct? {.set|%encoded-folder%|#frompath.}
Upon arrival at [server is busy] I need to copy the global variable to temporary/local variable (no#) instantly (so that other user probably doesn't change it). What is the macro to copy #frompath to frompath variable?
For that, should I use %encoded-folder% or %item-url% ?
At [server is busy] I intend to display a nice message and redirect-after-seconds to frompath variable so that the user is not on an island.
-
is this correct? {.set|%encoded-folder%|#frompath.}
always set | #variable | value
this variable will be initiated for all connections, if you plan to use it in an event, you should associate the ip and the name of the user to it in order to target the event
{.set|#frompath|%encoded-folder%;%ip%;%user%;.}
and separate the different fields in the event zone or the section considered
it may seem like a long process, but there is no simplified macro to do this kind of manipulation
this is an example you can verify in menu > debug > run script
{.set|#frompath|http://127.0.0.1/test;127.0.0.255;MyName;Private message to the user;/set.}
Start = "{.^#frompath.}"
{.comment|Use script only frome here.}
{.set|frompath|{.substring||;|{.^#frompath.}/substring.}/set.}
frompath = "{.^frompath.}"
{.set|#frompath|{.replace|{.^frompath.};||{.^#frompath.}/replace.}/set.}
{.set|ip|{.substring||;|{.^#frompath.}/substring.}/set.}
ip = "{.^ip.}"
{.set|#frompath|{.replace|{.^ip.};||{.^#frompath.}/replace.}/set.}
{.set|user|{.substring||;|{.^#frompath.}/substring.}/set.}
user = "{.^user.}"
{.set|#frompath|{.replace|{.^user.};||{.^#frompath.}/replace.}/set.}
{.set|reply|{.substring||;|{.^#frompath.}/substring.}/set.}
reply = "{.^reply.}"
{.set|#frompath|{.replace|{.^reply.};||{.^#frompath.}/replace.}/set.}
Final = "{.^#frompath.}"
result give
Start = "http://127.0.0.1/test;127.0.0.255;MyName;Private message to the user;"
frompath = "http://127.0.0.1/test"
ip = "127.0.0.255"
user = "MyName"
reply = "Private message to the user"
Final = ""
everything is based only on the manipulation of the chains
another form of use if the number of elements is not known in advance
{.set|sep|;.}
{.set|#frompath|http://127.0.0.1/test{.^sep.}127.0.0.255{.^sep.}MyName{.^sep.}Private message to the user{.^sep.}/set.}
Start = "{.^#frompath.}"
{.set|tot|{.count substring|{.^sep.}|{.^#frompath.}.}/set.}
{.set|#nb|{.^tot.}.}
{.while| #nb | {:
{.set|tab[{.^#nb.}]|{.substring||{.^sep.}|{.^#frompath.}/substring.}/set.} {.set|#frompath|{.replace|{.^tab[{.^#nb.}].}{.^sep.}||{.^#frompath.}/replace.}/set.} {.dec|#nb.}
:}/while.}
total items = {.^tot.}
{.for|x|{.^tot.}|1|-1|{:
tab[{.^x.}] = {.^tab[{.^x.}].}:}/for.}
Start = "http://127.0.0.1/test;127.0.0.255;MyName;Private message to the user;"
total items = 4
tab[4] = http://127.0.0.1/test
tab[3] = 127.0.0.255
tab[2] = MyName
tab[1] = Private message to the user
or to have it in the real order
{.set|sep|;.}
{.set|#frompath|http://127.0.0.1/test{.^sep.}127.0.0.255{.^sep.}MyName{.^sep.}Private message to the user{.^sep.}/set.}
Start = "{.^#frompath.}"
{.set|tot|{.count substring|{.^sep.}|{.^#frompath.}.}/set.}
{.set|#nb|{.^tot.}.}
{.while| #nb | {:
{.set|idx|{.calc|{.^tot.}-{.^#nb.}+1.}/set.}
{.set|tab[{.^idx.}]|{.substring||{.^sep.}|{.^#frompath.}/substring.}/set.}
{.set|#frompath|{.replace|{.^tab[{.^idx.}].}{.^sep.}||{.^#frompath.}/replace.}/set.}
{.dec|#nb.}
:}/while.}
total items = {.^tot.}
{.for|x|1|{.^tot.}|{:
tab[{.^x.}] = {.^tab[{.^x.}].}:}/for.}
Start = "http://127.0.0.1/test;127.0.0.255;MyName;Private message to the user;"
total items = 4
tab[1] = http://127.0.0.1/test
tab[2] = 127.0.0.255
tab[3] = MyName
tab[4] = Private message to the user
tab [ x ] is not a real array but a series of variables with different names "tab" + "[" + x + "]"
-
The idea goes something like this:
[max contemp downloads]
{.add header|Cache-Control: no-cache, max-age=0.}{.set|#frompath|%encoded-folder%.}{.redirect|/~server%20is%20busy.}
[server is busy|public]
{.set|backpath|{.^#frompath.}.}<!DOCTYPE html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<META HTTP-EQUIV="Refresh" CONTENT="3;URL={.^backpath.}"><TITLE>Busy</TITLE><link rel="icon" href="data:,"></head><body bgcolor="black" text="white" alink="white" link="white" vlink="white"><center><h2><br>Server is busy.</h2>Returning to previous page...</center></body></html>
Doesn't work as expected, because the url is only /
-
{.set | #frompath | %encoded-folder%.} cannot work because [max contemp downloads] is served as a new page from root and at this point the session is no longer the original page then %encoded- folder% returns / because some %var% are not defined, on the other hand I came across %url% which is the link of the loaded file that allows to deduce the original path
you can either use a global variable but which risks being changed by another user in the same situation or an url variable which will be valid only for the right recipient
[max contemp downloads]
{.add header|Cache-Control: no-cache, max-age=0.}
{.set|#frompath|{.encodeuri|%url%.}.}
{.redirect|/~server%20is%20busy?path={.encodeuri|{.filepath|%url%.}.}.}
[server is busy|public]
{.set|backpath|{.decodeuri|{.urlvar|path.}.}.}
<!DOCTYPE html><html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<META HTTP-EQUIV="Refresh" CONTENT="3;URL={.^backpath.}">
<TITLE>Busy</TITLE>
<link rel="icon" href="data:,"></head>
<body bgcolor="black" text="white" alink="white" link="white" vlink="white"><center><h2><br>Server is busy.</h2>Returning to previous page...</center>
</body></html>
-
... I came across %url% which is the link of the loaded file that allows to deduce the original path
Thanks!!! That works very well for the [max contemp downloads] section.
I found out that the approach doesn't apply to [overload] section because can't redirect when beyond the set connections limit.
So, here is the other option for preventing right click + save as corrupt file (if connections limits are used): [overload]
{.if|{.{.calc|{.get ini|max-connections.}.}<>0.}|{:{.set ini|max-connections=-1.}:}.}{.disconnect.}{.add to log|%ip% %user% disconnected at %connections% connections.}I've axed the Max connections limit so that 1 abuser can't knock out All users. And, then the remaining possibility is that an individual went over Max connections from single address limit, got disconnected and then their ip got logged. No corrupt downloads.
Edit: Can test-drive with the latest Stripes (http://rejetto.com/forum/index.php?topic=13415.msg1066850#msg1066850) template.
-
Is it possible to do something like?:
{.if|{.current connections|ip=%ip%.} > 9.}|{: busy excuse + animated recheck :}|{: do file list :}.}
Edit: The flaw with {.if|{.%connections% > 39.}|{: busy excuse + javascript recheck timer :}|{: do file list :}.} is DDOS, because 1 abuser can hinder All users. However, instead of global connections, if it were changed to connections-per-ip then the flaw is gone.
-
@Mars, I need some help with macro string processing to hit target of 1 folder higher, because some other use may have already deleted that folder...
[not found]
{.add header|Cache-Control: no-cache, max-age=0.}{.redirect|/~404?path={.encodeuri|{.filepath|%url%.}.}.}
[404|public]
{.set|backpath|{.decodeuri|{.urlvar|path.}.}.}<!DOCTYPE html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<META HTTP-EQUIV="Refresh" CONTENT="1;URL={.^backpath.}"><TITLE>404</TITLE><link rel="icon" href="data:,"></head><body><center><h2><br>Not Found.</h2>Returning to previous page...</center></body></html>If that had been entirely macro, it would be deadlock-loop.
Problem is, wrong target for 404 (if missing), because the need is parent of, aka 1 folder higher than that. Halp!
-
macro string processing to hit target of 1 folder higher...
In takeback 0.3 preview:
{.if|{.=|{.cut|-1||%url%.}|/.}|
<a href="../">⇦ {.!Back.}</a>
|
<a href="./">⇦ {.!Back.}</a>
.}
if what not found is a dir, then go to parent-dir; if a file, then go to its dir.
Is this what we need?
-
you must analyze the url which is sent by the path variable and check its validity by eliminating the non-existent parts or use a fallback url like the use with newurl in the following
[not found]
{set|newurl|/.}
{.add header|Cache-Control: no-cache, max-age=0.}
{.redirect|/~404?path={.if|{.exists|%url%.}|{:{.encodeuri|{.filepath|%url%.}.}:}|{:{.^newurl.}:}.}.}
the problem is that in section 404, the url is not that of the containing directory but that of the requested file "return to previous page" does not correspond to% url% as for [max contemp downloads]
use macro {. add to log | ..what you need to see.. .} in various places to control the transmitted data and more easily determine why you are having inconsistencies in the operation of your scripts
-
How do you like it?
[not found]
{.add header|Cache-Control: no-cache, max-age=0.}{.if|{.=|{.cut|-1||%url%.}|/.}
|{:{.redirect|../.}:}
|{:{.redirect|/~404?path={.encodeuri|{.filepath|%url%.}.}.}:}.}
[404|public]
<!DOCTYPE html><html><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<META HTTP-EQUIV="Refresh" CONTENT="1;URL={.decodeuri|{.urlvar|path.}.}"><TITLE>404</TITLE><link rel="icon" href="data:,">
</head><body><center><h2><br>File Not Found.</h2>Returning to previous page...</center></body></html>For missing Folder, a valid folder is quickly found. (the user is not lost)
For missing File, right-click+save-as does Not generate a corrupt file. (browser has 404.htm)
A complete set of quality-control error pages, can be found in the Throwback and Stripes templates.
Thanks to Mars and Naitlee
-
@Mars
For QOS purpose, I had few questions...
Are there some per ip-specific measurements?
How to make global variable apply to only 1 ip?
something like these:?
connections_this_ip
active_downloads_this_ip
set or get #^flag_this_ip
Some templates have extra features (such as thumbnails) that are not good in high-load conditions. It is good to make a bypass macro to temporarily streamline the template (without extras) to lighten the load. If global measure is used for traffic shaping, then 1 miscreant affects everyone (this is anti-QOS); so, for QOS function, it may be better if feature-bypass macro targeted the specific overdoing IP address.
I don't know; but, if the per-ip measures are available, then I could test it.
-
Are there some per ip-specific measurements?
no
How to make global variable apply to only 1 ip?
a variable is defined by its name which can include letters or numbers and preceded by # to become a global variable
just include %ip% in the name
{.set|#flag_%ip%| .... .}
global variables only exist while hfs is running
-
...a variable is defined by its name which can include letters or numbers and preceded by # to become a global variable
just include %ip% in the name
{.set|#flag_%ip%| .... .}
Thanks Mars!