Template/events for QOS or traffic shaping.

danny · **on:** March 23, 2021, 05:57:47 AM

This won't be complete because full equal-opposite metrics/status/measures aren't provisioned.

But, this is what I've 'cooked-up' so far:

{.if|{.{.calc|%number-addresses-downloading%*%speed-out% .}> 7500.}|{: busy :}|{: archive :}.}
can also be used for style display none or style display initial (to hide/show buttons)

or

{.if|{.{.calc|%number-addresses-downloading%*%speed-out% .} < 7500.}|{:
archive script
upload script
:}.}
If reasonable load, do script (else omit)

or

soft version {.if|{.%connections% > 39.}|{: busy excuse + javascript recheck timer :}|{: do file list :}.}
connection count management without high cpu load cost

hard version {.if|{.%connections% > 49.}|{:{.redirect|/~busyexcuse.}:}.}
new [busyexcuse|public] section is page for javascript recheck timer wait seconds, if less than x connections then back..
connection count management without high cpu load cost + connection desist (disconnect/redirect) kills downloader/scanner.

P.S.
Multi-user scalable: These above examples run condition detect on the server but the stalling waiting part runs on the Client cpu.
To make stronger logic, the scope was purposefully confined to doable. Just sayin' that hard road North got 2 more signposts, like Snacks > and < Else.

Mars · **Reply #1 on:** March 23, 2021, 01:17:45 PM

whether it is possible to perform a comparison test without going through a macro as it
{.if|{.8000 > 7500.}|{: busy :}|{: archive :}.}

it is strongly preferable to place the equivalent macro as a precautionary measure
{.if|{.>|8000|7500.}|{: busy :}|{: archive :}.}

this allows you to acquire good habits and write functional code, HFS does not support direct literal numerical calculations as javascript

{.%number-addresses-downloading%*%speed-out% > 7500.} is bad

{.>|{.calc|%number-addresses-downloading%*%speed-out%.}|7500.} is correct

{.256*3 < 10 .} is similar to "256*3" < "10" as strings comparaison

you can test this in menu>>debug>run script

{.if|{. 25.6*2 < 100.}|true|false.} ->> "25.6*2" > "100"

{.if|{. {.calc|25.6*2.} < 100.}|true|false.} ->> 56.2 < 100

danny · **Reply #2 on:** March 23, 2021, 08:19:35 PM

Thank you for doing the edits and the briefing. Awesome!!!

I wonder how to get upload speed/#/status when not on the upload page?

danny · **Reply #3 on:** April 03, 2021, 03:07:18 AM

@Mars

Would also like to streamline error handler for sections [overload] and [max contemp downloads].

Currently, I have:

Code: [Select]

[overload]
{.if not|%user%|{:{.if|{.%url% = /.}|{:{.disconnect.}:}.}:}.}{.add header|Cache-Control: no-cache, max-age=0.}<!DOCTYPE html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1"><META HTTP-EQUIV="Refresh" CONTENT="3;URL=./"><TITLE>Overload</TITLE><link rel="icon" href="data:,"></head><body bgcolor="black" text="white" alink="white" link="white" vlink="white"><center><h2><br>High traffic mode engaged.</h2>Returning to previous page after overload has cleared.</center></body>{.disconnect|{.current downloads|ip|file=this.}.}{.if|{.{.current downloads|ip=%ip%|file=this.} > 1.} |{: {.disconnection reason|knackered.} :}/if.}

[max contemp downloads]
{.add header|Cache-Control: no-cache, max-age=0.}<!DOCTYPE html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1"><META HTTP-EQUIV="Refresh" CONTENT="3;URL=./"><TITLE>Downloads</TITLE><link rel="icon" href="data:,"></head><body bgcolor="black" text="white" alink="white" link="white" vlink="white"><center><h2><br>There are ongoing downloads.</h2>More available <i>after</i> current downloads finish.</center></body>{.disconnect|{.current downloads|ip|file=this.}.}

This is bulky/verbose and partially successful. Main trouble is when in overdo condition, file downloaded consists of error page internally. It would be nice to Stop the download without filling the requested file internals with contents of error page.
Causing the client to stop a download requires either server-side redirect or disconnect, macros. So, these pages need to start with if download then disconnect; but, how to?

Mars · **Reply #4 on:** April 03, 2021, 01:40:03 PM

under normal conditions if a download is interrupted the server sends the client a response with Not served: 503 - Overload, this is what should be reproduced instead of performing a sudden disconnection, perhaps a special section with a macro add header, I don't know more, on the other hand there is an event [download] but I don't know how to manage it
you have to look at the list of working variables even if it means creating variables {.set | # var1.} in the sections of the template to reuse them in the event as {. ^ var1.}

danny · **Reply #5 on:** April 04, 2021, 11:19:43 PM

Quote from: Mars on April 03, 2021, 01:40:03 PM

under normal conditions if a download is interrupted the server sends the client a response with Not served: 503 - Overload, this is what should be reproduced instead of performing a sudden disconnection, perhaps a special section with a macro add header...

Some unexpected results:
Attempts to send those types of headers goes to Status=thinking, unresponsive console ui, stuck.
Problem: header does not reach client
Exception: sending a header that hfs and client don't recognize, reaches the client (but a meaningful header does not)

P.S. For preventing the "corrupt file generator" effect, (right-click+save-as with download limit set), it looks like the redirect macro is a working solution, because it changes the saved filename to section name with .htm extension (a proper file albeit not the one you clicked). This could provide better opportunity for downloading the correct file (instead of a corrupt file). Also new custom section name can be a message [server is busy], which would then look like user tried to download "server is busy.htm". Very clanky, but workable.

Quote from: Mars on April 03, 2021, 01:40:03 PM

...creating variables {.set | # var1.} in the sections of the template to reuse them in the event as {. ^ var1.}

For the error redirect macro, I need to set global-variable to %encoded-folder% (or url) Before to redirection to [server is busy]. I need it to remember the original location.
is this correct? ~~{.set|%encoded-folder%|#frompath.}~~
Upon arrival at [server is busy] I need to copy the global variable to temporary/local variable (no#) instantly (so that other user probably doesn't change it). What is the macro to copy #frompath to frompath variable?
For that, should I use %encoded-folder% or %item-url% ?
At [server is busy] I intend to display a nice message and redirect-after-seconds to frompath variable so that the user is not on an island.

Mars · **Reply #6 on:** April 05, 2021, 01:46:53 PM

is this correct? {.set|%encoded-folder%|#frompath.}

always set | #variable | value

this variable will be initiated for all connections, if you plan to use it in an event, you should associate the ip and the name of the user to it in order to target the event

{.set|#frompath|%encoded-folder%;%ip%;%user%;.}

and separate the different fields in the event zone or the section considered
it may seem like a long process, but there is no simplified macro to do this kind of manipulation

this is an example you can verify in menu > debug > run script

Quote

{.set|#frompath|http://127.0.0.1/test;127.0.0.255;MyName;Private message to the user;/set.}
Start = "{.^#frompath.}"

{.comment|Use script only frome here.}

{.set|frompath|{.substring||;|{.^#frompath.}/substring.}/set.}
frompath = "{.^frompath.}"

{.set|#frompath|{.replace|{.^frompath.};||{.^#frompath.}/replace.}/set.}
{.set|ip|{.substring||;|{.^#frompath.}/substring.}/set.}
ip = "{.^ip.}"

{.set|#frompath|{.replace|{.^ip.};||{.^#frompath.}/replace.}/set.}
{.set|user|{.substring||;|{.^#frompath.}/substring.}/set.}
user = "{.^user.}"

{.set|#frompath|{.replace|{.^user.};||{.^#frompath.}/replace.}/set.}
{.set|reply|{.substring||;|{.^#frompath.}/substring.}/set.}
reply = "{.^reply.}"

{.set|#frompath|{.replace|{.^reply.};||{.^#frompath.}/replace.}/set.}
Final = "{.^#frompath.}"

result give

Quote

Start = "http://127.0.0.1/test;127.0.0.255;MyName;Private message to the user;"

frompath = "http://127.0.0.1/test"
ip = "127.0.0.255"
user = "MyName"
reply = "Private message to the user"

Final = ""

everything is based only on the manipulation of the chains

another form of use if the number of elements is not known in advance

Quote

{.set|sep|;.}
{.set|#frompath|http://127.0.0.1/test{.^sep.}127.0.0.255{.^sep.}MyName{.^sep.}Private message to the user{.^sep.}/set.}

Start = "{.^#frompath.}"
{.set|tot|{.count substring|{.^sep.}|{.^#frompath.}.}/set.}
{.set|#nb|{.^tot.}.}

{.while| #nb | {:
{.set|tab[{.^#nb.}]|{.substring||{.^sep.}|{.^#frompath.}/substring.}/set.} {.set|#frompath|{.replace|{.^tab[{.^#nb.}].}{.^sep.}||{.^#frompath.}/replace.}/set.} {.dec|#nb.}
:}/while.}
total items = {.^tot.}
{.for|x|{.^tot.}|1|-1|{:
tab[{.^x.}] = {.^tab[{.^x.}].}:}/for.}

Quote

Start = "http://127.0.0.1/test;127.0.0.255;MyName;Private message to the user;"

total items = 4

tab[4] = http://127.0.0.1/test
tab[3] = 127.0.0.255
tab[2] = MyName
tab[1] = Private message to the user

or to have it in the real order

Quote

{.set|sep|;.}
{.set|#frompath|http://127.0.0.1/test{.^sep.}127.0.0.255{.^sep.}MyName{.^sep.}Private message to the user{.^sep.}/set.}
Start = "{.^#frompath.}"
{.set|tot|{.count substring|{.^sep.}|{.^#frompath.}.}/set.}
{.set|#nb|{.^tot.}.}
{.while| #nb | {:
{.set|idx|{.calc|{.^tot.}-{.^#nb.}+1.}/set.}
{.set|tab[{.^idx.}]|{.substring||{.^sep.}|{.^#frompath.}/substring.}/set.}
{.set|#frompath|{.replace|{.^tab[{.^idx.}].}{.^sep.}||{.^#frompath.}/replace.}/set.}
{.dec|#nb.}
:}/while.}
total items = {.^tot.}
{.for|x|1|{.^tot.}|{:
tab[{.^x.}] = {.^tab[{.^x.}].}:}/for.}

Quote

Start = "http://127.0.0.1/test;127.0.0.255;MyName;Private message to the user;"

total items = 4

tab[1] = http://127.0.0.1/test
tab[2] = 127.0.0.255
tab[3] = MyName
tab[4] = Private message to the user

tab [ x ] is not a real array but a series of variables with different names "tab" + "[" + x + "]"

danny · **Reply #7 on:** April 05, 2021, 03:43:59 PM

The idea goes something like this:

[max contemp downloads]
{.add header|Cache-Control: no-cache, max-age=0.}{.set|#frompath|%encoded-folder%.}{.redirect|/~server%20is%20busy.}

[server is busy|public]
{.set|backpath|{.^#frompath.}.}<!DOCTYPE html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<META HTTP-EQUIV="Refresh" CONTENT="3;URL={.^backpath.}"><TITLE>Busy</TITLE><link rel="icon" href="data:,"></head><body bgcolor="black" text="white" alink="white" link="white" vlink="white"><center><h2><br>Server is busy.</h2>Returning to previous page...</center></body></html>

Doesn't work as expected, because the url is only /

Mars · **Reply #8 on:** April 05, 2021, 08:42:24 PM

{.set | #frompath | %encoded-folder%.} cannot work because [max contemp downloads] is served as a new page from root and at this point the session is no longer the original page then %encoded- folder% returns / because some %var% are not defined, on the other hand I came across %url% which is the link of the loaded file that allows to deduce the original path

you can either use a global variable but which risks being changed by another user in the same situation or an url variable which will be valid only for the right recipient

Code: [Select]

[max contemp downloads]
{.add header|Cache-Control: no-cache, max-age=0.}
{.set|#frompath|{.encodeuri|%url%.}.}
{.redirect|/~server%20is%20busy?path={.encodeuri|{.filepath|%url%.}.}.}

[server is busy|public]
{.set|backpath|{.decodeuri|{.urlvar|path.}.}.}
<!DOCTYPE html><html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<META HTTP-EQUIV="Refresh" CONTENT="3;URL={.^backpath.}">
<TITLE>Busy</TITLE>
<link rel="icon" href="data:,"></head>
<body bgcolor="black" text="white" alink="white" link="white" vlink="white"><center><h2><br>Server is busy.</h2>Returning to previous page...</center>
</body></html>

danny · **Reply #9 on:** April 06, 2021, 10:22:47 PM

Quote from: Mars on April 05, 2021, 08:42:24 PM

... I came across %url% which is the link of the loaded file that allows to deduce the original path

Thanks!!! That works very well for the [max contemp downloads] section.

I found out that the approach doesn't apply to [overload] section because can't redirect when beyond the set connections limit.
So, here is the other option for preventing right click + save as corrupt file (if connections limits are used):

Code: [Select]

[overload]
{.if|{.{.calc|{.get ini|max-connections.}.}<>0.}|{:{.set ini|max-connections=-1.}:}.}{.disconnect.}{.add to log|%ip% %user% disconnected at %connections% connections.}

I've axed the Max connections limit so that 1 abuser can't knock out All users. And, then the remaining possibility is that an individual went over Max connections from single address limit, got disconnected and then their ip got logged. No corrupt downloads.

Edit: Can test-drive with the latest Stripes template.

danny · **Reply #10 on:** April 08, 2021, 02:59:25 PM

Is it possible to do something like?:
{.if|{.current connections|ip=%ip%.} > 9.}|{: busy excuse + animated recheck :}|{: do file list :}.}

Edit: The flaw with {.if|{.%connections% > 39.}|{: busy excuse + javascript recheck timer :}|{: do file list :}.} is DDOS, because 1 abuser can hinder All users. However, instead of global connections, if it were changed to connections-per-ip then the flaw is gone.

danny · **Reply #11 on:** April 13, 2021, 08:59:01 AM

@Mars, I need some help with macro string processing to hit target of 1 folder higher, because some other use may have already deleted that folder...

Code: [Select]

[not found]
{.add header|Cache-Control: no-cache, max-age=0.}{.redirect|/~404?path={.encodeuri|{.filepath|%url%.}.}.}
[404|public]
{.set|backpath|{.decodeuri|{.urlvar|path.}.}.}<!DOCTYPE html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<META HTTP-EQUIV="Refresh" CONTENT="1;URL={.^backpath.}"><TITLE>404</TITLE><link rel="icon" href="data:,"></head><body><center><h2><br>Not Found.</h2>Returning to previous page...</center></body></html>

If that had been entirely macro, it would be deadlock-loop.
Problem is, wrong target for 404 (if missing), because the need is parent of, aka 1 folder higher than that. Halp!

NaitLee · **Reply #12 on:** April 13, 2021, 09:15:17 AM

Quote from: danny on April 13, 2021, 08:59:01 AM

macro string processing to hit target of 1 folder higher...

In takeback 0.3 preview:

Code: [Select]

        {.if|{.=|{.cut|-1||%url%.}|/.}|
            <a href="../">⇦ {.!Back.}</a>
        |
            <a href="./">⇦ {.!Back.}</a>
        .}

if what not found is a dir, then go to parent-dir; if a file, then go to its dir.

Is this what we need?

Mars · **Reply #13 on:** April 13, 2021, 11:41:19 AM

you must analyze the url which is sent by the path variable and check its validity by eliminating the non-existent parts or use a fallback url like the use with newurl in the following

[not found]
{set|newurl|/.}
{.add header|Cache-Control: no-cache, max-age=0.}
{.redirect|/~404?path={.if|{.exists|%url%.}|{:{.encodeuri|{.filepath|%url%.}.}:}|{:{.^newurl.}:}.}.}

the problem is that in section 404, the url is not that of the containing directory but that of the requested file "return to previous page" does not correspond to% url% as for [max contemp downloads]

use macro {. add to log | ..what you need to see.. .} in various places to control the transmitted data and more easily determine why you are having inconsistencies in the operation of your scripts

danny · **Reply #14 on:** April 25, 2021, 05:48:20 AM

How do you like it?

Code: [Select]

[not found]
{.add header|Cache-Control: no-cache, max-age=0.}{.if|{.=|{.cut|-1||%url%.}|/.}
|{:{.redirect|../.}:}
|{:{.redirect|/~404?path={.encodeuri|{.filepath|%url%.}.}.}:}.}
[404|public]
<!DOCTYPE html><html><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<META HTTP-EQUIV="Refresh" CONTENT="1;URL={.decodeuri|{.urlvar|path.}.}"><TITLE>404</TITLE><link rel="icon" href="data:,">
</head><body><center><h2><br>File Not Found.</h2>Returning to previous page...</center></body></html>

For missing Folder, a valid folder is quickly found. (the user is not lost)
For missing File, right-click+save-as does Not generate a corrupt file. (browser has 404.htm)

A complete set of quality-control error pages, can be found in the Throwback and Stripes templates.
Thanks to Mars and Naitlee

Template/events for QOS or traffic shaping.

danny

Template/events for QOS or traffic shaping.

Mars

Re: Template/events for QOS or traffic shaping.

danny

Re: Template/events for QOS or traffic shaping.

danny

Re: Template/events for QOS or traffic shaping.

Mars

Re: Template/events for QOS or traffic shaping.

danny

Re: Template/events for QOS or traffic shaping.

Mars

Re: Template/events for QOS or traffic shaping.

danny

Re: Template/events for QOS or traffic shaping.

Mars

Re: Template/events for QOS or traffic shaping.

danny

Re: Template/events for QOS or traffic shaping.

danny

Re: Template/events for QOS or traffic shaping.

danny

Halp! missed the target for the 404

NaitLee

Re: Halp! missed the target for the 404

Mars

Re: Template/events for QOS or traffic shaping.

danny

Re: Template/events for QOS or traffic shaping.