rejetto forum

Software => HFS ~ HTTP File Server => Programmers corner => Topic started by: bacter on November 04, 2009, 04:20:37 PM

Title: Request for changing postvar (and urlvar) behaviour
Post by: bacter on November 04, 2009, 04:20:37 PM
When sending from a form with method 'get' or 'post', values that contain curlybraces with his points or the macro expression separator '|', the values seem to be sent to the server.. but hfs clears the whole content of this values that contain {., {: and .}, :} on arriving.!

I see this when the method is 'get' at least, and experience the same when the method is 'post'.

This is really annoying when you type  or modify a chunk of text and loose it this way. I suppose that for security reasons no macros (or elements of) can be accepted in this type of data... but perhaps it would be possible to 'escape' in some way the macro delimiters. Using html char values like { do not solve the problem.
This escape could be replacing the signs in the incoming postvar for further use in some way like this:
{. --> {+., 
.} --> .+},
{: --> {+:,
:} --> :+} , etc.
| --> ¦
or this way:
{  -->  ({)
}  -->  (})
|  -->  (or)
This would allow to save {.any macro|expr.} as {+.any macro¦expr.+}  or ({).any macro(or)expr.(}), something that can never be reverted to a executable macro. This would allow to use the macro signs {:{.another macro.}:} inside documents used in a wiki, forum or any cms.

When loading such a file, then we can then replace the codes with the original signs and no macro formed this way will be executed.

Thanks for a possible solution.
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: rejetto on November 05, 2009, 11:55:39 AM
can't you "escape" such values with javascript?
something like
string.replace("{.", "{+.")
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: bacter on November 05, 2009, 12:56:52 PM
I suppose it would be possible to use javascript to do this .. but if anyone use forms with method post and don't use such a script to do the conversion, any 'error' typing those sequences by accident (its sometimes nice to use | to separate some items) clearing the postvar may be loss of data.
So anyway, also if a workaround with javascript would be possible, i think it would be better to 'escape' the offending caracter(sequences)  or to substitute them with any allowed sign (so at least, data is not lost and the user may correct his sintax).

to simplify it could be:
|  -->  ¦
{.  -->  ¦. , {:  -->  ¦:
.}  -->  .¦ , :}  -->  :¦
or this way:
|  -->  ¦
{.  -->  ¿. , {:  -->  ¿:
.}  -->  .? , :}  -->  :?
As we can assume that  ¦ is never entered directly from keyboard and has no specific use, such a substitution would eliminate the risk of data loss and would also preserve the security of hfs against introducction of malicious code.
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: rejetto on November 05, 2009, 04:15:53 PM
i think we should avoid inventing an encoding, giving headache to someone.
i would go for an html encoding.
&#XXX;
do you want to test it?
http://www.sendspace.com/file/8dkbh5
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: bacter on November 05, 2009, 05:27:19 PM
Thanks Rejetto!

Your solutions to the problems i find, are always far better than my proposals to avoid them!  :D :D :D

Also i don't understand why you 'escape' the points of the end macro to #46 and #58,  instead of the ending brace } to #125, your solution works perfectly!

Thank you very much! ;) ;)
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: rejetto on November 06, 2009, 02:47:24 PM
happy to know it's working. i will keep it for the next release, hoping there's no unwanted side effect :P

Also i don't understand why you 'escape' the points of the end macro to #46 and #58,  instead of the ending brace } to #125, your solution works perfectly!

the method is generic, and will just encode the first character.
i find pointless to distinguish between symbols.
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: bacter on November 14, 2009, 01:39:41 AM
Hi Rejetto

I have found another problem with caracter in postvar and urlvar.

When a postvar contains some template simbols like %user% , %total-in% they are immediatly substituted, if it contains names of sections like %upload% nothing happens and the symbol remains remains intact, with other section names like %files%, they are substituded by some partial execution: some html from the files section plus spaces and CR-LF generated by macro executions is included.

I see two possible solutions to the problem:

A) The percent sign % should be replaced by % in postvar and urlvar.

B)  Assure that a postvar dont suffer any implicit interpretation, so that we can apply {.set|data|{.replace|%|%|{.postvar|fieldname.}.}.} before writing data to a file.

PD: Don't forget the breadcrumbs-from bug (still buggy) ;)
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: rejetto on November 14, 2009, 02:22:41 PM
Don't think %XXX% is for getting sections text (that's demanded to {.section.} command).
This is true only for SOME symbols, others have different meaning.
You will find a good list at http://www.rejetto.com/wiki/index.php/HFS:_Template_symbols
and you'll see there's no %upload%.

Symbols substitution is at the moment made in an initial step, before macros.
It cannot be done AFTER, because many macros uses %symbols% value to work, like {.if.}
To do it WHILE, i should change the parser deeply.
At the moment there's no way to stop %symbols% like for {:quoting:}.
I may even introduce a special way to do it, but i think that for such special case a javascript encoding should be enough.

Many of these cases, anyway, would be solved by just not having the text "expanded", but managed in a variable.
I'll try to be clearer: when you call {.postvar.} or {.urlvar.} you actually get its value made part of the output (the html), even just for a while.
If you do nothing more, you will get the value in your face, in the browser.
If you instead put such value as parameter of another command, like {.set.} or {.save.}, then it will not be displayed, but instead its used for further elaboration. That's often the case for an application like the one you are developing.

By having this value directly put in a variable, instead of being replaced in the output, we should nicely workaround the problem. And it would be even more efficient.
In the example you made, you would not need to {.set.} and maybe not even {.replace.}
Some commands may just be adapted to be able to work with variables, like i already did in the last year.

i'm sorry for the breadcrumbs, but i had truly no time lately.

this was long, but i hope it was clear enough.
next release.
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: bacter on November 15, 2009, 02:24:34 PM
Thanks Rejetto for yout long answer.

I understand that the % symbol is a rather complex question, as it implies different modes of treatment.
A) when applied to sections, %files% or {.$files.} for example are equivalent.
B) when applied to symbols like %user%, there is (at the moment) no alternative way to access this 'variable'.

Perhaps in a future - i know this may lead to compatibility problems with 'old' templates - this variables (from type B) could be substituted by pre-initialized variables: %user% ---> {.^user.}

You are right that the problem can be solved with a workaround in javascript. But as i want to get the wiki and forum things working initially without javascript, i will do some additional 'escaping' at the moment: .%. will be displayed as % . As the %-problem is not very destructive , this should be sufficient. I would reconsider the javascript in  future versions that uses a javascript editor instead of simple forms.

---
By the way, i don't know if there is a bug with numeric macros and real numbers, or if i don't use proper sintax: calculations (calc, add ..) dont work with real numbers ({.calc|8,7-6,4.} does not return anything {.calc|1+(5/2).} returns a real number!

Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: rejetto on November 15, 2009, 07:11:42 PM
i guess you should try by using a dot instead of a comma, as decimal separator.

about the javascript, you should consider having the js script degrade gracefully, having all others miss the encoding, and being unable to use such character.
you may even show a warning, "please enable javascript, or you won't be able to use the % sign in your writing."
javascript should not be disabled for every site. extra-security-minded people should be able to enable on a per-site basis.
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: bacter on November 15, 2009, 08:28:39 PM
Quote
I guess you should try by using a dot instead of a comma, as decimal separator.

Problem with real numbers seems more complex than this:
hfs is normally started with windows. Then an integer calc operation {.calc|1+(5/2).} returns 3,5 and all operations that involve real numbers - no matter if coma or point is used - return 0 or nothing.
When i stop hfs, then restart it with running windows, the calc operation {.calc|1+(5/2).} returns 3.5 !!, and calc with real numbers with decimal point dot works, but add,sub and mul dont either.


Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: rejetto on November 18, 2009, 11:37:55 PM
i didn't understand.
can you please tell me how to reproduce the problem?
next version, anyway, will support real numbers with add/sub/mul/div
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: bacter on November 19, 2009, 01:21:33 AM
Hi rejetto:

When i had problems using math functions, doubting to use decimal separator comma or point, i added the following code at the bottom of my template to test:

Code: [Select]
<br /> {.calc|1+(5/2).}
<br /> 8.7- 6.4 point:{.calc|8.7-6.4.} coma:{.calc|8,7-6,4.}
<br /> add 8.7 6.444 point:{.add|8.7|6.444.} coma:{.add|8,7|6,444.}
<br /> sub 8.7 6.444 point:{.sub|8.7|6.444.} coma:{.sub|8,7|6,444.}
<br /> mul 8.7 6.444 point:{.mul|8.7|6.444.} coma:{.mul|8,7|6,444.}
<br /> round 8.7345, 2 point: {.round|8.7345|2.} coma:  {.round|8,7345|2.}

So i observed the following results:
Quote
3,5
8.7- 6.4 point: coma:
add 8.7 6.444 point:0 coma:0
sub 8.7 6.444 point:0 coma:0
mul 8.7 6.444 point:0 coma:0
round 8.7345, 2 point: coma: 8,73
This is the output when hfs starts with windows after booting.

When i was doubting if it could be a problem of the special build i used, i tried with previous version and got the following output:
Quote
3.5
8.7- 6.4 point:2.3 coma:
add 8.7 6.444 point:0 coma:0
sub 8.7 6.444 point:0 coma:0
mul 8.7 6.444 point:0 coma:0
round 8.7345, 2 point: 8.73 coma:
So i first thought it would be a problem of the build, and returned to the newest special build. And it worked too!

Result is: After a new boot or reboot windows (shutdown and restart the computer), hfs uses the comma separator. Then i exit hfs, rerun it, and it works using the point separator!

I have checked that there is no other hfs.exe in any directory to be sure that its the same .exe is used on startup of the machine :)

EDIT
This is so with windows XPp SP3 , can't test it with W7 because HDD died this morning
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: rejetto on November 20, 2009, 09:59:02 AM
how is your hfs started at first?
is it in the "startup" folder? is it a service?
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: bacter on November 20, 2009, 11:50:12 AM
Rejetto:

there is a link to hfs.exe in the startup folder.

i have activated the menu-> start/exit -> run hfs when windows starts option.

I thought that perhaps it has to do with the country information that could be not ready when hfs starts (if it retrieves it ?), but the country information for spain says to use the point for thousands grouping and comma as decimal separator.
But this does not seem to be the reason, as coma is used just after booting, and is changed to point after exit and re-run.
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: bacter on November 20, 2009, 11:39:43 PM
Hi rejetto:
With new build 249 i get the following results of the above macros.

first run with windows booting:
Quote
3,5
8.7- 6.4 point: coma:
add 8.7 6.444 point:0 coma:15,144
sub 8.7 6.444 point:0 coma:2,256
mul 8.7 6.444 point:0 coma:56,0628
round 8.7345, 2 point: coma: 8,73
after re-run:
Quote
3.5
8.7- 6.4 point:2.3 coma:
add 8.7 6.444 point:15.144 coma:0
sub 8.7 6.444 point:2.256 coma:0
mul 8.7 6.444 point:56.0628 coma:0
round 8.7345, 2 point: 8.73 coma:
To avoid problems if somebody uses some floating point constants in the template, i suggest to use fixed point notation.
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: rejetto on November 21, 2009, 12:23:50 AM
thank you for your precious testing bacter.
i will think of a solution.
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: Mars on November 23, 2009, 12:01:08 AM
By looking at the calls of functions system, it is evident that it is the local format of the server that is used for all the calculations

Code: [Select]
     result:=floatToStr(parF(0) [color=brown]operator[/color] parF(1));
Code: [Select]
function FloatToStr(Value: Extended): string;
Quote
function FloatToStr(Value: Extended; const FormatSettings: TFormatSettings): string;

Quote
result:=floatToStr(parF(0) operator parF(1)), FormatSetting(par('format'));

adding a param as 'format' can solve the bacter' problem , we can take Microsoft Excel as example to define the format  by using chars as '#' '0' ',' '.'   

## ##0,00   
0.#

etc ....

Code: [Select]
  TFormatSettings = record
    CurrencyFormat: Byte;
    NegCurrFormat: Byte;
    ThousandSeparator: Char;
    DecimalSeparator: Char;
    CurrencyDecimals: Byte;
    DateSeparator: Char;
    TimeSeparator: Char;
    ListSeparator: Char;
    CurrencyString: string;
    ShortDateFormat: string;
    LongDateFormat: string;
    TimeAMString: string;
    TimePMString: string;
    ShortTimeFormat: string;
    LongTimeFormat: string;
    ShortMonthNames: array[1..12] of string;
    LongMonthNames: array[1..12] of string;
    ShortDayNames: array[1..7] of string;
    LongDayNames: array[1..7] of string;
    TwoDigitYearCenturyWindow: Word;
  end;


{ Currency and date/time formatting options

  The initial values of these variables are fetched from the system registry
  using the GetLocaleInfo function in the Win32 API. The description of each
  variable specifies the LOCALE_XXXX constant used to fetch the initial
  value.

  CurrencyString - Defines the currency symbol used in floating-point to
  decimal conversions. The initial value is fetched from LOCALE_SCURRENCY.

  CurrencyFormat - Defines the currency symbol placement and separation
  used in floating-point to decimal conversions. Possible values are:

    0 = '$1'
    1 = '1$'
    2 = '$ 1'
    3 = '1 $'

  The initial value is fetched from LOCALE_ICURRENCY.

  NegCurrFormat - Defines the currency format for used in floating-point to
  decimal conversions of negative numbers. Possible values are:

    0 = '($1)'      4 = '(1$)'      8 = '-1 $'      12 = '$ -1'
    1 = '-$1'       5 = '-1$'       9 = '-$ 1'      13 = '1- $'
    2 = '$-1'       6 = '1-$'      10 = '1 $-'      14 = '($ 1)'
    3 = '$1-'       7 = '1$-'      11 = '$ 1-'      15 = '(1 $)'

  The initial value is fetched from LOCALE_INEGCURR.

  ThousandSeparator - The character used to separate thousands in numbers
  with more than three digits to the left of the decimal separator. The
  initial value is fetched from LOCALE_STHOUSAND.  A value of #0 indicates
  no thousand separator character should be output even if the format string
  specifies thousand separators.

  DecimalSeparator - The character used to separate the integer part from
  the fractional part of a number. The initial value is fetched from
  LOCALE_SDECIMAL.  DecimalSeparator must be a non-zero value.

  CurrencyDecimals - The number of digits to the right of the decimal point
  in a currency amount. The initial value is fetched from LOCALE_ICURRDIGITS.

  DateSeparator - The character used to separate the year, month, and day
  parts of a date value. The initial value is fetched from LOCATE_SDATE.

  ShortDateFormat - The format string used to convert a date value to a
  short string suitable for editing. For a complete description of date and
  time format strings, refer to the documentation for the FormatDate
  function. The short date format should only use the date separator
  character and the  m, mm, d, dd, yy, and yyyy format specifiers. The
  initial value is fetched from LOCALE_SSHORTDATE.

  LongDateFormat - The format string used to convert a date value to a long
  string suitable for display but not for editing. For a complete description
  of date and time format strings, refer to the documentation for the
  FormatDate function. The initial value is fetched from LOCALE_SLONGDATE.

  TimeSeparator - The character used to separate the hour, minute, and
  second parts of a time value. The initial value is fetched from
  LOCALE_STIME.

  TimeAMString - The suffix string used for time values between 00:00 and
  11:59 in 12-hour clock format. The initial value is fetched from
  LOCALE_S1159.

  TimePMString - The suffix string used for time values between 12:00 and
  23:59 in 12-hour clock format. The initial value is fetched from
  LOCALE_S2359.

  ShortTimeFormat - The format string used to convert a time value to a
  short string with only hours and minutes. The default value is computed
  from LOCALE_ITIME and LOCALE_ITLZERO.

  LongTimeFormat - The format string used to convert a time value to a long
  string with hours, minutes, and seconds. The default value is computed
  from LOCALE_ITIME and LOCALE_ITLZERO.

  ShortMonthNames - Array of strings containing short month names. The mmm
  format specifier in a format string passed to FormatDate causes a short
  month name to be substituted. The default values are fecthed from the
  LOCALE_SABBREVMONTHNAME system locale entries.

  LongMonthNames - Array of strings containing long month names. The mmmm
  format specifier in a format string passed to FormatDate causes a long
  month name to be substituted. The default values are fecthed from the
  LOCALE_SMONTHNAME system locale entries.

  ShortDayNames - Array of strings containing short day names. The ddd
  format specifier in a format string passed to FormatDate causes a short
  day name to be substituted. The default values are fecthed from the
  LOCALE_SABBREVDAYNAME system locale entries.

  LongDayNames - Array of strings containing long day names. The dddd
  format specifier in a format string passed to FormatDate causes a long
  day name to be substituted. The default values are fecthed from the
  LOCALE_SDAYNAME system locale entries.

  ListSeparator - The character used to separate items in a list.  The
  initial value is fetched from LOCALE_SLIST.

  TwoDigitYearCenturyWindow - Determines what century is added to two
  digit years when converting string dates to numeric dates.  This value
  is subtracted from the current year before extracting the century.
  This can be used to extend the lifetime of existing applications that
  are inextricably tied to 2 digit year data entry.  The best solution
  to Year 2000 (Y2k) issues is not to accept 2 digit years at all - require
  4 digit years in data entry to eliminate century ambiguities.

  Examples:

  Current TwoDigitCenturyWindow  Century  StrToDate() of:
  Year    Value                  Pivot    '01/01/03' '01/01/68' '01/01/50'
  -------------------------------------------------------------------------
  1998    0                      1900     1903       1968       1950
  2002    0                      2000     2003       2068       2050
  1998    50 (default)           1948     2003       1968       1950
  2002    50 (default)           1952     2003       1968       2050
  2020    50 (default)           1970     2003       2068       2050
 }


Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: bacter on November 23, 2009, 01:21:45 AM
The problem is not the output (floatToStr). It might be ok that local settings of the server is used to format floating point values.
The problem is in the input, as macro does not use typed variables:
Suppose if you save in a #var the time: 40142,12345 with {.set|#t1|%now%.}
Then you want to calculate a time t2 32 hours or 1,5 days later.
In some systems you then should write: {.set|#t2|{.add|#t1|1,5.}.} and in another {.set|#t2|{.add|#t1|1.5.}.}

So it seems to me, the only solution is to make a full check if a varible is a number:

If first caracter is a digit, repeat for each of the following caracters if its a digit or a decimal separator ,(comma) OR .(point).
If a decimal separator is found, set decimal POINT and a flag and continue until a non digit value or end of string is found.

So:   1,234       -> 1.234
        1.234,547 -> 1.234
        1,23 GB    -> 1.23
Then calculations are done and the result can be output as now, using local settings.
Title: Re: Request for changing postvar (and urlvar) behaviour
Post by: rejetto on November 23, 2009, 06:01:26 PM
HFS sets comma as separator at start, ignoring local settings.
so, that's strange.
anway, i applied a workaround for next release.
let me know if it's enough.