Request for changing postvar (and urlvar) behaviour

[bacter]

When sending from a form with method 'get' or 'post', values that contain curlybraces with his points or the macro expression separator '|', the values seem to be sent to the server.. but hfs clears the whole content of this values that contain {., {: and .}, :} on arriving.!

I see this when the method is 'get' at least, and experience the same when the method is 'post'.

This is really annoying when you type  or modify a chunk of text and loose it this way. I suppose that for security reasons no macros (or elements of) can be accepted in this type of data... but perhaps it would be possible to 'escape' in some way the macro delimiters. Using html char values like { do not solve the problem.
This escape could be replacing the signs in the incoming postvar for further use in some way like this:
{. --> {+., 
.} --> .+},
{: --> {+:,
:} --> :+} , etc.
| --> ¦
or this way:
{  -->  ({)
}  -->  (})
|  -->  (or)
This would allow to save {.any macro|expr.} as {+.any macro¦expr.+}  or ({).any macro(or)expr.(}), something that can never be reverted to a executable macro. This would allow to use the macro signs {:{.another macro.}:} inside documents used in a wiki, forum or any cms.

When loading such a file, then we can then replace the codes with the original signs and no macro formed this way will be executed.

Thanks for a possible solution.

[rejetto]

can't you "escape" such values with javascript?
something like
string.replace("{.", "{+.")

[bacter]

I suppose it would be possible to use javascript to do this .. but if anyone use forms with method post and don't use such a script to do the conversion, any 'error' typing those sequences by accident (its sometimes nice to use | to separate some items) clearing the postvar may be loss of data.
So anyway, also if a workaround with javascript would be possible, i think it would be better to 'escape' the offending caracter(sequences)  or to substitute them with any allowed sign (so at least, data is not lost and the user may correct his sintax).

to simplify it could be:
|  -->  ¦
{.  -->  ¦. , {:  -->  ¦:
.}  -->  .¦ , :}  -->  :¦
or this way:
|  -->  ¦
{.  -->  ¿. , {:  -->  ¿:
.}  -->  .? , :}  -->  :?
As we can assume that  ¦ is never entered directly from keyboard and has no specific use, such a substitution would eliminate the risk of data loss and would also preserve the security of hfs against introducction of malicious code.

[rejetto]

i think we should avoid inventing an encoding, giving headache to someone.
i would go for an html encoding.
&#XXX;
do you want to test it?
http://www.sendspace.com/file/8dkbh5

[bacter]

Thanks Rejetto!

Your solutions to the problems i find, are always far better than my proposals to avoid them! 

Also i don't understand why you 'escape' the points of the end macro to #46 and #58,  instead of the ending brace } to #125, your solution works perfectly!

Thank you very much!

[rejetto]

happy to know it's working. i will keep it for the next release, hoping there's no unwanted side effect

Also i don't understand why you 'escape' the points of the end macro to #46 and #58,  instead of the ending brace } to #125, your solution works perfectly!

the method is generic, and will just encode the first character.
i find pointless to distinguish between symbols.

[bacter]

Hi Rejetto

I have found another problem with caracter in postvar and urlvar.

When a postvar contains some template simbols like %user% , %total-in% they are immediatly substituted, if it contains names of sections like %upload% nothing happens and the symbol remains remains intact, with other section names like %files%, they are substituded by some partial execution: some html from the files section plus spaces and CR-LF generated by macro executions is included.

I see two possible solutions to the problem:

A) The percent sign % should be replaced by % in postvar and urlvar.

B)  Assure that a postvar dont suffer any implicit interpretation, so that we can apply {.set|data|{.replace|%|%|{.postvar|fieldname.}.}.} before writing data to a file.

PD: Don't forget the breadcrumbs-from bug (still buggy)

[rejetto]

Don't think %XXX% is for getting sections text (that's demanded to {.section.} command).
This is true only for SOME symbols, others have different meaning.
You will find a good list at http://www.rejetto.com/wiki/index.php/HFS:_Template_symbols
and you'll see there's no %upload%.

Symbols substitution is at the moment made in an initial step, before macros.
It cannot be done AFTER, because many macros uses %symbols% value to work, like {.if.}
To do it WHILE, i should change the parser deeply.
At the moment there's no way to stop %symbols% like for {:quoting:}.
I may even introduce a special way to do it, but i think that for such special case a javascript encoding should be enough.

Many of these cases, anyway, would be solved by just not having the text "expanded", but managed in a variable.
I'll try to be clearer: when you call {.postvar.} or {.urlvar.} you actually get its value made part of the output (the html), even just for a while.
If you do nothing more, you will get the value in your face, in the browser.
If you instead put such value as parameter of another command, like {.set.} or {.save.}, then it will not be displayed, but instead its used for further elaboration. That's often the case for an application like the one you are developing.

By having this value directly put in a variable, instead of being replaced in the output, we should nicely workaround the problem. And it would be even more efficient.
In the example you made, you would not need to {.set.} and maybe not even {.replace.}
Some commands may just be adapted to be able to work with variables, like i already did in the last year.

i'm sorry for the breadcrumbs, but i had truly no time lately.

this was long, but i hope it was clear enough.
next release.

[bacter]

Thanks Rejetto for yout long answer.

I understand that the % symbol is a rather complex question, as it implies different modes of treatment.
A) when applied to sections, %files% or {.$files.} for example are equivalent.
B) when applied to symbols like %user%, there is (at the moment) no alternative way to access this 'variable'.

Perhaps in a future - i know this may lead to compatibility problems with 'old' templates - this variables (from type B) could be substituted by pre-initialized variables: %user% ---> {.^user.}

You are right that the problem can be solved with a workaround in javascript. But as i want to get the wiki and forum things working initially without javascript, i will do some additional 'escaping' at the moment: .%. will be displayed as % . As the %-problem is not very destructive , this should be sufficient. I would reconsider the javascript in  future versions that uses a javascript editor instead of simple forms.

---
By the way, i don't know if there is a bug with numeric macros and real numbers, or if i don't use proper sintax: calculations (calc, add ..) dont work with real numbers ({.calc|8,7-6,4.} does not return anything {.calc|1+(5/2).} returns a real number!

[rejetto]

i guess you should try by using a dot instead of a comma, as decimal separator.

about the javascript, you should consider having the js script degrade gracefully, having all others miss the encoding, and being unable to use such character.
you may even show a warning, "please enable javascript, or you won't be able to use the % sign in your writing."
javascript should not be disabled for every site. extra-security-minded people should be able to enable on a per-site basis.

[bacter]

I guess you should try by using a dot instead of a comma, as decimal separator.

Problem with real numbers seems more complex than this:
hfs is normally started with windows. Then an integer calc operation {.calc|1+(5/2).} returns 3,5 and all operations that involve real numbers - no matter if coma or point is used - return 0 or nothing.
When i stop hfs, then restart it with running windows, the calc operation {.calc|1+(5/2).} returns 3.5 !!, and calc with real numbers with decimal point dot works, but add,sub and mul dont either.

[rejetto]

i didn't understand.
can you please tell me how to reproduce the problem?
next version, anyway, will support real numbers with add/sub/mul/div

[bacter]

Hi rejetto:

When i had problems using math functions, doubting to use decimal separator comma or point, i added the following code at the bottom of my template to test:

Code: [Select]

<br /> {.calc|1+(5/2).}
<br /> 8.7- 6.4 point:{.calc|8.7-6.4.} coma:{.calc|8,7-6,4.}
<br /> add 8.7 6.444 point:{.add|8.7|6.444.} coma:{.add|8,7|6,444.}
<br /> sub 8.7 6.444 point:{.sub|8.7|6.444.} coma:{.sub|8,7|6,444.}
<br /> mul 8.7 6.444 point:{.mul|8.7|6.444.} coma:{.mul|8,7|6,444.}
<br /> round 8.7345, 2 point: {.round|8.7345|2.} coma:  {.round|8,7345|2.}

So i observed the following results:

3,5
8.7- 6.4 point: coma:
add 8.7 6.444 point:0 coma:0
sub 8.7 6.444 point:0 coma:0
mul 8.7 6.444 point:0 coma:0
round 8.7345, 2 point: coma: 8,73

This is the output when hfs starts with windows after booting.

When i was doubting if it could be a problem of the special build i used, i tried with previous version and got the following output:

3.5
8.7- 6.4 point:2.3 coma:
add 8.7 6.444 point:0 coma:0
sub 8.7 6.444 point:0 coma:0
mul 8.7 6.444 point:0 coma:0
round 8.7345, 2 point: 8.73 coma:

So i first thought it would be a problem of the build, and returned to the newest special build. And it worked too!

Result is: After a new boot or reboot windows (shutdown and restart the computer), hfs uses the comma separator. Then i exit hfs, rerun it, and it works using the point separator!

I have checked that there is no other hfs.exe in any directory to be sure that its the same .exe is used on startup of the machine

EDIT
This is so with windows XPp SP3 , can't test it with W7 because HDD died this morning

[rejetto]

how is your hfs started at first?
is it in the "startup" folder? is it a service?

[bacter]

Rejetto:

there is a link to hfs.exe in the startup folder.

i have activated the menu-> start/exit -> run hfs when windows starts option.

I thought that perhaps it has to do with the country information that could be not ready when hfs starts (if it retrieves it ?), but the country information for spain says to use the point for thousands grouping and comma as decimal separator.
But this does not seem to be the reason, as coma is used just after booting, and is changed to point after exit and re-run.