rejetto forum
Software => HFS ~ HTTP File Server => Topic started by: r][m on July 13, 2008, 03:37:03 PM
-
Could a delimiter character (maybe -) be added to the log similar to the example below?
And leave out the @.
(origional) 7/12/08 6:57:21 PM user@192.168.1.35:2433 Fully uploaded - 14.01 KB @ 0 KB/s
(changed) 7/12/08 - 6:57:21 PM - user - 192.168.1.35:2433 - Fully uploaded - 14.01 KB @ 0 KB/s
This would allow linking the log text file to a database for sorting things like most downloaded
file, or anything else needed. This info might eventually be available as a html file via load macro
for a stats page for remote viewing.
Apache style log is pretty lame. Aparrently it wont list uploads or downloads properly,
and does show some info of (from my point of view) questionable value.
r][m
HFS=no 3rd party !
-
at the moment, customization of log lines is supported through the apache-style log.
i can add more symbols easily if needed.
the change you are asking for the log is easily achieved through a script, python, php, anything.
here is for you a one-line solution in php, to run it you need to install php.exe (not the apache php module)
<? foreach (file('myfile.log') as $l) echo preg_replace('_(\S+) (.+ (AM|PM)) (([^@]+)@)?(\S+) (.+)_', '$1 - $2 - $5 - $6 - $7', $l) ?>
If you don't want to install php, you can try Sed (http://en.wikipedia.org/wiki/Sed), a command line utility, but it requires some syntax adjusting.
s/\([^ ]\+\) \(.\+ \(AM\|PM\)\) \(\([^@]\+\)@\)\?\([^ ]\+\) \(.\+\)/\1 - \2 - \5 - \6 - \7/g
I spent 30+ minutes on this solution, because i had to learn sed syntax. It seems quite shitty, only little better than my template macros. ;D
For your convenience, i will attach sed (the program itself)
-
So it can't be added?
Is there a configuration for apache log to show what files were
uploaded and downloaded?
So far I haven't found them listed anywhere.
Anyway, on second look, character " - " wouldn't work.
"|" would be best.
Sorry, didn't realiaze it would be a big problem to add a
seperator character.
Thanks for the reply and suggestions.
r][m
-
what's the problem with the "sed" solution?
the apache log problem is that it expects one line for one request. (am i wrong?)
uploading 3 files together is done through one request, so i could put all the files on the same line.
would this fit your needs?
-
Don't think I can cope with the sed solution.
PHP would likely be more useful, but I can't see installing it for just
this one thing.
Putting all uploaded files in one line might work if there was blank space
between each.
Apache log appears to be intended to track a web site, not a file server.
I find no info on how to set up the apache log to show uploads at all,
not just in HFS, but in general, and apache log for downloads leaves much
to be desired. I'd want to change the time format also.
The regular log format is far more useful. It does show all files uploaded
even several at once. Thats the one I'd prefer to use. If a "|" can't be
used to define sections, what about putting a blank space between them?
r][m
-
Actually, I could make this work well enough to be of some use if there
were just a blank space added before and after the @ like
"user @ 192.168.1.35:1443"
Is that possible?
Or...does anyone know of a log analizer prog that will read the regular
log? So far I've found none?
r][m
-
Don't think I can cope with the sed solution.
put the sed script in a file "script".
then run: sed -f script hfs_apache_log_file > new_format_log_file
Putting all uploaded files in one line might work if there was blank space
between each.
i could add a symbol that will expand to such list, so that you include this symbol in your apache-style log
I find no info on how to set up the apache log to show uploads at all, not just in HFS, but in general, and apache log for downloads leaves much to be desired.
indeed it doesn't, it would be a feature i would add (as %symbol)
I'd want to change the time format also.
it's likely you can do it by changing the sed script, but you'll have to learn regular expressions (if you don't know them yet)
The regular log format is far more useful. It does show all files uploaded even several at once. Thats the one I'd prefer to use. If a "|" can't be used to define sections, what about putting a blank space between them?
at the moment the "regular" log cannot be customized, but you can build a solution similar to the one i suggested above.
if you don't have necessary knowledge you can ask a programmer friend.
-
This would allow linking the log text file to a database for sorting things like most downloaded
file, or anything else needed.
I think it's problem for database, not HFS. What if rejetto adds deliimters and next days someone says that he needs to feed it to different database and they must be removed for that? :)
What db and how are you trying to use exactly? In my opinion regexp solutions from above are pretty fine (except I'd use languages I am more faimiliar with but that's going to be few lines of code anyway).
-
i could add a symbol that will expand to such list, so that you include this symbol in your apache-style log
If you will do this I'd be happy to give it a try.
Rarst
Thanks for your reply.
The problem is with hfs log. Most db prog I've used like M$ Access will link to
external data. Most I'm familar with use pretty much the same criteria to delimit
or seperate the sections for sorting such as "blank space" or various characters
that the developer may choose to use out of necessity. This criteria is then applied
to all data. Is not selectable by block.
This link is dynamic, so that any time a query or report is opened what you
see is up to date as of that moment. I have this type of setup running for other
types of programs that output logs or other data.
Trying to use something like sed to convert to a useable format would likely be static.
Calling sed by script from the db is possible, but in my experience, once the db link is
activated, the file linked to is locked. Once the link closes the log up dates on next hit.
Obviously the db can't be left open, but then it doesn't need to be.
Addition of a space or character to delimit shouldn't impact those who log hfs by other
criteria like user or date.
So far I've found only one log analyzer program that will read hfs's apache log and
its static. On a scale of 1 to 10 its usefullness compared to what I'm trying to do
is a 3 !
r][m
-
i could add a symbol that will expand to such list, so that you include this symbol in your apache-style log
If you will do this I'd be happy to give it a try.
Rarst
Thanks for your reply.
The problem is with hfs log. Most db prog I've used like M$ Access will link to
external data. Most I'm familar with use pretty much the same criteria to delimit
or seperate the sections for sorting such as "blank space" or various characters
that the developer may choose to use out of necessity. This criteria is then applied
to all data. Is not selectable by block.
Reading by separator only seems to be very limited functionality. I am not experienced in databases but most of programming languages have regular expressions support that allows highly customizable actions with text.
Addition of a space or character to delimit shouldn't impact those who log hfs by other
criteria like user or date.
Spaces won't work because they are possible in URLs and it is going to ruin reading by separators. Other characters are going to add clutter.
-
Reading by separator only seems to be very limited functionality. I am not experienced in databases but most of programming languages have regular expressions support that allows highly customizable actions with text.
Reading by delimiter character or even a blank works very well. I'm already doing this with data
output from various log files and data out put like down time from machines, and yes your are right,
the db has customizeable actions with text, and even via programing language, but like hfs
aparrently,theres only so much you can do.
Spaces won't work because they are possible in URLs and it is going to ruin reading by separators. Other characters are going to add clutter.
Spaces would work, because if you'll look, they already exist in the hfs log in places and I'm
presently using that method. One problem area is the @ that joins ip and user in the regular log.
I'm sure you must know that in the apache style log you can format it with delimiter now, as
%h | %u | %t | \"%r\" | %>s | %b | %{user-agent}i | "%{Referer}i -> %U
Regards
r][m
-
in next build you will be able to specify %z to get extra information about uploaded files.
What exactly are your needs about it?
i'm going to print filenames separated by pipes, but only those who have been successfully uploaded.
we can parameterize this by using the %{parameters}z standard syntax.
-
rejetto , just a question about uploaded files
actually , if we upload a very big file and if the file can not be stored, we must wait the end of upload before know this.
Is there a possiblity with the header to compare the upload-filename with hfs-internal mask, and stop the current upload for this file only, if the filename is forbiden, then ask to the next upload file sended by the browser?
-
Continued testing indicates apache log works best.
What would work best, is log output that indicates real downloads and
uploads, different from the pages. Kinda like the "fully uploaded/downloaded"
in the regular log.
Would it be possible to add %{format}t and maybe failed logins and bans to apache?
Also, could someone check the ban to regular log? It doesn't seem to be working here.
-
Is there a possiblity with the header to compare the upload-filename with hfs-internal mask, and stop the current upload for this file only, if the filename is forbiden, then ask to the next upload file sended by the browser?
not with the current method, based on a single http request.
in the future i can use something else, based on javascript or flash.
-
What would work best, is log output that indicates real downloads and uploads, different from the pages. Kinda like the "fully uploaded/downloaded" in the regular log.
that's quite possible, indeed %z will show you only fully uploaded, and i can make it show an information about the file being fully downloaded
Would it be possible to add %{format}t and maybe failed logins and bans to apache?
can you point me to documentation of the format?
Also, could someone check the ban to regular log? It doesn't seem to be working here.
can you tell me how to reproduce the problem?
-
can you point me to documentation of the format?
These probably won't be very helpful but its where I read of this.
http://www.cs.washington.edu/lab/sw/apache/manual/mod/mod_log_config.html (http://www.cs.washington.edu/lab/sw/apache/manual/mod/mod_log_config.html)
http://publib.boulder.ibm.com/httpserv/manual60/programs/rotatelogs.html (http://publib.boulder.ibm.com/httpserv/manual60/programs/rotatelogs.html)
can you tell me how to reproduce the problem?
I'm testing with my LAN Ip. I get the "Banned" page, and I have Banned checked in the
"Log What" menu, for regular log, but there was no log entry of the occurance. I'm using
my own template.
Can't think what else to check at the moment.
-
Log file update
I found a free log analyzer prog that works, then I found out why the
others I tried didn't. It was a apache log thing, not HFS.
I'm using WebLog Expert Lite from
http://www.weblogexpert.com/ (http://www.weblogexpert.com/)
Apache log must be like this.
%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"
Its out put is .htm so it can be included as a page, but it must be
updated manually. It shows a great deal of info but not user names, just IP.
Found another - Analog 6.0 - that is extremely configurable, and will show
and chart user.
I plan to test others.
Now I wonder if new symbols are added in apache style log, will these progs be able
to use them?
The bad news is that these analyzers need the quotes (") to work
so I'm stuck with a string like "GET /Main/ HTTP/1.1". The data base that I'm
also working with (Access) uses quotes in its language so it wont break it down furthur.
Looking for a Access work around for that.
-
i think today i will publish the new beta build, with enhanced apache log support
Posted on: 24 July 2008, 12:45:13
r][m check the new build and let me know
-
r][m check the new build and let me know
%>s "%{Referer}i" and "%{User-Agent}i" didn't work in 193 ?
All I seemed to get from %z was - No uploaded file info
Maybe I don't understand how to use %z?
Went back to 192 for the moment and everything works again.
-
sorry, recheck with latest one
-
sorry, recheck with latest one
No problem, part of developing and testing I think ;D
Apache log format in 194 works as
%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %z
Putting %z last allows log analyzer progs to still work. Any
where else in the format causes problems. Of course log analyzer prog
doesn't recognize %z, but Analog 6.0 can use a custom log format, so...I'll work on it,
db doesn't care as long as it finds a space.
Results look like:
192.168.1.140 - user [27/Jul/2008:13:01:56 -0500] "POST /Server/Up_Load/ HTTP/1.1" 200 550 "http.//192.168.1.140:xxxx/Server/Up_Load/~upload" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Avant Browser)" Blue Oyster Cult- Don't Fear The Reaper.ape
Thanks
r][m
-
I notice HFS only returns Status 200 in apache log, even though
a 401 or 403 error page had been returned ? (testing on lan)
Is this expected behavior, or would it be possible for HFS to log
errors?
r][m
-
it may be related to the "only served requests" in the log menu.
check it.
-
it may be related to the "only served requests" in the log menu.
check it.
Sorry, should have known that.
It works now. Many thanks.