rejetto forum

Software => HFS ~ HTTP File Server => Bug reports => Topic started by: Falcon4 on October 20, 2008, 03:18:09 PM

Title: memory usage with update check
Post by: Falcon4 on October 20, 2008, 03:18:09 PM
I didn't see it in the changelog, so I figure I should mention it here.

I use HFS to serve files from my dead-but-still-serving-old-files file hosting site, "hostfile.org" (go there and request a file, notice it redirects to hostfile.org:81), on my home DSL connection. HFS serves often over 1-2 requests per second, and I think that puts me in a bit of a unique situation to test long-term stability of HFS. ;)

HFS gives me the unique and incredible ability to see requests processed in realtime, and kill ones that are tying up the server. However, I've noticed one recurring problem. Memory usage sometimes hits a "snag" and spirals out of control.

I checked on the server this morning and found memory usage had doubled and began swapping like mad. HFS, as usual, was the cause of the problem. I activated HFS from the taskbar and found it asking me if I wanted to update to the latest version. HFS had been running stable for the past 4 or 5 days straight before this. This leads me to believe that the update check routine causes some sort of memory leak while it's got that dialog open. I'm all up for new versions and being automatically notified, but not if it causes the server to crash and burn.

I ended up locking up the server while trying to kill some processes and regain control of it (by cleanly shutting down HFS instead of terminating it), but when I set mmc.exe to Realtime (oops), to stop my Folding@Home service, which eats up a good chunk of memory, it crashed and I had to reboot. *ugh*

So just figured I'd point out that little leak... maybe take a look into that? Hey, other than that, thanks for providing Windows with such a great tool! :)
Title: Re: memory usage with update check
Post by: rejetto on October 20, 2008, 03:33:41 PM
Hi, and welcome.
I moved your post because it seemed to me unrelated to that specific build.
How many times did this happen? Over what time span?
Title: Re: memory usage with update check
Post by: Falcon4 on October 20, 2008, 06:07:14 PM
Ugh, it looked like when you moved it, I couldn't reply as a guest anymore, so I registered. It didn't tell me my password needed to be over 8 characters (my normal one is 7 and perfectly secure), which is a little annoying as well. Then I came back and noticed that one of the two buttons was actually "reply". *facepalm*

Anyway, it's happened a few times now, and each time, now that I've seen it for maybe a third time, it had the "You are invited to use the new version" message on the screen. So I'm inclined to think it has something to do with the update mechanism, or maybe the message... or something to do with the fact that it pops up, and memory usage goes up ;)
Title: Re: memory usage with update check
Post by: rejetto on October 20, 2008, 08:07:59 PM
A lot of memory...200MB ?

Anyway, you can download this file (http://www.rejetto.com/hfs/hfs.updateinfo.txt) in your hfs folder. By editing it, you can fake an update and test. If you can master the problem, it's likely i may fix it.

It's strange other people didn't report a similar problem. The difference may be on your system, or maybe in the fact you waited so long to answer to the dialog.
Title: Re: memory usage with update check
Post by: Falcon4 on October 20, 2008, 10:39:49 PM
200... 300... 400MB... I usually catch the problem by noticing that the web server (Apache) responds to requests, but connections to HFS (port 81) time out. I then check the server and head straight for Task Manager (before even trying to activate HFS from the taskbar), and see memory usage around 850-900mb when there's only 512mb RAM in the server, and memory usage is usually around 460mb (like it is now).

I'll try dumping that file in the folder and check the server again later tonight, trying my best to mimic a real update that I didn't see until hours later. Hopefully it does the same thing. :)

edit: It may also be relevant that on the server, I use Firefox (to write this post now), uTorrent, Apache, MySQL, "Macallan Mail Solution" mail server, and access the server completely by Remote Desktop (it has no monitor)
Title: Re: memory usage with update check
Post by: rejetto on October 20, 2008, 11:12:12 PM
How long since you started noticing this problem? ("months" is accurate enough)
How long since you started using HFS 2.3 ?
Title: Re: memory usage with update check
Post by: Falcon4 on October 20, 2008, 11:34:12 PM
I moved the site over to my home server less than a month ago (beginning of October), and noticed problems immediately, regarding the logging to screen, it does a very slow and poor job of dropping off old entries while adding new ones, often eating up 80-100% CPU (2.4GHz P4) and losing connections, after a couple hours of serving files unattended. I shut off all logging except "uploads", "other events", and "browsing" and that fixed that problem. Of course I also shut off persistent connections, that was a huge problem. I noticed the wild memory usage (with the update popup) shortly after starting to use it. But it didn't bug me much, it only happened once every few days and until today, I blamed it on being bottlenecked while 5 people were downloading large files (like FLVs) at once, and it turning down everyone's connections but maybe not releasing some memory. But today was different - after getting PO'd at FLV's tying up the connection for hours at a time, I decided to delete all FLVs, and that seems to have solved that problem (Apache handles the 410 errors by the MySQL database; links to files in HFS isn't linked directly anywhere on the internet, so I can change the URLs on the fly with zero downtime). But it still crashed this morning, which led me to realize it's probably related to that "use new version?" message.

FWIW, I noticed that memory usage has actually climbed from 460mb to 472mb with no new programs, but that margin in this amount of time isn't really in line with such a huge amount of memory being used that causes a crash. I also haven't seen it popup an update message either (I changed the latest build to "215"). Might have to wait until tomorrow, and if it does crash or hang, I'll know what to do in order to recover it. ;)

edit: This may be of some use to you:
Code: [Select]
window-max=yes
# default: no

easy=no
# default: yes

port=81
# default:

log-only-served=no
# default: yes

log-requests=no
# default: yes

log-full-downloads=no
# default: yes

tpl-file=C:\Users\Falcon\Desktop\HFS\templates\thunderchicken_of_glory-4.0.1.tpl
# default:

persistent-connections=no
# default: yes

minimize-to-tray=no
# default: yes

tray-icon-for-each-download=no
# default: yes

last-update-check=39741.4608662731
# default: 0

tray-shows=ips-ever
# default: downloads

max-contemporary-dls=5
# default: 0

flash-on=
# default: download

last-file-open=C:\Users\Falcon\Desktop\HFS\hostfile.vfs
# default:

recent-files=C:\Users\Falcon\Desktop\HFS\hostfile.vfs
# default:

trusted-files=C:\Users\Falcon\Desktop\HFS\hostfile.vfs
# default:

hints4newcomers=no
# default: yes

out-total=27352837114
# default: 0

in-total=299507688
# default: 0

hits-total=740146
# default: 0

downloads-total=376187
# default: 0

connections-columns=IP address;120|File;207|Status;180|Speed;60|Time left;55|Progress;403|
# default: IP address;120|File;180|Status;180|Speed;60|Time left;55|Progress;70|

edit edit: Oh, and by "memory usage", I'm not referencing that number for HFS' in-RAM usage, I'm referring to "PF Usage" in Task Manager, the quintessential figure of how much memory is in use on the system. :)
Title: Re: memory usage with update check
Post by: rejetto on October 21, 2008, 12:06:36 AM
Quote
I moved the site over to my home server less than a month ago (beginning of October),

that is the time when you started using HFS ?

Quote
regarding the logging to screen, it does a very slow and poor job of dropping off old entries while adding new ones,

i fear that job is not done at all :)
i understand it may sounds strange, but no one ever asked me such feature.
i will add it in next beta.
what default value do you suggest as max number of lines on screen? 5000 ?

Quote
I shut off all logging except "uploads", "other events", and "browsing" and that fixed that problem.

in next beta you may decide to re-enable something you disabled

Quote
Of course I also shut off persistent connections, that was a huge problem.

are they? pers.conn. are a good thing, as long as the system can handle them. So i would like to know more to see if i can do something about it.
Did you notice a problem with too many connections open? Like what?

Quote
it only happened once every few days and until today,

maybe because every few days i publish a new beta version :)
i'd be grateful if you persist investigating the problem, because i'd like to fix it.
when you'll be tired you can just disable the automatic check.

Quote
after getting PO'd

PO ?

---

Anyway, HFS was designed with few users in mind, no huge loads.
I'll try to improve it, for what i can, but it depends on the users feedback because i have no luck on testing.
Title: Re: memory usage with update check
Post by: Falcon4 on October 21, 2008, 12:50:56 AM
Heh, cool! I'd love to see people use HFS on a large scale. It's literally the only decent web server (on ANY platform) with real-time status information - which gives it a HUGE edge over shit like IIS or Apache. If you (or someone) can get PHP working in it - and I think I read somewhere that it'd already been done - then you've got yourself a full featured, under-one-hood, web server... and something seriously huge... on your hands ;)
(Bravo for that, by the way. Excellent freaking work. I'd've donated by now if it were actually making me any money, but I'm unemployed and dirt poor... :( )

As for logging, wow. I could've sworn I saw the list topping off and rotating off the top of the page when it reached a certain limit... maybe the log control itself was capping it off (on screen). I'd say to go for a similar number of lines as in a command prompt window, which appears to be 300 lines. 5000 is just way overkill, and would take far too much CPU power to maintain.

I would imagine you have more experience in this than I do, but if you're not... entirely... sure, I've got a basic idea of how to implement it without memory leaks through the system control or otherwise:
pointer (int) = current position in stack
stack (1) = line of log of a fixed size
stack (...) = lines of log of a fixed size
stack (300) = line of log of a fixed size
stack (301) = EOF
Write the newest log entry at the position in the log array indicated by pointer, and EOF the (pointer+1) with rollover, i.e. "2", or back to "1" if it was 301. On a timer, cycle through the log buffer (stack) to screen until it reaches EOF (no human needs to read each log entry in realtime; writing to screen on each request takes a lot more processing than just "thinking" it, then writing the screen on a timer). This way, no additional memory ever needs to be allocated to the log, which logically eliminates any possible memory leaks relating to the log ;)
(And it can also be adjustable, by setting an option for log buffer size)

As for persistent connections, I disabled that because people don't actually browse the site - they just connect to grab one file that was on some website they visited, and never need any additional files. I can see how this would come in handy if someone was browsing a website, like this forum, that actively requests content and would otherwise have to make lots of the same connections, but my site is rather different - nobody ever browses the actual site; most, if not all of the connections, are from websites with one of the files embedded from my site. Also, I think Apache is already holding persistent connections, so doing double work is just... euh. The benefit is extraordinarily minimal. But I love that my WRT54GL with Tomato can handle it without too much of a problem; its QoS is excellent. ;)

PO'd = Pissed Off :)

Oh, and yeah, I had been using HFS on my home server previously, but had no problems with it. It wasn't until I "hooked it up" to Hostfile, that the floodgates opened and I ended up straining the heck out of HFS. BTW, if you want to use Remote Desktop or VNC to check out how HFS is performing on the server, you're more than welcome to PM me for the details! :)
Title: Re: memory usage with update check
Post by: rejetto on October 21, 2008, 01:09:57 AM
(thank you for all the kind words)

Quote
If you (or someone) can get PHP working in it - and I think I read somewhere that it'd already been done

i never worked on the php part yet.
but before or later...

Quote
I'd say to go for a similar number of lines as in a command prompt window, which appears to be 300 lines. 5000 is just way overkill, and would take far too much CPU power to maintain.

not everyone logs to file. several people will rely on the log-on-screen.
i can lower the default to 2000, then you will lower it as much as you want.
when it reaches the max, it deletes the oldest 10%.

Quote
I would imagine you have more experience in this than I do, but if you're not... entirely... sure, I've got a basic idea of how to implement it without memory leaks through the system control or otherwise:

that's a classic, but i have no such control over the log widget buffer.
i didn't even found a way to delete multiple lines with a single operation -_-

Quote
As for persistent connections, I disabled that because people don't actually browse the site

ah, ok, that's clear

Quote
BTW, if you want to use Remote Desktop or VNC to check out how HFS is performing on the server, you're more than welcome to PM me for the details! :)

that would be very interesting!
Title: Re: memory usage with update check
Post by: Falcon4 on October 21, 2008, 01:22:33 AM
Heh, I haven't even gone to school or anything, so I didn't even know that was a classic ;)

I don't log to file either, I actually hate logging. I just figured logging to screen would actually take more processing power than logging to file (especially in the case of remote desktop)... I just figure that deleting a portion (after writing it) may not release all the allocated memory back to the OS. Then again, I don't really know how it's coded internally, so... hey.

There's nothing spectacular going on at the moment, but I'll PM you the VNC info and you can enjoy the fireworks show on the server as requests flash in and out of the queue in perfect symphony!
Title: Re: memory usage with update check
Post by: Falcon4 on October 21, 2008, 05:16:09 AM
Hey, I think I caught it... ever since I left the "Update info was loaded from local file" dialog untouched, I'd been watching memory usage. Memory usage has started going through the roof, and as requests are coming in, I watch Task Manager's "mem usage" for hfs.exe. It seems to increase by 16kb for each completed request!

Take a look at it on VNC if you're still around, I emailed and PM'd you the info  ;)
Title: Re: memory usage with update check
Post by: rejetto on October 21, 2008, 11:59:07 AM
I made some more testing and think the problem is in the on-screen log.
I sent you a possible test via e-mail.
Title: Re: memory usage with update check
Post by: Fysack on October 21, 2008, 05:16:09 PM
i`m madly in love with you rejetto, you just give me the rush  :-*
Title: Re: memory usage with update check
Post by: rejetto on October 21, 2008, 06:22:20 PM
here comes the rush
(http://blogs.laweekly.com/play/2008/06/13/Rush-band-1978.jpg)
Title: Re: memory usage with update check
Post by: Mars on October 21, 2008, 07:34:01 PM
Whouah, rejetto in various stages of his life :D
Title: Re: memory usage with update check
Post by: Fysack on October 26, 2008, 08:51:38 PM
ROFL  ;D ;D  hehe

No mars, he just enumerate e`m  ;)

lol, now i got it, sorry, hehe, so why did you lose the moustache rejetto  ;)