Messages

1

router & port problems / Re: Routers & ports firewalls - Oh, my!

« on: June 12, 2015, 11:05:29 AM »

Download via your hfs server link = .5 MB/s
Download via your ftp to my ftp = 1.6 MB/s

Thanks for the results! It may have been just "a fluke". Or it may actually be that HFS has a limitation. Or it may be that the web browser has a limitation (for example, Chromium). Both FileZilla and HFS are running on my computer in the same conditions. I swear by FileZilla myself and use it for reference (the buffer size is a very extreme example and not necessarily "correct"). FileZilla can also push gigabit speeds on LAN on Pentium-4 grade hardware. 1.6 MB/s looks about right.

Problem with FileZilla and "users" is that sometimes FTP isn't accessible for them. They try PORT mode with local IPs, request weird directories because they can't parse the file listing, have problems with special characters in file names, and so on. Or they "distrust" an FTP server. This is where a webserver comes in.

I am getting 127 ms to you today, but the value was around 145 when I looked at it on the day the test was carried out. I am in Latvia, which is approximately 10 ms from kst5-speedtest-1.tele2.net (note that the main address is AnyCasted, but by default resolves to kst5 for me). My path towards Verizon goes through "retn.net" and "ntt.net".

C:\>tracert 108.53.107.XXX

Tracing route to pool-108-53-107-XXX.nwrknj.fios.verizon.net [108.53.107.XXX]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  router.j7nh [192.168.15.254]
  2    <1 ms     2 ms    <1 ms  10.91.104.254
  3    <1 ms    <1 ms    <1 ms  te-2401-2-72-94.balticom.lv [82.193.72.94]
  4    <1 ms    <1 ms    <1 ms  balticom-193-69-199.balticom.lv [82.193.69.199]

  5    <1 ms    <1 ms    <1 ms  ge-te-s5-64-254.balticom.lv [82.193.64.254]
  6     4 ms     1 ms     1 ms  te1-2.501.tv.riga.globalcom.lv [85.254.1.135]
  7     3 ms    33 ms     1 ms  ae0-504.rt.rtc.rix.lv.retn.net [87.245.242.25]
  8    29 ms    28 ms    30 ms  ae0-4.rt.irx.fkt.de.retn.net [87.245.233.74]
  9    29 ms    29 ms    29 ms  213.198.77.213
 10     *       35 ms    32 ms  ae-2.r20.frnkge04.de.bb.gin.ntt.net [129.250.5.217]
 11   150 ms   119 ms   114 ms  ae-7.r22.asbnva02.us.bb.gin.ntt.net [129.250.3.20]
 12   120 ms   115 ms   115 ms  ae-44.r06.asbnva02.us.bb.gin.ntt.net [129.250.6.113]
 13   123 ms   130 ms   122 ms  ae-0.verizonbusiness.asbnva02.us.bb.gin.ntt.net [129.250.8.34]
 14     *        *        *     Request timed out.
 15     *        *        *     Request timed out.
 16   126 ms   127 ms   127 ms  pool-108-53-107-XXX.nwrknj.fios.verizon.net [108.53.107.XXX]

OOKLA opens 6 to 8 connections. You can observe that if you run TcpView. If you go to FileZilla Client's settings and set "maximum simultaneous transfers" to 8, and queue this many large files, the speeds would match up. And you would be able to set the length of the test by your own choosing.

20 Mbit to Australia cannot be possible over 1 connection. No way.

Tele2 are all close to me. The site says that, "TCP windows have been slightly tweaked to support higher throughput."

You could also ask over at the TestMy.net forums I linked. They seem to have the expertise but only seem to concern with web browsers, which have too many variables in them. I have shared my experience with FTP/FileZilla and Tele2 over there.

I'm very pleased with the tele2 service. I was also able to benchmark my FileZilla Server by initiating a FXP transfer between it and their box. No other site offers this flexibility.

I'm looking forward to a follow up when you get time.

2

HFS ~ HTTP File Server / Re: Uploading: Connection Reset - (1) (2) (3)

« on: June 12, 2015, 10:20:44 AM »

I feel stupid now, because I failed to reproduce the problem on Computer 2. 

I was able to upload both large and many small files at once without an issue, from the problematic computer. I copied over the exact configuration (INI file), except that I created a new VFS. But I also attempted to start over with a new VFS, and the problem persisted. I noticed that the size of the part-files that HFS creates is proportional to the total file size. Larger files usually get 256-512 KB up, but files smaller than that are still incomplete.

It's likely that I have an incompatible setting somewhere in the network drivers. TcpIp/AFD on both PCs have been tuned by myself with similar settings. Nothing obvious is standing out.

Uploading hundreds of small files in FileZilla (many connections per second) does not reveal a problem. I'm pleased that HFS works in Opera browser, even with the high level of interactivity that it has in file listings and forms.

3

HFS ~ HTTP File Server / Re: Uploading: Connection Reset - (1) (2) (3)

« on: June 10, 2015, 04:59:16 AM »

Thank you for the reply.

I installed HFS on one computer and tried to connect to it from two other client computers. By upload form, I meant the HFS' built-in web interface through which uploading happens. All computers happen to be running WinXP but used different browsers.

The results I got you can see here. (link removed)

For example, the file "08 - Mean Old Frisco.flac" never finished uploading. There are 8 parts. "36_Goodnight_Girl_(Adam_Street).mp3" eventually did upload in the (2) part, which is a complete file.

I've tried unticking "Experimental high speed handling" and restarted, which made no apparent difference.

I will try to put HFS onto another computer, and maybe we'll learn something more.

4

router & port problems / Re: Routers & ports firewalls - Oh, my!

« on: June 10, 2015, 04:34:46 AM »

Thank you for confirming that these values apply to Win Seven.

1 Megabyte is a rather large buffer and shouldn't be needed to reach these speeds. It seems that "something" is introducing latency into the connection. I don't have a ready explanation. I would keep the buffers in "reasonable" size, because otherwise some applications may start to report inaccurate, fluctuating speeds or register a timeout in extreme cases, as they fill the whole buffer at once and it then, for whatever reason, cannot drain for a while.

The formula for getting the buffer size is: Throughput = buffer / latency. Typical latency for East Coast to "deep" Europe is 120ms. West Coast to Europe would be approx. 200ms.

My system-wide settings are: receive 256960 (0x3ebc0), send 393216 (0x60000), and I found them to work generally well.

256960 / 0.120 = 2,141,333 (2 MB/s).

I've read conflicting information as to what the send buffer size should be rounded to (MSS or memory page). I think it should be rounded to full kilobytes because it is impossible to match it to MTUs of all users (some may negotiate full 1460 segments, others may connect via DSL, or some other VPN and have header overhead). CurrentControlSet, as a shortcut, points to the registry branch selected at boot time. If you put the value in a different "set" it will have no effect.

Maybe DynamicSendBufferDisable could help?

I would proceed with a process of elimination to determine if it's win7, the router, or the ISP that needs to be "fixed". Take a stable working computer with 2000/XP/2003, and no software that can slow it down, adjust the two window size parameters, and measure the speed. As the last step, connect the PC directly to the ISP's network (if possible), and try again.

As for the security of XP, I think it's okay if you purpose build a computer to do one or a few services only, and shut down everything else.

So far, only one user downloaded my test file:

19:52:26 213.114.xxx.xx:63966 Fully downloaded - 625.8 M @ 5.6 MB/s - /speedtest/656203776.iso

Our ping was approx. 17ms. Which means that the send window was at least 99824 bytes. I was afraid that HFS may have capped it smaller.

5.6 * 1024 * 1024 * 0.017 = 99824

The speed was definitely limited by congestion on my end at this hour (7PM). Downloads from America or Australia would give more useful statistics about the buffer.

5

HFS ~ HTTP File Server / Re: Multiple Websites with HFS like Vhost in apache

« on: June 05, 2015, 04:36:53 PM »

This starts 2 servers on different port numbers. They are more reliable: if one instance crashes, the other continues running.

But. Problem is that if a visitor doesn't want to type the port number or forgets it, he gets onto an unrelated web page.

6

HFS ~ HTTP File Server / Uploading: Connection Reset - (1) (2) (3)

« on: May 29, 2015, 11:26:56 AM »

I am currently testing the HFS server v2.3e.293. So far I it appears to beat Lighttpd in speed 10 MB/s vs 5 MB/s over Fast Ethernet. Lighttpd is really slow on Win32. I do not know the reason. HFS on the other hand uses more CPU.

I encountered a serious problem uploading files using the built in form. I am frequently getting a generic "connection reset" message from the browser. When this happens, several files are created on the server named with suffixes (1), (2), (3) .. (8). Only the last file in the sequence may be complete. If that is so, I get a success report from HFS. In case of connection reset, the files remain on the server, but none of them are complete.

1. test on the same computer

Connection reset immediately in Opera.
Firefox and Chromium works for the first file uploaded.
If I click the Back button on the success form, and try to upload another file, a few duplicates are created. If I reload the tab with the directory, I can upload 1 file successfully again.

2. test on another computer connected via Gigabit Ethernet

Neither Opera nor Chromium work. Connection reset immediately. (6) to (8) files are created on the server for each upload. Each partial file is 261,319 - 261,323 bytes long, and contains the beginning of the data.

3. test on another, older computer connected via Fast Ethernet

Uploading works in Opera most of the time, even if I click Back. Occasionally I get a "reset".

4. test on the older computer again. Bandwidth is throttled to 5 Mbit using a simple queue on the router.

All uploads are successuful. I can press the Back button and upload more files.

This sounds like some kind of a race condition in the program, and only occurs if the latency and bandwidth are excellent. It may be that my TCP network setup has a problem. But no other program is impaired to the same extent. FileZilla works. No connections were dropped while I used HFS. All downloads from HFS were successful. Only uploading has the problem.

7

router & port problems / Re: Routers & ports firewalls - Oh, my!

« on: May 29, 2015, 02:33:46 AM »

This is an interesting thread. I have been tackling this problem in the past, and have not completely solved it for HTTP servers. (Lighttpd is dog slow.)

Thank you for the link to speedtest.tele2.net I will make good use of it. Most other sites use heavy HTML or Flash, which might be the bottleneck. Another good site for download only is LeaseWeb bins. My ping to tele2 (Sweden) is 9ms, and I'm getting between 9 and 10 MB/s upload. What is your ping to it? What is your ping to the client whose address is obstructed?

TCP has two windows, the receive window and the send window (or congestion window). They are controlled by the client and server respectively. The amount of data in transit is limited by the smallest of these values. You have no direct control over the RWIN of your clients. They could choose to allocate more or less memory, enable or disable auto tuning. If the client has a receive window of 64K, he will download slowly regardless of your tunings. Your RWIN is only a factor when you download from overseas or when clients upload to you. The most you can do is control your transmit buffer.

The send window is set by the operating system and can also be limited by the application when it creates a socket. (For example, FileZilla Server has an option for "Socket buffer size"). I do not know how HFS manages this, because I've just started using this program.

In Win2K/XP the OS setting is (I am unsure if it has an effect in later versions):

Code: [Select]

REGEDIT4

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters]
"DefaultReceiveWindow"=dword:0003ebc0
"DefaultSendWindow"=dword:00040000
In WinNT 6 Vista/Seven/2008 the setting is:

Code: [Select]

netsh int tcp set global congestionprovider=ctcp
In WinNT 6.1 another setting controls how rapidly the send window can expand:

Code: [Select]

netsh interface tcp set supplemental template=custom icw=10 enablecwndrestart=disabled
netsh interface tcp set supplemental template=custom

All wan - intranet speed tests are showing 59 Mb/s download & 62 Mb/s upload.
Hard wired lan/router 10/100 set at 100.

This looks correct. Your router can pass through 60 Mbit as proven by speedtest.net. Although, it possible that a "duplex mismatch" could occur if you forced only one end of the link to a certain speed. Or, if you left the LAN at 1 Gbit, the traffic could get bursty reaching a gigabit speed for short periods of time and overwhelming the receive buffer in the router. Best results will be achieved if the whole system is either FE or GbE. It is not happening here, because the OS never reaches a high enough upload speed.

Maybe try temporarily disabling any security software like firewalls and antiviruses and test again. What was the socket buffer size in FileZilla when you tested the upload to tele2?

I was about to call and update to a gigabit router BUT than I stumbled across 46 pages of............

This is rather odd. A fiber connection should allow buffers with window scaling to reach good overseas speed. If I understand this correctly, the current RWIN and scalefactor is part of the packet which a firewall/router can read and react to. But the send window, which limits your upload, is not explicitly transmitted. It can only be inferred from the rate at which packets leave your server.

More than half the lan is on a switch (dynamic), while the server(s) and a few other work horses are static. Some of the machines have 10-100 nic's and a few have gigabit nic's. The work horses of course.

As a last resort, maybe you could run your server on a WinXP box, if that gets more stable speed. Does any other machine get a better speed to distant hosts (europe, australia)?

"It is amazing how the "Support Engineers" can't answer a smiple question: Can you manually set the RWIN or TCP Window Size on Vista 64? If so, how?"

I think they do not want to admit that the "Next Generation TCP Stack" is in some aspects worse, which is normal with any technology, and suggest to users to stay with 2000/XP/2003. They would rather tiptoe around the issue and deflect it on some other factor.

I personally don't have access to such speeds over here. But it's interesting that more HFS users with similar speeds, make further tests.

I have shared a file in both HFS and FileZilla, which is known to be working fairly well (expected speed 1MB-4MB depending on the time of day).

---- links removed ----

8

HFS ~ HTTP File Server / Re: Multiple Websites with HFS like Vhost in apache

« on: May 29, 2015, 01:01:59 AM »

Please consider adding support for VHOSTS maybe as multiple root / directories where one of them is a default. I could be wrong, but looks like the existing framework of the "Virtual" File System could be utilized to implement it.

Virtual Hosts is a rather basic function of a webserver, more important than uploading or interactive file listings with icons.