rejetto forum

Software => HFS ~ HTTP File Server => router & port problems => Topic started by: fdiskMBR on April 11, 2015, 03:21:59 PM

Title: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on April 11, 2015, 03:21:59 PM
I'm having a speed issue with those connecting to my http file server.
Version 2.3d
Fios
Dynamic IP
Dns address
Static lan addressing
All wan - intranet speed tests are showing 59 Mb/s download & 62 Mb/s upload.
Hard wired lan/router 10/100 set at 100.
Port 80 is open in both directions not throttled by provider.
Connection via wan - intranet = 160 KB/s downloads.
hfs screen showing 160 KB/s going out.
Lan connection speeds are absolutely off the charts.
Win7 firewall allowances in/out on 80 to static addressed lan server.

I've read everything & scanned all speed related posts for the last 8 years.
What am I missing? (scratching head for a while now)

Thanks

 

Title: Re: Routers & ports firewalls - Oh, my!
Post by: bmartino1 on April 12, 2015, 01:26:34 AM
sounds like a bandwidth issue form ISP?...(internet service provider?)
*some not all ISP limit http traffic...

http://www.speedtest.net/

*run a test...
what is you upload allotted (that is the conection that the HFS server side user are on and using...)

*Try changing the port on which hfs connects.

Windows firewall should be allowing the Program, not the port!...
Windows 7 Firewall port 80 is meant for Iss...

http://www.codeproject.com/Tips/365704/Install-IIS-on-Windows

http://stackoverflow.com/questions/11238830/how-to-enable-world-wide-services-http-in-the-firewall-using-command-line
Title: Re: Routers & ports firewalls - Oh, my!
Post by: LeoNeeson on April 13, 2015, 06:16:23 AM
I'm having a speed issue with those connecting to my http file server.
Version 2.3d
You may try to update HFS to v2.3e, and using another port, like 443 (even if you are no using SSL).

May be your ISP is throttling your internet connection too. You may take a test here:
Code: [Select]
http://www.howtogeek.com/165481/how-to-test-if-your-isp-is-throttling-your-internet-connection/
Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on April 13, 2015, 01:34:01 PM
Leo:
Updated to 2.3e
No throttling issue.
See speed test pic also.


bmartino1:
Exception for hfs in firewall.
Wan test from outside reports port not open to lan addy,  when hfs starts it reports port open.

Your links refer is to IIS. Is this necessary for hfs to operate at high speed?

Thanks
Title: Re: Routers & ports firewalls - Oh, my!
Post by: bmartino1 on April 14, 2015, 07:50:15 PM

bmartino1:
Exception for hfs in firewall.
Wan test from outside reports port not open to lan addy,  when hfs starts it reports port open.

Your links refer is to IIS. Is this necessary for hfs to operate at high speed?

Thanks

Thank you for the picture for speed test, you have plenty of bandwidth for upload to host a sever.

*No, Iss is not need for HFS. I post thoses links as refferences for iis and the colation with firewall port 80...

When you first launch HFS, a firewall message should appear asking the program for firewall permission...
I believe you are on a public network, so the program needs public and private traffic...

Resources:
http://windows.microsoft.com/en-us/windows/choosing-network-location#1TC=windows-7

win 8:
http://www.7tutorials.com/change-location-network-private-public-windows-81

------------------------------

?? lan addy... is hfs set to receive traffic form all ips ie: 0.0.0.0 ???
and is  your router port forward re setup to the correct port for HFS?

try to run hfs as administrator.
Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on April 15, 2015, 02:23:16 AM
Quote > When you first launch HFS, a firewall message should appear asking the program for firewall permission...
I believe you are on a public network, so the program needs public and private traffic...

looks like private - public
Open firewall in and out with exceptions see pic

Quote > ?? lan addy... is hfs set to receive traffic form all ips ie: 0.0.0.0
No, only static lan hfs machine address

Quote > and is  your router port forward re setup to the correct port for HFS?
Port 80

Like I said, I'm missing something but can't put my finger on it.

*I have found something interesting. Since I've updated to v2.3e and am able to allow multiple file downloads at once,
the combined client download of 3 files at once is much more than 3 times faster than a single file download. 700-800 combined for 3 files at once from 3 separate wan addresses.  o0
Title: Re: Routers & ports firewalls - Oh, my!
Post by: bmartino1 on April 15, 2015, 04:03:31 PM
huh... well, i'm just about stumped, are you at least able to connect to the server as it is?
everything you've posted, and the settings are correct...

*at this point, is HFS up and runing where you can connect to it?

SO, are you still experiencing a speed issue, but able to connect?

how big is the file you are serving?
i see in a previous pic that it is an cd rom ISO image.

it might be a HFS issue with serving large files...
in PHP here are possible solutions...
HFS doesn't native-ly support php, but can marco template (such as FHFS)to launch it

http://stackoverflow.com/questions/432713/serving-large-files-with-php

http://stackoverflow.com/questions/8600843/serving-large-files-with-high-loads-in-django

*I would have you look into this website to double check windows GPO settings:
http://www.rushinformation.com/how-to-boost-and-increase-your-internet-speed/
Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on April 16, 2015, 03:58:56 PM
huh... well, i'm just about stumped, are you at least able to connect to the server as it is?
everything you've posted, and the settings are correct...

*at this point, is HFS up and runing where you can connect to it?
I only put it up as needed

SO, are you still experiencing a speed issue, but able to connect?
Yes

how big is the file you are serving?
i see in a previous pic that it is an cd rom ISO image.
Most are 600mb + zips BUT even smaller files have the same speed issue.

it might be a HFS issue with serving large files...
I don't think so otherwise the smaller files would be faster
in PHP here are possible solutions...
HFS doesn't native-ly support php, but can marco template (such as FHFS)to launch it

http://stackoverflow.com/questions/432713/serving-large-files-with-php

http://stackoverflow.com/questions/8600843/serving-large-files-with-high-loads-in-django

*I would have you look into this website to double check windows GPO settings:
http://www.rushinformation.com/how-to-boost-and-increase-your-internet-speed/

I'll look at the links you mentioned for the illusive solution  ???
Thanks
Title: Re: Routers & ports firewalls - Oh, my!
Post by: LeoNeeson on April 17, 2015, 08:07:31 AM
How about using some "Download Accelerator"? (like the good old "FlashGet", or if you are using Firefox, using the open source extension "DownThemAll!"). After seeing your screenshots, I think that should solve your problem. Just give it a try, and post here the results...

Code: [Select]
http://www.downthemall.net/
Good luck... ;)
Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on April 17, 2015, 03:04:19 PM
How about using some "Download Accelerator"? (like the good old "FlashGet", or if you are using Firefox, using the open source extension "DownThemAll!"). After seeing your screenshots, I think that should solve your problem. Just give it a try, and post here the results...

Code: [Select]
http://www.downthemall.net/
Good luck... ;)

I've never tried to see if I have a problem downloading from an hfs server from my end.
The issue seems to lie with others downloading from my hfs server from what I can see.

Now if your saying that the  "Download Accelerators" you mentioned would help others to speed their downloads from my hfs server you could be right BUT I have no control over their machines and they would need to install an "accelerator".
That idea also defeats the purpose of using hfs at all. If that were the case, I would just set up an FTP server and have them download a client instead.

If your saying that a "Download Accelerator" will speed up the downloads from my hfs server to those connecting to it, I'll gladly try it ;)

Can you be a little more specific?
Thanks.
Title: Re: Routers & ports firewalls - Oh, my!
Post by: LeoNeeson on April 18, 2015, 04:36:42 AM
A "Download Accelerator" it's a "client" thing, not a "server" thing. You don't have to install a "Download Accelerator" in your server, it's "who" downloads files from you, that should install it.

I'm 90% sure this it's a ISP issue. "ISPs often employ deep packet inspection (DPI)", so, your SpeedTests will always have awesome speeds, but then, when you use a "bandwidth intensive application, such as a server, [your ISP] might limit, or throttle, the rate at which it accepts data" (explanation taken from Wikipedia). For example, Verizon (and other ISP), often do this.

Your best bet, it's trying another application, and see the results. You may try another HTTP server, or any other program, like FileZilla (FTP). Try sharing files over FTP using FileZilla, and see if it's working better.

Code: [Select]
https://filezilla-project.org/
http://en.wikipedia.org/wiki/Bandwidth_throttling#Application
Title: Re: Routers & ports firewalls - Oh, my!
Post by: bmartino1 on April 18, 2015, 11:41:25 PM
IF you are the client(test machine to download / test HFS) person as well, the lan machine might only be open to a download connection limit of 3-5 MBs... some routers to handle the bandwidth limit this connection... so you ISP is giving you a upload of 50 x MBs, but your router can only handle as example 10 Mbs upload... that also might be the issue here...

To continue on with what Leo means is to install this software on a client(a machine that is connecting to your machine to download, or something simlar to this software get the files from HFS...

https://www.internetdownloadmanager.com/

there have been problem reports and threads with issues with accelerators and this software working with HFS...

Try to DMZ the IP for the machine hosting HFS (less secure, but sometimes is a router limit workaround...

the only other option/last thing i can think of is your nic/networking devfices are using ethernet 10 MBs, not gigabit 1000Mbs....
with a connection form "verizon fiber optic", you should be using gigabit routers/switches... is there is an Ethernet switch in between, them? the clients/upload would then be liited to a max at the Ethernet speed...

also, a clint could have poor download connection to get file...
Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on April 20, 2015, 07:09:34 PM
Leo:
I'm thinking I'll give filezilla a shot after I try Bmartino1's suggestion to DMZ the server.

bmartino1:
Didn't even cross my mind to DMZ the server ip.  :o  That should eliminate any routing issues.
The nic is a gigabit but set to 100 because of the router The router is 10-100.
Everything is on cat5e. It hasn't been tested @ 1000. From what I understand, it should handle it.
More than half the lan is on a switch (dynamic), while the server(s) and a few other work horses are static. Some of the machines have 10-100 nic's and a few have gigabit nic's. The work horses of course.

I was about to call and update to a gigabit router BUT than I stumbled across 46 pages of............
https://forums.verizon.com/t5/FiOS-Internet/Verizon-FIOS-Router-Actiontec-MI424WR-GEN3I-Issue-Dropped/td-p/668969 (https://forums.verizon.com/t5/FiOS-Internet/Verizon-FIOS-Router-Actiontec-MI424WR-GEN3I-Issue-Dropped/td-p/668969)
Maybe I'll wait a while for gigabit (scans over 46 pages) and just see what I can do with what I have that at least somewhat works. 

Thanks for the tips, I'll give them a try and post with results.
Title: Re: Routers & ports firewalls - Oh, my!
Post by: bmartino1 on April 20, 2015, 11:16:21 PM
Goign over DMZ, t depends on the router...:
see DMZ host...
http://en.wikipedia.org/wiki/DMZ_%28computing%29

*Some home routers refer to a DMZ host... You want to becareful if you use it on the lan, as some routers DMZ will elt you see it, but not comunicate to it...

example:
client > internet > ISP mode > Router > DmZ > HFS server machine

will work, but depending on router....

isp modem > rotuer > lan machine (ping DMZ IP)

Router > DMZ (is it on the same subnet? is it allowed access?) > HFS server machine blocked transmission due to DMZ..../or Connected to HFS Server machine...

Cool, well, i though i would post some helpful stuff for the FTP server...
----------------------Filezila and HFS hybrid----------------------
and/or you can use FHFS...
http://www.rejetto.com/forum/fhfs/version-2-0-0-release-download/

FHFS download:
http://sourceforge.net/projects/fhfs/files/2.0.x/2.0.0/
-------------------------------


FTP server:
https://filezilla-project.org/download.php?type=server

in a webpage, you can still use a browser for ftp / or even windows explorer...
example:
ftp:\\192.168.1.x\folder
Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on April 21, 2015, 09:41:19 PM
It would seem that I will be able to allow an individual static address DMZ even though it's on the lan.
I understand your subnet insinuation also but from what I've read there shouldn't be any issue.

We'll see..........
Title: Re: Routers & ports firewalls - Oh, my!
Post by: LeoNeeson on April 23, 2015, 03:00:03 AM
@fdiskMBR: Enabling "DMZ" is a big security risk, since you are opening all ports, and any services or vulnerabilities that exist on that machine are fully exposed to the internet. Only enable "DMZ" for testings, then disable it. When using individual "Port Forwarding", you're only opening the port you need, for the purpose of hosting a service (ie: FTP server, file server).

I would like to agree with bmartino1, but I can't. I do not recommend enable DMZ, since that will NOT solve your speed problems (enabling DMZ may help if you are dealing with BitTorrent or eMule, but not with HFS). DMZ makes your PC fully and directly open to internet, but will not make your connection faster. If your server is working (and you can transfer files with friends over the internet), it means you have the correct ports open already, so you don't need to touch DMZ.

I seriously recommend you try another HTTP server (and not HFS based), because I'm 99% sure you will get the same speed, because it's an ISP problem. If you get the same speed with another software, you can be sure it's not your internal network fault.

I do not like to do this, because HFS is better than any other HTTP server (IMHO), but you can try "Easy File Sharing Web Server". It's not free, but you can test it for free, and see if your speed problem continues.

Code: [Select]
http://www.sharing-file.com/
Good luck... ;)
Title: Re: Routers & ports firewalls - Oh, my!
Post by: bmartino1 on April 24, 2015, 11:45:29 PM
I agree with Leo on this(only use DMZ for testing!, if its works better then its a port problem...), form the sounds of your last post...(I don't think it is the ISP, it could be thought!)... I think it is due to your network still being on Ethernet Speeds / purposing limiting your LAN bandwidth on the other machines.

Practically over the internet downloading an iso image: (lets say Ubuntu iso...)
A 600 MB file will still take about 15-30 MIN maybe even longer at your current allotted bandwidth form your ISP over the internet.

BUT -- Since you intranet (the network setup that all you machine connect to before the internet) is running at Ethernet speed, a 600 MB file can take up to 1-2 hours maybe a little bit faster... there go your "speed issue"

as for a second web server test, i'm not family with the program LEO mentioned, but her is a setup for a free opwn source project --(there are many variety of it!) XAMPP Apache + MySQL + PHP + Perl (Apache web server for windows) https://www.apachefriends.org/index.html
Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on April 28, 2015, 11:17:26 AM
Getting a little more detailed it would seem that the rwin value is automatically adjusted in windows 7. (not that it works)
That presents a serious problem to the hfs server. (in the case of vista & win 7) It limits uploading speed of your hsf server no matter how fast a broadband speed you have OR how fast the clients connection to the hsf server is.

Which now has me wondering about FTP and what the fix would be in win7 or vista since it's an OS limitation.

The rwin is limited to 65535.
The rwin should be about 1045440.

As you can see, win 7 & vista have automatic throttling and a max rwin size of 65535 which limits the servers functionality.
There are "go around's" but not for the common user as they're very technical & tedious.

The information is here if anyone cares to look but I would caution you about the optimizer. If you read "carefully" you'll see that the optimizer when set to "optimal" will adjust some settings contradictory to the statements made by the writer. If you dare, you'll need to "manually" choose all the settings for proper functionality.
http://www.speedguide.net/articles/windows-7-vista-2008-tweaks-2574 (http://www.speedguide.net/articles/windows-7-vista-2008-tweaks-2574)

Microsoft is aware of the issue and has made some patch's for 2008 Server, Vista and provided information for Win 7. None of these provide the fix for to the issue. From what I can find, speedguide's Optimizer is the only real option.  See the links.

Seems the corp techs are struggling with this also > http://serverfault.com/questions/608060/windows-tcp-window-scaling-hitting-plateau-too-early (http://serverfault.com/questions/608060/windows-tcp-window-scaling-hitting-plateau-too-early)

I'm a little surprised someone has not mentioned this here as it has a direct effect on hsf and it's functionality.
Did I miss this somewhere?

 

 
Title: Re: Routers & ports firewalls - Oh, my!
Post by: LeoNeeson on April 29, 2015, 08:23:54 AM
I personally don't have access to such speeds over here. But it's interesting that more HFS users with similar speeds, make further tests. I don't know if in Italy, Rejetto has similar speeds to make tests. Anyway, like you say, this is a Win7/Vista OS limitation, not directly a HFS problem.

I was searching for RWIN Tweaking tools, and I did found some interesting links about it (I didn't test any of these, because I don't need to):

DrTCP & Tweak Test:
Code: [Select]
http://www.dslreports.com/tweaks/RWIN
http://www.dslreports.com/tools
http://www.dslreports.com/faq/8159
http://www.dslreports.com/faq/5699
http://www.dslreports.com/faq/bellsouth/5.0_Connection_and_Tweaking
https://web.archive.org/web/*/http://www.dslreports.com/drtcp

SpeedGuide TCP Optimizer:
Code: [Select]
http://www.speedguide.net/downloads.php
http://www.neowin.net/forum/topic/1026386-speedguide-tcp-optimizer-308/

SpeedGuide TCP/IP Analyzer:
Code: [Select]
http://www.speedguide.net:8080/
http://www.speedguide.net/analyzer.php

How to increase your Network Speed or Internet Speed by over 20%:
Code: [Select]
http://www.addictivetips.com/windows-tips/how-to-increase-your-network-speed-or-internet-speed-by-over-20/
TCP tuning:
Code: [Select]
http://en.wikipedia.org/wiki/TCP_tuning
Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on April 29, 2015, 01:52:07 PM
LeoNeeson:
Some of the links you posted are for earlier Windows operating systems that do not have automatically capped RWIN scaling. 

I must caution those who are following the same rabbit trail as I have here.
The information being discussed here pertains to Vista & Win 7 (32/64) and possibly newer Windows os versions. 
You must look carefully to see which operating system any link or reference here may be discussing!


"Why is it that if I have 3 simultaneous connections from 3 different clients I'm able to see 3 times the upload speed in the hfs window as I would see with just 1 individual upload"  :o
To be more specific:   
It would then seem that either there is a cap on each individual client IP download from the hfs server OR the server window is not accurately showing the speed.   ie. 1 client downloads @ 230 while  3 simultaneous clients download @ 230 each. The total 3 client simultaneous download speed being 690.
Why is a single client unable to download from the server @ 690?
I'll try to do some more detailed testing in the next week and get back to this post.

 
Title: Re: Routers & ports firewalls - Oh, my!
Post by: bmartino1 on April 29, 2015, 04:12:02 PM
in regards to rwin:

it can be edited!...
(Create a SYSTEM RESTORE!) - this can break your machine!

http://answers.microsoft.com/en-us/windows/forum/windows_7-networking/how-do-you-reduce-tcp-rwin-in-windows-7-to/a46599a0-eb4f-40dd-a9c3-301c2e3c6b33

http://www.speedguide.net/articles/windows-7-vista-2008-tweaks-2574

* http://answers.microsoft.com/en-us/windows/forum/windows_vista-networking/rwin-tcp-window-size-problem/1544522d-386b-4ff5-92b7-dff1cc3d7a22

--------example of one!-------------
QoS Reserved Bandwidth
As with Windows XP, network adapters have a "QoS Packet Scheduler" enabled by default, which reserves 20% of bandwidth by default for QoS applications that request priority traffic. Note this only has effect in the presence of running QoS applications that request priority traffic. Registry value is undocumented for the Vista version of Windows. To customize this setting, in the Windows Registry:

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\Psched
NonBestEffortLimit=0 (DWORD, not present in the registry by default. Recommended: 0 , possible values between 0 and 100) - indicates the percentage value of reserved bandwidth for QoS applications. Set to 0 to disable.

Notes: This tweak applies only to Windows versions that have Qos Packet Scheduler enabled. It will ONLY have effect in the presence of running QoS applications.(windows update...etc...)

... Good luck!

in regards to your multiple downlaods/uplaods:
It is also due to rwin in the fact on waht allocation of bandwidth due they have...

RWIN is a multiple of MSS
Other RWIN values that might work well with your current MTU/MSS:
63888  (up to 2 Mbit lines, depending on latency(ping MS and/or ttl hops). MSS * 44)
127776 (1-5 Mbit lines, depending on latency. MSS * 44 * 2)
255552 (2-14 Mbit lines, depending on latency. MSS * 44 * 2^2)
511104 (8-30 Mbit lines, depending on latency. MSS * 44 * 2^3)
1022208 (25-60 Mbit lines depending on latency. MSS * 44 * 2^4)
Title: Re: Routers & ports firewalls - Oh, my!
Post by: bmartino1 on April 29, 2015, 04:43:41 PM
http://www.speedguide.net/articles/windows-7-vista-2008-tweaks-2574

(this here is all that you mainly need to test and edit....) (open good old comand prompt -- run as administrator!)

Check the TCP/IP state
To check the current status of the Vista TCP/IP tweakable parameters, in elevated command prompt type the following command:

netsh int tcp show global

You will be presented with something like the following:



The settings, as well as their default and recommended state are explained below. The two most important tweakable parameters are "Auto-Tuning Level" and "Congestion Control Provider".

When checking the TCP state with the "netsh int tcp show global" command, it is also possible to see the following message below all those parameters:

** The above autotuninglevel setting is the result of Windows Scaling heuristics overriding any local/policy configuration on at least one profile.

It is displayed when the "Receive Window Auto-Tuning Level" is not explicitly set, or if the system deemed it necessary to make a change because of user prompted "repairing" of your network connection, for example.



Disable Windows Scaling heuristics
Windows Vista/7 has the ability to automatically change its own TCP Window auto-tuning behavior to a more conservative state regardless of any user settings. It is possible for Windows to override the autotuninlevel even after an user sets their custom TCP auto-tuning level. When that behavior occurs, it can have a very noticeable negative impact on throughput, and it does not automatically correct itself. If auto-tuning gets limited, the "netsh int tcp show global" command displays the following message:

** The above autotuninglevel setting is the result of Windows Scaling heuristics
overriding any local/policy configuration on at least one profile.

To prevent that behavior and enforce any user-set TCP Window auto-tuning level, you should execute the following command:

netsh int tcp set heuristics disabled

possible settings are: disabled,enabled,default (sets to the Windows default state)
recommended: disabled (to retain user-set auto-tuning level)

Note this should be executed in elevated command prompt (with admin priviledges) before setting the autotuninlevel in next section. If the command is accepted by the OS you will see an "Ok." on a new line.

The corresponding Registry value (not necessary to edit if setting via netsh) is located in:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Parameters
EnableWsd=0   (default: 1, recommended: 0)

Note: This automatic limitation of the TCP Window usually occurs in the presence of some packet loss, which can be common in longer transfers and server applications.



TCP Auto-Tuning
To turn off the default RWIN auto tuning behavior, (in elevated command prompt) type:

netsh int tcp set global autotuninglevel=disabled

The default auto-tuning level is "normal", and the possible settings for the above command are:

disabled: uses a fixed value for the tcp receive window. Limits it to 64KB (limited at 65535).
highlyrestricted: allows the receive window to grow beyond its default value, very conservatively
restricted: somewhat restricted growth of the tcp receive window beyond its default value
normal: default value, allows the receive window to grow to accommodate most conditions
experimental: allows the receive window to grow to accommodate extreme scenarios (not recommended, it can degrade performance in common scenarios, only intended for research purposes. It enables RWIN values of over 16 MB)

Our recommendation: normal  (unless you're experiencing problems).

If you're experiencing problems with your NAT router or SPI firewall, try the "restricted", "highlyrestricted", or even "disabled" state.

Notes:
- Reportedly, some older residential NAT routers with a SPI firewall may have problems with enabled tcp auto-tuning in it's "normal" state, resulting in slow speeds, packet loss, reduced network performance in general.
- auto-tuning also causes problems with really old routers that do not support TCP Windows scaling. See MSKB 935400
- netsh set commands take effect immediately after executing, there is no need to reboot.
- sometimes when using "normal" mode and long lasting connections (p2p software / torrents), tcp windows can get very large and consume too much resources, if you're experiencing problems try a more conservative (restricted) setting.

If you're experiencing problems with Auto-Tuning, see also:
MSKB 835400 - email issues
MSKB 934430 - network connectivity behind firewall problems
MSKB 940646 - 3G WWAN throughput issues
MSKB 929868 - web browsing issues
MSKB 932170 - slow network file transfer



Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on April 29, 2015, 04:55:49 PM
The restore point is a given. A boot disk and acronis full system backup is a necessity ;)

On the microsoft link with the MVP, the poster never replied "if it worked OR iif it didn't".  The simplicity of this answer sums it up...
"It is amazing how the "Support Engineers" can't answer a smiple question: Can you manually set the RWIN or TCP Window Size on Vista 64? If so, how?"

As far as qos goes, I see by selecting "0" it will increase bandwidth by up to 20%.
Lets see..... 230 + 20% = 276 >>>>> That's NOT 630 although I'll implement that also if it doesn't impede the Optimizer settings implied.

Thank you for the input! 

After seeing your last post:
I saw that also and it was the direction I was about to take :)
Thanks again!
Title: Re: Routers & ports firewalls - Oh, my!
Post by: rejetto on May 01, 2015, 10:28:40 PM
1. did you already look if you have something enabled in Menu > limits ?
2. did you then test ftp speed?
Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on May 02, 2015, 02:58:43 PM
1. did you already look if you have something enabled in Menu > limits ?
2. did you then test ftp speed?

No limits set in menu.
No "FTP" transfer test, http transfer tests have been shown in this thread.

I'll detail and take pictures of "FTP" speeds and "http transfer speeds" along with differences each windows setting makes when I get time to focus on this issue only.
Like you, my time is also limited.

Thank you for your software work.

Title: Re: Routers & ports firewalls - Oh, my!
Post by: rejetto on May 03, 2015, 09:24:55 AM
ok, i didn't mean to focus on ftp, as any single-stream tcp test would be ok to me (torrent isn't single-stream for example).
I wanted to know if a download test was made with a different software than HFS. I didn't notice any such result reading your posts but i may have overlooked.
Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on May 11, 2015, 01:54:14 PM
Very frustrating.
Tweaked win 7 pro.
Same result on hfs and a like result with ftp which makes me believe it's not an hfs issue.

There's something very wrong here but I just can't put my finger on it yet.
I'll get back to this again when I have more time.

Thanks for the help.


Title: Re: Routers & ports firewalls - Oh, my!
Post by: j7n on May 29, 2015, 02:33:46 AM
This is an interesting thread. I have been tackling this problem in the past, and have not completely solved it for HTTP servers. (Lighttpd is dog slow.)

Thank you for the link to speedtest.tele2.net I will make good use of it. Most other sites use heavy HTML or Flash, which might be the bottleneck. Another good site for download only is LeaseWeb bins (https://www.leaseweb.com/platform/network). My ping to tele2 (Sweden) is 9ms, and I'm getting between 9 and 10 MB/s upload. What is your ping to it? What is your ping to the client whose address is obstructed?

TCP has two windows, the receive window and the send window (or congestion window). They are controlled by the client and server respectively. The amount of data in transit is limited by the smallest of these values. You have no direct control over the RWIN of your clients. They could choose to allocate more or less memory, enable or disable auto tuning. If the client has a receive window of 64K, he will download slowly regardless of your tunings. Your RWIN is only a factor when you download from overseas or when clients upload to you. The most you can do is control your transmit buffer.

The send window is set by the operating system and can also be limited by the application when it creates a socket. (For example, FileZilla Server has an option for "Socket buffer size"). I do not know how HFS manages this, because I've just started using this program.

In Win2K/XP the OS setting is (I am unsure if it has an effect in later versions):

Code: [Select]
REGEDIT4

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters]
"DefaultReceiveWindow"=dword:0003ebc0
"DefaultSendWindow"=dword:00040000

In WinNT 6 Vista/Seven/2008 the setting is:

Code: [Select]
netsh int tcp set global congestionprovider=ctcp
In WinNT 6.1 another setting controls how rapidly the send window can expand (http://smallvoid.com/article/winnt-tcp-slow-start.html):
Code: [Select]
netsh interface tcp set supplemental template=custom icw=10 enablecwndrestart=disabled
netsh interface tcp set supplemental template=custom


Quote from: fdiskMBR
All wan - intranet speed tests are showing 59 Mb/s download & 62 Mb/s upload.
Hard wired lan/router 10/100 set at 100.
This looks correct. Your router can pass through 60 Mbit as proven by speedtest.net. Although, it possible that a "duplex mismatch" could occur if you forced only one end of the link to a certain speed. Or, if you left the LAN at 1 Gbit, the traffic could get bursty reaching a gigabit speed for short periods of time and overwhelming the receive buffer in the router. Best results will be achieved if the whole system is either FE or GbE. It is not happening here, because the OS never reaches a high enough upload speed.

Maybe try temporarily disabling any security software like firewalls and antiviruses and test again. What was the socket buffer size in FileZilla when you tested the upload to tele2?

Quote
I was about to call and update to a gigabit router BUT than I stumbled across 46 pages of............ (https://forums.verizon.com/t5/FiOS-Internet/Verizon-FIOS-Router-Actiontec-MI424WR-GEN3I-Issue-Dropped/td-p/668969)
This is rather odd. A fiber connection should allow buffers with window scaling to reach good overseas speed. If I understand this correctly, the current RWIN and scalefactor is part of the packet which a firewall/router can read and react to. But the send window, which limits your upload, is not explicitly transmitted. It can only be inferred from the rate at which packets leave your server.

Quote
More than half the lan is on a switch (dynamic), while the server(s) and a few other work horses are static. Some of the machines have 10-100 nic's and a few have gigabit nic's. The work horses of course.
As a last resort, maybe you could run your server on a WinXP box, if that gets more stable speed. Does any other machine get a better speed to distant hosts (europe, australia)?

Quote
"It is amazing how the "Support Engineers" can't answer a smiple question: Can you manually set the RWIN or TCP Window Size on Vista 64? If so, how?"
I think they do not want to admit that the "Next Generation TCP Stack" is in some aspects worse, which is normal with any technology, and suggest to users to stay with 2000/XP/2003. They would rather tiptoe around the issue and deflect it on some other factor.

Quote from: LeoNeeson
I personally don't have access to such speeds over here. But it's interesting that more HFS users with similar speeds, make further tests.
I have shared a file in both HFS and FileZilla, which is known to be working fairly well (expected speed 1MB-4MB depending on the time of day).

---- links removed ----
Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on June 09, 2015, 04:20:42 PM
Thanks J7n

It would seem that the issue is within my machine and or router.
I have not had time to research further (just too busy) but your comment and info is helpful.

I did notice a change when increasing the send value. Speeds increased from 180kbs to 500kbs in both FTP and http transfers.

The location of this setting is as follows:
Regedit
Hkey_Local_Machine
System
ControlSet001
services
AFD
Peramiters

Change DefaultSendWindow value to what's needed.
If it's not there make a new D word name it DefaultSendWindow
put the number in.

I had changed the default from 261360 to 1045440.
A substantial change & an increase in transfer rate BUT obviously my upload-download speeds in both FTP and HFS should be in the 40 to 50 Mbs range rather than 500kbs on my service. My service is also not throttling as I've tested it.

I've tried local servers, checked hops and other defeating situations. No issues.
I've had others clock their up-downs with the same services and theirs are much much faster. It's an issue on my end that needs to be found.

Thanks again
Title: Re: Routers & ports firewalls - Oh, my!
Post by: j7n on June 10, 2015, 04:34:46 AM
Thank you for confirming that these values apply to Win Seven.

1 Megabyte is a rather large buffer and shouldn't be needed to reach these speeds. It seems that "something" is introducing latency into the connection. I don't have a ready explanation. I would keep the buffers in "reasonable" size, because otherwise some applications may start to report inaccurate, fluctuating speeds or register a timeout (https://forum.filezilla-project.org/viewtopic.php?t=15301) in extreme cases, as they fill the whole buffer at once and it then, for whatever reason, cannot drain for a while.

The formula for getting the buffer size is: Throughput = buffer / latency. Typical latency for East Coast to "deep" Europe is 120ms. West Coast to Europe would be approx. 200ms.

My system-wide settings are: receive 256960 (0x3ebc0), send 393216 (0x60000), and I found them to work generally well.

256960 / 0.120 = 2,141,333 (2 MB/s).

I've read conflicting information as to what the send buffer size should be rounded to (MSS or memory page). I think it should be rounded to full kilobytes because it is impossible to match it to MTUs of all users (some may negotiate full 1460 segments, others may connect via DSL, or some other VPN and have header overhead). CurrentControlSet, as a shortcut, points to the registry branch selected at boot time. If you put the value in a different "set" it will have no effect.

Maybe DynamicSendBufferDisable (http://forums.speedguide.net/showthread.php?285160-Parameters-of-Ancillary-Function-Driver-(AFD)-for-Windows-7) could help?

I would proceed with a process of elimination to determine if it's win7, the router, or the ISP that needs to be "fixed". Take a stable working computer with 2000/XP/2003, and no software that can slow it down, adjust the two window size parameters, and measure the speed. As the last step, connect the PC directly to the ISP's network (if possible), and try again.

As for the security of XP, I think it's okay if you purpose build a computer to do one or a few services only, and shut down everything else.

So far, only one user downloaded my test file:

Quote
19:52:26 213.114.xxx.xx:63966 Fully downloaded - 625.8 M @ 5.6 MB/s - /speedtest/656203776.iso

Our ping was approx. 17ms. Which means that the send window was at least 99824 bytes. I was afraid that HFS may have capped it smaller.

Quote
5.6 * 1024 * 1024 * 0.017 = 99824

The speed was definitely limited by congestion on my end at this hour (7PM). Downloads from America or Australia would give more useful statistics about the buffer.
Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on June 10, 2015, 02:26:26 PM
Thanks j7n

Results on this end:
6-10-15 @ 9:40 am EST

Download via your hfs server link = .5 MB/s

Download via your ftp to my ftp = 1.6 MB/s

I don't know where your server is but I'm located in central New Jersey on the coast.
My ping to Stockholm is between 95 & 135. Download is 60MB/s and upload is 3MB/s via ookla speed test.

hmmm, I could understand Sweden throttling download but not upload......  as I've said, I believe it's something with my windows 7 or my fios router.
I've also used a linux boot disk with services, same issue.
I have an xp server I'll give it a try when I get time.

I've looked at the Dynamic send buffer disable (good call!) but I've reverted to windows 7 stock installation settings for now until I can get back to testing.


Title: Re: Routers & ports firewalls - Oh, my!
Post by: bmartino1 on June 10, 2015, 07:41:01 PM
it may not work 100% on windows 7, but try tcp optimizer, it can chage just about all teh widnwos side lan settings...:

it a lan tcp thing...: http://www.speedguide.net/downloads.php

use tcp optimizer only if the bandwidth is capable!.. (look into the math and device and find out your bandwidth before changing the slider!)... TCP optimize auto does some of this!...

click optimal...
then move the slider to (12)... apply changes

make a backup of the changes either via program or system restore!... restart and you should be receiving the speed!...
(this does not gurantess you that speed!...

There are many factors... If your 100%set that it is your router, you may have to call your ISP and switch it out...
good luck...
if you odn't use the program to chage setting, use it for the cacultor for your windwos speeds...
Title: Re: Routers & ports firewalls - Oh, my!
Post by: j7n on June 12, 2015, 11:05:29 AM
Download via your hfs server link = .5 MB/s
Download via your ftp to my ftp = 1.6 MB/s
Thanks for the results! It may have been just "a fluke". Or it may actually be that HFS has a limitation. Or it may be that the web browser has a limitation (for example, Chromium (http://testmy.net/ipb/topic/32107-chromium-browser-caps-the-tcp-receive-window/)). Both FileZilla and HFS are running on my computer in the same conditions. I swear by FileZilla myself and use it for reference (http://www.sharetechnote.com/html/Throughput_Software.html) (the buffer size is a very extreme example and not necessarily "correct"). FileZilla can also push gigabit speeds on LAN on Pentium-4 grade hardware. 1.6 MB/s looks about right.

Problem with FileZilla and "users" is that sometimes FTP isn't accessible for them. They try PORT mode with local IPs, request weird directories because they can't parse the file listing, have problems with special characters in file names, and so on. Or they "distrust" an FTP server. This is where a webserver comes in.

I am getting 127 ms to you today, but the value was around 145 when I looked at it on the day the test was carried out. I am in Latvia, which is approximately 10 ms from kst5-speedtest-1.tele2.net (note that the main address is AnyCasted, but by default resolves to kst5 for me). My path towards Verizon goes through "retn.net" and "ntt.net".

C:\>tracert 108.53.107.XXX

Tracing route to pool-108-53-107-XXX.nwrknj.fios.verizon.net [108.53.107.XXX]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  router.j7nh [192.168.15.254]
  2    <1 ms     2 ms    <1 ms  10.91.104.254
  3    <1 ms    <1 ms    <1 ms  te-2401-2-72-94.balticom.lv [82.193.72.94]
  4    <1 ms    <1 ms    <1 ms  balticom-193-69-199.balticom.lv [82.193.69.199]

  5    <1 ms    <1 ms    <1 ms  ge-te-s5-64-254.balticom.lv [82.193.64.254]
  6     4 ms     1 ms     1 ms  te1-2.501.tv.riga.globalcom.lv [85.254.1.135]
  7     3 ms    33 ms     1 ms  ae0-504.rt.rtc.rix.lv.retn.net [87.245.242.25]
  8    29 ms    28 ms    30 ms  ae0-4.rt.irx.fkt.de.retn.net [87.245.233.74]
  9    29 ms    29 ms    29 ms  213.198.77.213
 10     *       35 ms    32 ms  ae-2.r20.frnkge04.de.bb.gin.ntt.net [129.250.5.217]
 11   150 ms   119 ms   114 ms  ae-7.r22.asbnva02.us.bb.gin.ntt.net [129.250.3.20]
 12   120 ms   115 ms   115 ms  ae-44.r06.asbnva02.us.bb.gin.ntt.net [129.250.6.113]
 13   123 ms   130 ms   122 ms  ae-0.verizonbusiness.asbnva02.us.bb.gin.ntt.net [129.250.8.34]
 14     *        *        *     Request timed out.
 15     *        *        *     Request timed out.
 16   126 ms   127 ms   127 ms  pool-108-53-107-XXX.nwrknj.fios.verizon.net [108.53.107.XXX]


OOKLA opens 6 to 8 connections. You can observe that if you run TcpView (https://technet.microsoft.com/en-us/library/bb897437.aspx?f=255&MSPPError=-2147217396). If you go to FileZilla Client's settings and set "maximum simultaneous transfers" to 8, and queue this many large files, the speeds would match up. And you would be able to set the length of the test by your own choosing.

20 Mbit (http://www.speedtest.net/result/4326757629.png) to Australia cannot be possible over 1 connection. No way.

Tele2 are all close to me. The site says that, "TCP windows have been slightly tweaked to support higher throughput."

You could also ask over at the TestMy.net forums I linked. They seem to have the expertise but only seem to concern with web browsers, which have too many variables in them. I have shared my experience with FTP/FileZilla and Tele2 over there.

I'm very pleased with the tele2 service. I was also able to benchmark my FileZilla Server by initiating a FXP (http://testmy.net/ipb/topic/32384-tele2-speed-test-ftp/) transfer between it and their box. No other site offers this flexibility.

I'm looking forward to a follow up when you get time.
Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on June 12, 2015, 01:02:42 PM
j7n:
You had mentioned a possible web browser limitation.......... hmmmmm....... network http pipelining...

https://www.google.com/#q=firefox+http+speed+2015 (https://www.google.com/#q=firefox+http+speed+2015)
Could Firefox be "another bottleneck" in http transfer performance? The plot thickens.
I use Firefox and it would seem that there may be a possibility to speed it up by 40%.  Is this an http issue, tuning or just snake oil. I'll need to look into it. Thanks you for the browser link, I didn't even consider that could be an issue.

I still feel like I'm circling the real issue and need to check out the router more carefully, maybe replacing it. If it's hardware, no tweak will fix it :(

The simplicity of HSF, requiring no extra software to install or download for clients is currently the optimal solution. We both know that TCP is high overhead  but other proprietary UDP ultra fast solutions are just too costly & too much trouble for the masses at this moment.
Thank you rejetto for HFS :)
 
will get back to this in a few days....
Thanks
 
Title: Re: Routers & ports firewalls - Oh, my!
Post by: fdiskMBR on June 27, 2015, 04:50:17 PM
I'm back to the file transfer grind stone once again.
Since some have been following this issue, I've tracked down some other information that gives a clearer picture of Windows 7 rwin and heuristics  issues along with firefox browser issues.
The following tests will verify if you have an issue with http & or file transfer speed.
https://www.duckware.com/blog/how-windows-is-killing-internet-download-speeds/index.html
Quote:
A quick test to see if you are running into the problem: Open one browser window and download a large file (test file) from a known fast location. Note this result as 'speed1'. Now open two browser windows and start the same download at the same time in each browser Window. Note this result as 'speed2'. If 'speed2' is faster than 2 × 'speed1', then then your computer may have this problem.

That would explain 2 downloads or uploads being much, much faster than 1 individual download or upload. 
ie. I can upload or download 2 files at the same time @ faster speeds than I can a single file.  It seems windows 7 (or something) is only throttling each individual file and not sum of many files at the same time. The idea of 2 files uploading or downloading at faster individual speeds than 1 file boggles my mind.
Yes, I have this problem.

The next issue is the FireFox browser.
There are 2 locations I've found that deal with a firefox browser issue.
The first:
Increasing the speed of firefox browser:
http://www.improgrammer.net/speed-up-firefox-browser/
The second:
http://www.mymobotips.com/2015/03/ultimate-guide-to-speed-up-mozilla-firefox-browsing/

It would seem that the second site above makes smaller changes to fiirefox.  I've tried both and they seem to be equal in increasing browsing speed. The results have been more than double but I've not put it all together yet to see the effect on ftp or http file transfer.

Unfortunately I can only work on this when I have ample time. (not at the moment)
Thanks for the browser tip j7n, if nothing else firefox is now ludicrous speed.