i tested the script and it worked fine here
[+download]
{.if|{.{.current downloads|ip=%ip%|file=this.} > 3.} |{: {.disconnection reason|too many threads.} :}/if.}
i advice to use {.disconnection reason.} in place of {.disconnect.}
All I need is to limit 3 connections per file but they still get more connection if they download a new file.
it's not clear what you are saying, "new file".
being the limit applied to a single file, any file is not affecting others.
if you download 2 files you'll have 3 + 3 available connections from the same IP address.