Author Topic: Parallel proxy transfer software (Read 13586 times)

electronixtar · « **on:** December 20, 2009, 12:24 AM »

Hi DC,

I have a remote server as a reverse proxy/http proxy, it's kinda slow, I hope there's some kind of software to speed it up a little by doing parallel data transfer.

Does any software like this exist?

My ideal version:

1. The server prefetches 3KB of data
2. The server send to client 3KB of data, but open multithreaded transfer, with 1KB of data per thread
3. The client receives 1KBx3, and combine them to the original 3KB data

The latency is not important as long as the transfer rate is good.

f0dder · « **Reply #1 on:** December 20, 2009, 05:50 AM »

Hm, I think you'll have to describe what you're trying to attempt in more detail.

First, at least if we're talking HTTP/FTP and not some custom protocol, the "multi" aspect is done by the client, not the server (or proxy server).

Second, threading isn't necessary to do parallel transfers

Third, if you have that tiny data amounts, doing parallel transfers is going to hurt your performance, not improve it, because of the way TCP works (starts out slow and gradually increases transfer window size).

electronixtar · « **Reply #2 on:** December 20, 2009, 06:12 PM »

First, at least if we're talking HTTP/FTP and not some custom protocol, the "multi" aspect is done by the client, not the server (or proxy server).

It's obvious not a conventional HTTP/FTP proxy server, it's some advanced stuff I hope someone already invented

Second, threading isn't necessary to do parallel transfers

OK, no more threading as long as you understand what I am trying to say, but threading is the only best way I can think of how to describe it

,

Third, if you have that tiny data amounts, doing parallel transfers is going to hurt your performance, not improve it, because of the way TCP works (starts out slow and gradually increases transfer window size).

Yes this kind of proxy is best done with UDP.

The basic idea is, if the server need to send 1000 packets one by one serialized, why not send it parallelized.

Stoic Joker · « **Reply #3 on:** December 20, 2009, 10:35 PM »

The basic idea is, if the server need to send 1000 packets one by one serialized, why not send it parallelized.
-electronixtar (December 20, 2009, 06:12 PM)

Because it doesn't really matter if you have one data stream or fifty. The (ISP Controlled) connection speed (data) pipe is only as big as it is, and trying to force more through it will only cause a flood of retransmits (think turbulence) that will slow down the overall transfer speed.

electronixtar · « **Reply #4 on:** December 21, 2009, 12:10 AM »

The basic idea is, if the server need to send 1000 packets one by one serialized, why not send it parallelized.
-electronixtar (December 20, 2009, 06:12 PM)
Because it doesn't really matter if you have one data stream or fifty. The (ISP Controlled) connection speed (data) pipe is only as big as it is, and trying to force more through it will only cause a flood of retransmits (think turbulence) that will slow down the overall transfer speed.
-Stoic Joker (December 20, 2009, 10:35 PM)

I didn't mean to break the ISP pipe, but some connections are really slow, like 10KB/s compared to my total bandwidth (2M). It's better to open more channels for communication.

Third party download tools (like FreeDownloadManager) are invented for the exactly same reason. What I need is not downloading the whole file by splitting Content-Range, but to transfer packets x3 with three threads.

Just want to know if anything like this exists.

f0dder · « **Reply #5 on:** December 21, 2009, 02:53 AM »

If it's a custom protocol, how on earth would a generic proxy server be able to automagically split the workload for you?

- you can't just say "take this data stream and split it into three new ones", that kind of thing requires support both in the client and server side programs.

If your UDP data rate is poor, it's either because one (or both) end(s) are rate-limited somehow, or because the protocol is designed poorly (waiting for ACKs). With TCP you can gain a little by multiple connections even if there's no rate-limiting going on, but UDP always goes full throttle and shouldn't benefit from parallelizing - au contraire.

electronixtar · « **Reply #6 on:** December 21, 2009, 01:05 PM »

If it's a custom protocol, how on earth would a generic proxy server be able to automagically split the workload for you? - you can't just say "take this data stream and split it into three new ones", that kind of thing requires support both in the client and server side programs.

If your UDP data rate is poor, it's either because one (or both) end(s) are rate-limited somehow, or because the protocol is designed poorly (waiting for ACKs). With TCP you can gain a little by multiple connections even if there's no rate-limiting going on, but UDP always goes full throttle and shouldn't benefit from parallelizing - au contraire.
-f0dder (December 21, 2009, 02:53 AM)

I can download BitTorrent fast enough. It's just one personal remote server (reverse proxy) which is transferring data really slow. What should I do? I can't wait all the http resources to be full downloaded then use multithreaded download tools to download them because they are requested on demand. So what's your guy's suggestion?

f0dder · « **Reply #7 on:** December 21, 2009, 02:27 PM »

Why should you worry about my udp transporting speed? I can download BitTorrent fast enough.
-electronixtar (December 21, 2009, 01:05 PM)

I don't really worry about your UDP speed, all I'm saying is that (unless rate-limited some way), UDP goes full throttle and pumps out packets as fast as your link can handle - unlike TCP which does congestion control, and has slow-start.

BitTorrent uses TCP and not UDP, but has massively parallel connections - but you probably already know that.

It's just one personal remote server (reverse proxy) which is transferring data really slow. What should I do? I can't wait all the http resources to be full downloaded then use multithreaded download tools to download them because they are requested on demand. So what's your guy's suggestion?
-electronixtar (December 21, 2009, 01:05 PM)

You really need to explain in more detail what it is you want to do, and which software is involved. As I've already mentioned, you can't just automagically make a proxy parallelize things without client+server support...

electronixtar · « **Reply #8 on:** December 21, 2009, 02:58 PM »

Why should you worry about my udp transporting speed? I can download BitTorrent fast enough.
-electronixtar (December 21, 2009, 01:05 PM)
I don't really worry about your UDP speed, all I'm saying is that (unless rate-limited some way), UDP goes full throttle and pumps out packets as fast as your link can handle - unlike TCP which does congestion control, and has slow-start.

BitTorrent uses TCP and not UDP, but has massively parallel connections - but you probably already know that.

It's just one personal remote server (reverse proxy) which is transferring data really slow. What should I do? I can't wait all the http resources to be full downloaded then use multithreaded download tools to download them because they are requested on demand. So what's your guy's suggestion?
-electronixtar (December 21, 2009, 01:05 PM)
You really need to explain in more detail what it is you want to do, and which software is involved. As I've already mentioned, you can't just automagically make a proxy parallelize things without client+server support...
-f0dder (December 21, 2009, 02:27 PM)

Ok, I am a bit confused, what else should I explain?

And again,

I have a remote server as a reverse proxy/http proxy, it's kinda slow, I hope there's some kind of software to speed it up a little by doing parallel data transfer.

Is this too hard?

electronixtar · « **Reply #9 on:** December 21, 2009, 03:00 PM »

Why should you worry about my udp transporting speed? I can download BitTorrent fast enough.
-electronixtar (December 21, 2009, 01:05 PM)
I don't really worry about your UDP speed, all I'm saying is that (unless rate-limited some way), UDP goes full throttle and pumps out packets as fast as your link can handle - unlike TCP which does congestion control, and has slow-start.

BitTorrent uses TCP and not UDP, but has massively parallel connections - but you probably already know that.

It's just one personal remote server (reverse proxy) which is transferring data really slow. What should I do? I can't wait all the http resources to be full downloaded then use multithreaded download tools to download them because they are requested on demand. So what's your guy's suggestion?
-electronixtar (December 21, 2009, 01:05 PM)
You really need to explain in more detail what it is you want to do, and which software is involved. As I've already mentioned, you can't just automagically make a proxy parallelize things without client+server support...
-f0dder (December 21, 2009, 02:27 PM)

Oh btw, BitTorrent is way faster in UDP, perhaps you are choosing the wrong client

f0dder · « **Reply #10 on:** December 21, 2009, 03:09 PM »

Ok, I am a bit confused, what else should I explain?
-electronixtar (December 21, 2009, 02:58 PM)

Which software is involved

There isn't really anything the proxy in and by itself can do to help you out, here - as long as it supports multiple concurrent connections. It's up to the client and server software on each ends of the proxy to do the parallel transfers.

Oh btw, BitTorrent is way faster in UDP, perhaps you are choosing the wrong client
-electronixtar (December 21, 2009, 03:00 PM)

Hm, I haven't seen peer-to-peer data transfers via UDP with BitTorrent? I thought UDP communications were limited to trackerless peer exchange and such, and not the actual data transfers?

Anyway, I'm getting 2MB/s with regular TCP connections on my 20Mbit line, so...

electronixtar · « **Reply #11 on:** December 21, 2009, 03:20 PM »

There isn't really anything the proxy in and by itself can do to help you out
-f0dder (December 21, 2009, 03:09 PM)

I am looking for this kind of software. Thus this post.

Anyway, I'm getting 2MB/s with regular TCP connections on my 20Mbit line, so...

I am so jealous

f0dder · « **Reply #12 on:** December 21, 2009, 03:25 PM »

There isn't really anything the proxy in and by itself can do to help you out
-f0dder (December 21, 2009, 03:09 PM)
I am looking for this kind of software. Thus this post.
-electronixtar (December 21, 2009, 03:20 PM)

And I'm saying it can't be done without client+server support

. If you're talking HTTP connections, then something could be done - the proxy could split client requests into multiple server requests with varying content-length... but that wouldn't buy you very much, and if the bottleneck is between client->proxy, then it buys you nothing at all.

Anyway, I'm getting 2MB/s with regular TCP connections on my 20Mbit line, so...
-f0dder (December 21, 2009, 03:09 PM)
I am so jealous
-electronixtar (December 21, 2009, 03:20 PM)

I started with a 512/128 line, and switched to 4096/256 when I figured out it was only $20/month more expensive. Since that, my ISP has been doubling my speed every now and then... too bad I've pretty much hit the limit of what ADSL2+ offers, hope they'll start reducing price soon, or move me to VDSL

electronixtar · « **Reply #13 on:** December 21, 2009, 03:44 PM »

There isn't really anything the proxy in and by itself can do to help you out
-f0dder (December 21, 2009, 03:09 PM)
I am looking for this kind of software. Thus this post.
-electronixtar (December 21, 2009, 03:20 PM)
And I'm saying it can't be done without client+server support . If you're talking HTTP connections, then something could be done - the proxy could split client requests into multiple server requests with varying content-length... but that wouldn't buy you very much, and if the bottleneck is between client->proxy, then it buys you nothing at all.

Anyway, I'm getting 2MB/s with regular TCP connections on my 20Mbit line, so...
-f0dder (December 21, 2009, 03:09 PM)
I am so jealous
-electronixtar (December 21, 2009, 03:20 PM)

I started with a 512/128 line, and switched to 4096/256 when I figured out it was only $20/month more expensive. Since that, my ISP has been doubling my speed every now and then... too bad I've pretty much hit the limit of what ADSL2+ offers, hope they'll start reducing price soon, or move me to VDSL
-f0dder (December 21, 2009, 03:25 PM)

Yes a client+server pair is needed. My remote server can prefetch http resources fast enough, it's just the proxy -> me part is slow if using a normal HTTP proxy.

f0dder · « **Reply #14 on:** December 21, 2009, 04:19 PM »

You mention reverse proxy - does this mean the proxy does connect-back to the client machine, rather than the client connecting to the proxy? If this is the case, the client probably isn't a standard web browser

If it's a regular HTTP client, like a browser, then there's not much more you can do on the proxy side, though - the client needs to be tweaked.

electronixtar · « **Reply #15 on:** December 21, 2009, 05:31 PM »

the client probably isn't a standard web browser
-f0dder (December 21, 2009, 04:19 PM)

yes there should also be a standard http proxy server on the client side for browsers.

Traditional way:

[Select]

HTTP resources -> prefetch server -> client -> browser

Parallel transfer:

[Select]

HTTP resources -> prefetch server ====> client -> local http server -> browser

The question I am asking is

Are there any software (client-server pair) has this ====> feature.

f0dder · « **Reply #16 on:** December 21, 2009, 05:42 PM »

are there any software that can do the ====> part.
-electronixtar (December 21, 2009, 05:31 PM)

Dunno - that would pretty much depend on the client in your diagram

- and the local http server as well.

As stated before, you aren't going to find software that can perform magic and parallelize stuff without support from it's endpoints... and you provide very little information on goals and involved software, so it's hard to give any concrete recommendations.

So, what's the deal? Is this understanding correct?

1) high-speed server preFETCHes HTTP resources
2) server PUSHes those resources to "client" machine
3) "client" machine runs a http proxy, and web browser on "client" machine access that local proxy

...and you want the PUSH to be faster than it currently is, and you don't want to wait for entire FETCH to be done before you PUSH?

electronixtar · « **Reply #17 on:** December 21, 2009, 05:56 PM »

are there any software that can do the ====> part.
-electronixtar (December 21, 2009, 05:31 PM)
Dunno - that would pretty much depend on the client in your diagram - and the local http server as well.

As stated before, you aren't going to find software that can perform magic and parallelize stuff without support from it's endpoints... and you provide very little information on goals and involved software, so it's hard to give any concrete recommendations.

So, what's the deal? Is this understanding correct?

1) high-speed server preFETCHes HTTP resources
2) server PUSHes those resources to "client" machine
3) "client" machine runs a http proxy, and web browser on "client" machine access that local proxy

...and you want the PUSH to be faster than it currently is, and you don't want to wait for entire FETCH to be done before you PUSH?
-f0dder (December 21, 2009, 05:42 PM)

Thanks for the pentiance, man.

Push or fetch doesn't matter, I just need some client-server softwarekit to replace the traditional standard HTTP proxy, which has a special mechanism to make transfer faster. By parallelly or by magic.

I knew there's space for speed up, because I can download from that remote proxy server in high speed (by using multithreaded download tools)

f0dder · « **Reply #18 on:** December 21, 2009, 06:21 PM »

Well, assuming you're using a browser to interface with the proxy, have you tried tweaking browser settings? For instance, disables HTTP pipelining when it can see it's dealing with a proxy server...

Apart from that, it's still hard to say much else - there's unfortunately no such thing as magic in computer software. How to speed things up depends on the rest of the software... you might be able to use a different proxy software (or not, depending on how it interfaces with the prefetching). Perhaps client actions can be anticipated and acted on, perhaps not. It also depends on whether you're going to transfer a few big files, or a lot of small ones.. there's lots and lots of parameters

electronixtar · « **Reply #19 on:** December 21, 2009, 06:32 PM »

Well, assuming you're using a browser to interface with the proxy, have you tried tweaking browser settings? For instance, disables HTTP pipelining when it can see it's dealing with a proxy server...

Apart from that, it's still hard to say much else - there's unfortunately no such thing as magic in computer software. How to speed things up depends on the rest of the software... you might be able to use a different proxy software (or not, depending on how it interfaces with the prefetching). Perhaps client actions can be anticipated and acted on, perhaps not. It also depends on whether you're going to transfer a few big files, or a lot of small ones.. there's lots and lots of parameters
-f0dder (December 21, 2009, 06:21 PM)

tweaking solves edge problem, not the main problem. What I need is a general pupose proxy toolkit to speed things up.

Still, thanks a lot man.

If you knew any stuff like this invented in the future, please inform me