Welcome Guest.   Make a donation to an author on the site November 01, 2014, 07:02:07 AM  *

Please login or register.
Or did you miss your validation email?


Login with username and password (forgot your password?)
Why not become a lifetime supporting member of the site with a one-time donation of any amount? Your donation entitles you to a ton of additional benefits, including access to exclusive discounts and downloads, the ability to enter monthly free software drawings, and a single non-expiring license key for all of our programs.


You must sign up here before you can post and access some areas of the site. Registration is totally free and confidential.
 
Read the Practical Guide to DonationCoder.com Forum Search Features
   
   Forum Home   Thread Marks Chat! Downloads Search Login Register  
Pages: [1]   Go Down
  Reply  |  New Topic  |  Print  
Author Topic: Parallel proxy transfer software  (Read 4649 times)
electronixtar
Member
**
Posts: 141


View Profile WWW Give some DonationCredits to this forum member
« on: December 20, 2009, 12:24:09 AM »

Hi DC,

I have a remote server as a reverse proxy/http proxy, it's kinda slow, I hope there's some kind of software to speed it up a little by doing parallel data transfer.

Does any software like this exist?

My ideal version:
Quote
1. The server prefetches 3KB of data
2. The server send to client 3KB of data, but open multithreaded transfer, with 1KB of data per thread
3. The client receives 1KBx3, and combine them to the original 3KB data

The latency is not important as long as the transfer rate is good.
« Last Edit: December 21, 2009, 03:22:13 PM by electronixtar » Logged

f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #1 on: December 20, 2009, 05:50:01 AM »

Hm, I think you'll have to describe what you're trying to attempt in more detail.

First, at least if we're talking HTTP/FTP and not some custom protocol, the "multi" aspect is done by the client, not the server (or proxy server).

Second, threading isn't necessary to do parallel transfers smiley

Third, if you have that tiny data amounts, doing parallel transfers is going to hurt your performance, not improve it, because of the way TCP works (starts out slow and gradually increases transfer window size).
Logged

- carpe noctem
electronixtar
Member
**
Posts: 141


View Profile WWW Give some DonationCredits to this forum member
« Reply #2 on: December 20, 2009, 06:12:24 PM »

Quote
First, at least if we're talking HTTP/FTP and not some custom protocol, the "multi" aspect is done by the client, not the server (or proxy server).

It's obvious not a conventional HTTP/FTP proxy server, it's some advanced stuff I hope someone already invented

Quote
Second, threading isn't necessary to do parallel transfers

OK, no more threading as long as you understand what I am trying to say, but threading is the only best way I can think of how to describe it smiley ,

Quote
Third, if you have that tiny data amounts, doing parallel transfers is going to hurt your performance, not improve it, because of the way TCP works (starts out slow and gradually increases transfer window size).

Yes this kind of proxy is best done with UDP.

The basic idea is, if the server need to send 1000 packets one by one serialized, why not send it parallelized.
« Last Edit: December 20, 2009, 06:39:02 PM by electronixtar » Logged

Stoic Joker
Honorary Member
**
Posts: 5,347



View Profile WWW Give some DonationCredits to this forum member
« Reply #3 on: December 20, 2009, 10:35:21 PM »

The basic idea is, if the server need to send 1000 packets one by one serialized, why not send it parallelized.
Because it doesn't really matter if you have one data stream or fifty. The (ISP Controlled) connection speed (data) pipe is only as big as it is, and trying to force more through it will only cause a flood of retransmits (think turbulence) that will slow down the overall transfer speed.
Logged
electronixtar
Member
**
Posts: 141


View Profile WWW Give some DonationCredits to this forum member
« Reply #4 on: December 21, 2009, 12:10:32 AM »

The basic idea is, if the server need to send 1000 packets one by one serialized, why not send it parallelized.
Because it doesn't really matter if you have one data stream or fifty. The (ISP Controlled) connection speed (data) pipe is only as big as it is, and trying to force more through it will only cause a flood of retransmits (think turbulence) that will slow down the overall transfer speed.

I didn't mean to break the ISP pipe, but some connections are really slow, like 10KB/s compared to my total bandwidth (2M). It's better to open more channels for communication.

Third party download tools (like FreeDownloadManager) are invented for the exactly same reason. What I need is not downloading the whole file by splitting Content-Range, but to transfer packets x3 with three threads.

Just want to know if anything like this exists.
« Last Edit: December 21, 2009, 12:13:24 AM by electronixtar » Logged

f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #5 on: December 21, 2009, 02:53:14 AM »

If it's a custom protocol, how on earth would a generic proxy server be able to automagically split the workload for you? smiley - you can't just say "take this data stream and split it into three new ones", that kind of thing requires support both in the client and server side programs.

If your UDP data rate is poor, it's either because one (or both) end(s) are rate-limited somehow, or because the protocol is designed poorly (waiting for ACKs). With TCP you can gain a little by multiple connections even if there's no rate-limiting going on, but UDP always goes full throttle and shouldn't benefit from parallelizing - au contraire.
Logged

- carpe noctem
electronixtar
Member
**
Posts: 141


View Profile WWW Give some DonationCredits to this forum member
« Reply #6 on: December 21, 2009, 01:05:38 PM »

If it's a custom protocol, how on earth would a generic proxy server be able to automagically split the workload for you? smiley - you can't just say "take this data stream and split it into three new ones", that kind of thing requires support both in the client and server side programs.

If your UDP data rate is poor, it's either because one (or both) end(s) are rate-limited somehow, or because the protocol is designed poorly (waiting for ACKs). With TCP you can gain a little by multiple connections even if there's no rate-limiting going on, but UDP always goes full throttle and shouldn't benefit from parallelizing - au contraire.

I can download BitTorrent fast enough. It's just one personal remote server (reverse proxy) which is transferring data really slow. What should I do? I can't wait all the http resources to be full downloaded then use multithreaded download tools to download them because they are requested on demand. So what's your guy's suggestion?
« Last Edit: December 21, 2009, 02:03:45 PM by electronixtar » Logged

f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #7 on: December 21, 2009, 02:27:27 PM »

Why should you worry about my udp transporting speed? I can download BitTorrent fast enough.
I don't really worry about your UDP speed, all I'm saying is that (unless rate-limited some way), UDP goes full throttle and pumps out packets as fast as your link can handle - unlike TCP which does congestion control, and has slow-start.

BitTorrent uses TCP and not UDP, but has massively parallel connections - but you probably already know that.

It's just one personal remote server (reverse proxy) which is transferring data really slow. What should I do? I can't wait all the http resources to be full downloaded then use multithreaded download tools to download them because they are requested on demand. So what's your guy's suggestion?
You really need to explain in more detail what it is you want to do, and which software is involved. As I've already mentioned, you can't just automagically make a proxy parallelize things without client+server support...
Logged

- carpe noctem
electronixtar
Member
**
Posts: 141


View Profile WWW Give some DonationCredits to this forum member
« Reply #8 on: December 21, 2009, 02:58:30 PM »

Why should you worry about my udp transporting speed? I can download BitTorrent fast enough.
I don't really worry about your UDP speed, all I'm saying is that (unless rate-limited some way), UDP goes full throttle and pumps out packets as fast as your link can handle - unlike TCP which does congestion control, and has slow-start.

BitTorrent uses TCP and not UDP, but has massively parallel connections - but you probably already know that.

It's just one personal remote server (reverse proxy) which is transferring data really slow. What should I do? I can't wait all the http resources to be full downloaded then use multithreaded download tools to download them because they are requested on demand. So what's your guy's suggestion?
You really need to explain in more detail what it is you want to do, and which software is involved. As I've already mentioned, you can't just automagically make a proxy parallelize things without client+server support...

Ok, I am a bit confused, what else should I explain?

And again,

I have a remote server as a reverse proxy/http proxy, it's kinda slow, I hope there's some kind of software to speed it up a little by doing parallel data transfer.

Is this too hard?
Logged

electronixtar
Member
**
Posts: 141


View Profile WWW Give some DonationCredits to this forum member
« Reply #9 on: December 21, 2009, 03:00:26 PM »

Why should you worry about my udp transporting speed? I can download BitTorrent fast enough.
I don't really worry about your UDP speed, all I'm saying is that (unless rate-limited some way), UDP goes full throttle and pumps out packets as fast as your link can handle - unlike TCP which does congestion control, and has slow-start.

BitTorrent uses TCP and not UDP, but has massively parallel connections - but you probably already know that.

It's just one personal remote server (reverse proxy) which is transferring data really slow. What should I do? I can't wait all the http resources to be full downloaded then use multithreaded download tools to download them because they are requested on demand. So what's your guy's suggestion?
You really need to explain in more detail what it is you want to do, and which software is involved. As I've already mentioned, you can't just automagically make a proxy parallelize things without client+server support...

Oh btw, BitTorrent is way faster in UDP, perhaps you are choosing the wrong client  tongue
Logged

f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #10 on: December 21, 2009, 03:09:26 PM »

Ok, I am a bit confused, what else should I explain?
Which software is involved smiley

There isn't really anything the proxy in and by itself can do to help you out, here - as long as it supports multiple concurrent connections. It's up to the client and server software on each ends of the proxy to do the parallel transfers.

Oh btw, BitTorrent is way faster in UDP, perhaps you are choosing the wrong client  tongue
Hm, I haven't seen peer-to-peer data transfers via UDP with BitTorrent? I thought UDP communications were limited to trackerless peer exchange and such, and not the actual data transfers?

Anyway, I'm getting 2MB/s with regular TCP connections on my 20Mbit line, so... smiley
Logged

- carpe noctem
electronixtar
Member
**
Posts: 141


View Profile WWW Give some DonationCredits to this forum member
« Reply #11 on: December 21, 2009, 03:20:39 PM »

There isn't really anything the proxy in and by itself can do to help you out

I am looking for this kind of software. Thus this post.  embarassed

Quote
Anyway, I'm getting 2MB/s with regular TCP connections on my 20Mbit line, so...

 ohmy I am so jealous
« Last Edit: December 21, 2009, 03:22:57 PM by electronixtar » Logged

f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #12 on: December 21, 2009, 03:25:57 PM »

There isn't really anything the proxy in and by itself can do to help you out
I am looking for this kind of software. Thus this post.
And I'm saying it can't be done without client+server support smiley. If you're talking HTTP connections, then something could be done - the proxy could split client requests into multiple server requests with varying content-length... but that wouldn't buy you very much, and if the bottleneck is between client->proxy, then it buys you nothing at all.

Anyway, I'm getting 2MB/s with regular TCP connections on my 20Mbit line, so...
I am so jealous
smiley

I started with a 512/128 line, and switched to 4096/256 when I figured out it was only $20/month more expensive. Since that, my ISP has been doubling my speed every now and then... too bad I've pretty much hit the limit of what ADSL2+ offers, hope they'll start reducing price soon, or move me to VDSL tongue
Logged

- carpe noctem
electronixtar
Member
**
Posts: 141


View Profile WWW Give some DonationCredits to this forum member
« Reply #13 on: December 21, 2009, 03:44:00 PM »

There isn't really anything the proxy in and by itself can do to help you out
I am looking for this kind of software. Thus this post.
And I'm saying it can't be done without client+server support smiley. If you're talking HTTP connections, then something could be done - the proxy could split client requests into multiple server requests with varying content-length... but that wouldn't buy you very much, and if the bottleneck is between client->proxy, then it buys you nothing at all.

Anyway, I'm getting 2MB/s with regular TCP connections on my 20Mbit line, so...
I am so jealous
smiley

I started with a 512/128 line, and switched to 4096/256 when I figured out it was only $20/month more expensive. Since that, my ISP has been doubling my speed every now and then... too bad I've pretty much hit the limit of what ADSL2+ offers, hope they'll start reducing price soon, or move me to VDSL tongue

Yes a client+server pair is needed. My remote server can prefetch http resources fast enough, it's just the proxy -> me part is slow if using a normal HTTP proxy.
Logged

f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #14 on: December 21, 2009, 04:19:19 PM »

You mention reverse proxy - does this mean the proxy does connect-back to the client machine, rather than the client connecting to the proxy? If this is the case, the client probably isn't a standard web browser smiley

If it's a regular HTTP client, like a browser, then there's not much more you can do on the proxy side, though - the client needs to be tweaked.
Logged

- carpe noctem
electronixtar
Member
**
Posts: 141


View Profile WWW Give some DonationCredits to this forum member
« Reply #15 on: December 21, 2009, 05:31:46 PM »

the client probably isn't a standard web browser

yes there should also be a standard http proxy server on the client side for browsers.

Traditional way:

[copy or print]
HTTP resources -> prefetch server   ->  client -> browser

Parallel transfer:

[copy or print]
HTTP resources -> prefetch server ====> client -> local http server -> browser

The question I am asking is

Are there any software (client-server pair) has this ====> feature.
« Last Edit: December 21, 2009, 05:36:57 PM by electronixtar » Logged

f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #16 on: December 21, 2009, 05:42:59 PM »

are there any software that can do the ====> part.
Dunno - that would pretty much depend on the client in your diagram smiley - and the local http server as well.

As stated before, you aren't going to find software that can perform magic and parallelize stuff without support from it's endpoints... and you provide very little information on goals and involved software, so it's hard to give any concrete recommendations.

So, what's the deal? Is this understanding correct?

1) high-speed server preFETCHes HTTP resources
2) server PUSHes those resources to "client" machine
3) "client" machine runs a http proxy, and web browser on "client" machine access that local proxy

...and you want the PUSH to be faster than it currently is, and you don't want to wait for entire FETCH to be done before you PUSH?
Logged

- carpe noctem
electronixtar
Member
**
Posts: 141


View Profile WWW Give some DonationCredits to this forum member
« Reply #17 on: December 21, 2009, 05:56:58 PM »

are there any software that can do the ====> part.
Dunno - that would pretty much depend on the client in your diagram smiley - and the local http server as well.

As stated before, you aren't going to find software that can perform magic and parallelize stuff without support from it's endpoints... and you provide very little information on goals and involved software, so it's hard to give any concrete recommendations.

So, what's the deal? Is this understanding correct?

1) high-speed server preFETCHes HTTP resources
2) server PUSHes those resources to "client" machine
3) "client" machine runs a http proxy, and web browser on "client" machine access that local proxy

...and you want the PUSH to be faster than it currently is, and you don't want to wait for entire FETCH to be done before you PUSH?

Thanks for the pentiance, man.

Push or fetch doesn't matter, I just need some client-server softwarekit to replace the traditional standard HTTP proxy, which has a special mechanism to make transfer faster. By parallelly or by magic.

I knew there's space for speed up, because I can download from that remote proxy server in high speed (by using multithreaded download tools)
Logged

f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #18 on: December 21, 2009, 06:21:08 PM »

Well, assuming you're using a browser to interface with the proxy, have you tried tweaking browser settings? For instance, disables HTTP pipelining when it can see it's dealing with a proxy server...

Apart from that, it's still hard to say much else - there's unfortunately no such thing as magic in computer software. How to speed things up depends on the rest of the software... you might be able to use a different proxy software (or not, depending on how it interfaces with the prefetching). Perhaps client actions can be anticipated and acted on, perhaps not. It also depends on whether you're going to transfer a few big files, or a lot of small ones.. there's lots and lots of parameters smiley
Logged

- carpe noctem
electronixtar
Member
**
Posts: 141


View Profile WWW Give some DonationCredits to this forum member
« Reply #19 on: December 21, 2009, 06:32:20 PM »

Well, assuming you're using a browser to interface with the proxy, have you tried tweaking browser settings? For instance, disables HTTP pipelining when it can see it's dealing with a proxy server...

Apart from that, it's still hard to say much else - there's unfortunately no such thing as magic in computer software. How to speed things up depends on the rest of the software... you might be able to use a different proxy software (or not, depending on how it interfaces with the prefetching). Perhaps client actions can be anticipated and acted on, perhaps not. It also depends on whether you're going to transfer a few big files, or a lot of small ones.. there's lots and lots of parameters smiley

tweaking solves edge problem, not the main problem. What I need is a general pupose proxy toolkit to speed things up.

Still, thanks a lot man.

If you knew any stuff like this invented in the future, please inform me  Grin
Logged

Pages: [1]   Go Up
  Reply  |  New Topic  |  Print  
 
Jump to:  
   Forum Home   Thread Marks Chat! Downloads Search Login Register  

DonationCoder.com | About Us
DonationCoder.com Forum | Powered by SMF
[ Page time: 0.047s | Server load: 0.25 ]