https://github.com/vkuznet/urlfetch2go

URL fetch proxy server (Go implementation)
https://github.com/vkuznet/urlfetch2go

Last synced: about 1 month ago
JSON representation

URL fetch proxy server (Go implementation)

• Host: GitHub
• URL: https://github.com/vkuznet/urlfetch2go
• Owner: vkuznet
• License: other
• Created: 2013-03-20T22:53:12.000Z (almost 12 years ago)
• Default Branch: master
• Last Pushed: 2013-04-18T13:03:42.000Z (almost 12 years ago)
• Last Synced: 2024-10-30T06:27:35.629Z (3 months ago)
• Language: Go
• Size: 191 KB
• Stars: 0
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
urlfetch2go

===========

URL fetch proxy server (Go implementation)

### Description

This is draft implementation of URL fetch proxy server. The main idea is to

have a proxy server for concurent data fetching from provided URL lists. It is

implemented as sinle HTTP server which accepts POST request from a client. The

POST method is choosen due to ability to send multiple (large) URLs to proxy

server for data retrieval.

### Installation & Usage

To compile the URL server you need a Go compiler, then perform the following:

```

go build

```

It will build ```urlfetch2go``` executable which you can fetch from UNIX shell.

By default it serves requests on localhost:8000/getdata,

feel free to modify code accoringly.

I also provided a simple python client to demonstrate the usage of proxy

server. Modify code accoringly for your favorite URLs and run it as following:

```

python ./urlfetch_client.py

```