HTTP_Request_Randomizer - Proxying Python Requests

  •        6

Proxying is very useful when conducting intensive web crawling/scrapping or when you just want to hide your identity (anonymization). In this project I am using public proxies to randomise http requests over a number of IP addresses and using a variety of known user agent headers these requests look to have been produced by different applications and operating systems.

http://pgaref.com/blog/python-proxy/
https://github.com/pgaref/HTTP_Request_Randomizer

Tags
Implementation
License
Platform

   




Related Projects

elite-proxy-finder - Finds public elite anonymity proxies and concurrently tests them

  •    Python

This script was not working for a few months. I just fixed it 7/6/15. Only uses gatherproxy.com now. Finds elite anonymity (L1) HTTP proxies then tests them all in parallel. Tests each proxy against 3 IP checking URLs including one which is HTTPS to make sure it can handle HTTPS requests. Then checks the proxy headers to confirm it's an elite L1 proxy that will not leak any extra info. By default the script will only print the proxy IP, request time, and country code of proxies that pass all four tests but you can see all the results including errors in any of the tests with the -a (--all) option.

curl - A command line tool and library for transferring data with URL syntax, supporting HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, FILE, IMAP, SMTP, POP3, RTSP and RTMP

  •    C

curl is used in command lines or scripts to transfer data. It is also used in cars, television sets, routers, printers, audio equipment, mobile phones, tablets, settop boxes, media players and is the internet transfer backbone for thousands of software applications affecting billions of humans daily.

goproxy - Proxy is a high performance HTTP(S), websocket, TCP, UDP,Secure DNS, Socks5 proxy server implemented by golang

  •    Go

Pull Request is welcomed. First, you need to clone the project to your account, and then modify the code on the dev branch. Finally, Pull Request to dev branch of goproxy project, and contribute code for efficiency. PR needs to explain what changes have been made and why you change them. This page is the v6.0 manual, and the other version of the manual can be checked by the following link.

php-restclient - A generic REST API client for PHP

  •    PHP

headers - An associative array of HTTP headers and values to be included in every request. parameters - An associative array of URL or body parameters to be included in every request. curl_options - cURL options to apply to every request; anything defined here: https://secure.php.net/manual/en/function.curl-setopt.php. These will override any automatically generated values. build_indexed_queries (bool) - http_build_query automatically adds an array index to repeated parameters which is not desirable on most systems. Use this option to enable the default behavior. Defaults to FALSE. user_agent - User agent string to use in requests. base_url - URL to use for the base of each request. format - Format string is appended to resource on request (extension), and used to determine which decoder to use on response; a request URL like "api.twitter.com/1.1/statuses/user_timeline.json" would be expected to return well-formed JSON. format_regex - Pattern to extract format from response Content-Type header, used to determine which decoder to use on response. decoders - Associative array of format decoders. See "Direct Iteration and Response Decoding". username - Username to use for HTTP basic authentication. Requires password. password - Password to use for HTTP basic authentication. Requires username. url (string) - URL of the resource you are requesting. Will be prepended with the value of the base_url option, if it has been configured. Will be appended with the value of the format option, if it has been configured.

express-http-proxy - Proxy middleware for express/connect

  •    Javascript

Express middleware to proxy request to another host and pass response back to original caller. Proxy requests and user responses are piped/streamed/chunked by default.


fetch-some-proxies - Simple Python script for fetching "some" (usable) proxies

  •    Python

Simple Python script for fetching "some" (usable) proxies. It fetches (daily) list of public proxies and automatically finds in a quick manner those usable in that same moment (Note: testing of SOCKS proxies is currently possible only on non-Windows platforms). Why should you use it? Well, if you've ever used free proxy lists around you'll know the pain of finding actually working proxies. This tool will automatically do the list fetching and proxy testing for you.

sniproxy - Proxies incoming HTTP and TLS connections based on the hostname contained in the initial request of the TCP session

  •    C

Proxies incoming HTTP and TLS connections based on the hostname contained in the initial request of the TCP session. This enables HTTPS name-based virtual hosting to separate backend servers without installing the private key on the proxy machine. For Debian or Fedora based Linux distributions see building packages below.

agent - Minimalistic Swift HTTP request agent for iOS and OS X

  •    Swift

Minimalistic Swift HTTP request agent for iOS and OS X. This is a tiny framework that gives you nice a API for crafting HTTP requests.

php-proxy - A PHP proxy script with https and post support

  •    PHP

This is a HTTP/HTTPS proxy script that forwards requests to a different server and returns the response. The Proxy class uses PSR7 request/response objects as input/output, and uses Guzzle to do the actual HTTP request. The following example creates a request object, based on the current browser request, and forwards it to example.com. The RemoveEncodingFilter removes the encoding headers from the original response so that the current webserver can set these correctly.

desproxy - a TCP tunnel for HTTP proxies

  •    C

Bored of having HTTP-only connection to the Internet? Tired of being unable to connect to irc through a HTTP proxy? desproxy (a TCP tunnel for HTTP proxies) just makes you able to tunnel all your TCP traffic through a HTTP proxy! Keywords: HTTP pr

zan-proxy - An extensible proxy for PC/Mobile/APP developer

  •    Vue

Zan Proxy is an HTTP proxy server written in Node.js, which can be used to modify requests and mock reponse data. It is also a tool for custom DNS resolving and requests monitoring. The proxy server can be easily configured by a user-friendly interface. In addition, a mechanism is provided for developers to customize the behavior of the server. The website for management will be opened automatically.

fetch

  •    C

fetch is a simple, fast, and flexible HTTP download tool built on the HTTP Fetcher library. fetch allows you to customize your request by changing the User Agent and Referer presented to the webserver, and also supports proxy servers.

nuster - A web caching proxy server based on HAProxy

  •    C

Nuster is a simple yet powerful web caching proxy server based on HAProxy. It is 100% compatible with HAProxy, and takes full advantage of the ACL functionality of HAProxy to provide fine-grained caching policy based on the content of request, response or server status. Nuster is very fast, some test shows nuster is almost three times faster than nginx when both using single core, and nearly two times faster than nginx and three times faster than varnish when using all cores.

requests-scala - A Scala port of the popular Python Requests HTTP client: flexible, intuitive, and straightforward to use

  •    Scala

Requests-Scala is a Scala port of the popular Python Requests HTTP client. Requests-Scala aims to provide the same API and user-experience as the original Requests: flexible, intuitive, and straightforward to use. Making your first HTTP request is simple: simply call requests.get with the URL you want, and requests will fetch it for you.

rotating-proxy - Rotating TOR proxy with Docker

  •    Ruby

Why: Lots of IP addresses. One single endpoint for your client. Load-balancing by HAproxy.Please note: Tor offers a SOCKS Proxy only. In order to allow communication from HAproxy to Tor, Polipo is used to translate from HTTP proxy to SOCKS proxy. HAproxy is able to talk to HTTP proxies only.

multitor - A tool that lets you create multiple TOR instances with a load-balancing traffic between them by HAProxy

  •    Shell

A tool that lets you create multiple TOR instances with a load-balancing traffic between them by HAProxy. It's provides one single endpoint for clients. Support socks protocol and http-proxy servers: polipo, privoxy and hpts. In addition, you can view previously running TOR processes and create a new identity for all or selected processes. Multitor was created with the aim of initialize many TOR processes as quickly as possible. I could use many instances for my daily use programs (web browsers, messangers and other). In addition, I was looking for a tool that would increase anonymity when conducting penetration tests and testing the security of infrastructure.

requests - Convenient http client for java, inspired by python request module

  •    Java

Requests is a http request lib with fluent api for java, inspired by the python request module. Requests requires JDK 1.8+, the last version support Java7 is 4.18.* . Requests is now in maven central repo.

jwtproxy - An HTTP-Proxy that adds AuthN through JWTs

  •    Go

The JWT proxy is intended to be used as a complementary service for authenticating, and possibly authorizing requests made between services. There is a forward proxy component, which can be configured to sign outgoing requests to another service, and a reverse proxy component, which can be used to authenticate incoming requests from another service.The JWT forward proxy is used to sign outgoing requests with a JWT using a private key.

Philtron

  •    PHP

Philtron is a proxomitron-like filtering HTTP proxy server writ entirely in PHP. Major design goals include removing unwanted/hostile content from web pages, better anonymity measures (HTTP headers control), user friendly, web-based configuration.