http request 빠른 모듈

나는 평소에 웹해킹 하면서

requests 모듈로 blind sql injection 같은 브루트포싱을 자주 쓴다.

하지만 가끔씩 database , table , column 등을 조회해야되서 for 를 여러번 돌려야될때가 있는데

속도가 너무 느리다고 생각하기도 한다.

몰론 python 한테 속도를 바라면은 안되지만 조금 더 빠르게 웹에 요청을 하고 스크래핑할 수 없을까 해서

구글링으로 다양한 모듈을 찾아봤다.

찾아본 모듈은 aiohttp , requests, httpli2 , http.client , faster_than_requests , urllib.request 이다.

 

import time

print('import module list [ aiohttp , requests , httplib2 , urllib3 , http.client , faster_than_requests , urllib.request ]')


import aiohttp
import asyncio

asynctime = time.time()

##########################

loop = asyncio.get_event_loop()

async def main():

    async with aiohttp.ClientSession() as session:

        async with session.get('https://google.com') as responese:

            return await responese.text()

for i in range(1,20):

    a = loop.run_until_complete(main())

print("aiohttp :",time.time() - asynctime,"s")

###############

import requests


start = time.time()

for j in range(1,20):

    a = requests.get('https://google.com').text

print("requests :",time.time() - start,"s")

################

import httplib2

httplib2time = time.time()


for k in range(1,20):
    resp, content = httplib2.Http().request("https://google.com")

print("httplib2 :",time.time() - httplib2time,"s")

################

import urllib3

urllib3time = time.time()

http = urllib3.PoolManager()

for u in range(1,20):
    responesee = http.request('GET','https://google.com')
    print("urllib3 :",time.time() - urllib3time,"s")

##############

httpclient = time.time()

import http.client

for i in range(1,20):
    conn = http.client.HTTPSConnection("google.com")

    conn.request("GET","/")

    r1 = conn.getresponse().read()
    conn.close()

print("http.client :",time.time() - httpclient,"s")

#########################

import faster_than_requests as request

requests_test_tiem = time.time()

for i in range(1,20):
    request.get("https://google.com")
print("faster_than_requests :",time.time() - requests_test_tiem,"s")

####################

import urllib.request

urllib_requesttime = time.time()

for i in range(1,20):
    f = urllib.request.urlopen('https://google.com')

print("urllib.request :",time.time() - urllib_requesttime,"s")

#######################

대충 이렇게 for 로 각 모듈마다 해당 http 요청을 20번씩 반복한후 시간을 비교해보았다.

[첫번째 테스트]

 

 

[두번째 테스트]

 

 

[세번째 테스트]

 

 

[네번째 테스트]

 

 

[다섯번째 테스트]

테스트 평균

↓

5번 테스트 후 시간 평균

aiohttp : 7.105390310287476 s
aiohttp : 6.995399236679077 s
aiohttp : 6.960367441177368 s
aiohttp : 6.637087821960449 s
aiohttp : 7.1516735553741455 s

ㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡ
aiohttp -> 6.9700

 

 

requests : 7.342324495315552 s
requests : 7.312094449996948 s
requests : 7.701656818389893 s
requests : 7.282943964004517 s
requests : 7.387377738952637 s
ㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡ
requests -> 7.4053

 

 

httplib2 : 7.072418928146362 s
httplib2 : 7.6918864250183105 s
httplib2 : 7.499775409698486 s
httplib2 : 7.305320739746094 s
httplib2 : 7.590466737747192 s

ㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡ
httplib2 -> 7.4320

 

 

urllib3 : 3.5603652000427246 s
urllib3 : 3.358464479446411 s
urllib3 : 3.429964303970337 s
urllib3 : 3.8262503147125244 s
urllib3 : 3.393043279647827 s

ㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡ
urllib3 -> 3.5136

 

 

http.client : 3.1202638149261475 s
http.client : 3.1688544750213623 s
http.client : 3.1942315101623535 s
http.client : 3.4707350730895996 s
http.client : 3.400688648223877 s

ㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡ
http.client -> 3.2710

 

 

faster_than_requests : 55.189573764801025 s
faster_than_requests : 54.283759355545044 s
faster_than_requests : 55.20519185066223 s
faster_than_requests : 54.72826957702637 s
faster_than_requests : 54.97745203971863 s

ㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡ

faster_than_requests -> 54.8768

 

 

urllib.request : 7.2166900634765625 s
urllib.request : 6.844275712966919 s
urllib.request : 7.112508535385132 s
urllib.request : 6.698997735977173 s
urllib.request : 6.64036750793457 s

ㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡ
urllib.request -> 6.9026

 

 

총 평균ㅡㅡㅡㅡ

 

aiohttp -> 6.9700
requests -> 7.4053
urllib3 -> 3.5136
httplib2 -> 7.4320
http.client -> 3.2710
faster_than_requests -> 54.8768
urllib.request -> 6.9026

 

테스트 해본 결과 

http.client => urllib3 => urllib.request => aiohttp => requests => httplib2 => faster_than_requests

빠른순으로 이렇게 되는것을 볼수가있다.

 

그러면 제일 빠른 http.client 는 앞으로 웹 요청을 주기적으로 돌린다면은

 

자주 사용될거같다.