• 欢迎访问挑战自我博客网站,安全研究,web渗透,推荐使用最新版火狐浏览器和Chrome浏览器访问本网站,欢迎加入挑战自我博客网站 网站主页

基于web的信息探测-Httpscan-1.6

httpscan 挑战自我 490次浏览 已收录 0个评论

代码已上传至github

https://github.com/linxi0428/httpscan

作者博客

http://www.tiaozhanziwo.com/

更新日志

Httpscan Version:1.6

修复针对域名列表的测试相关问题;

修复部分bug;

Httpscan Version:1.5

添加web页面测试时常见的漏洞目录等要素;

添加端口扫描模块,在测试的时候先进行端口扫描,然后针对指定端口开展web漏洞目录测试;

基于web的信息探测-Httpscan-1.6

Httpscan Version:1.4

测试的时候发现对https的页面测试效果非常差,修复忽略https请求的证书验证等问题;

添加web页面测试时常见的漏洞端口等要素;

Httpscan Version:1.3

增加对多线程处理中的中断响应,(ctrl+c)实现柔和退出

Httpscan Version:1.2

增加https页面的访问功能,原来的版本https页面访问从日志信息中发现一直报错

Httpscan Version:1.1

增添日志输出功能,输出到程序目录的httpscan_log.txt文件中

Httpscan Version:1.0

代码是网上综合改造的

功能:基于web的信息探测,支持从文件读取目标(IP或域名),支持IP地址探测,IP地址支持CIDR,文件中的目标不加http(s)://,程序会自动添加

演示:

python xxx.py -f file.txt -t 20

python xxx.py 1.1.1.0/24 -t 20

Install(安装需求库与程序)

1、pip install nmap

2、pip install python-nmap

3、pip install IPy

4、系统需要安装nmap程序(下载安装:https://nmap.org/)

To Do

有什么需求可以提交,我会着手改造!

1、https页面的访问问题,目前有些问题,其实就是证书相关的东西;(V-1.2版本已经解决)

2、增添对于访问目标后302跳转页面的输出以及日志记录功能;

3、添加对多线程处理中的中断响应(ctrl+c);(V-1.3版本已经解决)

4、添加域名解析、旁站解析、C段旁站解析;

5、测试的时候发现对https的页面测试效果非常不好,可能涉及到忽略证书等问题;(V-1.4版本已经解决)

6、在进行页面的测试时,简单添加常见的漏洞目录、漏洞端口等要素;(V-1.4版本初步构建功能)

7、收集了一个常见的网站目录,大概2000个,结合1.4版本构建的几千个常见web端口,完善一个测试策略。比如先进行一个端口扫描,再把开放的端口放入扫描队列,同时结合常见的漏洞目录列表进行,效率会有成倍的提升;(V-1.5版本构建功能)

8、添加网站爬虫方法,不断提升网站目录漏洞扫描精准度,目前目录的遍历探测仅仅针对根目录,后续将实现对其他的二级、三级乃至多级目录的探测;

9、不断简化功能分类以及功能模组,提升代码的可读性以及代码分工的简易性;

10、随着后期程序功能的不断扩充,替换部分效率低下的功能方法,去除不必要的接口、循环以及判断,提升程序的整体效率;

11、鉴于C段扫描过程太长,后续可以添加一个扫描进度显示,然后界面就不显示扫描结果了,这个可以从端口扫描结束的时候开始显示,端口扫描结束的时候,后续的整体request数量就定了;

12、目前的域名文件列表读取存在一些bug或者是问题,下一步着手解决;(V-1.6版本已经解决)

13、portscan扫描模块有时候会卡住(一些可能的原因python-nmap库,多线程解决方案不完善等问题),下一步着手解决;

14、扫描结果的优化处理:一是http的status为200,但是是一个特制的404页面,这种可以从title中做进一步判断;二是部分IP直接访问后,URL地址处会直接变成或者跳转到某个域名地址,结果可以把这一部分地址记录下来;

Q & A

1、错误’ValueError: IP(‘1.1.1.1/24’) has invalid prefix length (24)’的解决办法

这个是由于IP地址的CIDR格式引起的,根据IPy的库规定,第一位必须是所在IP段的网络号,正确的写法如下:

1.1.1.0/24 : 1.1.1.0~1.1.1.255

1.1.1.128/25 : 1.1.1.128~1.1.1.255

1.1.1.64/26 : 1.1.1.64~1.1.1.127

1.1.1.32/27 : 1.1.1.32~1.1.1.63

1.1.1.16/28 : 1.1.1.16~1.1.1.31

2、结果保存的相关问题

结果除了显示在cmd的控制台界面外,还会在程序当前目录生成httpscan_log.txt文件,保存扫描的结果

3、当遇到错误’module’ object has no attribute ‘PortScanner’时,尝试重新安装python-nmap

pip uninstall python-nmap

pip install python-nmap

 

 

 

#!/usr/bin/env python
#coding:utf-8
#Author: linxi0428
#Version: 1.6

import re
import os
import sys
import ssl
import time
import logging
import optparse
import requests
import signal
import socket
import nmap
import logging
import threading
import Queue


from IPy import IP
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager


#Config the default encoding
reload(sys)
sys.setdefaultencoding("utf8")

#Set the request in ssl with unverified cert and disable_warnings
ssl._create_default_https_context = ssl._create_unverified_context
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
#import requests.packages.urllib3.util.ssl_ 
#requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = 'ALL'

#Lock Screen Print
printLock = threading.Semaphore(1)

#Request Timeout
TimeOut = 5

#Log-Config( CRITICAL > ERROR > WARNING > INFO > DEBUG > NOTSET )
logging.basicConfig(
    level=logging.INFO,
    format="[%(asctime)s] %(levelname)s: %(message)s",
    filename = 'httpscan_log.txt',
    filemode = 'w')

#User-Agent
header = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 \
          (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36','Connection':'close'}

#Transport adapter" that allows us to use SSLv3
class Ssl3HttpAdapter(HTTPAdapter):
    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(num_pools=connections,
                                       maxsize=maxsize,
                                       block=block,
                                       ssl_version=ssl.PROTOCOL_SSLv3)

class httpscan():
    def __init__(self,cidr,threads_num,file_source,open_ports):
        self.threads_num = threads_num

        #build ip queue
        self.IPs = Queue.Queue()
        self.open_ports = open_ports

        #open the path dictionary
        self.dict_lists = open('dict.txt','r')

        for dict_line in self.dict_lists.readlines():
            dict_line = dict_line.strip('\n').strip('\r')
            if file_source == None:
                self.cidr = IP(cidr)
                for open_port in self.open_ports:
                    self.IPs.put("http://"+str(open_port)+str(dict_line))
                    self.IPs.put("https://"+str(open_port)+str(dict_line))
            else:
                self.file_source = file_source
                with open(self.file_source,"r") as file_ip:
                    for line in file_ip:
                        line = line.strip('\n').strip('\r')
                        for open_port in self.open_ports:
                            self.IPs.put("http://"+str(open_port)+str(dict_line))
                            self.IPs.put("https://"+str(open_port)+str(dict_line))
                    
    def request(self):
        with threading.Lock():
            while self.IPs.qsize() > 0:
                ip = self.IPs.get()
                #print ip
                try:
                    s = requests.Session()
                    s.mount('https:', Ssl3HttpAdapter())#Mount All Https to ssl.PROTOCOL_SSLV3
                    r = s.get(str(ip.strip()).strip('\n').strip('\r'),headers=header,timeout=TimeOut,verify=False)
                    status = r.status_code
                    title = re.search(r'(.*)', r.text) #get the title
                    if title:
                        title = title.group(1).strip().strip("\r").strip("\n")[:30]
                    else:
                        title = "None"
                    banner = ''
                    try:
                        banner += r.headers['Server'][:20] #get the server banner
                        printLock.acquire()
                        if (status != 400) and (status != 404):
                            print "|%-33s|%-6s|%-14s|%-20s|" % (ip.strip(),status,banner,title)
                            print "+---------------------------------+------+--------------+--------------------+"
                            logging.info("|%-66s|%-6s|%-14s|%-40s|" % (ip.strip(),status,banner,title))
                            #logging.info("+---------------------------------+------+--------------+--------------------+")
                    except Exception,e:
                        printLock.acquire()
                        #print e
                        if (status != 400) and (status != 404):
                            print "|%-33s|%-6s|%-14s|%-20s|" % (ip.strip(),status,banner,title)
                            print "+---------------------------------+------+--------------+--------------------+"
                            logging.info("|%-66s|%-6s|%-14s|%-40s|" % (ip.strip(),status,banner,title))
                            #logging.info("+---------------------------------+------+--------------+--------------------+")
                    finally:
                        printLock.release()
                except Exception,e:
                    #print e
                    pass
                finally:
                    printLock.release()
    
    def run(self):#Multi thread
        signal.signal(signal.SIGINT, quit)
        signal.signal(signal.SIGTERM, quit)
        for i in range(self.threads_num):
            t = threading.Thread(target=self.request)
            t.setDaemon(True)
            t.start()
        while True:
            if not t.isAlive():
                break

    def print_head(self):
        print "+---------------------------------+------+--------------+--------------------+"
        print "|            IP                   |Status|     Server   |         Title      |"
        print "+---------------------------------+------+--------------+--------------------+"


class portscan():
    def __init__(self,cidr,threads_num,file_source,ports):
        self.threads_num = threads_num
        self.ports = ports
        self.IPs = Queue.Queue()
        self.file_source = file_source

        #ip-port lists
        self.open_ports = set()

        if self.file_source == None:
            try:
                self.cidr = IP(cidr)
            except Exception,e:
                print e
            for ip in self.cidr:
                ip = str(ip)
                self.IPs.put(ip)
        else:
            with open(self.file_source,'r') as file_ip:
                for line in file_ip:
                    self.IPs.put(line)

    def nmapScan(self):
        with threading.Lock():
            while self.IPs.qsize() > 0:
                item = self.IPs.get()
                self.IPs.task_done()
                try:
                    nmScan = nmap.PortScanner()
                    #print 'Start PortScan'
                    nmScan.scan(item,arguments = self.ports.read())
                    for tgthost in nmScan.all_hosts():
                        for tgtport in nmScan[tgthost]['tcp']:
                            tgthost = tgthost.strip()
                            tgtport = int(tgtport)
                            if nmScan[tgthost]['tcp'][tgtport]['state'] == 'open':
                                if self.file_source ==None:
                                    open_list = str(tgthost) + ':' + str(tgtport)
                                    self.open_ports.add(open_list)
                                    print 'the target %s open the port %s \n' % (tgthost,tgtport)
                                    logging.info('the target %s open the port %s' % (tgthost,tgtport))
                                else:
                                    open_list = str(item.strip()) + ':' + str(tgtport)
                                    self.open_ports.add(open_list)
                                    print 'the target %s open the port %s \n' % (item.strip(),tgtport)
                                    logging.info('the target %s open the port %s' % (item.strip(),tgtport))                                
                except Exception, e:
                    print e

    def run(self):
        threads = [threading.Thread(target=self.nmapScan) for i in range(self.threads_num)]
        for thread in threads:
            thread.setDaemon(True)
            thread.start()
        for thread in threads:
            thread.join()

        while True:
            if not thread.isAlive():
                break
        return self.open_ports

def help():
    print "Example:"
    print "  python "+sys.argv[0]+" -f domain_list.txt"
    print "  python "+sys.argv[0]+" 1.1.1.0/24"

def quit(signum, frame):#Judge Child Thread's Statue(Exit or Not)!
    print '\nYou choose to stop me!!'
    sys.exit()

if __name__ == "__main__":
    parser = optparse.OptionParser("Usage: %prog [target or file] [options] ")
    parser.add_option("-t", "--thread", dest = "threads_num",\
                      default = 100, help = "number of theads,default=100")
    parser.add_option("-f", "--file", dest = "file_source",\
                      help = "source of file,default=domain_list.txt")
    (options, args) = parser.parse_args()

    if options.file_source == None:
        if len(args) < 1:
            parser.print_help()
            help()
            sys.exit(0)
        else:
            ports = open('port.txt','r')
            logging.info('Start Port Scan !!')
            print 'Port Scan start !!'
            scan_port = portscan(cidr=args[0],threads_num=3,file_source=None,ports=ports)
            open_ports = scan_port.run()
            print 'Port Scan Ends !!'
            print "------------------------------------------------------------------------------"
            print 'Start Http Scan !!'
            s = httpscan(cidr=args[0],threads_num=options.threads_num,file_source=None,open_ports=open_ports)
            s.print_head()
            s.run()
    else:
        ports = open('port.txt','r')
        logging.info('Start Port Scan !!')
        print 'Start Port Scan !!'
        print "------------------------------------------------------------------------------"
        scan_port = portscan(cidr=None,threads_num=3,file_source=options.file_source,ports=ports)
        open_ports = scan_port.run()
        print 'Port Scan Ends !!'
        print "------------------------------------------------------------------------------"
        print 'Start Http Scan !!'
        s = httpscan(cidr=None,threads_num=options.threads_num,file_source=options.file_source,open_ports=open_ports)
        s.print_head()
        s.run()


挑战自我博客, 版权所有丨如未注明 , 均为原创丨本网站采用BY-NC-SA协议进行授权 , 转载请注明基于web的信息探测-Httpscan-1.6
喜欢 (10)
支付宝[]
分享 (0)
发表我的评论
取消评论
表情 贴图 加粗 删除线 居中 斜体 签到

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址