IT博客汇
  • 首页
  • 精华
  • 技术
  • 设计
  • 资讯
  • 扯淡
  • 权利声明
  • 登录 注册

    存货重新发布shell反爬虫脚本手动版

    C1G发表于 2023-07-28 08:59:00
    love 0

    kickleech反爬虫

    介绍

    shell反爬虫脚本手动版,分析web日志找出访问量异常高的IP。

    • 手动版反爬虫,输出ip及相关信息供反爬虫判断;
    • 运行前请确保日志临时目录V_TMPFILE存在并可读写,调用IP所在地公网接口需要curl支持
    • 支持单独白名单文件,nginx白名单文件,支持ipv4,ipv6 的/24斜线写法,不支持255.255.255.0 写法
    • 支持取日志中remote_addr或http_x_forwarded_for左一(没有则取remote_addr)为判断IP
    • 可以指定提取全部或多少条日志进行分析,排序显示单IP或IP C段大于指定阈值的IP
    • 运行结果可以输出到指定日志文件
    • 命令行支持指定WEB日志文件/日志条数/单IP阈值/IP C段阈值/显示IP数量/显示当前IP白名单
    • 可以指定是否排除蜘蛛等特定UserAgent
    • 分析后显示单IP请求数/页面分布/HTTP状态分布/抽样/IP所在地/是否(白|黑)名单中/是否正常请求

    优势

    • 支持http_x_forwarded_for取IP
    • 支持IPV6地址
    • 支持IPC段统计
    • 支持IP黑白名单

    项目地址

    • Gitee(国内): https://gitee.com/c1g/kickleech
    • Blog: https://blog.c1gstudio.com/

    安装部署

    kickleech手动版

    wget -O kickleech_manual.sh https://gitee.com/c1g/kickleech/raw/master/kickleech/kickleech_manual.sh
    chmod +x ./kickleech_manual.sh
    #脚本会自动创建临时目录和日志目录,也可以手动创建
    mkdir -p /opt/nginx/logs/
    mkdir ./log

    ./kickleech_manual.sh

    使用说明

    Example: ./kickleech_manual.sh -f /var/log/nginx/access.log -n 500000 -t 1000 -c 3000 -m 100 -s -i

    OPTION:
    -f apache log file,default=/var/log/nginx/access.log
    -n output the last K lines {full|500000},default=full
    -t threshold for the number of requests from same IP address,default=1000
    -c threshold for the number of requests from same class C IP address,default=3000
    -m show number of IP,default=100
    -s show IP allow list
    -i show IP location
    -v show this information
    RESULT:
    IP Count: ./log/kickleech_manual_ip_count.log
    Class C IP Count: ./log/kickleech_manual_ipc_count.log
    IP Detail: ./log/kickleech_manual_ip_info.log
    IP Result: ./log/kickleech_manual_ip_bad.log

    也可以直接编辑脚本将常用参数写在变量里。  

    示例

    ./kickleech_manual.sh -f /var/log/nginx/bbs.c1gstudio.com.log -n 50000 -t 100 -c 300 -m 100 -s -i

    =====start:20230706 18:04:46=====
    LOAD nginx safe ip
    SAFE IP:
    127.0.0.1/32 192.168.0.0/24
    LINE number:50000
    364 64.62.252.163
    241 173.208.200.2
    139 182.148.222.121
    115 110.188.45.251
    101 183.146.253.6
    KICKLEECH BAD 64.62.252.163
    KICKLEECH BAD 173.208.200.2
    KICKLEECH BAD 182.148.222.121
    KICKLEECH BAD 110.188.45.251
    KICKLEECH BAD 183.146.253.6
    skip reload nginx!
    =====end:20230706 18:04:50=====

    结果查看

    cat log/kickleech_manual_ip_info.log

    get head line:

    220.185.30.0 – – [27/Jun/2023:13:08:06 +0800] “GET /forum.php?mod=forumdisplay&fid=56&filter=digest&digest=1&typeid=25&specialtype=trade&orderby=views HTTP/1.1” 200 7925 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.873.3 Safari/537.36”

    364 64.62.252.163
    get status layout:5
    364 200
    get page layout:7
    199 /space
    125 /thread
    40 /forum.php?mod=forumdisplay
    get sample:3
    64.62.252.163 – – [27/Jun/2023:13:08:16 +0800] “GET /space-uid-735.html HTTP/1.1” 200 4182 “-” “The Knowledge AI” 64.62.252.163 – – [27/Jun/2023:13:08:16 +0800] “GET /space-username-a.html HTTP/1.1” 200 4203 “-” “The Knowledge AI” 64.62.252.163 – – [27/Jun/2023:13:08:17 +0800] “GET /thread-935-1-1.html HTTP/1.1” 200 4959 “-” “The Knowledge AI” get host:
    Host 163.252.62.64.in-addr.arpa. not found: 3(NXDOMAIN)
    get ip location:
    {“ip”:”64.62.252.163″,”beginip”:”64.62.244.152″,”endip”:”64.62.255.255″,”country”:”美国”,”area”:”加利福尼亚州弗里蒙特市Hurricane Electric公司”}

    241 173.208.200.2
    get status layout:5
    241 200
    get page layout:7
    91 /forum.php?mod=forumdisplay
    38 /connect.php?mod=login
    6 /forum.php?mod=viewthread
    3 /forum.php?mod=post
    1 /forum.php?mod=redirect
    get sample:3
    173.208.200.2 – – [27/Jun/2023:13:09:32 +0800] “GET /forum.php?mod=forumdisplay&fid=94&orderby=lastpost HTTP/1.1” 200 7989 “-” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2545.102 Safari/537.36”
    173.208.200.2 – – [27/Jun/2023:13:09:34 +0800] “GET /forum.php?mod=forumdisplay&fid=34&filter=author HTTP/1.1” 200 8210 “-” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.1717.77 Safari/537.36”
    173.208.200.2 – – [27/Jun/2023:13:09:35 +0800] “GET /forum.php?mod=forumdisplay&fid=39&filter=author HTTP/1.1” 200 8250 “-” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.1349.50 Safari/537.36”
    get host:
    2.200.208.173.in-addr.arpa domain name pointer smtp101.florybeauty.com.
    get ip location:
    {“ip”:”173.208.200.2″,”beginip”:”173.208.128.0″,”endip”:”173.208.255.255″,”country”:”美国”,”area”:”密苏里州堪萨斯城WholeSale互联网股份有限公司”}

    目录结构

    .
    ├── kickleech_manual.sh
    ├── kickleechsafeip.conf
    └── log

    ├── kickleech_manual_ip_bad.log
    ├── kickleech_manual_ipc_count.log
    ├── kickleech_manual_ip_count.log
    ├── kickleech_manual_ip_info.log
    └── kickleech_manual_log_2023.log

    The post 存货重新发布shell反爬虫脚本手动版 first appeared on C1G军火库.



沪ICP备19023445号-2号
友情链接