- Bot Visits, 30 Days
- Today
- Honeypot Hits
- Unique Crawlers
- Unique IPs

Most Visited Paths

Loading path data...

IP Analysis

Loading IP data...

Detected Crawlers

Loading crawlers...

Honeypot Hits

Suspicious visitors that triggered honeypot traps

Loading honeypot data...

Bot detection dashboard and free IP blocklist API

Crawler Exposer is a live view of the automated traffic that reaches techtools.cz. Requests that hit the site's application server are checked against known crawler, scraper and scanner signatures, and every request for a trap path such as /wp-login.php or /.env is recorded with its IP address, path and method. The dashboard above turns those records into counters, a chart of the most requested paths, a per-IP breakdown, a list of named crawlers and the latest honeypot hits.

The data comes from this one site, not from yours, and static pages served straight by the web server never pass through the detector, so it is a sample rather than a full traffic log. It is still a fair picture of what any public server sees: search engines and AI crawlers fetching pages, uptime monitors checking in, and a steady stream of scripts probing for WordPress logins, leaked environment files and database dumps on a server that runs none of them.

It is made for web developers, sysadmins and anyone curious about who reads their site besides people. If you run a server, the free API turns the honeypot data into deny lists for nginx, Apache, iptables or fail2ban.

What you get

  • Live counters. Bot visits, honeypot hits, unique crawlers and unique IPs over the last 30 days, plus visits today, reloaded every 10 seconds. Visits older than 30 days are deleted.
  • Top paths. The 20 most requested URLs, which shows at a glance what scanners are hunting for. The site's own endpoints (tool APIs and pages that TechTools and well-behaved crawlers call on purpose) are left out.
  • IP analysis. The 100 busiest addresses with their visits, distinct paths and trap hits, each linked to IP Lookup.
  • Crawler list. Every named bot with its visits, unique IPs, honeypot hits and when it was last seen.
  • Blocklist API. IPs that hit a trap in the last 30 days as JSON, plain text, nginx, Apache or iptables rules, plus a detailed list with a threat level and a 7-day threat feed.
  • Community reports. A free API key lets you report bots from your own site into a combined, confidence-scored list.

Crawler Types

Search Engine

Google, Bing, DuckDuckGo

AI Crawler

GPTBot, ClaudeBot, Perplexity

Social Media

Facebook, Twitter, LinkedIn

SEO Tool

Semrush, Ahrefs, Moz

Scanner

Nmap, Nuclei, Shodan

Scraper

Python, curl, wget

Monitor

UptimeRobot, Pingdom

Generic Bot

Other automated visitors

How It Works

Bot Detection

We analyze User-Agent strings and request patterns to identify crawlers, bots, and automated scripts visiting the site.

Honeypot Traps

Hidden endpoints like /wp-admin, /.env, and /backup.sql attract scanners looking for vulnerabilities.

Analytics

Track which crawlers visit most, what paths they access, and identify potentially malicious scanning activity.

Public Blocklist API

Free threat intelligence API - no authentication required. Use these endpoints to protect your websites from malicious bots.

GET
/crawler-api/blocklist

Get malicious IPs. Supports formats: json, nginx, apache, iptables, plain

GET
/crawler-api/blocklist/user-agents

Get malicious User-Agent strings detected by honeypots

GET
/crawler-api/blocklist/full

Full threat intelligence with threat levels, paths attempted, timestamps

GET
/crawler-api/threat-feed

Real-time honeypot hits (last 24h by default, use ?hours=N)

Quick Integration Examples

Nginx blocklist: curl -s 'https://techtools.cz/crawler-api/blocklist?format=nginx' >> /etc/nginx/blocklist.conf
iptables firewall: curl -s 'https://techtools.cz/crawler-api/blocklist?format=iptables' | bash
fail2ban: curl -s 'https://techtools.cz/crawler-api/blocklist?format=plain' > /etc/fail2ban/ip.blacklist

Community Threat Sharing (P2P)

Help protect everyone! Report bots you detect on your website and contribute to our community threat intelligence.

1

Get Free API Key

Register with no email required

POST /crawler-api/register
2

Report Bots

Submit IPs of bots hitting your honeypots

POST /crawler-api/report
3

Build Reputation

Verified reports increase your trust score

4

Community Blocklist

Access combined threat intelligence

GET /crawler-api/community-blocklist
- Contributors
- Reports
- Verified
- Threats

How to block scanners with the blocklist

  1. Look at what the traps catch

    Open the Honeypots tab for the latest 50 trap hits and the IPs tab for the 100 busiest addresses. Click an IP to open it in IP Lookup.

  2. Pick a format and a threshold

    Add ?format=nginx, apache, iptables, plain or json to /crawler-api/blocklist. Add &min_hits=3 to keep only repeat offenders.

  3. Download the list and read it

    Save it to a file, for example curl -s -o blocklist.conf 'https://techtools.cz/crawler-api/blocklist?format=nginx', and skim it before you use it.

  4. Load it into your server

    In nginx, include the file in an http or server block, then run nginx -t and reload. Apache 2.4 reads the Deny from lines through mod_access_compat. The plain format suits fail2ban and your own scripts.

  5. Refresh it on a schedule

    A cron job that overwrites the file once an hour keeps it current. Overwrite it with -o instead of appending with >>, or the file grows duplicates.

Good to know before you block

  • The iptables output uses -A, so running it twice adds every rule twice. Keep the rules in a chain of their own and flush it before each reload.
  • Requests from the same IP to the same path within five minutes are counted once, so the numbers show distinct probes rather than raw request bursts.
  • A crawler name is only what the User-Agent says. To be sure a "Googlebot" really is Google, check its IP with a reverse DNS lookup.
  • The threat feed returns the latest 100 trap hits from the last 24 hours by default; ?hours=N goes back up to 168 hours.

Frequently asked questions

Does Crawler Exposer monitor my own website?

No. It shows the bots that reach techtools.cz, and there is nothing to install. What you can take to your own server is the blocklist built from our traps, and you can send the bots you catch to the community API.

Is the blocklist API free, and do I need a key?

Reading is free and needs no account and no key. The blocklist, user-agents, full, threat-feed and community-blocklist endpoints are plain GET requests with open CORS, so a server script or a browser page can call them. A key is needed only to submit reports.

Which formats does the blocklist support?

json (the default), plain with one IP per line, nginx with deny IP; lines, apache with Deny from IP lines and iptables with iptables -A INPUT -s IP -j DROP commands. Add min_hits=N to keep only addresses with at least N trap hits.

How does an IP end up on the blocklist?

By requesting a trap path on techtools.cz: WordPress login and admin URLs, /.env, /.git, database dumps such as /backup.sql, web shells such as /shell.php, /.aws, /cgi-bin and similar. The site runs none of that software, so a normal visitor has no reason to ask for those URLs. Most traps answer with a fake login page or a fake file, the rest with an empty error, and every request is recorded.

Can the blocklist block real visitors?

It can. With the default min_hits=1 a single trap request is enough, and the list covers the last 30 days, so an address that was used by a scanner once stays on it for 30 days. IP addresses are shared and reassigned, so use a higher min_hits, review the list and never block your own addresses.

What does the threat level in the full blocklist mean?

It is based only on the number of trap hits from that IP: 1 to 2 is low, 3 to 5 medium, 6 to 10 high and more than 10 critical. The full list also shows the paths each IP tried, the user agents it sent and when it was last seen.

How are crawlers like Googlebot or GPTBot recognised?

By patterns in the User-Agent header, grouped into search engines, AI crawlers, social previews, SEO tools, monitors, scanners and scrapers, plus generic bot, spider and crawler names. Any client can fake its User-Agent, so a name on the dashboard is what the bot claims to be.

How does community reporting work?

Send POST /crawler-api/register to get an API key; no e-mail is needed and the key is shown only once. Then send reports with the X-API-Key header to /crawler-api/report or /report/bulk (up to 100 per request and 100 per hour). Only public IPv4 addresses are accepted. A report is verified when the IP is already in our honeypot data, the path is a known trap path or the user agent matches a known scanner, and verified reports raise your reputation and the weight of your reports in the combined list at /crawler-api/community-blocklist.

How fresh are the numbers on the dashboard?

The counters reload every 10 seconds while the page is open, and the server caches them for up to a minute. Each tab loads when you open it, and the Refresh button reloads the counters and the open tab.