Automating Reconnaissance with Custom Python Scripts

In the world of penetration testing and red teaming, reconnaissance is arguably the most critical phase of an engagement. The more you know about your target's external footprint—their subdomains, open ports, exposed APIs, and technology stack—the higher your chances of finding a vulnerable entry point.

However, manually querying search engines, parsing certificate transparency logs, and running individual port scans is incredibly tedious and doesn't scale. To be effective, you must automate.

Python is the undisputed language of choice for security automation due to its massive ecosystem of networking and web libraries. In this post, we’ll explore how to build custom, modular Python scripts to automate your reconnaissance workflow.

Why Write Custom Scripts?

There are already hundreds of fantastic open-source recon tools out there (like Amass, Sublist3r, and Recon-ng). Why reinvent the wheel?

1. Workflow Integration: Monolithic tools often produce output in complex formats. A custom Python script allows you to tailor the output exactly how you want it (e.g., piping it directly into a specific database, a Slack channel, or another tool in your chain).
2. Stealth and OPSEC: Popular tools have predictable signatures. By writing your own HTTP requests and tweaking user agents, you can blend in with normal web traffic more effectively.
3. API Customization: Many modern recon techniques rely on querying APIs (like Shodan, Censys, or GitHub). Custom scripts allow you to integrate your personal API keys securely and handle rate-limiting logic on your own terms.

Script 1: Subdomain Enumeration via Certificate Transparency

One of the fastest ways to find valid subdomains is by querying Certificate Transparency (CT) logs. When a company registers an SSL/TLS certificate for a domain, it's recorded publicly. We can query databases like crt.sh to extract these subdomains.

Here is a quick Python script leveraging the requests library to automate this:

import requests
import json
import sys

def get_subdomains(domain):
    print(f"[*] Querying crt.sh for {domain}...")
    url = f"https://crt.sh/?q=%25.{domain}&output=json"
    
    try:
        response = requests.get(url, timeout=10)
        if response.status_code == 200:
            data = response.json()
            subdomains = set() # Use a set to avoid duplicates
            
            for entry in data:
                name = entry['name_value']
                # Handle wildcard certificates
                if not name.startswith("*"):
                    subdomains.add(name)
                    
            for sub in sorted(subdomains):
                print(f"[+] Found: {sub}")
                
        else:
            print(f"[-] Failed to fetch data. Status code: {response.status_code}")
            
    except Exception as e:
        print(f"[!] Error: {e}")

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python3 ct_recon.py <domain.com>")
        sys.exit(1)
    get_subdomains(sys.argv[1])

Script 2: Concurrent Directory Brute-Forcing

Once you have a list of subdomains and IP addresses, the next step is discovering hidden directories (like /admin, /api/v1, or /.git/). Doing this sequentially is too slow. We can use Python's concurrent.futures module to implement multithreading, making our script significantly faster.

import requests
import concurrent.futures

TARGET_URL = "http://example.com"
WORDLIST = "common_dirs.txt" # A local text file with directory names
THREADS = 20

def check_url(directory):
    url = f"{TARGET_URL}/{directory.strip()}"
    try:
        # Using a custom User-Agent to avoid generic blocks
        headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
        response = requests.get(url, headers=headers, timeout=5)
        
        if response.status_code in [200, 301, 302, 401, 403]:
            print(f"[+] Discovered ({response.status_code}): {url}")
    except requests.exceptions.RequestException:
        pass

with open(WORDLIST, 'r') as file:
    directories = file.readlines()

print(f"[*] Starting brute-force on {TARGET_URL} with {THREADS} threads...")
with concurrent.futures.ThreadPoolExecutor(max_workers=THREADS) as executor:
    executor.map(check_url, directories)

The Importance of Rate Limiting and Error Handling

When automating recon, you must consider the operational impact.

• Do not accidentally DoS your target. While the threading script above is fast, pointing 200 concurrent threads at a fragile legacy application might crash it. Always implement configurable delays or back-off logic.
• Handle Timeouts and Exceptions: Network connectivity during a pentest is often unreliable. Notice how both scripts use try/except blocks and timeouts. Without these, a single dropped packet could crash your entire automation script midway through a massive scan.

Conclusion

Building a customized reconnaissance framework in Python gives you unparalleled flexibility. By stitching together API queries, threaded directory brute-forcing, and data parsing, you can spend less time waiting for scans to finish and more time actively hunting for vulnerabilities.