# Introduction ... or what secator is all about. `secator` is a task and workflow runner used for security assessments. It supports dozens of well-known security tools and is designed to improve productivity for pentesters and security researchers. *** ## Quick demo

Example of running secator tasks and workflows

*** ## Features * [Philosophy & design](/in-depth/philosophy-and-design#curated-list-of-tools) * [Philosophy & design](/in-depth/philosophy-and-design#unified-input-options) * [Philosophy & design](/in-depth/philosophy-and-design#unified-output-schema) * [Philosophy & design](/in-depth/philosophy-and-design#cli-and-library-usage) * [Philosophy & design](/in-depth/philosophy-and-design#distributed-options) * [Philosophy & design](/in-depth/philosophy-and-design#from-simple-tasks-to-complex-workflows) * [Philosophy & design](/in-depth/philosophy-and-design#customizable) *** ## Supported tools `secator` integrates the following tools: | Name | Description | Category | | ------------------------------------------------------------- | -------------------------------------------------------------------------------------- | -------------- | | [httpx](https://github.com/projectdiscovery/httpx) | Fast HTTP prober. | `http` | | [cariddi](https://github.com/edoardottt/cariddi) | Fast crawler and endpoint secrets / api keys / tokens matcher. | `http/crawler` | | [gau](https://github.com/lc/gau) | Offline URL crawler (Alien Vault, The Wayback Machine, Common Crawl, URLScan). | `http/crawler` | | [gospider](https://github.com/jaeles-project/gospider) | Fast web spider written in Go. | `http/crawler` | | [katana](https://github.com/projectdiscovery/katana) | Next-generation crawling and spidering framework. | `http/crawler` | | [dirsearch](https://github.com/maurosoria/dirsearch) | Web path discovery. | `http/fuzzer` | | [feroxbuster](https://github.com/epi052/feroxbuster) | Simple, fast, recursive content discovery tool written in Rust. | `http/fuzzer` | | [ffuf](https://github.com/ffuf/ffuf) | Fast web fuzzer written in Go. | `http/fuzzer` | | [h8mail](https://github.com/khast3x/h8mail) | Email OSINT and breach hunting tool. | `osint` | | [dnsx](https://github.com/projectdiscovery/dnsx) | Fast and multi-purpose DNS toolkit designed for running DNS queries. | `recon/dns` | | [dnsxbrute](https://github.com/projectdiscovery/dnsx) | Fast and multi-purpose DNS toolkit designed for running DNS queries (bruteforce mode). | `recon/dns` | | [subfinder](https://github.com/projectdiscovery/subfinder) | Fast subdomain finder. | `recon/dns` | | [fping](https://fping.org/) | Find alive hosts on local networks. | `recon/ip` | | [mapcidr](https://github.com/projectdiscovery/mapcidr) | Expand CIDR ranges into IPs. | `recon/ip` | | [naabu](https://github.com/projectdiscovery/naabu) | Fast port discovery tool. | `recon/port` | | [maigret](https://github.com/soxoj/maigret) | Hunt for user accounts across many websites. | `recon/user` | | [gf](https://github.com/tomnomnom/gf) | A wrapper around grep to avoid typing common patterns. | `tagger` | | [grype](https://github.com/anchore/grype) | A vulnerability scanner for container images and filesystems. | `vuln/code` | | [dalfox](https://github.com/hahwul/dalfox) | Powerful XSS scanning tool and parameter analyzer. | `vuln/http` | | [msfconsole](https://docs.rapid7.com/metasploit/msf-overview) | CLI to access and work with the Metasploit Framework. | `vuln/http` | | [wpscan](https://github.com/wpscanteam/wpscan) | WordPress Security Scanner | `vuln/multi` | | [nmap](https://github.com/nmap/nmap) | Vulnerability scanner using NSE scripts. | `vuln/multi` | | [nuclei](https://github.com/projectdiscovery/nuclei) | Fast and customisable vulnerability scanner based on simple YAML based DSL. | `vuln/multi` | {% hint style="info" %} Feel free to request new tools integrations by [opening an issue](https://github.com/freelabz/secator/issues/new) on the repo, but please check that the tool complies with our selection criterias before doing so (read [Philosophy & design](/in-depth/philosophy-and-design#curated-list-of-tools)). If it doesn't but you still want to integrate it into `secator`, you can plug it in (read [Writing tasks](/for-developers/writing-tasks)). {% endhint %} *** # Installation ... or how to install secator and it's dependencies on different platforms. *** ## Installing secator {% tabs %} {% tab title="Pipx" %} ```bash pipx install secator ``` {% endtab %} {% tab title="Pip" %} ```bash pip install secator ``` {% endtab %} {% tab title="Bash" %} ```bash wget -O - https://raw.githubusercontent.com/freelabz/secator/main/scripts/install.sh | sh ``` {% endtab %} {% tab title="Docker" %} ```bash docker run -it --rm --net=host -v ~/.secator:/root/.secator freelabz/secator --help ``` {% hint style="info" %} The volume mount `-v` is necessary to save all `secator` reports to your host machine, and`--net=host` is recommended to grant full access to the host network. {% endhint %} You can alias this command to run it easier: ``` alias secator="docker run -it --rm --net=host -v ~/.secator:/root/.secator freelabz/secator" ``` Now you can run `secator` like if it was installed on baremetal: ``` secator --help ``` {% endtab %} {% tab title="Docker Compose" %} 
git clone https://github.com/freelabz/secator
cd secator
docker-compose up -d
docker-compose exec secator secator --help
{% endtab %} {% endtabs %} {% hint style="success" %} If you chose the Bash, Docker, or Docker Compose installation methods, you can jump straight to [CLI Usage](/getting-started/cli-usage). {% endhint %} *** ## Installing languages (optional) `secator` uses external tools, so you might need to install languages used by those tools assuming they are not already installed on your system. We provide a subcommand to install required languages if you don't manage them externally: 
secator install langs go   # install Go
secator install langs ruby # install Ruby
*** ## Installing tools (optional) `secator` can install tools automatically at runtime (provided `security.auto_install_commands` is enabled, which is the default), but you can also do it manually. We provide a subcommand to install or update each supported tool which should work on all systems supporting `apt`: ```bash secator install tools httpx # install httpx secator install tools # install all supported tools ``` *** ## Installing addons (optional) `secator` comes installed with the minimum amount of dependencies. We provide a subcommand to install additional addons which are required for various features: {% tabs %} {% tab title="worker" %} Add support for Celery (see [Distributed runs with Celery](/in-depth/distributed-runs-with-celery)). ```sh secator install addons worker ``` {% endtab %} {% tab title="google" %} Add support for Google Drive exporter (see [Exporters](/in-depth/concepts/exporters)). ```sh secator install addons google ``` {% endtab %} {% tab title="mongodb" %} Add support for MongoDB driver (see [Drivers](/in-depth/concepts/drivers#mongodb-driver)). ```sh secator install addons mongodb ``` {% endtab %} {% tab title="redis" %} Add support for Celery Redis broker / backend. ```sh secator install addons redis ``` {% endtab %} {% tab title="dev" %} Add development tools like `coverage` and `flake8` required for running tests. ```sh secator install addons dev ``` {% endtab %} {% tab title="trace" %} Add tracing tools like `memray` and `pyinstrument` required for tracing functions. ```sh secator install addons trace ``` {% endtab %} {% endtabs %} *** ## Checking installation health To figure out which languages or tools are installed on your system (along with their version): ```bash secator health ```

Secator Health CLI Output

*** # CLI Usage ... or how you can use secator as your pentesting swiss-knife. `secator` is first and foremost a command-line interface (CLI). This page describes how to use it in-depth. *** ## Usage ```bash secator --help # General help secator x # List available tasks secator w # List available workflows secator s # List available scans secator u # List available utilities ``` *** ## Running tasks You can run any of the supported tasks out-of-the box using the `secator x` (execute) subcommand: {% tabs %} {% tab title="subfinder" %} Find subdomains of a domain using offline sources with `subfinder`: ```bash secator x subfinder wikipedia.org ``` {% endtab %} {% tab title="httpx" %} Find information about an URL with `httpx`: ```bash secator x httpx wikipedia.org ``` {% endtab %} {% tab title="ffuf" %} Fuzz URLs with `ffuf` with max 100 requests / second and matching select HTTP codes: ```bash secator x ffuf http://testphp.vulnweb.com/FUZZ -rl 100 -mc 200,201,300,500 ``` {% endtab %} {% tab title="nmap" %} Find open ports and associated vulnerabilities with `nmap` using proxychains as a proxy: ```bash secator x nmap myhost.com -p 443,80,8080,8081,21 -proxy proxychains ``` {% endtab %} {% tab title="maigret" %} Find user accounts with `maigret`: ```bash secator x maigret elonmusk ``` {% endtab %} {% endtabs %} {% hint style="info" %} Use **`secator x --help`** to list options for a specific task. {% endhint %} *** ## Running workflows A workflow is a set of pre-defined tasks. You can run some pre-written workflows using the `secator w` (workflow) subcommand: {% tabs %} {% tab title="Host recon" %} To perform a basic host recon (open ports, network + HTTP vulnerabilities): ```bash secator w host_recon 192.168.1.18 ``` {% endtab %} {% tab title="Subdomain recon" %} To perform a basic subdomain discovery (subdomain + root URLs): ```bash secator w subdomain_recon mydomain.com ``` {% endtab %} {% tab title="URL crawl" %} To perform URL crawling: ```bash secator w url_crawl https://mydomain.com/start/crawling/from/here/ ``` {% endtab %} {% tab title="URL fuzz" %} To perform URL fuzzing: ```bash secator w url_fuzz https://mydomain.com/start/fuzzing/from/here/ ``` {% endtab %} {% tab title="Code scan" %} To perform code vulnerability scan: ```bash secator w code_scan /path/to/code/repo ``` {% endtab %} {% tab title="User hunt" %} To find user accounts for a username: ```bash secator w user_hunt elonmusk ``` {% endtab %} {% endtabs %} {% hint style="info" %} Use **`secator w --help`** to list options for a specific workflow. {% endhint %} *** ## Running scans A scan is a set of workflows that run one after the other. You can run some pre-written scans using the `secator s` subcommand: {% tabs %} {% tab title="Domain scan" %} ``` secator s domain example.com ``` {% endtab %} {% tab title="Subdomain scan" %} ``` secator s subdomain sub.example.com ``` {% endtab %} {% tab title="Network scan" %} ``` secator s network 192.168.1.0/24 ``` {% endtab %} {% tab title="URL Scan" %} ``` secator s url http://testphp.vulnweb.com ``` {% endtab %} {% endtabs %} {% hint style="info" %} Use **`secator s --help`** to list a options for a specific scan. {% endhint %} *** ## Running utils `secator` provides a number of utilities that can be useful when doing pentesting. ### **Proxy** You can get a random proxy: ```bash secator u proxy # print a random proxy secator u proxy -n 5 --timeout 1 # print 5 proxies with 1s max timeout ``` ### **Reverse shells** You can spawn reverse shells in any language, and optional netcat listener: ```bash secator u revshell # list all reverse shells secator u revshell bash # show a Bash reverse shell secator u revshell javascript -h -p # show a Javascript reverse shell to connect to LHOST / LPORT secator u revshell javascript -h -p -l # ... also spawn a netcat listener ``` ### Serve You can run an HTTP server to serve payloads: ```sh secator u serve ``` ### **Recording** You can record pentesting sessions as a GIF: ```bash secator u record -i # record an interactive session secator u record --script test.sh # put your commands in a script and record the execution ``` *** ## Configuring secator To configure `secator`, use the following commands: ```bash secator c get # get full config (with defaults) secator c get --user # get user config secator c get wordlists.defaults.http # get default wordlist path secator c set wordlists.defaults.http rockyou.txt # set default wordlist secator c edit # edit user config yaml secator c default # get default config ``` To see the full available configuration options, get the default configuration using `secator c default`. *** ## Running a worker \[optional] You can enable enable distributed runs by starting `secator` workers. All tasks / workflows / scans will be sent to the workers for execution. You can run a worker using the file system as a broker and result backend: ```bash secator install addons worker secator worker ``` {% hint style="info" %} Learn more about [Distributed runs with Celery](/in-depth/distributed-runs-with-celery) {% endhint %} *** # Library usage ... or how you can use secator as a foundation to build powerful security software. `secator` can also be used as a Python library. {% hint style="info" %} We recommend using `secator` as a library when building complex systems around `secator` to overcome CLI limitations. {% endhint %} *** ## **Running tasks, workflows, and scans** You can run any task supported by `secator` by simply importing it by name from `secator.tasks`. You can run any workflow or scan by importing it from `secator.workflows` or `secator.scans`. ```python from secator.template import TemplateLoader from secator.runners import Workflow from secator.tasks import subfinder, httpx, naabu from secator.workflows import host_recon from secator.scans import host # Run simple tasks, chain them together target = 'wikipedia.org' subdomains = subfinder(target).run() hosts = set(_.host for _ in subdomains if _._type == 'subdomain'] ports_open = naabu(hosts).run() to_probe = set(f'{_.host}:{_.port}' for _ in ports_open if _._type == 'port'] alive_urls = httpx(to_probe).run() # ... or run a workflow results = host_recon(target).run() # ... or run a scan results = host(target).run() # ... or run any custom template by loading it dynamically config = TemplateLoader('/path/to/my/workflow.yaml') results = Workflow(config, target).run() ``` *** ## Consuming results live All runners yield results in real-time, which means you can run them as generators to consume their results: For instance, you can consume results lazily using threads or a Celery task: {% tabs %} {% tab title="Using threads" %} ```python from threading import Thread from secator.tasks import feroxbuster from secator.workflows import url_crawl from secator.output_types import Url, Tag from .models import Urls, Tags def process_url(url): Urls.objects.create(**url) print(f'Saved {url.url} [{url.status_code}] to database') def process_tag(tag): Tags.objects.create(**tag) print(f'Found tag {tag.name} for target {tag.match}') # Set the initial host host = 'http://testphp.vulnweb.com' # Use a task as a generator for url in feroxbuster(host, rate_limit=100): Thread(target=process_url, args=(url,)) # Use a workflow as a generator threads = [] for result in url_crawl(host, rate_limit=100): if isinstance(result, Url): thread = Thread(target=process_url, args=(result,)) elif isinstance(result, Tag): thread = Thread(target=process_tag, args=(result,)) threads.append(thread) thread.start() for thread in threads: thread.join() ``` {% endtab %} {% tab title="Using a Celery task" %} ```python from celery import Celery from secator.tasks import ffuf from secator.output_types import Url, Tag from .models import Urls, Tags from secator.workflows import url_crawl app = Celery(__name__) @app.task def process_url(url): Urls.objects.create(**url) print(f'Saved {url.url} [{url.status_code}] to database') @app.task def process_tag(tag): Tags.objects.create(**tag) print(f'Found tag {tag.name} for target {tag.match}') # Set the initial host host = 'http://testphp.vulnweb.com' # Use a task as a generator for url in feroxbuster(host, rate_limit=100): process_url.delay(url) # Use a workflow as a generator for result in url_crawl(host, rate_limit=100): if isinstance(result, Url): process_url.delay(result) elif isinstance(result, Tag): process_tag.delay(result) ``` {% endtab %} {% endtabs %} {% hint style="info" %} All tasks support being run like generators, but some of them have to wait for the command to finish before outputting results (e.g: `nmap`). {% endhint %} *** ## **Overriding global options** Options specified with the name of the command name prefixed will override global options for that specific command. For instance, if you want a global rate limit of `1000` (reqs/s), but for ffuf you want it to be `100` you can do so: ```python from secator.tasks import ffuf, gau, gospider, katana host = 'wikipedia.org' options = { 'rate_limit': 1000, # reqs/s 'ffuf.rate_limit': 100, 'katana.rate_limit': 30 } for tool in [ffuf, gau, gospider, katana]: tool(host, **options) ``` {% hint style="info" %} In the example above: * `gau`, and `gospider` will have a rate limit of `1000` requests / second. * `ffuf` will have a rate limit of `100` requests / second. * `katana` will have a rate limit of `30` requests / second. {% endhint %} *** ## **Disabling default options** Sometimes you might wish to omit passing the option and use the command defaults. You can set the option to `False` in order to do this. ```python options = { 'rate_limit': 1000, # reqs/s 'ffuf.rate_limit': False, # explicitely disabling `rate_limit` option, will use ffuf defaults } ``` *** # Configuration ... how to configure every aspect of how secator operates. *** ## Default configuration `secator` is configured using a YAML config file. The default configuration is as follow: ```yaml dirs: bin: ~/.local/bin share: ~/.local/share data: ~/.secator templates: ~/.secator/templates reports: ~/.secator/reports wordlists: ~/.secator/wordlists cves: ~/.secator/cves payloads: ~/.secator/payloads performance: ~/.secator/performance revshells: ~/.secator/revshells celery: ~/.secator/celery celery_data: ~/.secator/celery/data celery_results: ~/.secator/celery/results debug: level: 0 component: '' celery: broker_url: filesystem:// broker_pool_limit: 10 broker_connection_timeout: 4.0 broker_visibility_timeout: 3600 broker_transport_options: '' override_default_logging: true result_backend: file://~/.secator/celery/results result_backend_transport_options: '' result_expires: 86400 task_acks_late: false task_send_sent_event: false task_reject_on_worker_lost: false task_max_timeout: -1 task_memory_limit_mb: -1 worker_max_tasks_per_child: 20 worker_prefetch_multiplier: 1 worker_send_task_events: false worker_kill_after_task: false worker_kill_after_idle_seconds: -1 worker_command_verbose: false cli: github_token: '' record: false stdin_timeout: 1000 show_http_response_headers: false show_command_output: false exclude_http_response_headers: - connection - content_type - content_length - date - server runners: input_chunk_size: 100 progress_update_frequency: 20 stat_update_frequency: 20 backend_update_frequency: 5 poll_frequency: 5 skip_cve_search: false skip_exploit_search: false skip_cve_low_confidence: false remove_duplicates: false threads: 50 prompt_timeout: 20 http: socks5_proxy: socks5://127.0.0.1:9050 http_proxy: https://127.0.0.1:9080 store_responses: true response_max_size_bytes: 100000 proxychains_command: proxychains freeproxy_timeout: 1 default_header: User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 tasks: exporters: - json - csv - txt workflows: exporters: - json - csv - txt scans: exporters: - json - csv - txt payloads: templates: lse: https://github.com/diego-treitos/linux-smart-enumeration/releases/latest/download/lse.sh linpeas: https://github.com/carlospolop/PEASS-ng/releases/latest/download/linpeas.sh sudo_killer: https://github.com/TH3xACE/SUDO_KILLER/archive/refs/heads/V3.zip wordlists: defaults: http: bo0m_fuzz dns: combined_subdomains http_params: burp-parameter-names templates: bo0m_fuzz: https://raw.githubusercontent.com/Bo0oM/fuzz.txt/master/fuzz.txt combined_subdomains: https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/DNS/combined_subdomains.txt directory_list_small: https://gist.githubusercontent.com/sl4v/c087e36164e74233514b/raw/c51a811c70bbdd87f4725521420cc30e7232b36d/directory-list-2.3-small.txt burp-parameter-names: https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Discovery/Web-Content/burp-parameter-names.txt lists: {} addons: gdrive: enabled: false drive_parent_folder_id: '' credentials_path: '' gcs: enabled: false bucket_name: '' credentials_path: '' worker: enabled: false mongodb: enabled: false url: mongodb://localhost update_frequency: 60 max_pool_size: 10 server_selection_timeout_ms: 5000 max_items: null duplicate_main_copy_fields: - screenshot_path - stored_response_path - is_false_positive - is_acknowledged - verified vulners: enabled: false api_key: '' profiles: defaults: [] drivers: defaults: [] providers: defaults: cve: circl exploit: exploitdb ghsa: ghsa security: allow_local_file_access: true auto_install_commands: true force_source_install: false offline_mode: false ``` To get `secator`'s default configuration, run: ```bash secator config default ``` *** ## Custom configuration It is possible to override `secator`'s default configuration using a **user configuration file** located at `~/.secator/config.yml` . The default user configuration is empty. ### Get config To get `secator`'s user config, run: ```bash secator config get ``` ### Edit config #### Editing whole config To modify `secator`'s user config, run: ```bash secator config edit ``` This will open the YAML config in your default editor (using the`$EDITOR` env variable). Upon saving your changes, the config will be validated by `pydantic` and saved to disk if it's valid. {% hint style="warning" %} If validation errors occur (for instance a wrong key, or wrong value type), you will get an error message such as: ```bash ❌ 1 validation error for SecatorConfig offline_test Extra inputs are not permitted For further information visit https://errors.pydantic.dev/2.7/v/extra_forbidden Hint: Run "secator config edit --resume" to edit your patch and fix issues. ``` The invalid patch will not be saved in the user config. This ensures the actual user config is always valid. Run `secator config edit --resume` to resume the edit to fix the issues, or skip the `--resume` flag to start over. {% endhint %} #### Editing specific config keys To edit specific keys, use: ```bash secator config set ``` For instance, to set the debug component to `celery`: ```bash secator config set debug.component celery ``` ... or to set the `mongodb` addon URL: ```bash secator config set addons.mongodb.url mongodb://mymongodbhost ``` ### Env overrides Values in the `secator` config can be overriden using environment variables. Environment variables are prefixed with `SECATOR_`and use dotted path notation. For instance, to override `debug.component`, run: ```bash export SECATOR_DEBUG_COMPONENT=celery ``` To override the default HTTP wordlist: ```bash export SECATOR_WORDLISTS_DEFAULTS_HTTP=/path/to/wordlist.txt ``` and so on. *** # Examples ... or concrete use cases for secator. *** ### **Find subdomains using `subfinder` and run HTTP probes using `httpx`** {% tabs %} {% tab title="CLI" %} ```bash secator x subfinder -raw alibaba.com | secator x httpx -rl 10 -ss ``` {% endtab %} {% tab title="Python" %} ```python from secator.tasks import subfinder, httpx target = 'alibaba.com' results = subfinder(target).run() hosts = [_.host for _ in results if _._type == 'subdomain'] for probe in httpx(hosts, rate_limit=10, screenshot=True): print('Found alive subdomain URL {url}[{status_code}]'.format(**probe)) ``` {% endtab %} {% endtabs %} *** ### **Run host reconnaissance workflow** {% tabs %} {% tab title="CLI" %} ```bash secator w host_recon cnn.com ``` {% endtab %} {% tab title="Python" %} ```python from secator.workflows import host_recon target = 'cnn.com' for result in host_recon(target): # consume results live print(result) ``` {% endtab %} {% endtabs %} *** ### **Fuzz URLs with multiple fuzzers and a custom wordlist** {% tabs %} {% tab title="CLI" %} ```bash secator w url_fuzz example.com -mc 200,302 -rl 1 -w dicc.txt -o table -quiet ``` {% endtab %} {% tab title="Python" %} ```python from secator.workflows import url_fuzz target = 'example.com' opts = { 'match_codes': '200, 302', 'rate_limit': 1 # req/s 'quiet': True, 'ffuf.wordlist': 'dicc.txt' # ffuf wordlist } # Print results live and a summary table at the end of the run for result in url_fuzz(target, exporters=['table']): print(result) ``` {% endtab %} {% endtabs %} *** # 5 minutes secator session or how you can hack faster than ever before... This quickstart will be focused on using `secator` to find vulnerabilities on the website . We will start by using simple `secator` tasks, and then show how to use workflows to considerably speed up the session. *** ## Using tasks
Step 1: Run katana on the root URL We'll start by using a crawler to find some URLs that could be interesting to exploit for vulnerabiliites. We most often use `katana`, a tool by ProjectDiscovery. We'll save our results to a `.txt` file: 
secator x katana http://testphp.vulnweb.com -o txt
                         __            
   ________  _________ _/ /_____  _____
  / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/
 (__  /  __/ /__/ /_/ / /_/ /_/ / /    
/____/\___/\___/\__,_/\__/\____/_/     v0.0.1

                    freelabz.com

katana -silent -jc -js-crawl -known-files all -u http://testphp.vulnweb.com -json -concurrency 50                                                                                                                                _base.py:614
🔗 http://testphp.vulnweb.com [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu, DreamWeaver]
🔗 http://testphp.vulnweb.com/high [404] [Nginx:1.19.0]
🔗 http://testphp.vulnweb.com/index.php
🔗 http://testphp.vulnweb.com/style.css [200] [Nginx:1.19.0]
🔗 http://testphp.vulnweb.com/privacy.php [404] [PHP:5.6.40, Ubuntu, Nginx:1.19.0]
🔗 http://testphp.vulnweb.com/AJAX/index.php [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu]
🔗 http://testphp.vulnweb.com/categories.php [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu, DreamWeaver]
🔗 http://testphp.vulnweb.com/cart.php [200] [PHP:5.6.40, Ubuntu, DreamWeaver, Nginx:1.19.0]
🔗 http://testphp.vulnweb.com/artists.php [200] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu]
🔗 http://testphp.vulnweb.com/Mod_Rewrite_Shop/ [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu]
🔗 http://testphp.vulnweb.com/hpp/ [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu]
🔗 http://testphp.vulnweb.com/disclaimer.php [200] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu]
🔗 http://testphp.vulnweb.com/login.php [200] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu]
🔗 http://testphp.vulnweb.com/Templates/main_dynamic_template.dwt.php [200] [Ubuntu, Nginx:1.19.0, PHP:5.6.40]
🔗 http://testphp.vulnweb.com/guestbook.php [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu, DreamWeaver]
🔗 http://testphp.vulnweb.com/userinfo.php
🔗 http://testphp.vulnweb.com/Mod_Rewrite_Shop/Details/color-printer/3/ [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu]
🔗 http://testphp.vulnweb.com/comment.php?aid=3 [200] [Ubuntu, Nginx:1.19.0, PHP:5.6.40]
🔗 http://testphp.vulnweb.com/search.php?test=query [200] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu]
🔗 http://testphp.vulnweb.com/Mod_Rewrite_Shop/Details/network-attached-storage-dlink/1/ [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu]
🔗 http://testphp.vulnweb.com/Mod_Rewrite_Shop/Details/web-camera-a4tech/2/ [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu]
🔗 http://testphp.vulnweb.com/Templates/high
🔗 http://testphp.vulnweb.com/comment.php?aid=2
🔗 http://testphp.vulnweb.com/artists.php?artist=1 [200] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu]
🔗 http://testphp.vulnweb.com/hpp/?pp=12 [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu]
🔗 http://testphp.vulnweb.com/signup.php [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu, DreamWeaver]
🔗 http://testphp.vulnweb.com/comment.php?aid=1
🔗 http://testphp.vulnweb.com/AJAX/showxml.php [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu]
🔗 http://testphp.vulnweb.com/AJAX/styles.css [200] [Nginx:1.19.0]
🔗 http://testphp.vulnweb.com/artists.php?artist=3 [200] [PHP:5.6.40, Ubuntu, DreamWeaver, Nginx:1.19.0]
🔗 http://testphp.vulnweb.com/listproducts.php?cat=3 [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu, DreamWeaver]
🔗 http://testphp.vulnweb.com/listproducts.php?cat=2 [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu, DreamWeaver]
🔗 http://testphp.vulnweb.com/showimage.php?file= [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu]
🔗 http://testphp.vulnweb.com/artists.php?artist=2 [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu, DreamWeaver]
🔗 http://testphp.vulnweb.com/listproducts.php?cat=4
🔗 http://testphp.vulnweb.com/listproducts.php?cat=1 [200] [Nginx:1.19.0, PHP:5.6.40, Ubuntu, DreamWeaver]
🗄 Saved TXT reports to 
   • /home/vagrant/.secator/reports/default/tasks/task_katana_target_2023_07_04-01_33_13_152782_PM.txt
   • /home/vagrant/.secator/reports/default/tasks/task_katana_url_2023_07_04-01_33_13_152782_PM.txt
`katana` found some pretty interesting results, including some PHP files that could potentially be vulnerable.
Step 2: Run httpx on found URLs to alive URLs Crawlers usually find URLs from HTML response bodies, which means we have no ideas if those URLs will actually respond to HTTP requests or not. In order to filter only the URLs that will give a valid HTTP status code, we can use `httpx` on the previous results (txt file).\ \ We'll add some rate limiting (`-rl`) in order to respect the server and not DDoS it for no reason. We want to keep only some HTTP codes (`-mc`) for instance `200`, `301`, and `500` in case we have there are errors we can take advantage of: ```bash secator x httpx /home/vagrant/.secator/reports/default/tasks/task_katana_target_2023_07_04-01_33_13_152782_PM.txt -rl 10 -mc 200,301,500 __ ________ _________ _/ /_____ _____ / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/ (__ / __/ /__/ /_/ / /_/ /_/ / / /____/\___/\___/\__,_/\__/\____/_/ v0.0.1 freelabz.com httpx -silent -td -asn -cdn -l /tmp/httpx_2023_07_04-01_19_35_686343_PM.txt -json -threads 50 -match-code 200,301,500 _base.py:614 🔗 http://testphp.vulnweb.com/showimage.php?file [200] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [image/jpeg] [196] 🔗 http://testphp.vulnweb.com/hpp [200] [HTTP Parameter Pollution Example] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [203] 🔗 http://testphp.vulnweb.com/cart.php [200] [you cart] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [4903] 🔗 http://testphp.vulnweb.com/categories.php [200] [picture categories] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [6115] 🔗 http://testphp.vulnweb.com/artists.php?artist=3 [200] [artists] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [6193] 🔗 http://testphp.vulnweb.com/comment.php?aid=3 [200] [comment on artist] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [1252] 🔗 http://testphp.vulnweb.com/comment.php?aid=2 [200] [comment on artist] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [1252] 🔗 http://testphp.vulnweb.com/AJAX/showxml.php [200] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [11] 🔗 http://testphp.vulnweb.com/style.css [200] [nginx/1.19.0] [Nginx:1.19.0] [text/css] [5482] 🔗 http://testphp.vulnweb.com/artists.php?artist=1 [200] [artists] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [6251] 🔗 http://testphp.vulnweb.com/Templates/main_dynamic_template.dwt.php [200] [Document titleg] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [4697] 🔗 http://testphp.vulnweb.com/Mod_Rewrite_Shop/Details/color-printer/3 [200] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [313] 🔗 http://testphp.vulnweb.com/AJAX/styles.css [200] [nginx/1.19.0] [Nginx:1.19.0] [text/css] [562] 🔗 http://testphp.vulnweb.com/hpp/?pp=12 [200] [HTTP Parameter Pollution Example] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [383] 🔗 http://testphp.vulnweb.com/login.php [200] [login page] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [5523] 🔗 http://testphp.vulnweb.com/guestbook.php [200] [guestbook] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [5390] 🔗 http://testphp.vulnweb.com/signup.php [200] [signup] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [6033] 🔗 http://testphp.vulnweb.com/Mod_Rewrite_Shop/Details/network-attached-storage-dlink/1 [200] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [319] 🔗 http://testphp.vulnweb.com/search.php?test=query [200] [search] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [4732] 🔗 http://testphp.vulnweb.com/listproducts.php?cat=3 [200] [pictures] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [4699] 🔗 http://testphp.vulnweb.com/AJAX/index.php [200] [ajax test] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [4236] 🔗 http://testphp.vulnweb.com/disclaimer.php [200] [disclaimer] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [5524] 🔗 http://testphp.vulnweb.com/artists.php?artist=2 [200] [artists] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [6193] 🔗 http://testphp.vulnweb.com/listproducts.php?cat=4 [200] [pictures] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [4699] 🔗 http://testphp.vulnweb.com/artists.php [200] [artists] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [5328] 🔗 http://testphp.vulnweb.com/listproducts.php?cat=2 [200] [pictures] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [5311] 🔗 http://testphp.vulnweb.com/Mod_Rewrite_Shop [200] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [975] 🔗 http://testphp.vulnweb.com/listproducts.php?cat=1 [200] [pictures] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [7880] 🔗 http://testphp.vulnweb.com/comment.php?aid=1 [200] [comment on artist] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [1252] 🔗 http://testphp.vulnweb.com/Mod_Rewrite_Shop/Details/web-camera-a4tech/2 [200] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [279] 🔗 http://testphp.vulnweb.com [200] [Home of Acunetix Art] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [4958] 🔗 http://testphp.vulnweb.com/index.php [200] [Home of Acunetix Art] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [4958] 🗄 Saved TXT reports to • /home/vagrant/.secator/reports/default/tasks/task_httpx_target_2023_07_04-01_34_22_081275_PM.txt • /home/vagrant/.secator/reports/default/tasks/task_httpx_url_2023_07_04-01_34_22_081275_PM.txt ```
Step 3: Run gf on found URLs `gf` allows to run patterns on URLs, we can use it to quickly detect potential interesting URLs including XSS, LFIs, SSRFs, RCEs, Interesting params, or Insecure Direct Object references.\ Let's hunt for potential XSS: 
secator x gf --pattern xss /home/vagrant/.secator/reports/default/tasks/task_httpx_target_2023_07_04-01_34_22_081275_PM.tx
🏷️ [xss] http://testphp.vulnweb.com/comment.php?aid=1 []
🏷️ [xss] http://testphp.vulnweb.com/comment.php?aid=2 []
🏷️ [xss] http://testphp.vulnweb.com/comment.php?aid=3 []
🏷️ [xss] http://testphp.vulnweb.com/hpp/?pp=12 []
Mmmh, it seems like we might have some interesting XSS targets here. Let's use `dalfox` to find out !
Step 4: Run dalfox on potential XSS links `dalfox` is a pretty thorough XSS checker, let's run it on the targets we've identified: ``` secator x dalfox http://testphp.vulnweb.com/hpp/?pp=12 __ ________ _________ _/ /_____ _____ / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/ (__ / __/ /__/ /_/ / /_/ /_/ / / /____/\___/\___/\__,_/\__/\____/_/ v0.0.1 freelabz.com dalfox --silence url 'http://testphp.vulnweb.com/hpp/?pp=12' --format json --worker 50 _base.py:614 🚨 [Verified XSS] [high] http://testphp.vulnweb.com/hpp/ [CWE-83] [inject_type:inATTR-double(3)-URL, poc_type:plain, method:GET, data:http://testphp.vulnweb.com/hpp/?pp=12%22id%3Dx+tabindex%3D1+style%3D%22display%3Ablock%3Btransition%3Aoutline+1s%3B%22+ontransitionend%3Dalert.apply%28null%2C1%29+class%3Ddalfox+, param:pp, payload:"id=x tabindex=1 style="display:block;transition:outline 1s;" ontransitionend=alert.apply(null,1) class=dalfox , evidence:4 line: ms.php?p=valid&pp=12"id=x tabindex=1 style="display:block;transition:outline 1s;, message_id:1103, message_str:Triggered XSS Payload (found DOM Object): pp="id=x tabindex=1 style="display:block;transition:outline 1s;" ontransitionend=alert.apply(null,1) class=dalfox ] ``` **We have found a Verified XSS !** We could verify this XSS in our browser to validate it, but `dalfox` already gave us evidence that it works...
*** ## Using a task pipe `secator` supports UNIX pipes out-of-the-box. You can write a `secator` pipe that can automate the 4 previous steps: ```bash secator x katana http://testphp.vulnweb.com | secator x httpx | secator x gf --pattern lfi | secator x dalfox ``` You don't have to specify any additional flag than when running normally, since `secator` will detect that you are running a UNIX pipe and automagically pass the proper inputs between task invocations: * By default, it will pass on `stdin` the raw string results from the previous task. If the task can output multiple [Output types](/in-depth/concepts/output-types), then the first one in the class definition `output_types` attribute is picked. * We can specify which fields we want to use when passing raw strings using the `-fmt` option.
Command output ``` katana -silent -jc -js-crawl -known-files all -u http://testphp.vulnweb.com -json -concurrency 50 _base.py:614 ... httpx -silent -td -asn -cdn -l /tmp/httpx_2023_07_04-01_50_43_044323_PM.txt -json -threads 50 _base.py:614 ... cat /tmp/gf_2023_07_04-01_50_49_787397_PM.txt | gf xss _base.py:614 dalfox --silence file /tmp/dalfox_2023_07_04-01_50_49_855768_PM.txt --format json --worker 50 _base.py:614 🚨 [Verified XSS] [high] http://testphp.vulnweb.com/hpp/ [CWE-83] [inject_type:inATTR-double(3)-URL, poc_type:plain, method:GET, data:http://testphp.vulnweb.com/hpp/?pp=12%22%26%2339%3B%3E%3Caudio+controls+ondurationchange%3Dprompt%281%29+id%3Ddalfox%3E%3Csource+src%3D1.mp3+type%3Daudio%2Fmpeg%3E%3C%2Faudio%3E, param:pp, payload:"'>, evidence:4 line: ms.php?p=valid&pp=12"'>
This is already a good time improvement over the previous lengthy set of tasks. *** ## Using a workflow Task pipes are good to quickly find things, but workflows are much better to repeat the same set of tasks over separate sets of targets, with the same input options for all tasks, while filtering final results and reacting to live results, etc... Here is a `secator` workflow corresponding to the previous set of tasks: {% code title="\~/.secator/templates/xss\_finder.yaml" %} ```yaml type: workflow name: xss_finder description: XSS Finder tasks: katana: description: Crawling root URL httpx: description: Finding alive URLs targets_: - type: url field: url condition: item.status_code == 0 gf: description: Identifying XSS pattern: xss targets_: - url.url dalfox: description: Verifying XSS targets_: - type: tag field: match condition: item.name == 'xss' ``` {% endcode %} And here is the workflow run using `secator`: ``` secator w xss_finder.yaml http://testphp.vulnweb.com ```
Command output ``` __ ________ _________ _/ /_____ _____ / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/ (__ / __/ /__/ /_/ / /_/ /_/ / / /____/\___/\___/\__,_/\__/\____/_/ v0.0.1 freelabz.com _base.py:614 🔧 Crawling root URL ... katana -silent -jc -js-crawl -known-files all -u http://testphp.vulnweb.com -json -concurrency 50 _base.py:614 🔗 http://testphp.vulnweb.com [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40, DreamWeaver] 🔗 http://testphp.vulnweb.com/high [404] [Nginx:1.19.0] 🔗 http://testphp.vulnweb.com/Mod_Rewrite_Shop/ [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40] 🔗 http://testphp.vulnweb.com/index.php 🔗 http://testphp.vulnweb.com/style.css [200] [Nginx:1.19.0] 🔗 http://testphp.vulnweb.com/guestbook.php [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40, DreamWeaver] 🔗 http://testphp.vulnweb.com/Templates/main_dynamic_template.dwt.php [200] [PHP:5.6.40, Nginx:1.19.0, Ubuntu] 🔗 http://testphp.vulnweb.com/AJAX/index.php [200] [PHP:5.6.40, Nginx:1.19.0, Ubuntu] 🔗 http://testphp.vulnweb.com/login.php [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40, DreamWeaver] 🔗 http://testphp.vulnweb.com/cart.php [200] [PHP:5.6.40, Nginx:1.19.0, Ubuntu, DreamWeaver] 🔗 http://testphp.vulnweb.com/privacy.php [404] [Nginx:1.19.0, Ubuntu, PHP:5.6.40] 🔗 http://testphp.vulnweb.com/hpp/ [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40] 🔗 http://testphp.vulnweb.com/categories.php [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40, DreamWeaver] 🔗 http://testphp.vulnweb.com/artists.php [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40, DreamWeaver] 🔗 http://testphp.vulnweb.com/disclaimer.php [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40, DreamWeaver] 🔗 http://testphp.vulnweb.com/userinfo.php 🔗 http://testphp.vulnweb.com/listproducts.php?cat=3 🔗 http://testphp.vulnweb.com/comment.php?aid=1 [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40] 🔗 http://testphp.vulnweb.com/listproducts.php?cat=4 [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40, DreamWeaver] 🔗 http://testphp.vulnweb.com/comment.php?aid=3 🔗 http://testphp.vulnweb.com/artists.php?artist=2 [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40, DreamWeaver] 🔗 http://testphp.vulnweb.com/comment.php?aid=2 🔗 http://testphp.vulnweb.com/signup.php [200] [PHP:5.6.40, DreamWeaver, Nginx:1.19.0, Ubuntu] 🔗 http://testphp.vulnweb.com/artists.php?artist=3 [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40, DreamWeaver] 🔗 http://testphp.vulnweb.com/artists.php?artist=1 [200] [Ubuntu, PHP:5.6.40, DreamWeaver, Nginx:1.19.0] 🔗 http://testphp.vulnweb.com/listproducts.php?cat=1 [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40, DreamWeaver] 🔗 http://testphp.vulnweb.com/AJAX/showxml.php [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40] 🔗 http://testphp.vulnweb.com/showimage.php?file= [200] [Nginx:1.19.0, Ubuntu, PHP:5.6.40] 🔗 http://testphp.vulnweb.com/hpp/?pp=12 [200] [Ubuntu, PHP:5.6.40, Nginx:1.19.0] 🔗 http://testphp.vulnweb.com/AJAX/styles.css [200] [Nginx:1.19.0] 🔗 http://testphp.vulnweb.com/Templates/high 🔗 http://testphp.vulnweb.com/Mod_Rewrite_Shop/Details/network-attached-storage-dlink/1/ [200] [PHP:5.6.40, Nginx:1.19.0, Ubuntu] 🔗 http://testphp.vulnweb.com/Mod_Rewrite_Shop/Details/web-camera-a4tech/2/ [200] [PHP:5.6.40, Nginx:1.19.0, Ubuntu] 🔗 http://testphp.vulnweb.com/listproducts.php?cat=2 [200] [PHP:5.6.40, Nginx:1.19.0, Ubuntu, DreamWeaver] 🔗 http://testphp.vulnweb.com/Mod_Rewrite_Shop/Details/color-printer/3/ [200] [PHP:5.6.40, Nginx:1.19.0, Ubuntu] 🔗 http://testphp.vulnweb.com/search.php?test=query [200] [Nginx:1.19.0, Ubuntu, DreamWeaver, PHP:5.6.40] 🔧 Finding alive URLs ... httpx -silent -td -asn -cdn -l /tmp/httpx_2023_07_04-02_03_45_807367_PM.txt -json -threads 50 _base.py:614 🔗 http://testphp.vulnweb.com/Templates/high [404] [404 Not Found] [nginx/1.19.0] [Nginx:1.19.0] [text/html] [153] 🔗 http://testphp.vulnweb.com/index.php [200] [Home of Acunetix Art] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [4958] 🔗 http://testphp.vulnweb.com/comment.php?aid=2 [200] [comment on artist] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [1252] 🔗 http://testphp.vulnweb.com/userinfo.php [302] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [14] 🔗 http://testphp.vulnweb.com/comment.php?aid=3 [200] [comment on artist] [nginx/1.19.0] [Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [1252] 🔗 http://testphp.vulnweb.com/listproducts.php?cat=3 [200] [pictures] [nginx/1.19.0] [DreamWeaver, Nginx:1.19.0, PHP:5.6.40, Ubuntu] [text/html] [4699] 🔧 Identifying XSS ... cat /tmp/gf_2023_07_04-02_03_52_398624_PM.txt | gf xss _base.py:614 🏷️ [xss] http://testphp.vulnweb.com/comment.php?aid=1 [] 🏷️ [xss] http://testphp.vulnweb.com/comment.php?aid=2 [] 🏷️ [xss] http://testphp.vulnweb.com/comment.php?aid=3 [] 🏷️ [xss] http://testphp.vulnweb.com/hpp/?pp=12 [] 🔧 Verifying XSS ... dalfox --silence file /tmp/dalfox_2023_07_04-02_03_52_411553_PM.txt --format json --worker 50 _base.py:614 🚨 [Verified XSS] [high] http://testphp.vulnweb.com/hpp/ [CWE-83] [inject_type:inATTR-double(3)-URL, poc_type:plain, method:GET, data:http://testphp.vulnweb.com/hpp/?pp=12%22onmouseenter%3Dprompt.call%28null%2C1%29+class%3Ddalfox+, param:pp, payload:"onmouseenter=prompt.call(null,1) class=dalfox , evidence:4 line: ms.php?p=valid&pp=12"onmouseenter=prompt.call(null,1) class=dalfox ">link2`.
Example - export live results to MongoDB First, install the `mongodb` addon using `secator install addons mongodb` Then, use the `-driver` flag route your results: ```bash secator x httpx mydomain.com -driver mongodb secator w host_recon mydomain.com -driver mongodb secator s domain mydomain.com -driver mongodb ```
{% hint style="info" %} Learn more about [Drivers](/in-depth/concepts/drivers). {% endhint %} *** # Meta options ... or options that are mutualized among task categories for efficiency, speed, and user-friendliness. **Meta options** apply to **tasks**, **workflows**, or **scans**. When passed to **workflows** or **scans**, they will be passed to each task contained in the runner. {% hint style="warning" %} Some tasks, workflows, or scans do not support some of the options mentioned below. Run**`secator x/w/s --help`** to get the complete list of supported options. {% endhint %} *** ## Execution Options ### Threads (`-threads`) Number of threads to use. Applies to all tasks supporting threads (or concurrency).
Example: set 50 threads ```bash secator w host_recon mydomain.com -threads 50 ```
*** ## Requests Options The following options will apply to tasks making network requests (if they implement it), no matter the protocol used (HTTP, TCP, UDP, DNS, FTP, ...). ### Proxy (`-proxy`) Proxy (HTTP, Socks5, ...) to use when communicating with the targets.
Example: set proxies in config and -proxy to auto ```bash secator config set http.http_proxy http://localhost:8080 secator config set http.socks5_proxy socks5://localhost:9050 secator w host_recon mydomain.com -proxy auto # auto choose the right proxy ```
{% hint style="info" %} Learn more about [Proxies](/in-depth/concepts/proxies). {% endhint %} *** ### Rate limit (`-rl`) Rate limit is an upper limit on the number of requests per second.
Example: set a rate limit of 50 requests/second ```bash secator w host_recon mydomain.com -rl 50 ```
*** ### Timeout (`-timeout`) Timeout is the time to wait (in seconds) before giving up on the request.
Example: set a request timeout of 10 seconds ```bash secator w host_recon mydomain.com -timeout 10 ```
*** ### Retries (`-retries`) Number of retries for failed requests.
Example: set 5 retries for all requests ```bash secator w host_recon mydomain.com -retries 5 ```
*** ### Delay (`-d`) Delay to add between each request (in seconds).
Example: add a 0.5 second delay between requests ```bash secator w host_recon mydomain.com -d 0.5 ```
*** ## HTTP Options The following options will apply to tasks making HTTP requests (if they implement it). ### Header (`-H`) Custom header to add to each request in the form "KEY1:VALUE1;; KEY2:VALUE2".
Example: set an Authorization and an Accept header ```bash secator x cariddi mydomain.com -H "Authorization: Basic ;; Accept: application/json" ```
*** ### Method (`-X`) HTTP method to use for request GET, POST, PUT, DELETE, etc...
Example: use POST method for fuzzing ```bash secator x ffuf mydomain.com -X POST ```
*** ### Data (`-data`) Data to send in the request body.
Example: send JSON data in POST request ```bash secator x ffuf mydomain.com/api -X POST -data '{"key":"value"}' ```
*** ### User-agent (`-ua`) Custom user-agent to use for request.
Example: use secator as a user agent value ```bash secator x dalfox mydomain.com -ua secator ```
*** ### **Match regex (`-mr)`** Keep responses which body content match the input.
Example: keep responses which match the regexMySQLError.* ```bash secator x ffuf mydomain.com -mr MySQLError.* ```
*** ### Match size (`-ms`) Keep responses which body size (in bytes) match the input.
Example: keep responses with 1025 bytes ```bash secator x katana mydomain.com -ms 1026 # bytes ```
*** ### Match-words (`-mw)` Keep responses which body word count match the input.
Example: keep responses with 10 words ```bash secator x katana mydomain.com -mw 10 ```
*** ### Match code (`-mc`) Keep responses which HTTP status codes match the input.
Example: keep responses matching HTTP statuses 200,400,501 ```bash secator x katana mydomain.com -mc 200,400,501 ```
*** ### Filter regex (`-fr`) Filter out responses which body content match the input.
Example: filter out responses containing the string LoginPage ```bash secator x ffuf mydomain.com -fr LoginPage.* ```
*** ### Filter codes (`-fc`) Filter out responses which HTTP status codes match the input.
Example: filter out responses matching HTTP status 500 ```bash secator x ffuf mydomain.com -fc 500 ```
*** ### Filter size (`-fs)` Filter out responses which body size (in bytes) match the input.
Example: filter out responses with 1025 bytes ```bash secator x ffuf mydomain.com -fs 1025 ```
*** ### Filter words (`-fw`) Filter out responses which body word count match the input.
Example: filter out responses with 10 words ```bash secator x ffuf mydomain.com -fw 10 ```
*** ### Follow redirect (`-frd`) Follow all http redirects.
Example: follow HTTP redirects ```bash secator x katana mydomain.com -frd ```
*** ### Depth (`-depth`) Scan depth for crawling tasks.
Example: set crawl depth to 3 ```bash secator x gospider mydomain.com -depth 3 ```
*** ### Replay proxy (`-P`) Proxy to use for replay requests (useful for fuzzing tasks).
Example: use a proxy for replay requests ```bash secator x ffuf mydomain.com/FUZZ -P http://localhost:8080 ```
*** ### Wordlist (`-w`) Custom wordlist to use.
Example: use fuzz-Bo0oM wordlist ```bash secator x ffuf mydomain.com/FFUF/ -w /usr/share/seclists/Fuzzing/fuzz-Bo0oM.txt ```
*** ## Port Scanning Options The following options apply to port scanning tasks (e.g., `naabu`, `nmap`). ### Ports (`-p`) Only scan specific ports. Accepts a comma-separated list of ports, or `-` for all ports.
Example: scan ports 80, 443, and 8080 ```bash secator x naabu mydomain.com -p 80,443,8080 ```
*** ### Top ports (`-tp`) Scan the N most common ports.
```bash secator x naabu mydomain.com -tp 100 ```
*** # Input formats ... or how to pass targets to secator. `secator` is built to be flexible in terms of input formats. *** ## **Direct input** Inputs can be passed directly as an argument to the command / workflow / scan you wish to run: ```sh secator x httpx example.com # single input secator x httpx example.com,example2.com,example3.com # multiple comma-separated inputs ``` *** ## **File input** Input can also be passed from a file containing one item per line: ```sh secator x httpx urls.txt ``` *** ## **Stdin input** Input can also be passed directly from `stdin`: ```sh cat urls.txt | secator x httpx ``` You can build basic workflow using UNIX pipes: ```bash secator x subfinder vulnweb.com | secator x nmap | secator x httpx ``` {% hint style="info" %} For more complex workflows, we highly recommend using the YAML-based workflow definitions or the code-based workflow definitions (see [Writing workflows](/for-developers/writing-workflows)). {% endhint %} *** # Output options ... or how to change secator's console output. `secator` is built to be flexible in terms of output options. *** ### Console The default `secator` output is the [Output types](/in-depth/concepts/output-types) `repr` function. It is supposed to be pretty and readable to quickly understand `secator`'s findings:

Console output

Unicode icons are printed before each result to distinguish each output type: * [Output types](/in-depth/concepts/output-types#exploit) * [Output types](/in-depth/concepts/output-types#ip) * [Output types](/in-depth/concepts/output-types#port) * [Output types](/in-depth/concepts/output-types#record) * [Output types](/in-depth/concepts/output-types#subdomain) * [Output types](/in-depth/concepts/output-types#tag) * [Output types](/in-depth/concepts/output-types#url) * [Output types](/in-depth/concepts/output-types#useraccount) * [Output types](/in-depth/concepts/output-types#vulnerability) When an output type has a low `confidence`, the output will be dimmed:
*** ### JSON lines (`-json`) You can use `-json` to output live results as JSON lines: ```bash secator x httpx example.com -json ``` {% hint style="info" %} JSON lines output is pipeable / streameable to other tools like `jq`. {% endhint %} *** ### Raw (`-raw`) You can use `-raw`to output live results in plaintext format: ``` secator x httpx example.com -raw ``` {% hint style="info" %} Raw output is saveable to txt files or can be used for chaining tasks using UNIX pipes. {% endhint %} *** ### Custom format (`-fmt`) You can use `-fmt`to output live results in a format of your choice: ```bash secator x naabu example.com -fmt '{host}:{port} -> {service_name}' ``` {% hint style="info" %} Custom formatting is based on the [Output types](/in-depth/concepts/output-types) fields. {% endhint %} *** # Philosophy & design ... or the core concepts behind the pillars of secator. *** ## Why do we need another tool ? Traditional pentesting sessions can be a pain: * Use of dozens of commands (each very good at what it does) * Each command has different options * Each command outputs results in a different format Here is for instance a basic host scan that we run day-to-day on many targets: {% tabs %} {% tab title="Standard" %} 
naabu -Pn -silent -host example.com -json -rate 10 -c 50 > open_ports.txt
nuclei -silent -sj -si 20 -hm -u example.com -jsonl -tags network,ssl -proxy socks5://tor-privoxy:9050 -rate-limit 10 -c 50
httpx -silent -td -asn -cdn -l open_ports.txt -json -proxy socks5://tor-privoxy:9050 -rate-limit 10 -threads 50 -match-code 200,204,301,302,307,401,403,405,500
katana -silent -jc -js-crawl -known-files all -u example.com -json -proxy socks5://tor-privoxy:9050 -rate-limit 10 -concurrency 50
echo https://example.com | cariddi -info -s -err -e -ext 1 -json -proxy socks5://tor-privoxy:9050 -c 50
{% endtab %} {% tab title="Secator" %} ```bash secator s host example.com -json -proxy socks5://tor-privoxy:9050 -rl 10 -threads 50 -mc 200,204,301,302,307,401,403,405,500 ``` {% endtab %} {% endtabs %} With `secator` the goal is to unify all these awesome tools by creating an abstract layer for input and output, such that all tools "speak" the same language and we can mutualize options that will apply to all tools, unlocking the ability to run complex workflows. *** ## Who is it made for ? * [x] Bug-bounty hunters * [x] Pentesters * [x] Security researchers * [x] Companies of any size *** ## Why is the license BSL ? We believe in open-source 100%. **Freelabz** founders have written code for open-source for as long as they started working in the tech industry. `secator` is and will remain free-to-use, forkable and open to community contributions forever, and we believe in the collective to make it one of the de-facto tools in the security world. However, **Freelabz** is a young company that is seeking to make a living and pay its workers correctly. We are working on a paid product derived from `secator` and BSL allows us to restrict commercial uses of `secator` by big tech actors who could profit from its success. All standard OSS rules still apply, which means you can still read / fork / modify it, use it in other OSS projects, and even use it in production **as long as you don't sell a service based on it**. Feel free to reach out to us if you want to use `secator` in a commercial tool, and we will review your request on a case-by-case basis. *** ## Design principles ### **Curated list of tools** Tools integrated to `secator` **MUST** be **fast**, **efficient**, **well-maintained**, and have **structured output** (either `JSON`, `JSON lines`, `CSV`, or `XML`). {% hint style="info" %} We do make exceptions for really awesome tools and write custom parsers (e.g: `nmap`). {% endhint %} ### **Unified input options** `secator` tools belonging to the same category (eg: fuzzers) **MUST** end up with **mutualized** [**input options**](/runner-options/meta-options), while still retaining the capability to use unique options for each command. ### **Unified output schema** Tools belonging to the same category **MUST** have **unified** [**output types**](/in-depth/concepts/output-types), allowing you to run multiple commands and aggregate results quickly. ### **CLI and library usage** When `secator` is called as a library from other Python code, the output **MUST** be **structured** (list of dicts). Results **MUST** also be yielded in **realtime**. When `secator` is called as a CLI, various [Exporters](/in-depth/concepts/exporters) **MUST** be available, such as **csv**, **json**, **txt**, or **table**. ### **Distributed options** `secator` **MUST** work in both **synchronous** mode (default) and **distributed** mode. Switch from synchronous to distributed when you want to increase the scanning speed **MUST** be easy, by simply configuring Celery worker with your broker and results backend of choice. ### **From simple tasks to complex workflows** `secator` **MUST** be useful for **running simple tasks** like in CTFs, bug-bounties or hackathon, or to **automate entire workflows**. ### **Customizable** `secator` **MUST** be **customizable**, so that the community can contribute tasks, workflows, and scans to the repo if they can serve the greater good. *** # Distributed runs with Celery ... or how you can 10x your scanning speed and massively parallelize your workflows. By default, `secator` runs all tasks synchronously. This guide shows how to enable distributed runs using Celery workers, which unlocks [Writing workflows](/for-developers/writing-workflows#concurrent-tasks). *** ## Prerequisite To use distributed runs, make sure the `worker` addon is installed: ```bash secator install addons worker ``` ## Step 1: Configure a broker and result backend \[optional] {% hint style="info" %} This step is **optional**. If you do not configure a broker, the **file system** will be used as a broker and result backend. Note that this works only if the client and worker run on the same VM. {% endhint %} You can set up a task queue using Celery with the broker and a results backend of your choice, and run Celery workers to execute tasks from the broker queue. The following is an example using `redis`, but you can use any [supported Celery broker and backend](https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/index.html). **Install `redis` addon:** ```bash secator install addons redis ``` **Install `redis`:** ```sh sudo apt install redis ``` **Start `redis` and enable at boot:** ```sh sudo systemctl enable redis sudo systemctl start redis ``` **Configure `secator` to use Redis:** 
secator config set celery.broker_url redis://<REDIS_IP>:6379/0
secator config set celery.result_backend redis://<REDIS_IP>:6379/0
{% hint style="warning" %} Make sure you replace `` in the variables above with the IP of your Redis server. {% endhint %} *** ## **Step 2: Start a Celery worker** ```sh secator worker ``` *** ## **Step 3: Run a task, workflow or scan** {% tabs %} {% tab title="CLI" %} ```bash secator x httpx wikipedia.org secator w host_recon wikipedia.org secator s host wikipedia.org ``` {% endtab %} {% tab title="Python" %} ```python from secator.tasks impor httpx from secator.workflows import host_recon from secator.scans import host target = 'testphp.vulnweb.com' for result in httpx(target, sync=False): print(result) for result in host_recon(target, sync=False): print(result) for result in host(target, sync=False): print(result) ``` {% endtab %} {% endtabs %} {% hint style="info" %} If you want to run synchronously (bypassing the broker), you can use the `--sync` flag (CLI) or the `sync` kwarg (Python). {% endhint %} *** # Concepts ... or foundational blocks that you must understand in order to go deeper. This page presents various `secator` concepts: * [Output types](/in-depth/concepts/output-types) * [Proxies](/in-depth/concepts/proxies) * [Exporters](/in-depth/concepts/exporters) * [Runners](/in-depth/concepts/runners) * [Drivers](/in-depth/concepts/drivers) * [Profiles](/in-depth/concepts/profiles) *** # Output types ... or how secator unifies all output into common schemas. `secator` uses the notion of output types to uniformize tasks output. A task can output one or multiple output types. The currently available output types are [#certificate](#certificate "mention"), [#domain](#domain "mention"), [#exploit](#exploit "mention"), [#ip](#ip "mention"), [#port](#port "mention"), [#record](#record "mention"), [#subdomain](#subdomain "mention"), [#tag](#tag "mention"), [#url](#url "mention"), [#useraccount](#useraccount "mention"), [#vulnerability](#vulnerability "mention"). {% hint style="info" %} Feel free to request more output types by [opening an issue](https://github.com/freelabz/secator/issues/new/choose) on the GitHub repo. {% endhint %} *** ## 📜 Certificate {% code title="secator/output\_types/certificate.py" %} ```python @dataclass class Certificate(OutputType): host: str fingerprint_sha256: str = field(default='') ip: str = field(default='', compare=False) raw_value: str = field(default='', compare=False) subject_cn: str = field(default='', compare=False) subject_an: list[str] = field(default_factory=list, compare=False) not_before: datetime = field(default=None, compare=False) not_after: datetime = field(default=None, compare=False) issuer_dn: str = field(default='', compare=False) issuer_cn: str = field(default='', compare=False) issuer: str = field(default='', compare=False) self_signed: bool = field(default=True, compare=False) trusted: bool = field(default=False, compare=False) status: str = field(default=CERTIFICATE_STATUS_UNKNOWN, compare=False) keysize: int = field(default=None, compare=False) serial_number: str = field(default='', compare=False) ciphers: list[str] = field(default_factory=list, compare=False) ``` {% endcode %} *** ## 🪪 Domain {% code title="secator/output\_types/domain.py" %} ```python @dataclass class Domain(OutputType): domain: str registrar: str = '' alive: bool = False creation_date: str = '' expiration_date: str = '' registrant: str = '' extra_data: dict = field(default_factory=dict, compare=False) ``` {% endcode %} *** ## ⍼ Exploit {% code title="secator/output\_types/exploit.py" %} ```python @dataclass class Exploit(OutputType): name: str provider: str id: str matched_at: str = '' ip: str = '' confidence: str = 'low' reference: str = field(default='', repr=True, compare=False) cves: list = field(default_factory=list, compare=False) tags: list = field(default_factory=list, compare=False) extra_data: dict = field(default_factory=dict, compare=False) ``` {% endcode %} *** ## 💻 Ip {% code title="secator/output\_types/ip.py" %} ```python @dataclass class Ip(OutputType): ip: str host: str = field(default='', repr=True, compare=False) alive: bool = False protocol: str = field(default=IpProtocol.IPv4) extra_data: dict = field(default_factory=dict, compare=False) ``` {% endcode %} *** ## 🔓 Port {% code title="secator/output\_types/port.py" %} ```python @dataclass class Port(OutputType): port: int ip: str state: str = 'UNKNOWN' service_name: str = field(default='', compare=False) cpes: list = field(default_factory=list, compare=False) host: str = field(default='', repr=True, compare=False) protocol: str = field(default='tcp', repr=True, compare=False) extra_data: dict = field(default_factory=dict, compare=False) confidence: str = field(default='low', repr=False, compare=False) ``` {% endcode %} *** ## 🎤 Record {% code title="secator/output\_types/record.py" %} ```python @dataclass class Record(OutputType): name: str type: str host: str = '' extra_data: dict = field(default_factory=dict, compare=False) ``` {% endcode %} *** ## 🏰 Subdomain {% code title="secator/output\_types/subdomain.py" %} ```python @dataclass class Subdomain(OutputType): host: str domain: str verified: bool = field(default=False, compare=False) sources: List[str] = field(default_factory=list, compare=False) extra_data: dict = field(default_factory=dict, compare=False) ``` {% endcode %} *** ## 🏷️ Tag {% code title="secator/output\_types/tag.py" %} ```python @dataclass class Tag(OutputType): name: str value: str match: str category: str = field(default='general') extra_data: dict = field(default_factory=dict, repr=True, compare=False) stored_response_path: str = field(default='', compare=False) ``` {% endcode %} *** ## 🔗 Url {% code title="secator/output\_types/url.py" %} ```python @dataclass class Url(OutputType): url: str host: str = field(default='', compare=False) verified: bool = field(default=False, compare=False) status_code: int = field(default=0, compare=False) title: str = field(default='', compare=False) protocol: str = field(default='', compare=False) webserver: str = field(default='', compare=False) tech: list = field(default_factory=list, compare=False) content_type: str = field(default='', compare=False) content_length: int = field(default=0, compare=False) time: str = field(default='', compare=False) method: str = field(default='', compare=False) words: int = field(default=0, compare=False) lines: int = field(default=0, compare=False) screenshot_path: str = field(default='', compare=False) stored_response_path: str = field(default='', compare=False) confidence: str = field(default='high', compare=False) response_headers: dict = field(default_factory=dict, repr=True, compare=False) request_headers: dict = field(default_factory=dict, repr=True, compare=False) extra_data: dict = field(default_factory=dict, compare=False) is_directory: bool = field(default=False, compare=False) is_root: bool = field(default=False, compare=False) is_redirect: bool = field(default=False, compare=False) ``` {% endcode %} *** ## 👤 UserAccount {% code title="secator/output\_types/user\_account.py" %} ```python @dataclass class UserAccount(OutputType): username: str url: str = '' email: str = '' site_name: str = '' extra_data: dict = field(default_factory=dict, compare=False) ``` {% endcode %} *** ## 🚨 Vulnerability {% code title="secator/output\_types/vulnerability.py" %} ```python @dataclass class Vulnerability(OutputType): name: str provider: str = '' id: str = '' matched_at: str = '' ip: str = field(default='', compare=False) confidence: str = 'low' severity: str = 'unknown' cvss_score: float = 0 cvss_vec: str = '' epss_score: float = 0 tags: List[str] = field(default_factory=list, compare=False) extra_data: dict = field(default_factory=dict, compare=False) description: str = field(default='', compare=False) references: List[str] = field(default_factory=list, compare=False) reference: str = field(default='', compare=False) confidence_nb: int = 0 severity_nb: int = 0 ``` {% endcode %} *** # Proxies ... or how to control how secator connects to targets. `secator` provides a range of proxy options to choose from, by passing the `-proxy` option to any runner (task, workflow, scan). *** ## Auto Using `-proxy auto` for a `secator` run will result in auto-detecting the right proxy to pass to each task. This is the recommended option for most runs. The defaults to use for proxies in `auto` mode are set through config variables: * 
secator config set http.socks5_proxy socks5://tor-privoxy:9050
* ```bash secator config set http.http_proxy http://tor-privoxy:8118 ``` #### **Example** Running `secator w host_recon -proxy auto` will result in the following behavior: * `naabu` supports Socks5 / HTTP proxy, but not proxychains, so it will use the first available SOCKS5 / HTTP proxy. * `nmap` supports proxy through `proxychains4` but has no good support for proxychains, HTTP or Socks5 proxy, so it will use `proxychains4` for the execution. * ... *** ## HTTP You can pass an HTTP proxy to a task / workflow / scan by using `-proxy http://:`. *** ## Socks5 You can pass a SOCKS5 proxy to a task / workflow / scan by using `-proxy socks5://:`. *** ## Proxychains You can pass `-proxy proxychains` to a task / workflow / scan by using `-proxy proxychains`. Remember to configure your `/etc/proxychains.conf` in that configuration, and test it prior to running `secator` tasks. *** # Exporters ... or how to export reports to different destinations. `secator` exporters allow exporting **reports** at the end of the run. {% hint style="info" %} To export **results** in **real-time**, see [Drivers](/in-depth/concepts/drivers) instead. {% endhint %} Available exporters out-of-the-box are: * **`txt`**: exports results as TXT file. * **`csv`**: exports results as CSV file. * **`json`**: exports results as JSON file. * **`console`**: exports results to console/stdout (similar to default output but as an exporter). * **`gdrive`**: exports results to Google Drive. Set `addons.gdrive.credentials_path` and `addons.gdrive.drive_parent_folder_id` in your config for this exporter to work. * **`table`**: prints results as a table in the terminal. *** ## Using exporters {% tabs %} {% tab title="CLI" %} To use exporters from the CLI, use the `--output` or `-o` flag: ```bash secator x httpx mydomain.com -o txt secator w host_recon mydomain.com -o gdrive,table secator s url http://testphp.vulnweb.com -o csv,json ``` {% endtab %} {% tab title="Python" %} To use exporters from the library, you can pass the `exporters` kwarg to any runner: ```python from secator.workflows import host_recon host = 'mydomain.com' workflow = host_recon(host, exporters=['csv', 'json']) workflow.run() ``` {% endtab %} {% endtabs %} *** # Runners ... or how secator's internals work. A runner is at the core of `secator` live processing capabilities. It handles the parsing, converting and processing of input options (CLI and library) and output items. All runners inherit from `secator.runners._base.Runner`. *** ## Supported runners Some built-in runners are available out-of-the-box:
RunnerDescriptionAdditional features
CommandRun an external command and stream it's output.
  • Automatic command install.
  • Priviledged mode (sudo).
TaskRun a task.
  • Remote mode (Celery).
  • Chunking on big inputs.
  • Direct calling from library.
WorkflowRun a DAG of tasks, defined in a YAML config file.
  • Remote mode (Celery).
  • Distributed (Celery).
  • Task chaining and parallel.
  • Re-use previous results as task inputs.
ScanRun a DAG of workflows, defined in a YAML config file.
  • Distributed (Celery).
  • Workflow chaining.
  • Re-use previous results as workflow inputs.
*** ## Lifecyle hooks Here is an overview of how a runner's lifecycle:

Runner lifecycle

The `Runner` lifecycle contains hooks that a user can plug into: **Base hooks:** * `before_init`: executed before the base runner's init starts. * `on_init`: executed when the base runner's init is completed. * `on_start` : executed when the runner has started running. * `on_iter`: executed when the runner iterates. * `on_end` : executed when the runner has finished running. * `on_cmd`: runs when the mapped command is built **\[**** ****`Command`**** ****runner only ]**. * `on_cmd_done`: runs when the command has finished running **\[**** ****`Command`**** ****runner only ]**. * `on_line`: executed when a line is output to stdout or stderr **\[**** ****`Command`**** ****runner only ]**. **Item hooks:** * `on_item_pre_convert`: executed before an item is converted to an output type. * `on_item`: executed when the runner emits an item. * `on_duplicate`: runs after an item has been marked as a duplicate. * `on_line`: executed when a line is emitted to `stdout` or `stderr` **\[**** ****`Command`**** ****runner only ]**. * `on_error`: executed when an error is emitted by the command **\[**** ****`Command`**** ****runner only ]** . * `on_``{serializer}``_loaded`: executed when a serializer has finished running. For instance, `on_json_loaded` after the `JSONSerializer` has finished running **\[**** ****`Command`**** ****runner only ]** . {% hint style="info" %} **All hooks** are defined with the `@staticmethod` decorator and take **`self`** as the **first** argument so that you can use the runner data in your hook implementation.\ \ **Item hooks** take **`item`** as the **second** argument and expect you to return the modified item. {% endhint %} *** ## Using hooks There are two different ways of specifying hooks: **static hooks** (in the task definition class), **dynamic hooks** (passed to a runner at runtime), or **drivers (collection of hooks).** ### Static hooks Static hooks **are** specified in the task specification class as `staticmethod`s: ```python from secator.runners import Command from secator.decorators import task from mylib import send_to_aws_s3 @task() class mytool(Command): # ... @staticmethod def on_item(self, item): if item._type == 'url': send_to_aws_s3(item.stored_response_path) return item ``` ### Dynamic hooks Dynamic hooks are specified at runtime by passing them to a runner. {% hint style="info" %} Dynamic hooks are a **library-only** feature, they are not available in the CLI. {% endhint %} Here are examples of specifying dynamic hooks: {% tabs %} {% tab title="Task" %} 
from secator.task import mytool

api_url = 'https://myapi.com'
hooks = {
    'on_item': lambda self, item: requests.post(api_url, json=item.toDict())
}
mytool('TARGET', hooks=hooks).run()
{% endtab %} {% tab title="Workflow / Scan" %} ```python from secator.runners import Workflow, Task from secator.template import TemplateLoader from secator.hooks.mongodb import update_runner, update_finding config = TemplateLoader(path='/path/to/my/workflow.yaml') workflow = Workflow( config, hooks={ Workflow: { 'on_init': [update_runner], 'on_start': [update_runner], 'on_iter': [update_runner], 'on_end': [update_runner] }, Task: { 'on_init': [update_runner], 'on_item': [update_finding], 'on_duplicate': [update_finding], 'on_iter': [update_runner], 'on_end': [update_runner] } } ) workflow.run() ``` {% endtab %} {% endtabs %} ### **Drivers** See [Drivers](/in-depth/concepts/drivers). *** # Drivers ... or how to route live results to a destination. A driver is a set of hooks that constitute a full integration to route live results to certain destination. Unlike hooks, you can use a driver from the CLI by using the `-driver` option. *** ## MongoDB driver To export live results to MongoDB database: 
secator install addons mongodb
secator config set addons.mongodb.url mongodb://localhost
secator w host_recon example.com -driver mongodb
Results will be added to the database in real-time as results come through from the various tools supported by `secator`. A MongoDB collection is created for each [Output types](/in-depth/concepts/output-types) supported by `secator`. *** ## GCS driver To upload files (screenshots and stored responses) from URL results to Google Cloud Storage: 
secator install addons gcs
secator config set addons.gcs.bucket_name my-bucket-name
secator w host_recon example.com -driver gcs
Files will be uploaded to Google Cloud Storage in real-time as results come through. The local file paths (`screenshot_path`, `stored_response_path`) in URL results will be replaced with GCS URLs (`gs://bucket-name/blob-name`). {% hint style="info" %} **Authentication**: The GCS driver uses the Google Cloud Storage client library which automatically detects credentials. You can authenticate using one of the following methods: * Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to point to your credentials JSON file * Use Application Default Credentials if running on Google Cloud Platform * The client library will automatically detect credentials from your environment The `addons.gcs.credentials_path` config option is available but optional, as the client library handles credential detection automatically. {% endhint %} *** # Profiles ... or how to manage different runner option sets effectively. `secator` profiles are a way to mutualize sets of options to be quickly re-used in tasks, workflows, and scans. Profiles allow you to apply predefined option configurations with a single flag, making it easy to switch between different scanning strategies. *** ## Using profiles You can use profiles with any task, workflow, or scan by using the `-pf` or `--profiles` flag: {% tabs %} {% tab title="Task" %} ```bash secator x httpx example.com -pf aggressive,full ``` {% endtab %} {% tab title="Workflow" %} ```bash secator w host_recon example.com -pf aggressive,full ``` {% endtab %} {% tab title="Scan" %} ```bash secator s host example.com -pf aggressive,full ``` {% endtab %} {% endtabs %} You can combine multiple profiles by separating them with commas. Options from profiles are merged, with later profiles taking precedence over earlier ones. *** ## Listing available profiles To see all available profiles and their options: ```bash secator p list ``` *** ## Built-in profiles `secator` comes with several built-in profiles organized by category: ### Speed profiles These profiles control the aggressiveness and speed of scans by adjusting rate limits, delays, timeouts, and retries. #### Aggressive (`aggressive`) Optimized for internal networks or time-sensitive scans with no rate limiting. ```yaml rate_limit: 10000 delay: 0 timeout: 1 retries: 1 ``` **Use case**: Fast scanning on trusted internal networks where you need results quickly. #### Insane (`insane`) Maximum speed for local LAN scanning or stress testing. ```yaml rate_limit: 100000 delay: 0 timeout: 1 retries: 0 ``` **Use case**: Local network scanning or stress testing where network impact is not a concern. #### Paranoid (`paranoid`) Maximum stealth with very conservative settings. ```yaml rate_limit: 5 delay: 5 timeout: 15 retries: 5 ``` **Use case**: Scanning sensitive networks where stealth is critical and you can afford to wait. #### Polite (`polite`) Balanced settings to avoid overloading the network. ```yaml rate_limit: 100 delay: 0 timeout: 10 retries: 5 ``` **Use case**: General-purpose scanning where you want to be respectful of network resources. *** ### Evasion profiles These profiles help evade detection systems and maintain anonymity. #### Sneaky (`sneaky`) IDS/IPS evasion for sensitive networks using packet fragmentation. ```yaml fragment: true nmap_light_fragment: true ``` **Use case**: Scanning networks with active IDS/IPS systems that you want to evade. #### Stealth (`stealth`) Stealth scan using TCP SYN scanning. ```yaml tcp_syn_stealth: true nmap_light_tcp_syn_stealth: true scan_type: s ``` **Use case**: Port scanning where you want to minimize detection by using stealth techniques. #### Tor (`tor`) Anonymous scanning using the Tor network. ```yaml proxy: auto ``` **Use case**: Scanning where you need anonymity and want to route traffic through Tor. {% hint style="info" %} Make sure you have Tor configured and running, and set up your proxy settings in the config (see [Proxies](/in-depth/concepts/proxies)). {% endhint %} *** ### General profiles These profiles control the overall scanning approach and feature activation. #### Active (`active`) Active scanning only - no passive sources are used. This profile enforces active-only mode. ```yaml active: true domain_recon_active: true subdomain_recon_active: true host_recon_active: true url_crawl_active: true url_vuln_active: true ``` **Use case**: When you want to ensure all scanning is done actively (making requests to targets) and avoid passive data sources. #### Passive (`passive`) Passive scanning only - no requests are made to targets. This profile enforces passive-only mode. ```yaml passive: true domain_recon_passive: true subdomain_recon_passive: true host_recon_passive: true url_crawl_passive: true url_vuln_passive: true ``` **Use case**: When you want to gather information without making any requests to the target (e.g., using only OSINT sources). #### Full (`full`) Activates all optional features for comprehensive scanning. ```yaml # Task options headless: true system_chrome: true no_sandbox: true screenshot: true juicy_extensions: 3 server_defaults: false # Workflow options ports: "-" nuclei: true brute_dns: true brute_http: true hunt_secrets: true test_ssl: true # Scan options host_recon_nuclei: true host_recon_nmap_ports: "-" domain_recon_testssl_server_defaults: null subdomain_recon_hunt_secrets: true subdomain_recon_test_ssl: true subdomain_recon_testssl_server_defaults: null url_crawl_hunt_secrets: true url_vuln_nuclei: true url_crawl_cariddi_juicy_extensions: 3 ``` **Use case**: Comprehensive security assessments where you want to enable all available features including headless browsing, screenshots, vulnerability scanning, secret hunting, and SSL testing. *** ### Network profiles These profiles configure network-specific scanning options. #### All Ports (`all_ports`) Scans all ports instead of just common ones. ```yaml ports: "-" host_recon_nmap_ports: "-" ``` **Use case**: When you need a complete port scan of all 65535 ports (warning: this can be very slow). #### HTTP Headless (`http_headless`) Enables headless browser mode for HTTP tasks. ```yaml headless: true system_chrome: true no_sandbox: true ``` **Use case**: When you need to render JavaScript or interact with modern web applications that require a browser. #### HTTP Record (`http_record`) Records HTTP requests/responses and takes screenshots. ```yaml screenshot: true store_responses: true system_chrome: true no_sandbox: true ``` **Use case**: When you need to capture full HTTP interactions and visual evidence of web pages. *** ## Profile enforcement Some profiles have the `enforce: true` setting, which means they override any conflicting options you might pass directly. Profiles with enforcement are: * `active` * `passive` * `full` * `all_ports` When using enforced profiles, the profile options take precedence over command-line options. *** ## Combining profiles You can combine multiple profiles to create a custom configuration: ```bash # Combine speed and evasion profiles secator w host_recon example.com -pf aggressive,stealth # Combine feature activation with speed secator s host example.com -pf full,polite # Combine network and evasion secator x naabu example.com -pf all_ports,tor ``` When combining profiles, options are merged with later profiles taking precedence over earlier ones. *** ## Setting default profiles You can set default profiles in your configuration that will be applied automatically: ```bash secator config set profiles.defaults aggressive,polite ``` Default profiles are automatically applied to all runs unless you explicitly override them with the `-pf` flag. *** ## Creating custom profiles To create your own profile, create a YAML file in `~/.secator/templates/profiles/`: ```yaml type: profile name: my_custom_profile category: general description: "My custom profile description" opts: rate_limit: 500 delay: 0.5 timeout: 5 retries: 3 ``` After creating the profile file, it will be automatically available for use with the `-pf` flag. *** # Deployment ... or how to run secator anywhere. Once you have started using `secator` locally, you can deploy it on remote instances. *** ## Amazon Web Services (AWS) *Amazon Web Services* (AWS) is a subsidiary of Amazon that provides on-demand cloud computing platforms and APIs to individuals, companies. ### Elastic Compute Cloud #### Deploy on a single EC2 instance

Deployment on a single EC2 instance

To deploy `secator` on AWS on a single EC2 instance, we will install RabbitMQ, Redis and a Celery worker running `secator` on the instance.
Step 1: Create an EC2 instance with AMI Ubuntu * Go to the AWS Management Console * Create an EC2 instance using the Ubuntu AMI * Configure **Security Groups:** * Allow port 6379 (Redis) * Allow port 5672 (RabitMQ) * SSH to your created instance
Step 2: Install RabbitMQ as a task broker Celery needs a task broker to send tasks to remote workers. ```bash sudo apt-get install curl gnupg apt-transport-https -y curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A36E6026DFCA" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/com.rabbitmq.team.gpg > /dev/null curl -1sLf "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xf77f1eda57ebb1cc" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/net.launchpad.ppa.rabbitmq.erlang.gpg > /dev/null curl -1sLf "https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/io.packagecloud.rabbitmq.gpg > /dev/null sudo apt-get update -y sudo apt-get install -y erlang-base \ erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets \ erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \ erlang-runtime-tools erlang-snmp erlang-ssl \ erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl sudo apt-get install rabbitmq-server -y --fix-missing sudo rabbitmq-plugins enable rabbitmq_management sudo rabbitmqctl add_user secator sudo rabbitmqctl set_user_tags secator administrator sudo rabbitmqctl set_permissions -p / secator ".*" ".*" ".*" ``` **Make sure your replace the \ by a strong password that you generate.**
Step 3: Install Redis as a storage backend Celery needs a storage backend to store results. `secator` uses the storage backend to print results in real-time. ```bash sudo apt install redis-server sudo vi /etc/redis/redis.conf # set requirepass to # comment the "bind 127.0.0.1 ::1" line # change "protected-mode" to "no" sudo /etc/init.d/redis-server restart ``` **Make sure your replace the \ by a strong password that you generate.**
Step 4: Deploy a secator worker First, setup `secator`using the all-in-one bash setup script: ``` wget -O - https://raw.githubusercontent.com/freelabz/secator/main/scripts/install.sh | sh ``` Then, set the RabbitMQ and Redis connection details in `secator`'s config: ```bash secator config set celery.broker_url amqp://secator:@localhost:5672/ secator config set celery.result_backend redis://default:@localhost:6379/0 ``` Finally, run a `secator worker`: ``` nohup secator worker > worker.log 2>&1 & # start in background and save logs ```
Step 5: Run a task from your local machine Let's configure the worker with RabbitMQ and Redis connection details: ```bash secator config set celery.broker_url amqp://secator:@:5672/ secator config set celery.result_backend redis://default:@:6379/0 ``` Run a test task: ``` secator x httpx wikipedia.org ``` You should get an output like the following: ```bash __ ________ _________ _/ /_____ _____ / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/ (__ / __/ /__/ /_/ / /_/ /_/ / / /____/\___/\___/\__,_/\__/\____/_/ v0.0.1 freelabz.com Celery worker is alive ! ╭──────── Task httpx ─────────╮ │ 📜 Description: DotMap() │ │ 👷 Workspace: default │ │ 🍐 Targets: │ │ • wikipedia.org │ │ 📌 Options: │ │ • follow_redirect: False │ │ • threads: 50 │ │ • debug_resp: False │ ╰─────────────────────────────╯ [10:20:54] 🎉 Task httpx sent to Celery worker... _base.py:614 🏆 Live results: 🔗 https://wikipedia.org [301] [301 Moved Permanently] [mw1415.eqiad.wmnet] [HSTS] [text/html] [234] ```
#### Deploy on multiple EC2 instances If you want a more scaleable architecture, we recommend deploying RabbitMQ, Redis, and Celery workers on different EC2 instances. The steps are exactly the same as for [#deploy-on-a-single-ec2-instance](#deploy-on-a-single-ec2-instance "mention"), expect that steps 2, 3, and **4** will each be run on **separate EC2 instance**. You can repeat step 4 on more instances to increase the number of workers. *** ## Google Cloud Platform (GCP) The *Google Cloud Platform* (GCP) is a suite of cloud services that offers server space on virtual machines, internal networks, VPN connections, disk storage, ... ### Google Compute Engine #### Deploy on a single GCE instance

Deployment on a single EC2 instance

To deploy `secator` on a single GCE machine, we will install RabbitMQ, Redis and a Celery worker running `secator` on the instance.
Step 1: Create a GCE instance * Go to the Google Cloud Console * Create a GCE instance using the Debian image * Create firewall rules in **Network** > **Firewall:** * Allow port 6379 (Redis) * Allow port 5672 (RabbitMQ) * SSH to your created instance
Step 2: Install RabbitMQ as a task broker Celery needs a task broker to send tasks to remote workers. ```bash sudo apt-get install curl gnupg apt-transport-https -y curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A36E6026DFCA" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/com.rabbitmq.team.gpg > /dev/null curl -1sLf "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xf77f1eda57ebb1cc" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/net.launchpad.ppa.rabbitmq.erlang.gpg > /dev/null curl -1sLf "https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/io.packagecloud.rabbitmq.gpg > /dev/null sudo apt-get update -y sudo apt-get install -y erlang-base \ erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets \ erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \ erlang-runtime-tools erlang-snmp erlang-ssl \ erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl sudo apt-get install rabbitmq-server -y --fix-missing sudo rabbitmq-plugins enable rabbitmq_management sudo rabbitmqctl add_user secator sudo rabbitmqctl set_user_tags secator administrator sudo rabbitmqctl set_permissions -p / secator ".*" ".*" ".*" ``` **Make sure your replace the \ by a strong password that you generate.**
Step 3: Install Redis as a storage backend Celery needs a storage backend to store results. `secator` uses the storage backend to print results in real-time. ```bash sudo apt install redis-server sudo vi /etc/redis/redis.conf # set requirepass to # comment the "bind 127.0.0.1 ::1" line # change "protected-mode" to "no" sudo /etc/init.d/redis-server restart ``` **Make sure your replace the \ by a strong password that you generate.**
Step 4: Deploy a secator worker First, setup `secator`using the all-in-one bash setup script: ``` wget -O - https://raw.githubusercontent.com/freelabz/secator/main/scripts/install.sh | sh ``` Then, set the RabbitMQ and Redis connection details in `secator`'s config: ```bash secator config set celery.broker_url amqp://secator:@localhost:5672/ secator config set celery.result_backend redis://default:@localhost:6379/0 ``` Finally, run a `secator worker`: ``` nohup secator worker > worker.log 2>&1 & # start in background and save logs ```
Step 5: Run a task from your local machine First, set the RabbitMQ and Redis connection details in `secator`'s config: ```bash secator config set celery.broker_url amqp://secator:@:5672/ secator config set celery.result_backend redis://default:@:6379/0 ``` Run a test task: ``` secator x httpx wikipedia.org ``` You should get an output like the following: ```bash __ ________ _________ _/ /_____ _____ / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/ (__ / __/ /__/ /_/ / /_/ /_/ / / /____/\___/\___/\__,_/\__/\____/_/ v0.0.1 freelabz.com Celery worker is alive ! ╭──────── Task httpx ─────────╮ │ 📜 Description: DotMap() │ │ 👷 Workspace: default │ │ 🍐 Targets: │ │ • wikipedia.org │ │ 📌 Options: │ │ • follow_redirect: False │ │ • threads: 50 │ │ • debug_resp: False │ ╰─────────────────────────────╯ [10:20:54] 🎉 Task httpx sent to Celery worker... _base.py:614 🏆 Live results: 🔗 https://wikipedia.org [301] [301 Moved Permanently] [mw1415.eqiad.wmnet] [HSTS] [text/html] [234] ```
#### Deploy on multiple GCE instances If you want a more scaleable architecture, we recommend deploying RabbitMQ, Redis, and Celery workers on different machines. The steps are exactly the same as for the previous section, except that steps 2, 3, and **4** will each be run on **separate GCE instance**. You can repeat step 4 on more instances to increase the number of workers. ### Google Kubernetes Engine \[WIP] ### Cloud Run \[WIP] *** ## Axiom \[WIP] **Axiom is a dynamic infrastructure framework** to efficiently work with multi-cloud environments, build and deploy repeatable infrastructure focused on offensive and defensive security. *** ## Bare metal ### Kubernetes A Helm chart is available in the repository: . ### Docker-compose \[WIP] *** # Development setup ... or how to setup a development environment for secator. *** ## Install a development build To install `secator` in development mode, first make sure `pip` and `virtualenv` are installed, and run the following steps: ```bash git clone https://github.com/freelabz/secator # clone the repository cd secator # go to the repository folder virtualenv .venv # create a virtualenv source .venv/bin/activate # load the virtualenv pip install -e .[dev] # install secator and dev dependencies ``` {% hint style="info" %} Use`secator health` to verify your installation. You can install addons / tools using the `secator install` command (choose what to install based on what you want to dev on). {% endhint %} *** ## Running tests ### Unit tests 
secator test unit
If you want to run unit tests to test a specific task only: ```sh secator test unit --test test_task --task ``` *** ### Integration tests ```bash secator test integration ``` If you want to run integration tests to test a specific task only: ```sh secator test integration --test test_tasks --tasks ``` *** ### Lint tests ```bash secator test lint ``` *** # Writing tasks ... or how to integrate new tasks with secator. Now that you have used `secator` for a while, you might regret that the scripts or CLI tools you use daily are not supported yet. No panic, just follow the guides to integrate them yourself ! To get started: * Read [Integrating an external command](/for-developers/writing-tasks/integrating-an-external-command) if you plan on integrating a new external command. * Read [Integrate custom Python code \[WIP\]](/for-developers/writing-tasks/integrate-custom-python-code-wip) if you plan on integrating embedded Python code. *** # Integrating an external command ... or how to turn a command that you use daily into an overpowered machine. *** ### Creating a task file Imagine we have a tool named `mytool` that we want to integrate with `secator`. Start by creating a file named `mytool.py`: ```python from secator.decorators import task # required for `secator` to recognize tasks from secator.runners import Command # the `secator` runner to use @task() class mytool(Command): # make sure class name is lowercase and matches the filename. cmd = 'mytool' # ... or whatever the name of your external command is. ``` {% hint style="info" %} The `task` decorator is required for `secator` to recognize class-based definition that need to be loaded at runtime. {% endhint %} Move this file over to: * `~/.secator/templates/` (or whatever your `dirs.templates` in [Configuration](/getting-started/configuration) points to) **OR** * `secator/tasks/` if you have a [Development setup](/for-developers/development-setup) and want to contribute your task implementation to the official `secator` repository. *** ### Adding an input flag \[optional] If your tool requires an input flag or a list flag to take its targets, for instance: * `mytool -u TARGET` * `mytool -l TXT_FILE` You need to set the `input_flag` and `file_flag` class options: ```python from secator.decorators import task from secator.runners import Command @task() class mytool(Command): cmd = 'mytool' input_flag = '-u' file_flag = '-l' ``` Setting these attributes allows us to run `mytool` with `secator` like: ```bash secator x mytool TARGET # will run mytool -u TARGET secator x mytool TXT_FILE # will run mytool -l TXT_FILE ``` ### Parsing a command's output Now that you have a basic implementation working, you need to convert your command's output into structured output (JSON). Find out what your command's output looks like and pick the corresponding guide: * Read [Parsing JSON lines](/for-developers/writing-tasks/integrating-an-external-command/parsing-json-lines) if your tool has an option to stream JSON lines (**preferred**). * Read [Parsing output files](/for-developers/writing-tasks/integrating-an-external-command/parsing-output-files) if your tool has an option to output to a file (e.g JSON or CSV). * Read [Parsing raw standard output](/for-developers/writing-tasks/integrating-an-external-command/parsing-raw-standard-output) if your tools **only** outputs to `stdout` . *** ### Adding more options \[optional] To support more options, you can use the `opt_prefix`, `opts` , `opt_key_map` and `opt_value_map` attributes. Assuming `mytool` has the `--delay`, `--debug` and `--include-tags` options, we would support them this way: ```python @task() class mytool(Command): # ... opt_prefix = '--' # default is '-' opts = { 'tags': {'type': str, 'short': 't', 'help': 'Tags'}, 'delay': {'type': int, 'short': 'dbg', 'help': 'Delay'}, 'debug': {'is_flag': True, 'short': 'd', 'help': 'Debug mode'}, } opt_key_map = { # to map input options 'tags': 'include-tags', } opt_value_map = { # to transform options values 'delay': lambda x: x * 1000 # convert seconds to milliseconds } ``` With this config, running either of: ```bash secator x mytool --tags tag1,tag2 --debug --delay 5 TARGET # long option format secator x mytool -t tag1,tag2 -dbg -d 5 TARGET # short option format ``` will result in running `mytool` like: ```bash mytool --include-tags tag1,tag2 --debug --delay 5000 -u TARGET ``` *** ### Adding an install command \[optional] To support installing your tool with secator, you can set the `install_cmd` , and / or `install_github_handle` attributes: ```python @task() class mytool(Command): # ... install_cmd = "sudo apt install -y mytool" install_github_handle = "myorg/mytool" ``` {% hint style="info" %} If `install_github_handle` is set, `secator` will try to fetch a binary from GitHub releases specific to your platform, and fallback to `install_cmd` if it cannot find a suitable release, or if the API rate limit is reached. {% endhint %} Now you can install `mytool` using: ```bash secator install tools mytool ``` *** ### Using a category \[optional] If your tool fits into one of `secator`'s built-in command categories, you can inherit from it's option set: * `Http`: A tool that makes HTTP requests. * `HttpCrawler`: A command that crawls URLs (subset of `Http`). * `HttpFuzzer`: A command that fuzzes URLs (subset of `Http`). You can inherit from these categories and map their options to your command. {% hint style="info" %} Categories are defined in `secator/tasks/_categories.py` {% endhint %} For instance, if `mytool` is an HTTP fuzzer, we would change it's implementation like: ```python from secator.tasks._categories import HTTPFuzzer from secator.definitions import OPT_NOT_SUPPORTED @task() class mytool(Command, HTTPFuzzer): # ... opt_key_map = { # HTTPFuzzer options mapping HEADER: 'header', DELAY: 'delay', DEPTH: OPT_NOT_SUPPORTED, FILTER_CODES: OPT_NOT_SUPPORTED, FILTER_REGEX: OPT_NOT_SUPPORTED, FILTER_SIZE: OPT_NOT_SUPPORTED, FILTER_WORDS: OPT_NOT_SUPPORTED, FOLLOW_REDIRECT: OPT_NOT_SUPPORTED, MATCH_CODES: OPT_NOT_SUPPORTED, MATCH_REGEX: OPT_NOT_SUPPORTED, MATCH_SIZE: OPT_NOT_SUPPORTED, MATCH_WORDS: OPT_NOT_SUPPORTED, METHOD: OPT_NOT_SUPPORTED, PROXY: OPT_NOT_SUPPORTED, RATE_LIMIT: OPT_NOT_SUPPORTED, RETRIES: OPT_NOT_SUPPORTED, THREADS: OPT_NOT_SUPPORTED, TIMEOUT: 'timeout', USER_AGENT: 'user-agent', # my tool specific options 'tags': 'include-tags', } opt_value_map = { 'delay': lambda x: x * 1000 # convert seconds to milliseconds } ``` {% hint style="info" %} Make sure you map **all** the options from the **`HTTPFuzzer`** category. If some options are not supported by your tool, mark them with `OPT_NOT_SUPPORTED`. {% endhint %} With this config, running: ```bash secator x mytool --help ``` would list: * The `meta`options in the `HTTPFuzzer` category that are supported by `mytool`. * The options only usable by `mytool`. For instance, running: {% tabs %} {% tab title="CLI" %} ```bash secator x mytool \ -header "Authorization: Bearer MYTOKEN" \ -delay 1 \ -ua "secator/0.6 (Debian)" \ -timeout 5 \ -t tag1,tag2 \ TARGET ``` {% endtab %} {% tab title="Python" %} ```python from secator.tasks import mytool task = mytool( 'TARGET', header="Authorization: Bearer MYTOKEN", delay=1, user_agent="secator/0.6 (Debian)", timeout=5, tags='tag1,tag2' ) for item in task: print(item) ``` {% endtab %} {% endtabs %} would result in running `mytool` like: ```bash mytool \ --header "Authorization: Bearer MYTOKEN" \ --delay 1000 \ --user-agent "secator/0.6 (Debian)" \ --timeout 5 \ --include-tags tag1,tag2 -u TARGET ``` *** ### Supporting proxies \[optional] If your tool supports proxies, `secator` has first-class support for **`proxychains`**, **`HTTP`** and **`SOCKS5`** proxies, and can dynamically choose the type of proxy to use based on the following attributes: * `proxy_socks5` : boolean indicating if your command supports `SOCKS5` proxies. * `proxy_http` : boolean indicating if your command supports `HTTP` / `HTTPS` proxies. * `proxychains`: boolean indicating if your command supports being run with `proxychains`. {% hint style="warning" %} If your proxy supports `SOCKS5`or `HTTP` proxies, make sure to have an option called **`proxy`** in your **`opts`** definition or it won't be picked up. If your proxy supports `proxychains`, `secator` will use the local `proxychains` binary and `proxychains.conf` configuration, so make sure those are functional. {% endhint %} {% hint style="info" %} Read [Proxies](/in-depth/concepts/proxies)for more details on how proxies work and how to configure them properly. {% endhint %} #### Example: Assuming `mytool` does not support HTTP or SOCKS5 proxies, but works with `proxychains`, you can update your task definition like: ```python @task() class mytool(Command): # ... proxychains = True proxy_socks5 = False proxy_http = True ``` With the above configuration, running with `-proxy ` would result in the following behaviour: {% tabs %} {% tab title="proxychains / auto" %} ```bash secator x mytool -proxy proxychains TARGET secator x mytool -proxy auto TARGET # auto-pick from proxychains > socks5 > http ``` becomes: ```bash proxychains mytool -u TARGET ``` {% endtab %} {% tab title="http" %} ```bash secator x mytool -proxy "http://testmyproxy.com" ``` becomes: ```bash mytool --use-proxy "http://testmyproxy.com" -u ``` {% endtab %} {% tab title="random" %} ```bash secator x mytool -proxy random ``` becomes: ```bash mytool --use-proxy "http://30.10.23.42" -u # random proxy (FreeProxy) ``` {% endtab %} {% endtabs %} *** ### Hooking onto runner lifecycle You can hook onto any part of the runner lifecycle by override the hooks methods (read [Runners](/in-depth/concepts/runners#lifecyle-hooks) to know more). #### Example: {% tabs %} {% tab title="CLI" %} ```python @task() class mytool(Command): # ... @staticmethod def on_line(self, line): return line.rstrip(',') # strip trailing comma of stdout lines @staticmethod def on_item_pre_convert(self, item): item['extra_data'] = { 'version': '2.0' # add extra data to items } return item ``` {% endtab %} {% tab title="Python" %} ```python from secator.tasks import mytool hooks = { 'on_line': lambda x: x.strip(','), 'on_item_pre_convert': lambda x: x | {'extra_data': {'version': '2.0'}}, } task = mytool('TARGET', hooks=hooks) for item in task: print(item) ``` {% endtab %} {% endtabs %} *** ### Chunking `secator` allows to chunk a task into multiple children tasks when the length of the input grows, or some other specific requirements (e.g: your command only takes one target at a time). {% hint style="warning" %} Chunking only works when [Distributed runs with Celery](/in-depth/distributed-runs-with-celery) are enabled. {% endhint %} You can specify the chunk size using the `input_chunk_size` attribute: ```python @task() class mytool(Command): # ... input_chunk_size = 10 # additional tasks will be spawned every 10 targets ``` With this config, running: ```bash secator x mytool tasks.txt # tasks.txt contains 20 targets ``` would result in: ```bash mytool -l /tmp/task_0_9.txt mytool -l /tmp/task_10_19.txt ``` {% hint style="info" %} If `mytool` did not support file input (i.e: `file_flag` not defined in the task definition class), the above would still work with an `input_chunk_size = 1`, thus splitting into one command per target passed. {% endhint %} *** # Parsing JSON lines ... or how to integrate tools that output JSON lines. If your tool outputs JSON lines, it's very easy to integrate it with `secator`. Based on how your tool's output maps will map to `secator` output types, read: * [#one-to-one-mapping](#one-to-one-mapping "mention") (1 JSON line = 1 `secator` output type). * [#one-to-many](#one-to-many "mention") (1 JSON line = many`secator` output types). *** ## One-to-one mapping **Steps:** * Use the `JSONSerializer` item loader. * Add an `output_map` to map your tool's output to one of secator's [Output types](/in-depth/concepts/output-types)**.** #### **Example:** For instance, `mytool` outputs JSON lines when we run it like: ```bash mytool -jsonl -u mytarget.com {"url": "https://mytarget.com/api", "status": 200, {"details": {"ct": "application/json"}} {"url": "https://mytarget.com/api/metrics", "status": 403, "details": {}} ``` An integration of `mytool` with `secator` would look like: {% code title="secator/tasks/mytool.py" %} ```python from secator.decorators import task from secator.runners import Command from secator.output_types import Url from secator.serializers import JSONSerializer from secator.definitions import URL, STATUS_CODE, CONTENT_TYPE @task() class mytool(Command): input_flag = '-u' json_flag = '-jsonl' output_types = [Url] item_loaders = [JSONSerializer()] output_map = { Url: { URL: 'url', STATUS_CODE: 'status', CONTENT_TYPE: lambda x: x['details'].get('ct', '') } } ``` {% endcode %} **Steps taken:** * Import the desired runner (`Command`). * Import the desired output type(s) and the field names that we want to map. * Set `input_flag` to `mytool`'s input flag `-u`. * Set `json_flag` to `mytool`'s JSON line flag `jsonl`. * Set `output_types` to our desired output types \[[`Url`](/in-depth/concepts/output-types#url)]. * Set `output_map` to map `mytool`'s output fields to each [`Url`](/in-depth/concepts/output-types#url)'s fields (you need to map at least the `required=True` fields). That's it ! You can now run `mytool` with `secator` and enjoy all the features it brings on top of it: {% tabs %} {% tab title="CLI" %} ```bash secator x mytool mytarget.com __ ________ _________ _/ /_____ _____ / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/ (__ / __/ /__/ /_/ / /_/ /_/ / / /____/\___/\___/\__,_/\__/\____/_/ v0.6.0 freelabz.com mytool -u mytarget.com -jsonl 🔗 https://mytarget.com/api [200] [application/json] 🔗 https://mytarget.com/api/metrics [403] ``` {% endtab %} {% tab title="Python" %} ```python from secator.tasks import mytool task = mytool('mytarget.com') for item in task: print(item) # item is now a secator output type like Vulnerability or Port ``` {% endtab %} {% endtabs %} *** ## One-to-many **Steps:** * Add a static method `on_json_loaded` to hook onto the JSON Serializer output. * Modify the data and yield `secator` [Output types](/in-depth/concepts/output-types). #### **Example:** For instance, if `mytool` outputs urls in batch: ```bash mytool -jsonl -u mytarget.com {"urls": [{"url": "https://mytarget.com/api", "status": 200, "content-type": "application/json"}, {"url": "https://mytarget.com/api/metrics", "status": 403, "content-type": "application/text"}] ``` An integration of `mytool` with `secator` would look like: 
from secator.decorators import task
from secator.runners import Command
from secator.output_types import Url
from secator.definitions import URL, STATUS_CODE, CONTENT_TYPE


@task()
class mytool(Command):
  input_flag = '-u'
  json_flag = '-jsonl'
  output_types = [Url]

  @staticmethod
  def on_json_loaded(self, data):
      for item in data['urls']:
          yield Url(
            URL: item['url'],
            STATUS_CODE: item['status'],
            CONTENT_TYPE: item.get('content-type', '')
          )
{% hint style="info" %} See [Runners](/in-depth/concepts/runners#lifecyle-hooks) for more details on which hooks you can use. {% endhint %} That's it ! You can now run `mytool` with `secator` and enjoy all the features it brings on top of it: {% tabs %} {% tab title="CLI" %} ```bash secator x mytool mytarget.com __ ________ _________ _/ /_____ _____ / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/ (__ / __/ /__/ /_/ / /_/ /_/ / / /____/\___/\___/\__,_/\__/\____/_/ v0.6.0 freelabz.com mytool -u mytarget.com -jsonl 🔗 https://mytarget.com/api [200] [application/json] 🔗 https://mytarget.com/api/metrics [403] ``` {% endtab %} {% tab title="Python" %} ```python from secator.tasks import mytool task = mytool('mytarget.com') for item in task: print(item) # item is now a secator output type like Vulnerability or Port ``` {% endtab %} {% endtabs %} # Parsing raw standard output ... or how to integrate tools that prints to stdout and do not support JSON lines. If your tool does not output JSON lines / JSON files, it's requires a bit more effort to integrate it with `secator`. Depending on how you want to parse the output, read: * [#using-regular-expressions](#using-regular-expressions "mention") * [#writing-a-custom-item-loader](#writing-a-custom-item-loader "mention") *** ## Using regular expressions **Steps:** * Use the `RegexSerializer` item loader. * Use the `on_regex_loaded` hook to yield `secator` output types. #### **Example:** Assume `mytool` outputs on stdout like: ```bash mytool -u mytarget.com [INF] This is an info message [ERR] This is an error message [FOUND] https://mytarget.com/api [type=url] [status=200] [content_type=application/json] [title=MyAwesomeWebPage] [FOUND] https://mytarget.com/api/metrics [type=url] [status=403] [FOUND] A3TBABCD1234EFGH5678 [type=aws_api_key] [matched_at=https://mytarget/api/.aws_key.json] [FOUND] <-- an HTML comment --> [type=aws_api_key] [matched_at=https://mytarget/api/.aws_key.json] [FOUND] CVE-2021-44228 [type=vulnerability] [matched_at=https://mytarget/api/sensitive_api_path] ``` First we need to find a regular expression that will match the items marked with `[FOUND]` and get the individual values using a named regex (you can use [Pythex](https://pythex.org) for this). Here is the one we came up with: {% code overflow="wrap" %} ```python OUTPUT_REGEX = r'\[\w+]\s(?P.*)\s\[type=(?P[\w_]+)\](\s\[status=(?P\d+)\])?(\s\[content_type=(?P[\w\/]+)\])?(\s\[title=(?P.*)\])?(\s\[matched_at=(?P<matched_at>.*)\])?' ``` {% endcode %} An integration of `mytool` with `secator` would look like: {% code title="secator/tasks/mytool.py" %} ```python from secator.decorators import task from secator.runners import Command from secator.output_types import Url, Tag, Vulnerability from secator.serializers import RegexSerializer from secator.tasks._categories import Vuln OUTPUT_REGEX = r'\[\w+]\s(?P<value>.*)\s\[type=(?P<type>[\w_]+)\](\s\[status=(?P<status>\d+)\])?(\s\[content_type=(?P<content_type>[\w\/]+)\])?(\s\[title=(?P<title>.*)\])?(\s\[matched_at=(?P<matched_at>.*)\])?' @task() class mytool(Command): cmd = '/home/osboxes/.local/bin/mytool' input_flag = '-u' json_flag = '-jsonl' output_types = [Url, Tag, Vulnerability] # Use the RegexSerializer to load the stdout input item_loaders = [ RegexSerializer( OUTPUT_REGEX, fields=['value', 'type', 'status', 'content_type', 'title', 'matched_at'] ) ] # React to items loaded by the RegexSerializer, and yield secator output types # like Url, Vulnerability, and Tag. @staticmethod def on_regex_loaded(self, item): # this is called after the regex serializer runs, # so we can expect item to be a dict with the matched regex values if (item['type'] == 'url'): yield Url( url=item['value'], status_code=int(item['status']), content_type=item['content_type'], title=item['title'] ) elif (item['type'] == 'vulnerability'): cve_id = item['value'] lookup_data = Vuln.lookup_cve(cve_id) # perform vulnerability search vuln = { 'matched_at': item['matched_at'] } if lookup_data: vuln.update(**lookup_data) yield Vulnerability(**vuln) else: yield Tag( name=item['type'], match=item['matched_at'], extra_data={ 'secret': item['value'] } ) ``` {% endcode %} Run it with `secator`: {% tabs %} {% tab title="CLI" %} ```bash $ secator x mytool mytarget.com __ ________ _________ _/ /_____ _____ / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/ (__ / __/ /__/ /_/ / /_/ /_/ / / /____/\___/\___/\__,_/\__/\____/_/ v0.6.0 freelabz.com No Celery worker alive. /home/osboxes/.local/bin/mytool -u mytarget.com -jsonl [INF] This is an info message [ERR] This is an error message 🔗 https://mytarget.com/api [200] [MyAwesomeWebPage] [application/json] 🔗 https://mytarget.com/api/metrics [403] 🏷️ aws_api_key found @ https://mytarget/api/.aws_key.json secret: A3TBABCD1234EFGH5678 🚨 [Object Injection 🡕] [critical] https://mytarget/api/sensitive_api_path ``` {% endtab %} {% tab title="Python" %} ```python from secator.tasks import mytool task = mytool('mytarget.com') for item in task: print(item) # this will output Url, Vulnerability, or Tag items. ``` {% endtab %} {% endtabs %} *** ### Writing a custom item loader **Steps:** * Override the `item_loader` static method to parse the standard output with custom code. **Example:** Assume `mytool` outputs on stdout like: ```bash mytool -u mytarget.com https://mytarget.com/api | url | 200 | application/json | MyAwesomePage https://mytarget.com/api/metrics | url | 403 A3TBABCD1234EFGH5678 | aws_api_key | http://mytarget/api/.aws_key.json <-- an HTML comment --> | html_comment | http://mytarget/api/.aws_key.json CVE-2021-44228 | vulnerability | http://mytarget/api/sensitive_ap ``` ```python from secator.decorators import task from secator.runners import Command from secator.output_types import Url, Tag, Vulnerability @task() class mytool(Command): cmd = '/home/osboxes/.local/bin/mytool' input_flag = '-u' json_flag = '-jsonl' output_types = [Url, Tag, Vulnerability] @staticmethod def item_loader(self, line): items = [c.strip() for c in line.split('|')] value, item_type = tuple(items[0:2]) if item_type == 'url': yield Url( url=value, status_code=items[3], content_type=items[4] if len(items) > 3 else '', title=items[5] if len(items) > 4 else '' ) elif item_type == 'vulnerability': cve_id = value lookup_data = Vuln.lookup_cve(cve_id) # perform vulnerability search vuln = { 'matched_at': items[2] } if lookup_data: vuln.update(**lookup_data) yield Vulnerability(**vuln) else: # tag yield Tag( name=item_type, match=items[2], extra_data={ 'value': value } ) ``` Run it with `secator`: {% tabs %} {% tab title="CLI" %} ```bash $ secator x mytool mytarget.com __ ________ _________ _/ /_____ _____ / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/ (__ / __/ /__/ /_/ / /_/ /_/ / / /____/\___/\___/\__,_/\__/\____/_/ v0.6.0 freelabz.com No Celery worker alive. /home/osboxes/.local/bin/mytool -u mytarget.com -jsonl [INF] This is an info message [ERR] This is an error message 🔗 https://mytarget.com/api [200] [MyAwesomeWebPage] [application/json] 🔗 https://mytarget.com/api/metrics [403] 🏷️ aws_api_key found @ https://mytarget/api/.aws_key.json secret: A3TBABCD1234EFGH5678 🚨 [Object Injection 🡕] [critical] https://mytarget/api/sensitive_api_path ``` {% endtab %} {% tab title="Python" %} ```python from secator.tasks import mytool task = mytool('mytarget.com') for item in task: print(item) # this will output Url, Vulnerability, or Tag items. ``` {% endtab %} {% endtabs %} *** # Parsing output files ... or how to integrate tools that save their output to a file. If your tool supports output as individual files, you can easily integrate it with `secator`. * If you can set the output file path in advance, read [#setting-the-file-path](#setting-the-file-path "mention"). * If you cannot know, set, or guess the output file path in advance, read [#using-a-regular-expression-to-get-the-file-path](#using-a-regular-expression-to-get-the-file-path "mention"). *** ## Setting the file path **Steps:** * Set the file path before the command runs using the `on_init` hook * Use the `on_cmd_done` hook to read the file and yield `secator` output types. #### **Example:** Assume `mytool` outputs to a JSON file like: ```bash mytool -u mytarget.com -format json -o result.json ... ``` and the corresponding JSON would look like: ```json [ { "type": "url", "url": "http://mytarget.com", "status": 200 }, { "type": "vulnerability", "target": "https://mytarget.com", "cve_id": "CVE-XXXX-XXXX", "name": "Bad vuln", "severity": "CRITICAL"}, ] ``` An integration of `mytool` with `secator` would look like: {% code title="secator/tasks/mytool.py" %} ```python from secator.decorators import task from secator.runners import Command from secator.output_types import Url, Tag, Vulnerability from secator.serializers import RegexSerializer from secator.tasks._categories import Vuln OUTPUT_REGEX = r'\[INF\] JSON report will be saved to (?P<output_path>)' @task() class mytool(Command): cmd = '/home/osboxes/.local/bin/mytool' input_flag = '-u' json_flag = '-format json' output_types = [Url, Vulnerability] # Set the output file path @staticmethod def on_init(self): self.output_path = self.get_opt_value(OUTPUT_PATH) if not self.output_path: self.output_path = f'{self.reports_folder}/.outputs/{self.unique_name}.json' self.cmd += f' -o {self.output_path}' # When the command completes, load the JSON file and yield secator output types @staticmethod def on_cmd_done(self): with open(self.output_path, 'r') as f: results = f.read() for r in results: if r['type'] == 'url': yield Url( url=r['url'], status_code=r['status'], ) elif r['type'] == 'vulnerability': yield Vulnerability( id=r['cve_id'], name=r['name'], matched_at=r['target'], severity=r['severity'].lower() ) ``` {% endcode %} Run it with `secator`: {% tabs %} {% tab title="CLI" %} ```bash $ secator x mytool mytarget.com __ ________ _________ _/ /_____ _____ / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/ (__ / __/ /__/ /_/ / /_/ /_/ / / /____/\___/\___/\__,_/\__/\____/_/ v0.6.0 freelabz.com No Celery worker alive. /home/osboxes/.local/bin/mytool -u mytarget.com -format json -o <OUTPUT_FILE_PATH> 🔗 https://mytarget.com [200] 🚨 [Bad vuln 🡕] [critical] https://mytarget.com ``` {% endtab %} {% endtabs %} *** ## Using a regular expression to get the file path **Steps:** * Use the `RegexSerializer` to read and set the output file path. * Use the `on_cmd_done` hook to read the file and yield `secator` output types. #### **Example:** Assume `mytool` outputs to a JSON file like: ```bash mytool -u mytarget.com -o json [INF] JSON report will be saved to /path/to/custom/path.json ... ``` and the corresponding JSON would look like: ```json [ { "type": "url", "url": "http://mytarget.com", "status": 200 }, { "type": "vulnerability", "target": "https://mytarget.com", "cve_id": "CVE-XXXX-XXXX", "description": "Bad vuln" }, ] ``` First we need to find a named regular expression that will match the filename. Here is the one we came up with: {% code overflow="wrap" %} ```python OUTPUT_REGEX = r'\[INF\] JSON report will be saved to (?P<output_path>)' ``` {% endcode %} An integration of `mytool` with `secator` would look like: {% code title="secator/tasks/mytool.py" %} ```python from secator.decorators import task from secator.runners import Command from secator.output_types import Url, Tag, Vulnerability from secator.serializers import RegexSerializer from secator.tasks._categories import Vuln OUTPUT_REGEX = r'\[INF\] JSON report will be saved to (?P<output_path>)' @task() class mytool(Command): cmd = '/home/osboxes/.local/bin/mytool' input_flag = '-u' json_flag = '-o json' output_types = [Url, Vulnerability] # Override the default item loader (JSONSerializer) with the RegexSerializer item_loaders = [ RegexSerializer( OUTPUT_REGEX, fields=['output_path'] ) ] # React to items loaded by the RegexSerializer, and set the output path @staticmethod def on_regex_loaded(self, item): self.output_path = item['output_path'] return # When the command completes, load the JSON file and yield secator output types @staticmethod def on_cmd_done(self): with open(self.output_path, 'r') as f: results = f.read() for r in results: if r['type'] == 'url': yield Url( url=r['url'], status_code=r['status'], ) elif r['type'] == 'vulnerability': yield Vulnerability( id=r['cve_id'], name=r['name'], matched_at=r['target'], severity=r['severity'].lower() ) ``` {% endcode %} Run it with `secator`: {% tabs %} {% tab title="CLI" %} ```bash $ secator x mytool mytarget.com __ ________ _________ _/ /_____ _____ / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/ (__ / __/ /__/ /_/ / /_/ /_/ / / /____/\___/\___/\__,_/\__/\____/_/ v0.6.0 freelabz.com No Celery worker alive. /home/osboxes/.local/bin/mytool -u mytarget.com -o json ... [INF] JSON report will be saved to /path/to/custom/path.json 🔗 https://mytarget.com [200] 🚨 [Bad vuln 🡕] [critical] https://mytarget.com ``` {% endtab %} {% tab title="Python" %} ```python from secator.tasks import mytool task = mytool('mytarget.com') for item in task: print(item) # this will output Url, Vulnerability, or Tag items. ``` {% endtab %} {% endtabs %} *** # Example: integrating ls ... or how to integrate a command without JSON output. This section will present a relatively simple (but complete) use case of integrating a **real-world** command into `secator`. We picked the `ls` command because: * It is relatively simple. * It has no JSON output (we will need to fabricate it). * It has no direct secator output type mapping. *** ## Writing the task file Start by creating a file named `ls.py`: {% code title="ls.py" %} ```python from secator.runners import Command from secator.decorators import task @task() class ls(Command): cmd = 'ls' ``` {% endcode %} Move this file over to `~/.secator/templates/` (if you modified the default `dirs.templates`, move it to the corresponding location instead). You can test the initial implementation like so: <pre class="language-bash"><code class="lang-bash"><strong>$ secator x ls . </strong>ls . ls.py __pycache__ tasks 🗄 Saved JSON report to ~/.secator/reports/default/tasks/94/report.json 🗄 Saved CSV reports to ~/.secator/reports/default/tasks/94/report_target.csv </code></pre> Okay, this works ! {% hint style="warning" %} <mark style="color:red;">**YES**</mark><mark style="color:red;">, but we don't have enough details in the results !</mark> {% endhint %} *** ## Getting detailed output To add more details to the results, we should add more beef to the ls command, let's try with `ls -al` instead as the default `cmd` instead: {% code title="\~/.secator/templates/ls.py" %} ```python from secator.runners import Command from secator.decorators import task @task() class ls(Command): cmd = 'ls -al' ``` {% endcode %} and the output: ```bash $ secator x ls . ls -al . total 16 drwxr-xr-x 3 osboxes osboxes 4096 May 2 04:50 . drwxr-xr-x 4 osboxes osboxes 4096 May 2 04:49 .. -rw-r--r-- 1 osboxes osboxes 119 May 2 04:51 ls.py drwxr-xr-x 2 osboxes osboxes 4096 May 2 04:51 __pycache__ 🗄 Saved JSON report to ~/.secator/reports/default/tasks/1/report.json 🗄 Saved CSV reports to ~/.secator/reports/default/tasks/1/report_target.csv ``` Okay, this gives more information already ! {% hint style="warning" %} <mark style="color:red;">**YES**</mark><mark style="color:red;">, but we don't have any structured output ! I can't use -json and pipe the results to my super awesome CLI tool ...</mark> {% endhint %} *** ## Adding JSON output For this step we need to parse the `ls` command line text output by writing the `item_loader` method. The `item_loader` method takes a **line** as input and yield the **desired structured output** (dict). Here is how to implement it for the `ls` command: {% code title="\~/.secator/templates/ls.py" %} ```python from secator.runners import Command from secator.decorators import task @task() class ls(Command): cmd = 'ls -al' @staticmethod def item_loader(self, line): fields = ['permissions', 'link_count', 'owner', 'group', 'size', 'month', 'day', 'hour', 'path'] result = [c for c in line.split(' ') if c] if len(result) != len(fields): return None data = {} for ix, value in enumerate(result): data[fields[ix]] = value yield data ``` {% endcode %} and the output: ```bash $ secator x ls . --json ls -al . total 16 ❌ Failed to load item as output type: {'permissions': 'drwxr-xr-x', 'link_count': '3', 'owner': 'osboxes', 'group': 'osboxes', 'size': '4096', 'month': 'May', 'day': '2', 'hour': '04:50', 'path': '.', '_type': 'unknown', '_context': {'workspace_name': 'default'}, '_source': 'ls', '_uuid': 'fa90033a-56a4-4df1-b280-bb6f7cfdcd37'} ❌ Failed to load item as output type: {'permissions': 'drwxr-xr-x', 'link_count': '4', 'owner': 'osboxes', 'group': 'osboxes', 'size': '4096', 'month': 'May', 'day': '2', 'hour': '04:49', 'path': '..', '_type': 'unknown', '_context': {'workspace_name': 'default'}, '_source': 'ls', '_uuid': 'a76c01e4-8e07-4bdd-a2cf-66e96f1cd112'} ❌ Failed to load item as output type: {'permissions': '-rw-r--r--', 'link_count': '1', 'owner': 'osboxes', 'group': 'osboxes', 'size': '508', 'month': 'May', 'day': '2', 'hour': '04:55', 'path': 'ls.py', '_type': 'unknown', '_context': {'workspace_name': 'default'}, '_source': 'ls', '_uuid': '7189352a-fd06-493e-9bf8-de8e4a083782'} ❌ Failed to load item as output type: {'permissions': 'drwxr-xr-x', 'link_count': '2', 'owner': 'osboxes', 'group': 'osboxes', 'size': '4096', 'month': 'May', 'day': '2', 'hour': '04:55', 'path': '__pycache__', '_type': 'unknown', '_context': {'workspace_name': 'default'}, '_source': 'ls', '_uuid': '2c1d78b6-0c4d-4fa8-90c8-d08f8eda4922'} 🗄 Saved JSON report to ~/.secator/reports/default/tasks/2/report.json 🗄 Saved CSV reports to ~/.secator/reports/default/tasks/2/report_target.csv ❗Found 0 results. ``` The JSON objects are properly output, but they fail to convert with an existing [Output types](/in-depth/concepts/output-types). To get the original JSON output, run with `--orig` flag: ```bash $ secator x ls . --json --orig ls -al . total 16 {"permissions": "drwxr-xr-x", "link_count": "3", "owner": "osboxes", "group": "osboxes", "size": "4096", "month": "May", "day": "2", "hour": "04:50", "filename": ".", "_context": {"workspace_name": "default"}, "_source": "ls", "_uuid": "4aafe790-286a-4d59-9376-e6867d9bed6d", "_type": {}} {"permissions": "drwxr-xr-x", "link_count": "4", "owner": "osboxes", "group": "osboxes", "size": "4096", "month": "May", "day": "2", "hour": "04:49", "filename": "..", "_context": {"workspace_name": "default"}, "_source": "ls", "_uuid": "733a0336-eef4-4601-a420-8de68315c3e3", "_type": {}} {"permissions": "-rw-r--r--", "link_count": "1", "owner": "osboxes", "group": "osboxes", "size": "508", "month": "May", "day": "2", "hour": "04:55", "filename": "ls.py", "_context": {"workspace_name": "default"}, "_source": "ls", "_uuid": "f06836e1-844b-4e5d-9978-9fbbdba50d9c", "_type": {}} {"permissions": "drwxr-xr-x", "link_count": "2", "owner": "osboxes", "group": "osboxes", "size": "4096", "month": "May", "day": "2", "hour": "04:56", "filename": "__pycache__", "_context": {"workspace_name": "default"}, "_source": "ls", "_uuid": "9eb6b94a-4765-4764-9d48-89f2fa72e327", "_type": {}} 🗄 Saved JSON report to ~/.secator/reports/default/tasks/3/report.json 🗄 Saved CSV reports to ~/.secator/reports/default/tasks/3/report_target.csv ``` Ok, in a few lines of code we successfully managed to turn the `ls` output into structured JSON lines. {% hint style="warning" %} <mark style="color:red;">**YES**</mark><mark style="color:red;">, but we don't have anything in the JSON reports !</mark> {% endhint %} *** ## Mapping output types To get some useful results that secator reports understand, we need to map this arbitrary JSON output to one of the existing output type that `secator` provides. For instance, the `Vulnerability` output type ! For instance, we could consider as a vulnerability any path that is executable by the public. That's the final `w` in the permission string. Let's change the implementation to output objects of type `Vulnerability`: {% code title="\~/.secator/templates/ls.py" %} ```python from secator.runners import Command from secator.decorators import task from secator.output_types import Vulnerability @task() class ls(Command): cmd = 'ls -al' output_types = [Vulnerability] @staticmethod def item_loader(self, line): fields = ['permissions', 'link_count', 'owner', 'group', 'size', 'month', 'day', 'hour', 'path'] result = [c for c in line.split(' ') if c] if len(result) != len(fields): return None data = {} for ix, value in enumerate(result): data[fields[ix]] = value # Output vulnerabilities permissions = data['permissions'] path = data['path'] full_path = f'{self.input}/{path}' if permissions[-2] == 'w': # found a vulnerability ! yield Vulnerability( name='World-writeable path', severity='high', confidence='high', provider='ls', matched_at=full_path, extra_data={k: v for k, v in data.items() if k != 'path'} ) ``` {% endcode %} Let's make the `ls.py` file world-writeable with `chmod a+w ls.py` to create a vulnerability, and re-run our command: ```bash $ secator x ls . ls -al . total 16 drwxr-xr-x 3 osboxes osboxes 4096 May 2 06:14 . drwxr-xr-x 4 osboxes osboxes 4096 May 2 04:49 .. 🚨 [World-writeable path 🡕] [high] ./ls.py [permissions:-rw-rw-rw-, link_count:1, owner:osboxes, group:osboxes, size:1015, month:May, day:2, hour:06:14] drwxr-xr-x 2 osboxes osboxes 4096 May 2 06:14 __pycache__ 🗄 Saved JSON report to /home/osboxes/.secator/reports/default/tasks/140/report.json 🗄 Saved CSV reports to • /home/osboxes/.secator/reports/default/tasks/140/report_target.csv • /home/osboxes/.secator/reports/default/tasks/140/report_vulnerability.csv ✔ Found 1 vulnerability. ``` {% hint style="success" %} We have successfully integrated the command `ls` with `secator` ! {% endhint %} *** # Example: cat hunters ... or how to integrate groups of tasks with similar options. This section will present a more complex use case where we have three commands: `bigdog`, `catkiller` and `eagle` which purpose is to find cats. We will start by integrating `bigdog` to `secator` before realizing that most options can be mutualized between the three tools, and a common output type `Cat` can be created for all three. *** ## Bigdog Let's suppose we have a **fictional** utility called `bigdog` which purpose is to hunt cats on the internet. We want to add `bigdog` to `secator`. Here are some of `bigdog`'s options: {% tabs %} {% tab title="-site" %} `bigdog` can be run on a single site using `-site`: ```sh $ bigdog -site loadsofcats.com / \__ ( @\___ ============= / O BIGDOG v1.0.0 / (_____/ ============= /_____/ garfield [boss, 14] tony [admin, 18] ``` {% endtab %} {% tab title="-list" %} `bigdog` can be run on a list of sites using `-list`: ```sh $ bigdog -list sites.txt -json / \__ ( @\___ ============= / O BIGDOG v1.0.0 / (_____/ ============= /_____/ garfield [boss, 14] romuald [minion, 5] tony [admin, 18] ``` {% endtab %} {% tab title="-json" %} `bigdog` can output JSON lines using `-json`: ```sh $ bigdog -site loadsofcats.com -json / \__ ( @\___ ============= / O BIGDOG v1.0.0 / (_____/ ============= /_____/ {"name": "garfield", "age": 14, "host": "loadsofcat.com", "position": "boss"} {"name": "tony", "age": 18, "host": "loadsofcats.com", "position": "admin"} ``` {% endtab %} {% endtabs %} A basic definition of `bigdog` using basic `secator` concepts will be: {% code title="secator/tasks/bigdog.py" lineNumbers="true" %} ```py from secator.runners import Command from secator.decorators import task @task class bigdog(Command): cmd = 'bigdog' json_flag = '-json' input_flag = '-site' file_flag = '-list' ``` {% endcode %} You can now run `bigdog` from the CLI or the library: {% tabs %} {% tab title="CLI" %} ```bash secator x bigdog --help secator x bigdog loadsofcats.com ``` {% endtab %} {% tab title="Python" %} ```python from secator.tasks import bigdog # Get all results as a list, blocks until command has finished running bigdog('loadsofcats.com').run() [ {"name": "garfield", "age": 14, "host": "loadofcats.com", "position": "boss"}, {"name": "tony", "age": 18, "host": "loadsofcats.com", "position": "admin"} ] # Get result items in real-time as they arrive to stdout for cat in bigdog('loadsofcats.com'): print(cat['name'] + '(' + cat['age'] + ')') # Will print garfield (14) tony (18) ``` {% endtab %} {% endtabs %} Okay, this is a good start. Now what if the `bigdog` command has some more options that you would like to integrate ? * `-timeout` allows to specify a request timeout. * `-rate` allows to specify the max requests per minute. You can add the `opts` parameter to your `Command` object to define the cmd options: <pre class="language-py" data-line-numbers><code class="lang-py"><strong>from secator.runners import Command </strong>from secator.decorators import task @task class bigdog(Command): cmd = 'bigdog' json_flag = '-json' input_flag = '-site' file_flag = '-list' opt_prefix = '-' opts = { 'timeout': {'type': int, 'help': 'Timeout (in seconds)'}, 'rate': {'type': int, 'help': 'Max requests per minute'} } </code></pre> You can now use `bigdog` with this set of options: {% tabs %} {% tab title="CLI" %} ```bash secator x bigdog --help secator x bigdog loadsofcats.com -json secator x bigdog loadsofcats.com -timeout 1 -rate 100 -o table,csv,txt,gdrive ``` {% endtab %} {% tab title="Python" %} ```python from secator.tasks import bigdog bigdog('loadsofcats.com', rate=100, timeout=3).run() # adding rate and timeout options ``` {% endtab %} {% endtabs %} *** ## Cat hunters category One advantage of having class-based definitions is that we can group similar tools together in categories. Let's assume we have 2 other tools that can hunt cats: `catkiller` and `eagle`... ... but each of those tools might be written by a different person, and so the interface and output is different for each of them: {% tabs %} {% tab title="catkiller" %} <pre class="language-bash"><code class="lang-bash"><strong>$ catkiller --host loadsofcats.com --max-wait 1000 --max-rate 10 --json </strong>Starting catkiller session ... {"_info": {"name": "tony", "years": 18}, "site": "loadsofcats.com", "job": "admin"} {"_info": {"name": "garfield", "years": 14}, "site": "loadsofcats.com", "job": "boss"} # or to pass multiple hosts, it needs to be called like: $ cat hosts.txt | catkiller --max-wait 1000 --max-rate 10 --json </code></pre> **Inputs:** * `--host` is equivalent to `bigdog`'s `-site`. * `--max-wait` is equivalent to `bigdog`'s `-timeout`, but in milliseconds instead of seconds. * `--max-rate` is equivalent to `bigdog`'s `-rate`. * `--json` is equivalent to `bigdog's` `-json` option, but uses a different option character "`--`". * `cat hosts.txt | catkiller` is the equivalent to`bigdog`'s `-list`. **Output:** * `_info` has the data for `name` and `age`, but `age` is now `years`. * `site` is the equivalent of `bigdog`'s `host`. * `job` is the equivalent of `bigdog`'s `position`. {% endtab %} {% tab title="eagle" %} <pre class="language-bash"><code class="lang-bash"><strong>$ eagle -u loadsofcats.com -timeexpires 1 -jsonl </strong> _ | | ___ __ _ __ _| | ___ / _ \/ _` |/ _` | |/ _ \ | __/ (_| | (_| | | __/ v2.2.0 \___|\__,_|\__, |_|\___| __/ | |___/ {"alias": "tony", "occupation": "admin", "human_age": 105} # or to pass multiple hosts, it needs to be called like: $ eagle -l hosts.txt -timeexpires 1 -jsonl _ | | ___ __ _ __ _| | ___ / _ \/ _` |/ _` | |/ _ \ | __/ (_| | (_| | | __/ v2.2.0 \___|\__,_|\__, |_|\___| __/ | |___/ {"alias": "tony", "occupation": "admin", "human_age": 105, "host": "loadsofcats.com"} </code></pre> **Inputs:** * `-u` is equivalent to `bigdog`'s `-site`. * `-l` is equivalent to `bigdog`'s `-list`. * `-timeexpires` is equivalent to `bigdog`'s `-timeout`. * `eagle` **does not support** setting the maximum requests per seconds (`bigdog`'s `-rate`). * `-jsonl` is the flag to output JSON lines, instead of `bigdog`'s `-json`. **Output:** * `alias` is the equivalent of `bigdog`'s `name`. * `occupation` is the equivalent of `bigdog`'s `job`. * `human_age` is the human age conversion of the cat age. {% endtab %} {% endtabs %} ### Cat output type We first define a base `Cat` dataclass to define the common output schema and a `CatHunter` category as an input interface. We take `bigdog`'s output schema as reference to create the `Cat` output type: {% code title="secator/output\_types/cat.py" %} ```py from secator.definitions import OPT_NOT_SUPPORTED from secator.output_types import OutputType from secator.decorators import task from dataclasses import dataclass, field @dataclass class Cat(OutputType): name: str age: int alive: bool = False _source: str = field(default='', repr=True) _type: str = field(default='cat', repr=True) _uuid: str = field(default='', repr=True, compare=False) _table_fields = [name, age] _sort_by = (name, age) def __str__(self) -> str: return self.ip ``` {% endcode %} ### CatHunter category We take `bigdog`'s options names as reference and add the ones that can be mutualized to the `CatHunter` category: {% code title="secator/tasks/\_categories.py" %} ```python from secator.output_types import Cat # ... class CatHunter(Command): meta_opts = { 'timeout': {'type': int, 'default': 1, 'help': 'Timeout (in seconds)'}, 'rate': {'type': int, 'default': 1000, 'help': 'Max requests per minute'}, } output_types = [Cat] ``` {% endcode %} ### Tools implementation Finally we inherit all commands implementation from `CatHunter` and write the option mapping for the remaining cat-hunter commands: {% tabs %} {% tab title="bigdog" %} {% code title="secator/tasks/bigdog.py" %} ```python from secator.categories import CatHunter from secator.decorators import task @task() class bigdog(CatHunter): cmd = 'bigdog' json_flag = '-json' input_flag = '-site' file_flag = '-list' opt_prefix = '-' ``` {% endcode %} {% endtab %} {% tab title="catkiller" %} {% code title="secator/tasks/catkiller.py" %} ```python from secator.categories import CatHunter from secator.decorators import task from secator.output_types import Cat @task() class catkiller(CatHunter): cmd = 'catkiller' json_flag = '--json' input_flag = '--host' # stdin-like input using 'cat <FILE> | <COMMAND>' file_flag = None # catkiller options start with "--" unlike the other tools opt_prefix = '--' # Map `catkiller` options to CatHunter.meta_opts opt_key_map = { 'rate': 'max-rate' 'timeout': 'max-wait' } opt_value_map = { 'timeout': lambda x: x / 1000 # converting milliseconds to seconds } # Map `catkiller` output schema to Cat schema output_map = { Cat: { 'name': lambda x: x['_info']['name'], # note: you can use any function, we use 'age': lambda x: x['_info']['age'], # lambdas for readability here 'host': 'site', # 1:1 mapping 'job': 'job' # 1:1 mapping } } ``` {% endcode %} {% endtab %} {% tab title="eagle" %} <pre class="language-python" data-title="secator/tasks/eagle.py"><code class="lang-python">from secator.categories import CatHunter from secator.decorators import task from secator.definitions import OPT_NOT_SUPPORTED from secator.output_types import Cat <strong> </strong><strong> </strong><strong>@task() </strong>class eagle(CatHunter): cmd = 'eagle' json_flag = '-jsonl' input_flag = '-u' file_flag = '-l' # Map `eagle` input options to CatHunter.meta_opts opt_key_map = { 'rate': 'timeexpires', 'timeout': OPT_NOT_SUPPORTED # explicitely state that this option not supported by the target tool } # Map `eagle` output schema to Cat schema: output_map = { Cat: { 'name': 'alias', 'age': lambda x: human_to_cat_age(x['human_age']), 'job': 'occupation', } } # Add 'host' key dynamically after the item has been converted to the output schema, # since `eagle` doesn't return the host systematically. @staticmethod def on_item(self, item): item['host'] = item.get('host') or self.input return item def human_to_cat_age(human_age): cat_age = 0 if human_age <= 22: cat_age = human_age // 11 else: cat_age = (human_age - 22) // 5 + 2 return cat_age </code></pre> {% endtab %} {% endtabs %} Using these definitions, we can now use all the cat-hunter commands with a common interface (input options & output schema): {% tabs %} {% tab title="CLI" %} ```bash secator x bigdog loadsofcats.com -rate 1000 -timeout 1 -json secator x eagle loadsofcats.com -rate 1000 -timeout 1 -json secator x catkiller loadsofcats.com -rate 1000 -timeout 1 -json ``` {% endtab %} {% tab title="Python" %} ```python >>> from secator.tasks import bigdog, catkiller, eagle >>> meta_opts = {'timeout': 1, 'rate': 1000, 'json': True} >>> bigdog('loadsofcats.com', **meta_opts).run() [ Cat(name="garfield", age=14, host="loadsofcats.com", position="boss", _source="bigdog"), Cat(name="tony", age=18, host="loadsofcats.com", position="admin", _source="bigdog") ] >>> catkiller('catrunner.com', **meta_opts).run() [ Cat(name=fred, age=12, host="catrunner.com", position="minion", _source="catkiller"}, Cat(name=mark, age=20, host="catrunner.com", position="minion", _source="catkiller"} ] >>> eagle('allthecats.com', **meta_opts).run() [ Cat(name="marcus", age=4, host="allthecats.com", position="minion", _source="eagle"}, Cat(name="rafik", age=7, host="allthecats.com", position="minion", _source="eagle"} ] ``` {% endtab %} {% endtabs %} *** # Integrate custom Python code \[WIP] How to create custom tasks using pure Python code without external commands. The `PythonRunner` class allows you to create custom Secator tasks using pure Python code, without needing to integrate external command-line tools. This is useful when you want to: * Process data using Python libraries * Implement custom logic that doesn't require external commands * Create lightweight tasks that perform transformations or analysis * Build tasks that interact with APIs or databases *** ## Overview To create a custom Python task, you need to: 1. Create a Python file in `~/.secator/templates/` (for user tasks) or `secator/tasks/` (for development) 2. Inherit from the `PythonRunner` class 3. Decorate your class with the `@task()` decorator 4. Define `input_types` and `output_types` 5. Implement the `yielder()` method to yield results *** ## Basic structure Here's the minimal structure of a Python task: {% code title="\~/.secator/templates/mytask.py" %} ```python from secator.decorators import task from secator.definitions import HOST from secator.output_types import Tag, Url from secator.runners import PythonRunner @task() class mytask(PythonRunner): """Description of what this task does.""" input_types = [HOST] output_types = [Tag, Url] def yielder(self): for target in self.inputs: yield Url(url=f"http://{target}") yield Tag(name="scanned", match=target) ``` {% endcode %} **Important notes:** * The class name must match the filename (without `.py`) * The `@task()` decorator is required * You must inherit from `PythonRunner` * The `yielder()` method is where your task logic goes * Use `yield` to return results as output types *** ## Available input types You can specify which input types your task accepts using `input_types`. Available input types include: * `HOST` - Hostnames or domains * `URL` - URLs * `IP` - IP addresses * `CIDR_RANGE` - CIDR ranges * `EMAIL` - Email addresses * `PATH` - File paths * `STRING` - Generic strings * `None` - Accept any input type Example: ```python from secator.definitions import URL, IP, HOST @task() class mytask(PythonRunner): # Accept multiple input types input_types = [URL, IP, HOST] # ... ``` *** ## Available output types Your task can yield various output types. Common ones include: * `Info` - Informational messages * `Url` - URLs * `Ip` - IP addresses * `Port` - Network ports * `Subdomain` - Subdomains * `Vulnerability` - Security vulnerabilities * `Tag` - Tags/metadata * `Domain` - Domains * `Record` - DNS records * `Certificate` - SSL certificates * `UserAccount` - User accounts * `Warning` - Warning messages * `Error` - Error messages Example: ```python from secator.output_types import Url, Vulnerability, Tag, Info @task() class mytask(PythonRunner): output_types = [Url, Vulnerability, Tag, Info] # ... ``` *** ## Example: Simple URL processor This example processes URLs and extracts information: {% code title="\~/.secator/templates/urlprocessor.py" %} ```python from urllib.parse import urlparse from secator.decorators import task from secator.definitions import URL from secator.output_types import Tag, Url from secator.runners import PythonRunner @task() class urlprocessor(PythonRunner): """Extract components from URLs.""" input_types = [URL] output_types = [Tag, Url] tags = ['url', 'parsing'] def yielder(self): for url in self.inputs: parsed = urlparse(url) # Yield the original URL yield Url(url=url) # Yield tags with extracted information yield Tag( name='url_scheme', value=parsed.scheme, match=url, category='info' ) yield Tag( name='url_domain', value=parsed.netloc, match=url, category='info' ) yield Tag( name='url_path', value=parsed.path, match=url, category='info' ) ``` {% endcode %} Usage: ```bash secator x urlprocessor https://example.com/path?query=1 ``` *** ## Example: Vulnerability scanner This example demonstrates yielding vulnerabilities: {% code title="\~/.secator/templates/customscanner.py" %} ```python import requests from secator.decorators import task from secator.definitions import URL from secator.output_types import Vulnerability, Info, Warning from secator.runners import PythonRunner @task() class customscanner(PythonRunner): """Custom vulnerability scanner.""" input_types = [URL] output_types = [Vulnerability, Info, Warning] tags = ['vuln', 'web'] def yielder(self): yield Info(message="Starting custom scan") for url in self.inputs: try: response = requests.get(url, timeout=5) # Check for security headers if 'X-Frame-Options' not in response.headers: yield Vulnerability( name="Missing X-Frame-Options header", severity="medium", confidence="high", matched_at=url, provider="customscanner" ) # Check for exposed server information if 'Server' in response.headers: server = response.headers['Server'] yield Warning( message=f"Server header exposed: {server}", matched_at=url ) except requests.RequestException as e: yield Warning( message=f"Failed to scan {url}: {str(e)}", matched_at=url ) yield Info(message="Scan complete") ``` {% endcode %} *** ## Example: Task with custom options You can define custom options that users can pass to your task: {% code title="\~/.secator/templates/tagger.py" %} ```python from secator.decorators import task from secator.definitions import HOST from secator.output_types import Tag from secator.runners import PythonRunner @task() class tagger(PythonRunner): """Tag hosts with custom metadata.""" input_types = [HOST] output_types = [Tag] tags = ['tagging'] opts = { 'tag_name': { 'type': str, 'default': 'custom_tag', 'help': 'Name of the tag to apply' }, 'tag_value': { 'type': str, 'default': 'scanned', 'help': 'Value for the tag' }, 'category': { 'type': str, 'default': 'info', 'help': 'Tag category' } } def yielder(self): tag_name = self.run_opts.get('tag_name', 'custom_tag') tag_value = self.run_opts.get('tag_value', 'scanned') category = self.run_opts.get('category', 'info') for host in self.inputs: yield Tag( name=tag_name, value=tag_value, match=host, category=category ) ``` {% endcode %} Usage: ```bash secator x tagger example.com --tag-name "environment" --tag-value "production" ``` *** ## Example: Task without inputs Some tasks don't require inputs. Use `default_inputs` to make inputs optional: {% code title="\~/.secator/templates/netdetect.py" %} ```python import ifaddr import ipaddress from secator.decorators import task from secator.output_types import Tag, Ip from secator.runners import PythonRunner @task() class netdetect(PythonRunner): """Detect local network CIDR ranges.""" output_types = [Tag, Ip] tags = ['network', 'recon'] default_inputs = '' # No inputs required input_flag = None def yielder(self): adapters = ifaddr.get_adapters() for adapter in adapters: if adapter.name == 'lo' or adapter.name.lower().startswith('loopback'): continue yield Tag( name='net_interface', match='localhost', value=adapter.nice_name, category='info', ) for ip in adapter.ips: if ip.is_IPv4: try: network = ipaddress.IPv4Network( f"{ip.ip}/{ip.network_prefix}", strict=False ) yield Ip( ip=ip.ip, host='localhost', alive=True, ) yield Tag( name='net_cidr', match='localhost', value=str(network), category='info', ) except ValueError: continue ``` {% endcode %} Usage: ```bash secator x netdetect ``` *** ## Accessing runner properties In your `yielder()` method, you have access to several useful properties: * `self.inputs` - List of input values * `self.run_opts` - Dictionary of run options (including custom `opts`) * `self.name` - Task name * `self.config` - Task configuration * `self.context` - Runner context Example: ```python def yielder(self): yield Info(message=f"Task name: {self.name}") yield Info(message=f"Processing {len(self.inputs)} inputs") # Access custom options timeout = self.run_opts.get('timeout', 10) for input in self.inputs: # Process each input pass ``` *** ## Advanced: Multiple output types You can yield different output types based on your logic: {% code title="\~/.secator/templates/multiplexer.py" %} ```python from secator.decorators import task from secator.definitions import HOST from secator.output_types import Url, Vulnerability, Tag, Info from secator.runners import PythonRunner @task() class multiplexer(PythonRunner): """Process hosts and yield different output types.""" input_types = [HOST] output_types = [Url, Vulnerability, Tag, Info] def yielder(self): yield Info(message="Starting multiplex scan") for host in self.inputs: # Always yield a URL yield Url(url=f"https://{host}") # Conditionally yield vulnerabilities if "test" in host.lower(): yield Vulnerability( name="Test environment detected", severity="low", confidence="high", matched_at=host, provider="multiplexer" ) # Yield tags yield Tag( name="scanned_host", value=host, match=host, category="info" ) yield Info(message="Scan complete") ``` {% endcode %} *** ## Task discovery Secator automatically discovers tasks from: * **User tasks**: `~/.secator/templates/*.py` * **Development tasks**: `secator/tasks/*.py` (when developing Secator itself) The task class name must match the filename (case-sensitive). For example: * File: `~/.secator/templates/mytask.py` → Class: `mytask` * File: `secator/tasks/urlparser.py` → Class: `urlparser` After creating your task file, Secator will automatically discover it on the next run. *** ## Best practices 1. **Use descriptive class names**: Choose names that clearly indicate what the task does 2. **Add docstrings**: Document what your task does in the class docstring 3. **Set appropriate tags**: Use `tags` to categorize your task for better discoverability 4. **Handle errors gracefully**: Use `Warning` or `Error` output types for error conditions 5. **Yield progress information**: Use `Info` messages to provide feedback during long-running tasks 6. **Validate inputs**: Check input validity before processing 7. **Use appropriate output types**: Choose the most specific output type for your results *** ## Testing your task You can test your task from the command line: ```bash # Run the task secator x mytask example.com # Get help secator x mytask --help # Run with JSON output secator x mytask example.com --json # Run with custom options secator x mytask example.com --option1 value1 ``` Or use it programmatically: ```python from secator.tasks import mytask # Run the task results = mytask('example.com').run() # Access results for result in results: print(f"{result._type}: {result}") ``` *** ## Common patterns ### Pattern: Processing with external libraries ```python import some_library @task() class mytask(PythonRunner): def yielder(self): for input in self.inputs: result = some_library.process(input) yield Tag(name="result", value=result, match=input) ``` ### Pattern: API integration ```python import requests @task() class apitask(PythonRunner): def yielder(self): for input in self.inputs: response = requests.get(f"https://api.example.com/{input}") data = response.json() yield Tag(name="api_data", value=str(data), match=input) ``` ### Pattern: Data transformation ```python @task() class transform(PythonRunner): def yielder(self): for input in self.inputs: # Transform the input transformed = input.upper().replace("-", "_") yield Tag(name="transformed", value=transformed, match=input) ``` *** ## Troubleshooting ### Task not discovered * Ensure the filename matches the class name exactly * Check that the file is in `~/.secator/templates/` or `secator/tasks/` * Verify the `@task()` decorator is present * Make sure the class inherits from `PythonRunner` ### Validation errors * If you get "Input is empty" errors, set `default_inputs = ''` for tasks that don't need inputs * For tasks that accept multiple inputs, ensure you're running with a worker (multiple inputs aren't supported in non-worker mode) ### Import errors * Ensure all required Python packages are installed * Check that imports are correct and available in your environment # Advanced options There are additional class options and functions you can specify in the task definition class, based on which runner is used. *** ## `Command` runner * `shell` (`bool`, `default: False`): Run `subprocess.Popen` with `shell=True` (**dangerous !**). * `cwd` (`str`, `default: None`): Command current working directory. * `encoding` (`dict`, `default: utf-8`): Output encoding. * `opt_prefix` (`str`, `default: -`): Change the prefix used to specify command options. * `version_flag` (`str`, `default: None`): The version flag. Defaults to `{opt_prefix}version`. * `ignore_return_code` (`bool`, `default: False`): Ignore the command return code (useful if your command has non-standard return codes). *** # Writing workflows ... or how to integrate new workflows with secator. New `secator` workflows should be easy to write to promote contributions from the community. Eventually we aim for our workflow library to become a reference in cyber-security much like Nuclei templates have become a reference for vulnerability searching. *** ## Basic YAML definition `secator` workflows are defined through YAML configs: {% code title="secator/configs/workflows/url\_finder.yaml" %} ```yaml type: workflow name: url_finder alias: ufind description: URL finder and tagger tags: [http] input_types: - url tasks: katana: description: Find URLs rate_limit: 100 timeout: 1 gf: description: Tag URLs pattern: xss ``` {% endcode %} *** ## Dynamic targets You can specify dynamic targets for tasks from current run results, by using the `targets_` key in your template like: ```yaml ... tasks: ... gf: description: Tag URLs found by the previous task targets_: # use previously found URLs - type: url field: url condition: item.status_code == 200 ``` {% hint style="info" %} The dynamic format keys are: * `type` is the output type, lower-case (see [Broken mention](broken://pages/qP8jXQz9GLFqQAaBiJHX) for the whole list) * `field` is the JSON field to use as target * `condition` is the filtering condition. {% endhint %} *** ## Concurrent tasks You can specify tasks that run in parallel using the `_group` key: ```yaml ... tasks: ... katana: _group: gf: ... another_task: description: Runs concurrently with the `gf` task ``` In this configuration, the `katana`task will begin the run, followed by a grouped execution of `gf` and `another_task`. {% hint style="info" %} Concurrent tasks require`secator` to be setup in worker mode (see [Distributed runs with Celery](/in-depth/distributed-runs-with-celery)). {% endhint %} *** ## Result filtering You can customize which results you want to keep for workflows and scans by adding the `results` key to the respective runner YAML configuration: <pre class="language-yaml"><code class="lang-yaml"><strong>... </strong><strong>results: </strong>- type: ip condition: item.alive - type: url condition: item.status_code == 200 </code></pre> {% hint style="warning" %} Filters apply to final run results sent to [Exporters](/in-depth/concepts/exporters), but do not apply to real-time results sent to [Drivers](/in-depth/concepts/drivers). {% endhint %} *** # Writing scans \[WIP] ... or how to integrate new scans with secator.