Integrating an external command

... or how to turn a command that you use daily into an overpowered machine.

Creating a task file

Imagine we have a tool named mytool that we want to integrate with secator.

Start by creating a file named mytool.py:

from secator.decorators import task  # required for `secator` to recognize tasks
from secator.runners import Command  # the `secator` runner to use


@task()
class mytool(Command): # make sure class name is lowercase and matches the filename.
    cmd = 'mytool'  # ... or whatever the name of your external command is.

The task decorator is required for secator to recognize class-based definition that need to be loaded at runtime.

Move this file over to:

~/.secator/templates/ (or whatever your dirs.templates in Configuration points to)

secator/tasks/ if you have a Development setup and want to contribute your task implementation to the official secator repository.

Adding an input flag [optional]

If your tool requires an input flag or a list flag to take its targets, for instance:

mytool -u TARGET
mytool -l TXT_FILE

You need to set the input_flag and file_flag class options:

from secator.decorators import task
from secator.runners import Command


@task()
class mytool(Command):
    cmd = 'mytool'
    input_flag = '-u'
    file_flag = '-l'

Setting these attributes allows us to run mytool with secator like:

secator x mytool TARGET    # will run mytool -u TARGET
secator x mytool TXT_FILE  # will run mytool -l TXT_FILE

Parsing a command's output

Now that you have a basic implementation working, you need to convert your command's output into structured output (JSON).

Find out what your command's output looks like and pick the corresponding guide:

Read Parsing JSON lines if your tool has an option to stream JSON lines (preferred).
Read Parsing output files if your tool has an option to output to a file (e.g JSON or CSV).
Read Parsing raw standard output if your tools only outputs to stdout .

Adding more options [optional]

To support more options, you can use the opt_prefix, opts , opt_key_map and opt_value_map attributes.

Assuming mytool has the --delay, --debug and --include-tags options, we would support them this way:

@task()
class mytool(Command):
    # ...
    opt_prefix = '--'  # default is '-'
    opts = {
        'tags': {'type': str, 'short': 't', 'help': 'Tags'},
        'delay': {'type': int, 'short': 'dbg', 'help': 'Delay'},
        'debug': {'is_flag': True, 'short': 'd', 'help': 'Debug mode'},
    }
    opt_key_map = {   # to map input options
        'tags': 'include-tags',
    }
    opt_value_map = {  # to transform options values
        'delay': lambda x: x * 1000  # convert seconds to milliseconds
    }

With this config, running either of:

secator x mytool --tags tag1,tag2 --debug --delay 5 TARGET  # long option format
secator x mytool -t tag1,tag2 -dbg -d 5 TARGET              # short option format

will result in running mytool like:

mytool --include-tags tag1,tag2 --debug --delay 5000 -u TARGET

Adding an install command [optional]

To support installing your tool with secator, you can set the install_cmd , and / or install_github_handle attributes:

@task()
class mytool(Command):
    # ...
    install_cmd = "sudo apt install -y mytool"
    install_github_handle = "myorg/mytool"

If install_github_handle is set, secator will try to fetch a binary from GitHub releases specific to your platform, and fallback to install_cmd if it cannot find a suitable release, or if the API rate limit is reached.

Now you can install mytool using:

secator install tools mytool

Using a category [optional]

If your tool fits into one of secator's built-in command categories, you can inherit from it's option set:

Http: A tool that makes HTTP requests.
HttpCrawler: A command that crawls URLs (subset of Http).
HttpFuzzer: A command that fuzzes URLs (subset of Http).

You can inherit from these categories and map their options to your command.

Categories are defined in secator/tasks/_categories.py

For instance, if mytool is an HTTP fuzzer, we would change it's implementation like:

from secator.tasks._categories import HTTPFuzzer
from secator.definitions import OPT_NOT_SUPPORTED


@task()
class mytool(Command, HTTPFuzzer):
    # ...
    opt_key_map = {
        # HTTPFuzzer options mapping
        HEADER: 'header',
        DELAY: 'delay',
        DEPTH: OPT_NOT_SUPPORTED,
        FILTER_CODES: OPT_NOT_SUPPORTED,
        FILTER_REGEX: OPT_NOT_SUPPORTED,
        FILTER_SIZE: OPT_NOT_SUPPORTED,
        FILTER_WORDS: OPT_NOT_SUPPORTED,
        FOLLOW_REDIRECT: OPT_NOT_SUPPORTED,
        MATCH_CODES: OPT_NOT_SUPPORTED,
        MATCH_REGEX: OPT_NOT_SUPPORTED,
        MATCH_SIZE: OPT_NOT_SUPPORTED,
        MATCH_WORDS: OPT_NOT_SUPPORTED,
        METHOD: OPT_NOT_SUPPORTED,
        PROXY: OPT_NOT_SUPPORTED,
        RATE_LIMIT: OPT_NOT_SUPPORTED,
        RETRIES: OPT_NOT_SUPPORTED,
        THREADS: OPT_NOT_SUPPORTED,
        TIMEOUT: 'timeout',
        USER_AGENT: 'user-agent',

        # my tool specific options
        'tags': 'include-tags',      
    }
    opt_value_map = {
        'delay': lambda x: x * 1000  # convert seconds to milliseconds
    }

Make sure you map all the options from the HTTPFuzzer category. If some options are not supported by your tool, mark them with OPT_NOT_SUPPORTED.

With this config, running:

secator x mytool --help

would list:

The metaoptions in the HTTPFuzzer category that are supported by mytool.
The options only usable by mytool.

For instance, running:

secator x mytool \
  -header "Authorization: Bearer MYTOKEN" \
  -delay 1 \
  -ua "secator/0.6 (Debian)" \
  -timeout 5 \
  -t tag1,tag2 \
  TARGET

from secator.tasks import mytool

task = mytool(
  'TARGET',
  header="Authorization: Bearer MYTOKEN",
  delay=1,
  user_agent="secator/0.6 (Debian)",
  timeout=5,
  tags='tag1,tag2'
)
for item in task:
     print(item)

would result in running mytool like:

mytool \
  --header "Authorization: Bearer MYTOKEN" \
  --delay 1000 \
  --user-agent "secator/0.6 (Debian)" \
  --timeout 5 \
  --include-tags tag1,tag2
  -u TARGET

Supporting proxies [optional]

If your tool supports proxies, secator has first-class support for proxychains, HTTP and SOCKS5 proxies, and can dynamically choose the type of proxy to use based on the following attributes:

proxy_socks5 : boolean indicating if your command supports SOCKS5 proxies.
proxy_http : boolean indicating if your command supports HTTP / HTTPS proxies.
proxychains: boolean indicating if your command supports being run with proxychains.

If your proxy supports SOCKS5or HTTP proxies, make sure to have an option called proxy in your opts definition or it won't be picked up.

If your proxy supports proxychains, secator will use the local proxychains binary and proxychains.conf configuration, so make sure those are functional.

Read Proxiesfor more details on how proxies work and how to configure them properly.

Example:

Assuming mytool does not support HTTP or SOCKS5 proxies, but works with proxychains, you can update your task definition like:

@task()
class mytool(Command):
    # ...
    proxychains = True
    proxy_socks5 = False
    proxy_http = True

With the above configuration, running with -proxy <VALUE> would result in the following behaviour:

secator x mytool -proxy proxychains TARGET
secator x mytool -proxy auto TARGET # auto-pick from proxychains > socks5 > http

becomes:

proxychains mytool -u TARGET

secator x mytool -proxy "http://testmyproxy.com" <TARGET>

becomes:

mytool --use-proxy "http://testmyproxy.com" -u <TARGET>

secator x mytool -proxy random <TARGET>

becomes:

mytool --use-proxy "http://30.10.23.42" -u <TARGET> # random proxy (FreeProxy)

Hooking onto runner lifecycle

You can hook onto any part of the runner lifecycle by override the hooks methods (read Lifecyle hooks to know more).

Example:

@task()
class mytool(Command):
    # ...
    @staticmethod
    def on_line(self, line):
        return line.rstrip(',')  # strip trailing comma of stdout lines

    @staticmethod
    def on_item_pre_convert(self, item):
        item['extra_data'] = {
            'version': '2.0'  # add extra data to items
        }
        return item

from secator.tasks import mytool

hooks = {
    'on_line': lambda x: x.strip(','),
    'on_item_pre_convert': lambda x: x | {'extra_data': {'version': '2.0'}},
}
task = mytool('TARGET', hooks=hooks)
for item in task:
     print(item)

Chunking

secator allows to chunk a task into multiple children tasks when the length of the input grows, or some other specific requirements (e.g: your command only takes one target at a time).

Chunking only works when Distributed runs with Celery are enabled.

You can specify the chunk size using the input_chunk_size attribute:

@task()
class mytool(Command):
    # ...
    input_chunk_size = 10  # additional tasks will be spawned every 10 targets

With this config, running:

secator x mytool tasks.txt  # tasks.txt contains 20 targets

would result in:

mytool -l /tmp/task_0_9.txt
mytool -l /tmp/task_10_19.txt

If mytool did not support file input (i.e: file_flag not defined in the task definition class), the above would still work with an input_chunk_size = 1, thus splitting into one command per target passed.

PreviousWriting tasks NextParsing JSON lines

Last updated 8 months ago

Was this helpful?