Example: cat hunters

... or how to integrate groups of tasks with similar options.

This section will present a more complex use case where we have three commands: bigdog, catkiller and eagle which purpose is to find cats.

We will start by integrating bigdog to secator before realizing that most options can be mutualized between the three tools, and a common output type Cat can be created for all three.

Bigdog

Let's suppose we have a fictional utility called bigdog which purpose is to hunt cats on the internet. We want to add bigdog to secator.

Here are some of bigdog's options:

bigdog can be run on a single site using -site:

$ bigdog -site loadsofcats.com
   / \__
  (    @\___   =============
  /         O  BIGDOG v1.0.0
 /   (_____/   =============
/_____/
garfield [boss, 14]
tony [admin, 18]

A basic definition of bigdog using basic secator concepts will be:

secator/tasks/bigdog.py
from secator.runners import Command
from secator.decorators import task


@task
class bigdog(Command):
    cmd = 'bigdog'
    json_flag = '-json'
    input_flag = '-site'
    file_flag = '-list'

You can now run bigdog from the CLI or the library:

secator x bigdog --help
secator x bigdog loadsofcats.com

Okay, this is a good start.

Now what if the bigdog command has some more options that you would like to integrate ?

  • -timeout allows to specify a request timeout.

  • -rate allows to specify the max requests per minute.

You can add the opts parameter to your Command object to define the cmd options:

from secator.runners import Command
from secator.decorators import task


@task
class bigdog(Command):
    cmd = 'bigdog'
    json_flag = '-json'
    input_flag = '-site'
    file_flag = '-list'
    opt_prefix = '-'
    opts = {
        'timeout': {'type': int, 'help': 'Timeout (in seconds)'},
        'rate': {'type': int, 'help': 'Max requests per minute'}
    }

You can now use bigdog with this set of options:

secator x bigdog --help
secator x bigdog loadsofcats.com -json
secator x bigdog loadsofcats.com -timeout 1 -rate 100 -o table,csv,txt,gdrive

Cat hunters category

One advantage of having class-based definitions is that we can group similar tools together in categories.

Let's assume we have 2 other tools that can hunt cats: catkiller and eagle...

... but each of those tools might be written by a different person, and so the interface and output is different for each of them:

$ catkiller --host loadsofcats.com --max-wait 1000 --max-rate 10 --json
Starting catkiller session ...
{"_info": {"name": "tony", "years": 18}, "site": "loadsofcats.com", "job": "admin"}
{"_info": {"name": "garfield", "years": 14}, "site": "loadsofcats.com", "job": "boss"}

# or to pass multiple hosts, it needs to be called like:
$ cat hosts.txt | catkiller --max-wait 1000 --max-rate 10 --json

Inputs:

  • --host is equivalent to bigdog's -site.

  • --max-wait is equivalent to bigdog's -timeout, but in milliseconds instead of seconds.

  • --max-rate is equivalent to bigdog's -rate.

  • --json is equivalent to bigdog's -json option, but uses a different option character "--".

  • cat hosts.txt | catkiller is the equivalent tobigdog's -list.

Output:

  • _info has the data for name and age, but age is now years.

  • site is the equivalent of bigdog's host.

  • job is the equivalent of bigdog's position.

Cat output type

We first define a base Cat dataclass to define the common output schema and a CatHunter category as an input interface.

We take bigdog's output schema as reference to create the Cat output type:

secator/output_types/cat.py
from secator.definitions import OPT_NOT_SUPPORTED
from secator.output_types import OutputType
from secator.decorators import task
from dataclasses import dataclass, field


@dataclass
class Cat(OutputType):
    name: str
    age: int
    alive: bool = False
    _source: str = field(default='', repr=True)
    _type: str = field(default='cat', repr=True)
    _uuid: str = field(default='', repr=True, compare=False)

    _table_fields = [name, age]
    _sort_by = (name, age)

    def __str__(self) -> str:
        return self.ip

CatHunter category

We take bigdog's options names as reference and add the ones that can be mutualized to the CatHunter category:

secator/tasks/_categories.py
from secator.output_types import Cat
# ...

class CatHunter(Command):
    meta_opts = {
        'timeout': {'type': int, 'default': 1, 'help': 'Timeout (in seconds)'},
        'rate': {'type': int, 'default': 1000, 'help': 'Max requests per minute'},
    }
    output_types = [Cat]

Tools implementation

Finally we inherit all commands implementation from CatHunter and write the option mapping for the remaining cat-hunter commands:

secator/tasks/bigdog.py
from secator.categories import CatHunter
from secator.decorators import task


@task()
class bigdog(CatHunter):
    cmd = 'bigdog'
    json_flag = '-json'
    input_flag = '-site'
    file_flag = '-list'
    opt_prefix = '-'

Using these definitions, we can now use all the cat-hunter commands with a common interface (input options & output schema):

secator x bigdog loadsofcats.com -rate 1000 -timeout 1 -json
secator x eagle loadsofcats.com -rate 1000 -timeout 1 -json
secator x catkiller loadsofcats.com -rate 1000 -timeout 1 -json
PreviousExample: integrating lsNextIntegrate custom Python code [WIP]

Last updated

Was this helpful?