Secator docs
  • GETTING STARTED
    • Introduction
    • Installation
    • CLI Usage
    • Library usage
    • Configuration
    • Examples
      • 5 minutes secator session
  • RUNNER OPTIONS
    • Global options
    • Meta options
    • Input formats
    • Output options
  • IN-DEPTH
    • Philosophy & design
    • Distributed runs with Celery
    • Concepts
      • Output types
      • Proxies
      • Exporters
      • Runners
      • Drivers
      • Profiles
    • Deployment
  • For developers
    • Development setup
    • Writing tasks
      • Integrating an external command
        • Parsing JSON lines
        • Parsing raw standard output
        • Parsing output files
        • Example: integrating ls
        • Example: cat hunters
      • Integrate custom Python code [WIP]
      • Advanced options
    • Writing workflows
    • Writing scans [WIP]
Powered by GitBook
On this page
  • Using regular expressions
  • Writing a custom item loader

Was this helpful?

  1. For developers
  2. Writing tasks
  3. Integrating an external command

Parsing raw standard output

... or how to integrate tools that prints to stdout and do not support JSON lines.

If your tool does not output JSON lines / JSON files, it's requires a bit more effort to integrate it with secator.

Depending on how you want to parse the output, read:

  • Using regular expressions

  • Writing a custom item loader


Using regular expressions

Steps:

  • Use the RegexSerializer item loader.

  • Use the on_regex_loaded hook to yield secator output types.

Example:

Assume mytool outputs on stdout like:

mytool -u mytarget.com
[INF] This is an info message
[ERR] This is an error message
[FOUND] https://mytarget.com/api [type=url] [status=200] [content_type=application/json] [title=MyAwesomeWebPage]
[FOUND] https://mytarget.com/api/metrics [type=url] [status=403]
[FOUND] A3TBABCD1234EFGH5678 [type=aws_api_key] [matched_at=https://mytarget/api/.aws_key.json]
[FOUND] <-- an HTML comment --> [type=aws_api_key] [matched_at=https://mytarget/api/.aws_key.json]
[FOUND] CVE-2021-44228 [type=vulnerability] [matched_at=https://mytarget/api/sensitive_api_path]

Here is the one we came up with:

OUTPUT_REGEX = r'\[\w+]\s(?P<value>.*)\s\[type=(?P<type>[\w_]+)\](\s\[status=(?P<status>\d+)\])?(\s\[content_type=(?P<content_type>[\w\/]+)\])?(\s\[title=(?P<title>.*)\])?(\s\[matched_at=(?P<matched_at>.*)\])?'

An integration of mytool with secator would look like:

secator/tasks/mytool.py
from secator.decorators import task
from secator.runners import Command
from secator.output_types import Url, Tag, Vulnerability
from secator.serializers import RegexSerializer
from secator.tasks._categories import Vuln

OUTPUT_REGEX = r'\[\w+]\s(?P<value>.*)\s\[type=(?P<type>[\w_]+)\](\s\[status=(?P<status>\d+)\])?(\s\[content_type=(?P<content_type>[\w\/]+)\])?(\s\[title=(?P<title>.*)\])?(\s\[matched_at=(?P<matched_at>.*)\])?'


@task()
class mytool(Command):
  cmd = '/home/osboxes/.local/bin/mytool'
  input_flag = '-u'
  json_flag = '-jsonl'
  output_types = [Url, Tag, Vulnerability]

  # Use the RegexSerializer to load the stdout input
  item_loaders = [
    RegexSerializer(
      OUTPUT_REGEX,
      fields=['value', 'type', 'status', 'content_type', 'title', 'matched_at']
    )
  ]

  # React to items loaded by the RegexSerializer, and yield secator output types
  # like Url, Vulnerability, and Tag.
  @staticmethod
  def on_regex_loaded(self, item):
    # this is called after the regex serializer runs,
    # so we can expect item to be a dict with the matched regex values
    if (item['type'] == 'url'):
      yield Url(
        url=item['value'],
        status_code=int(item['status']),
        content_type=item['content_type'],
        title=item['title']
      )
    elif (item['type'] == 'vulnerability'):
      cve_id = item['value']
      lookup_data = Vuln.lookup_cve(cve_id)  # perform vulnerability search
      vuln = {
        'matched_at': item['matched_at']
      }
      if lookup_data:
        vuln.update(**lookup_data)
      yield Vulnerability(**vuln)
    else:
      yield Tag(
        name=item['type'],
        match=item['matched_at'],
        extra_data={
          'secret': item['value']
        }
      )

Run it with secator:

$ secator x mytool mytarget.com

                         __            
   ________  _________ _/ /_____  _____
  / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/
 (__  /  __/ /__/ /_/ / /_/ /_/ / /    
/____/\___/\___/\__,_/\__/\____/_/     v0.6.0

                        freelabz.com

No Celery worker alive.
/home/osboxes/.local/bin/mytool -u mytarget.com -jsonl
[INF] This is an info message
[ERR] This is an error message
🔗 https://mytarget.com/api [200] [MyAwesomeWebPage] [application/json]
🔗 https://mytarget.com/api/metrics [403]
🏷️ aws_api_key found @ https://mytarget/api/.aws_key.json
    secret: A3TBABCD1234EFGH5678
🚨 [Object Injection 🡕] [critical] https://mytarget/api/sensitive_api_path
from secator.tasks import mytool
task = mytool('mytarget.com')
for item in task:
    print(item)  # this will output Url, Vulnerability, or Tag items.

Writing a custom item loader

Steps:

  • Override the item_loader static method to parse the standard output with custom code.

Example:

Assume mytool outputs on stdout like:

mytool -u mytarget.com
https://mytarget.com/api | url | 200 | application/json | MyAwesomePage
https://mytarget.com/api/metrics | url | 403
A3TBABCD1234EFGH5678 | aws_api_key | http://mytarget/api/.aws_key.json
<-- an HTML comment --> | html_comment | http://mytarget/api/.aws_key.json
CVE-2021-44228 | vulnerability | http://mytarget/api/sensitive_ap
from secator.decorators import task
from secator.runners import Command
from secator.output_types import Url, Tag, Vulnerability


@task()
class mytool(Command):
  cmd = '/home/osboxes/.local/bin/mytool'
  input_flag = '-u'
  json_flag = '-jsonl'
  output_types = [Url, Tag, Vulnerability]

  @staticmethod
  def item_loader(self, line):
      items = [c.strip() for c in line.split('|')]
      value, item_type = tuple(items[0:2])
      if item_type == 'url':
          yield Url(
              url=value,
              status_code=items[3],
              content_type=items[4] if len(items) > 3 else '',
              title=items[5] if len(items) > 4 else ''
          )
      elif item_type == 'vulnerability':
          cve_id = value
          lookup_data = Vuln.lookup_cve(cve_id)  # perform vulnerability search
          vuln = {
            'matched_at': items[2]
          }
          if lookup_data:
            vuln.update(**lookup_data)
          yield Vulnerability(**vuln)
      else: # tag
          yield Tag(
              name=item_type,
              match=items[2],
              extra_data={
                  'value': value
              }
          )
 

Run it with secator:

$ secator x mytool mytarget.com

                         __            
   ________  _________ _/ /_____  _____
  / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/
 (__  /  __/ /__/ /_/ / /_/ /_/ / /    
/____/\___/\___/\__,_/\__/\____/_/     v0.6.0

                        freelabz.com

No Celery worker alive.
/home/osboxes/.local/bin/mytool -u mytarget.com -jsonl
[INF] This is an info message
[ERR] This is an error message
🔗 https://mytarget.com/api [200] [MyAwesomeWebPage] [application/json]
🔗 https://mytarget.com/api/metrics [403]
🏷️ aws_api_key found @ https://mytarget/api/.aws_key.json
    secret: A3TBABCD1234EFGH5678
🚨 [Object Injection 🡕] [critical] https://mytarget/api/sensitive_api_path
from secator.tasks import mytool
task = mytool('mytarget.com')
for item in task:
    print(item)  # this will output Url, Vulnerability, or Tag items.

PreviousParsing JSON linesNextParsing output files

Last updated 7 months ago

Was this helpful?

First we need to find a regular expression that will match the items marked with [FOUND] and get the individual values using a named regex (you can use for this).

Pythex