Parsing JSON lines

... or how to integrate tools that output JSON lines.

If your tool outputs JSON lines, it's very easy to integrate it with secator.

Based on how your tool's output maps will map to secator output types, read:

One-to-one mapping

Steps:

  • Use the JSONSerializer item loader.

  • Add an output_map to map your tool's output to one of secator's Output types.

Example:

For instance, mytool outputs JSON lines when we run it like:

mytool -jsonl -u mytarget.com
{"url": "https://mytarget.com/api", "status": 200, {"details": {"ct": "application/json"}}
{"url": "https://mytarget.com/api/metrics", "status": 403, "details": {}}

An integration of mytool with secator would look like:

secator/tasks/mytool.py
from secator.decorators import task
from secator.runners import Command
from secator.output_types import Url
from secator.serializers import JSONSerializer
from secator.definitions import URL, STATUS_CODE, CONTENT_TYPE


@task()
class mytool(Command):
  input_flag = '-u'
  json_flag = '-jsonl'
  output_types = [Url]
  item_loaders = [JSONSerializer()]
  output_map = {
    Url: {
     URL: 'url',
     STATUS_CODE: 'status',
     CONTENT_TYPE: lambda x: x['details'].get('ct', '')
    }
  }

Steps taken:

  • Import the desired runner (Command).

  • Import the desired output type(s) and the field names that we want to map.

  • Set input_flag to mytool's input flag -u.

  • Set json_flag to mytool's JSON line flag jsonl.

  • Set output_types to our desired output types [Url].

  • Set output_map to map mytool's output fields to each Url's fields (you need to map at least the required=True fields).

That's it ! You can now run mytool with secator and enjoy all the features it brings on top of it:

secator x mytool mytarget.com
                         __            
   ________  _________ _/ /_____  _____
  / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/
 (__  /  __/ /__/ /_/ / /_/ /_/ / /    
/____/\___/\___/\__,_/\__/\____/_/     v0.6.0

                        freelabz.com
mytool -u mytarget.com -jsonl
🔗 https://mytarget.com/api [200] [application/json]
🔗 https://mytarget.com/api/metrics [403]

One-to-many

Steps:

  • Add a static method on_json_loaded to hook onto the JSON Serializer output.

  • Modify the data and yield secator Output types.

Example:

For instance, if mytool outputs urls in batch:

mytool -jsonl -u mytarget.com
{"urls": [{"url": "https://mytarget.com/api", "status": 200, "content-type": "application/json"}, {"url": "https://mytarget.com/api/metrics", "status": 403, "content-type": "application/text"}]

An integration of mytool with secator would look like:

secator/tasks/mytool.py
from secator.decorators import task
from secator.runners import Command
from secator.output_types import Url
from secator.definitions import URL, STATUS_CODE, CONTENT_TYPE


@task()
class mytool(Command):
  input_flag = '-u'
  json_flag = '-jsonl'
  output_types = [Url]

  @staticmethod
  def on_json_loaded(self, data):
      for item in data['urls']:
          yield Url(
            URL: item['url'],
            STATUS_CODE: item['status'],
            CONTENT_TYPE: item.get('content-type', '')
          )

See Lifecyle hooks for more details on which hooks you can use.

That's it ! You can now run mytool with secator and enjoy all the features it brings on top of it:

secator x mytool mytarget.com
                         __            
   ________  _________ _/ /_____  _____
  / ___/ _ \/ ___/ __ `/ __/ __ \/ ___/
 (__  /  __/ /__/ /_/ / /_/ /_/ / /    
/____/\___/\___/\__,_/\__/\____/_/     v0.6.0

                        freelabz.com
mytool -u mytarget.com -jsonl
🔗 https://mytarget.com/api [200] [application/json]
🔗 https://mytarget.com/api/metrics [403]
PreviousIntegrating an external commandNextParsing raw standard output

Last updated

Was this helpful?