search

Parsing JSON lines

... or how to integrate tools that output JSON lines.

If your tool outputs JSON lines, it's very easy to integrate it with secator.

Based on how your tool's output maps will map to secator output types, read:

hashtag
One-to-one mapping

Steps:

  • Use the JSONSerializer item loader.

  • Add an output_map to map your tool's output to one of secator's Output types.

hashtag
Example:

For instance, mytool outputs JSON lines when we run it like:

mytool -jsonl -u mytarget.com
{"url": "https://mytarget.com/api", "status": 200, {"details": {"ct": "application/json"}}
{"url": "https://mytarget.com/api/metrics", "status": 403, "details": {}}

An integration of mytool with secator would look like:

secator/tasks/mytool.py
from secator.decorators import task
from secator.runners import Command
from secator.output_types import Url
from secator.serializers import JSONSerializer
from secator.definitions import URL, STATUS_CODE, CONTENT_TYPE


@task()
class mytool(Command):
  input_flag = '-u'
  json_flag = '-jsonl'
  output_types = [Url]
  item_loaders = [JSONSerializer()]
  output_map = {
    Url: {
     URL: 'url',
     STATUS_CODE: 'status',
     CONTENT_TYPE: lambda x: x['details'].get('ct', '')
    }
  }

Steps taken:

  • Import the desired runner (Command).

  • Import the desired output type(s) and the field names that we want to map.

  • Set input_flag to mytool's input flag -u.

  • Set json_flag to mytool's JSON line flag jsonl.

  • Set output_types to our desired output types [Url].

  • Set output_map to map mytool's output fields to each Url's fields (you need to map at least the required=True fields).

That's it ! You can now run mytool with secator and enjoy all the features it brings on top of it:

hashtag
One-to-many

Steps:

  • Add a static method on_json_loaded to hook onto the JSON Serializer output.

  • Modify the data and yield secator Output types.

hashtag
Example:

For instance, if mytool outputs urls in batch:

An integration of mytool with secator would look like:

circle-info

See Lifecyle hooks for more details on which hooks you can use.

That's it ! You can now run mytool with secator and enjoy all the features it brings on top of it:

PreviousIntegrating an external commandNextParsing raw standard output

Last updated