Class JSONDictField

st2common/st2common/fields.py:332–510

Custom field types which stores dictionary as JSON serialized strings. This is done because storing large objects as JSON serialized strings is much more fficient on the serialize and unserialize paths compared to used EscapedDictField which needs to escape all the special values (

class JSONDictField(BinaryField):
    """
    Custom field types which stores dictionary as JSON serialized strings.

    This is done because storing large objects as JSON serialized strings is much more fficient
    on the serialize and unserialize paths compared to used EscapedDictField which needs to escape
    all the special values ($, .).

    Only downside is that to MongoDB those values are plain raw strings which means you can't query
    on actual dictionary field values. That's not an issue for us, because in places where we use
    it, those values are already treated as plain binary blobs to the database layer and we never
    directly query on those field values.

    In micro benchmarks we have seen speed ups for up to 10x on write path and up to 6x on read
    path. Change also scaled down which means it didn't add any additional overhead for very small
    results - in fact, it was also faster for small results dictionaries

    More context and numbers are available at https://github.com/StackStorm/st2/pull/4846.

    NOTES, LIMITATIONS:

    This field type can only be used on dictionary values on which we don't perform direct database
    queries (aka filter on a specific dictionary item value or similar).

    Good examples of those are "result" field on ExecutionDB, LiveActionDB and TaskExecutionDB,
    "output" on WorkflowExecutionDB, etc.

    IMPLEMENTATION DETAILS:

    If header is used, values are stored in the following format:
        <compression type>:<serialization type>:<serialized binary data>.

    For example:

        n:o:... - No compression, (or)json serialization
        z:o:... - Zstandard compression, (or)json serialization

    If header is not used, value is stored as a serialized JSON string of the input dictionary.
    """

    def __init__(self, *args, **kwargs):
        # True if we should use field header which is more future proof approach and also allows
        # us to support optional per-field compression, etc.
        # This option is only exposed so we can benchmark different approaches and how much overhead
        # using a header adds.
        self.use_header = kwargs.pop("use_header", False)
        self.compression_algorithm = kwargs.pop("compression_algorithm", "none")

        super(JSONDictField, self).__init__(*args, **kwargs)

    def to_mongo(self, value):
        if not isinstance(value, dict):
            raise ValueError(
                "value argument must be a dictionary (got: %s)" % type(value)
            )

        data = self._serialize_field_value(value)
        return data