Currently StreamBase offers following primitive field types:
The primitive types can be used in higher order structures:
There are also 2 auxiliary structures:
The scope of this idea is key/value higher order structure. This would be similar to list, but the index would be string not int.
With increased role of semi-design time model creation using loosely typed languages like Python or R (TERR) and flexible structure data, like datasets obtained with runtime defined SQL, the StreamBase tuple model is to rigid. There is a need to express bags with data addressed using labels that are unknown during design time. It is possible to achieve similar logic even now by defining list of tuples containing name and actual associated value.
There are major problems related with this approach:
The idea is to use similar structure to Avro map:
https://avro.apache.org/docs/1.8.1/spec.html#Maps
This structure looks like list of key value pairs, but offers much better runtime performance. On the wire it is indeed list of entries, but in memory it is typically a hash table (in the specific language binding flavor, i.e. in Java HashMap, in Python dict).
The dictionary would have design time declared type of any valid SB field type.
Type name: dict
Type declaration in functions: dict(<value type>), example dict(string), dict((a string, b int))
Access to the data: use [] operator with string argument, for example features["age"]
In-place creation: dict(key1: value1, key2: value2[...]), example dict("age": 23, "count": 2)
Null dict declaration: dict(value), note no key, like dict(string()) or dict(0), the latter producing null of type dictionary of integer values
Basic functions: