Arrays#
New in version 1.8.0: from Jonny Saunders, Ryan Ly, and Chris Mungall #1887
, linkml-model#181
We can divide data types into a few abstract forms (see “Recognizing Structural Forms”). Linked data tools regularly handle scalars, lists, tables, graphs, and trees, but less commonly handle multidimensional arrays. The LinkML metamodel has first-class support to specify them, and its generators are growing first-class support to make those specifications work with array formats and libraries that people actually use.
Types#
There are two types of array specification in LinkML:
NDArrays - “regular” dense multidimensional arrays with shape and dtype constraints
Coming Soon Labeled Arrays - arrays that use additional arrays as their indices (eg. a set of temperature measurements indexed by latitude and longitude).
NDArrays#
Show code cell content
from linkml.generators import PydanticGenerator
from linkml_runtime.loaders.yaml_loader import YAMLLoader
from linkml_runtime.dumpers.yaml_dumper import YAMLDumper
from IPython.display import display, Markdown
from pathlib import Path
import numpy as np
from rich.console import Console
from rich.theme import Theme
from rich.style import Style
from rich.color import Color
theme = Theme({
"repr.call": Style(color=Color.from_rgb(110,191,38), bold=True),
"repr.attrib_name": Style(color="slate_blue1"),
"repr.number": Style(color="deep_sky_blue1"),
})
console = Console(theme=theme)
schemas = Path('.').resolve().parent / '_includes' / 'arrays'
COMPARISON = """
:::::{{tab-set}}
::::{{tab-item}} LinkML
:::{{code-block}} yaml
{linkml}
:::
::::
::::{{tab-item}} pydantic - LoL
:::{{code-block}} python
{pydantic_lol}
:::
::::
::::{{tab-item}} numpydantic
:::{{code-block}} python
{pydantic_npd}
:::
::::
:::::
"""
def render_module(path, representation='list'):
generator = PydanticGenerator(str(path), array_representations=[representation])
module = generator.render()
return module
def compile_module(path, representation='list'):
generator = PydanticGenerator(str(path), array_representations=[representation])
module = generator.compile_module()
return module
def render_class(path, cls, representation='list') -> str:
module = render_module(path, representation)
cls = module.classes[cls]
code = cls.render(black=True)
return code
def render_comparison(path, cls, string=False) -> str:
if not isinstance(cls, list):
cls = [cls]
path = str(path)
sch = YAMLLoader().load_as_dict(path)
class_strs = []
pydantic_strs = []
npd_strs = []
for a_cls in cls:
class_def = sch['classes'][a_cls]
class_def = {a_cls: class_def}
class_strs.append(YAMLDumper().dumps(class_def))
pydantic_strs.append(render_class(path, a_cls))
npd_strs.append(render_class(path, a_cls, representation='numpydantic'))
class_str = "\n".join(class_strs)
pydantic_str = "\n".join(pydantic_strs)
npd_str = "\n".join(npd_strs)
md = COMPARISON.format(linkml=class_str, pydantic_lol=pydantic_str, pydantic_npd=npd_str)
if string:
return md
else:
display(Markdown(md))
NDArrays are a slot-level feature - they augment the usual slot range
syntax with an array
property.
For example, A class with a data
slot which consists of array of integers between 2 and 5 dimensions and its
PydanticGenerator
array
representations look like:
Show code cell source
sch = schemas / 'example.yaml'
render_comparison(sch, 'MyClass')
example_mod = compile_module(sch)
MyClass = getattr(example_mod, 'MyClass')
MyClass:
attributes:
data:
range: integer
array:
minimum_number_dimensions: 3
maximum_number_dimensions: 5
class MyClass(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "example"})
data: Optional[Union[List[List[List[int]]], List[List[List[List[int]]]], List[List[List[List[List[int]]]]]]] = (
Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "data",
"array": {"maximum_number_dimensions": 5, "minimum_number_dimensions": 3},
"domain_of": ["MyClass"],
}
},
)
)
class MyClass(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "example"})
data: Optional[
Union[NDArray[Shape["*, *, *"], int], NDArray[Shape["*, *, *, *"], int], NDArray[Shape["*, *, *, *, *"], int]]
] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "data",
"array": {"maximum_number_dimensions": 5, "minimum_number_dimensions": 3},
"domain_of": ["MyClass"],
}
},
)
The generated pydantic model can validate all the constraints of the array, serialize it to JSON, and everything else that you’d expect from a pydantic model.
This model correctly validates:
model = MyClass(data=np.ones((5,4,3), dtype=int))
console.print(model)
MyClass( data=[ [[1, 1, 1], [1, 1, 1], [1, 1, 1], [1, 1, 1]], [[1, 1, 1], [1, 1, 1], [1, 1, 1], [1, 1, 1]], [[1, 1, 1], [1, 1, 1], [1, 1, 1], [1, 1, 1]], [[1, 1, 1], [1, 1, 1], [1, 1, 1], [1, 1, 1]], [[1, 1, 1], [1, 1, 1], [1, 1, 1], [1, 1, 1]] ] )
But an array with the wrong shape doesn’t:
try:
MyClass(data=np.ones((1,)))
except Exception as e:
console.print(e)
Show code cell output
3 validation errors for MyClass data.list[list[list[int]]].0 Input should be a valid list [type=list_type, input_value=1.0, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].0 Input should be a valid list [type=list_type, input_value=1.0, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].0 Input should be a valid list [type=list_type, input_value=1.0, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type
Nor does an array with the wrong type
try:
MyClass(data=np.random.rand(5,4,3))
except Exception as e:
console.print(e)
Show code cell output
180 validation errors for MyClass data.list[list[list[int]]].0.0.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.10744261404877498, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].0.0.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.9333621764443865, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].0.0.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.414650557157473, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].0.1.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.6597958884230131, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].0.1.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.2333983906606204, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].0.1.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.2175119616223935, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].0.2.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.5643278810493227, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].0.2.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.15406614487348447, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].0.2.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.5236952550403481, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].0.3.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.5329689534799679, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].0.3.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.5017448200429743, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].0.3.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.5637835816096316, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].1.0.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.6777963500856549, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].1.0.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.994343500724518, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].1.0.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.5592114271194469, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].1.1.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.4567991483894588, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].1.1.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.8626759595590467, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].1.1.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.5397405193530868, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].1.2.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.4014201542429888, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].1.2.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.889214234096264, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].1.2.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.6341168652876745, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].1.3.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.08934656178696809, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].1.3.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.23155534742689543, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].1.3.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.6976514217480655, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].2.0.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.8234641713650437, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].2.0.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.561526368970976, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].2.0.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.656773573041047, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].2.1.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.571943849776279, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].2.1.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.05307516069204654, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].2.1.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.4351207036453304, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].2.2.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.7183296913615145, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].2.2.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.9493964224238899, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].2.2.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.7825848762016842, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].2.3.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.045630110297058324, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].2.3.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.2725064125541202, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].2.3.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.013359079379341554, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].3.0.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.18020054484117265, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].3.0.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.8542145445377802, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].3.0.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.09985555269860635, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].3.1.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.032184119302260394, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].3.1.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.7997445597713113, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].3.1.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.012981376849515724, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].3.2.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.9110107698593675, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].3.2.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.547745417422265, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].3.2.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.9122941044984904, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].3.3.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.700388616856826, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].3.3.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.9880721487656106, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].3.3.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.7910053641158942, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].4.0.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.38429176526941644, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].4.0.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.6554818666539063, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].4.0.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.8890491562954038, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].4.1.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.11096982014804224, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].4.1.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.5698875028417416, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].4.1.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.8935708089316191, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].4.2.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.9636922843628621, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].4.2.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.6936973774253865, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].4.2.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.06806945011912247, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].4.3.0 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.6564173959581124, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].4.3.1 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.39494275919578503, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[int]]].4.3.2 Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=0.9387129385211177, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/int_from_float data.list[list[list[list[int]]]].0.0.0 Input should be a valid list [type=list_type, input_value=0.10744261404877498, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].0.0.1 Input should be a valid list [type=list_type, input_value=0.9333621764443865, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].0.0.2 Input should be a valid list [type=list_type, input_value=0.414650557157473, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].0.1.0 Input should be a valid list [type=list_type, input_value=0.6597958884230131, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].0.1.1 Input should be a valid list [type=list_type, input_value=0.2333983906606204, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].0.1.2 Input should be a valid list [type=list_type, input_value=0.2175119616223935, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].0.2.0 Input should be a valid list [type=list_type, input_value=0.5643278810493227, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].0.2.1 Input should be a valid list [type=list_type, input_value=0.15406614487348447, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].0.2.2 Input should be a valid list [type=list_type, input_value=0.5236952550403481, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].0.3.0 Input should be a valid list [type=list_type, input_value=0.5329689534799679, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].0.3.1 Input should be a valid list [type=list_type, input_value=0.5017448200429743, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].0.3.2 Input should be a valid list [type=list_type, input_value=0.5637835816096316, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].1.0.0 Input should be a valid list [type=list_type, input_value=0.6777963500856549, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].1.0.1 Input should be a valid list [type=list_type, input_value=0.994343500724518, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].1.0.2 Input should be a valid list [type=list_type, input_value=0.5592114271194469, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].1.1.0 Input should be a valid list [type=list_type, input_value=0.4567991483894588, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].1.1.1 Input should be a valid list [type=list_type, input_value=0.8626759595590467, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].1.1.2 Input should be a valid list [type=list_type, input_value=0.5397405193530868, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].1.2.0 Input should be a valid list [type=list_type, input_value=0.4014201542429888, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].1.2.1 Input should be a valid list [type=list_type, input_value=0.889214234096264, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].1.2.2 Input should be a valid list [type=list_type, input_value=0.6341168652876745, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].1.3.0 Input should be a valid list [type=list_type, input_value=0.08934656178696809, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].1.3.1 Input should be a valid list [type=list_type, input_value=0.23155534742689543, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].1.3.2 Input should be a valid list [type=list_type, input_value=0.6976514217480655, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].2.0.0 Input should be a valid list [type=list_type, input_value=0.8234641713650437, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].2.0.1 Input should be a valid list [type=list_type, input_value=0.561526368970976, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].2.0.2 Input should be a valid list [type=list_type, input_value=0.656773573041047, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].2.1.0 Input should be a valid list [type=list_type, input_value=0.571943849776279, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].2.1.1 Input should be a valid list [type=list_type, input_value=0.05307516069204654, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].2.1.2 Input should be a valid list [type=list_type, input_value=0.4351207036453304, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].2.2.0 Input should be a valid list [type=list_type, input_value=0.7183296913615145, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].2.2.1 Input should be a valid list [type=list_type, input_value=0.9493964224238899, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].2.2.2 Input should be a valid list [type=list_type, input_value=0.7825848762016842, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].2.3.0 Input should be a valid list [type=list_type, input_value=0.045630110297058324, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].2.3.1 Input should be a valid list [type=list_type, input_value=0.2725064125541202, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].2.3.2 Input should be a valid list [type=list_type, input_value=0.013359079379341554, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].3.0.0 Input should be a valid list [type=list_type, input_value=0.18020054484117265, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].3.0.1 Input should be a valid list [type=list_type, input_value=0.8542145445377802, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].3.0.2 Input should be a valid list [type=list_type, input_value=0.09985555269860635, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].3.1.0 Input should be a valid list [type=list_type, input_value=0.032184119302260394, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].3.1.1 Input should be a valid list [type=list_type, input_value=0.7997445597713113, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].3.1.2 Input should be a valid list [type=list_type, input_value=0.012981376849515724, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].3.2.0 Input should be a valid list [type=list_type, input_value=0.9110107698593675, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].3.2.1 Input should be a valid list [type=list_type, input_value=0.547745417422265, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].3.2.2 Input should be a valid list [type=list_type, input_value=0.9122941044984904, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].3.3.0 Input should be a valid list [type=list_type, input_value=0.700388616856826, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].3.3.1 Input should be a valid list [type=list_type, input_value=0.9880721487656106, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].3.3.2 Input should be a valid list [type=list_type, input_value=0.7910053641158942, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].4.0.0 Input should be a valid list [type=list_type, input_value=0.38429176526941644, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].4.0.1 Input should be a valid list [type=list_type, input_value=0.6554818666539063, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].4.0.2 Input should be a valid list [type=list_type, input_value=0.8890491562954038, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].4.1.0 Input should be a valid list [type=list_type, input_value=0.11096982014804224, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].4.1.1 Input should be a valid list [type=list_type, input_value=0.5698875028417416, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].4.1.2 Input should be a valid list [type=list_type, input_value=0.8935708089316191, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].4.2.0 Input should be a valid list [type=list_type, input_value=0.9636922843628621, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].4.2.1 Input should be a valid list [type=list_type, input_value=0.6936973774253865, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].4.2.2 Input should be a valid list [type=list_type, input_value=0.06806945011912247, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].4.3.0 Input should be a valid list [type=list_type, input_value=0.6564173959581124, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].4.3.1 Input should be a valid list [type=list_type, input_value=0.39494275919578503, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[int]]]].4.3.2 Input should be a valid list [type=list_type, input_value=0.9387129385211177, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].0.0.0 Input should be a valid list [type=list_type, input_value=0.10744261404877498, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].0.0.1 Input should be a valid list [type=list_type, input_value=0.9333621764443865, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].0.0.2 Input should be a valid list [type=list_type, input_value=0.414650557157473, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].0.1.0 Input should be a valid list [type=list_type, input_value=0.6597958884230131, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].0.1.1 Input should be a valid list [type=list_type, input_value=0.2333983906606204, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].0.1.2 Input should be a valid list [type=list_type, input_value=0.2175119616223935, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].0.2.0 Input should be a valid list [type=list_type, input_value=0.5643278810493227, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].0.2.1 Input should be a valid list [type=list_type, input_value=0.15406614487348447, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].0.2.2 Input should be a valid list [type=list_type, input_value=0.5236952550403481, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].0.3.0 Input should be a valid list [type=list_type, input_value=0.5329689534799679, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].0.3.1 Input should be a valid list [type=list_type, input_value=0.5017448200429743, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].0.3.2 Input should be a valid list [type=list_type, input_value=0.5637835816096316, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].1.0.0 Input should be a valid list [type=list_type, input_value=0.6777963500856549, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].1.0.1 Input should be a valid list [type=list_type, input_value=0.994343500724518, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].1.0.2 Input should be a valid list [type=list_type, input_value=0.5592114271194469, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].1.1.0 Input should be a valid list [type=list_type, input_value=0.4567991483894588, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].1.1.1 Input should be a valid list [type=list_type, input_value=0.8626759595590467, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].1.1.2 Input should be a valid list [type=list_type, input_value=0.5397405193530868, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].1.2.0 Input should be a valid list [type=list_type, input_value=0.4014201542429888, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].1.2.1 Input should be a valid list [type=list_type, input_value=0.889214234096264, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].1.2.2 Input should be a valid list [type=list_type, input_value=0.6341168652876745, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].1.3.0 Input should be a valid list [type=list_type, input_value=0.08934656178696809, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].1.3.1 Input should be a valid list [type=list_type, input_value=0.23155534742689543, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].1.3.2 Input should be a valid list [type=list_type, input_value=0.6976514217480655, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].2.0.0 Input should be a valid list [type=list_type, input_value=0.8234641713650437, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].2.0.1 Input should be a valid list [type=list_type, input_value=0.561526368970976, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].2.0.2 Input should be a valid list [type=list_type, input_value=0.656773573041047, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].2.1.0 Input should be a valid list [type=list_type, input_value=0.571943849776279, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].2.1.1 Input should be a valid list [type=list_type, input_value=0.05307516069204654, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].2.1.2 Input should be a valid list [type=list_type, input_value=0.4351207036453304, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].2.2.0 Input should be a valid list [type=list_type, input_value=0.7183296913615145, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].2.2.1 Input should be a valid list [type=list_type, input_value=0.9493964224238899, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].2.2.2 Input should be a valid list [type=list_type, input_value=0.7825848762016842, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].2.3.0 Input should be a valid list [type=list_type, input_value=0.045630110297058324, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].2.3.1 Input should be a valid list [type=list_type, input_value=0.2725064125541202, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].2.3.2 Input should be a valid list [type=list_type, input_value=0.013359079379341554, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].3.0.0 Input should be a valid list [type=list_type, input_value=0.18020054484117265, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].3.0.1 Input should be a valid list [type=list_type, input_value=0.8542145445377802, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].3.0.2 Input should be a valid list [type=list_type, input_value=0.09985555269860635, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].3.1.0 Input should be a valid list [type=list_type, input_value=0.032184119302260394, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].3.1.1 Input should be a valid list [type=list_type, input_value=0.7997445597713113, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].3.1.2 Input should be a valid list [type=list_type, input_value=0.012981376849515724, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].3.2.0 Input should be a valid list [type=list_type, input_value=0.9110107698593675, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].3.2.1 Input should be a valid list [type=list_type, input_value=0.547745417422265, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].3.2.2 Input should be a valid list [type=list_type, input_value=0.9122941044984904, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].3.3.0 Input should be a valid list [type=list_type, input_value=0.700388616856826, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].3.3.1 Input should be a valid list [type=list_type, input_value=0.9880721487656106, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].3.3.2 Input should be a valid list [type=list_type, input_value=0.7910053641158942, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].4.0.0 Input should be a valid list [type=list_type, input_value=0.38429176526941644, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].4.0.1 Input should be a valid list [type=list_type, input_value=0.6554818666539063, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].4.0.2 Input should be a valid list [type=list_type, input_value=0.8890491562954038, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].4.1.0 Input should be a valid list [type=list_type, input_value=0.11096982014804224, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].4.1.1 Input should be a valid list [type=list_type, input_value=0.5698875028417416, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].4.1.2 Input should be a valid list [type=list_type, input_value=0.8935708089316191, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].4.2.0 Input should be a valid list [type=list_type, input_value=0.9636922843628621, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].4.2.1 Input should be a valid list [type=list_type, input_value=0.6936973774253865, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].4.2.2 Input should be a valid list [type=list_type, input_value=0.06806945011912247, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].4.3.0 Input should be a valid list [type=list_type, input_value=0.6564173959581124, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].4.3.1 Input should be a valid list [type=list_type, input_value=0.39494275919578503, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type data.list[list[list[list[list[int]]]]].4.3.2 Input should be a valid list [type=list_type, input_value=0.9387129385211177, input_type=float64] For further information visit https://errors.pydantic.dev/2.9/v/list_type
Array Library Integration#
The most basic kind of array annotations that the pydantic generator can produce are “list of list” style arrays, but most practical uses of arrays require specialized array libraries.
The PydanticGenerator
also supports generating array annotations with numpydantic
, which allows
you to use a single model with an extensible set of array libraries.
To generate numpydantic-style array annotations:
PydanticGenerator('my_schema.yaml', array_representations=['numpydantic'])
Which yields a model like this (and see the examples throughout the rest of this page)
from pydantic import BaseModel
from numpydantic import NDArray, Shape
class MyModel(BaseModel):
array: NDArray[Shape["3 x, 4 y, * z"], int]
Then use it as you please:
import numpy as np
import dask.array as da
import zarr
# numpy
model = MyModel(array=np.zeros((3, 4, 5), dtype=int))
# dask
model = MyModel(array=da.zeros((3, 4, 5), dtype=int))
# hdf5 datasets
model = MyModel(array=('data.h5', '/nested/dataset'))
# zarr arrays
model = MyModel(array=zarr.zeros((3,4,5), dtype=int))
model = MyModel(array='data.zarr')
model = MyModel(array=('data.zarr', '/nested/dataset'))
# video files
model = MyModel(array="data.mp4")
This makes it possible for your schema to be extremely implementation general - data formats and standards can support many array backends out of the box, reducing coupling between the abstract standard and its concrete implementation. More humble array users get powerful array modeling tools right from a yaml schema.
Tip
To use numpydantic arrays, you’ll need to add it as a dependency in your project.
Schema rendered with numpydantic arrays will include a module-level constant NUMPYDANTIC_VERSION
which specifies the
minimum version that it was designed for, but if you already depend on linkml
in your project you can add the
numpydantic
extra (eg. pip install linkml[numpydantic]
) which will give you a numpydantic
dependency that matches
the version supported by linkml
.
Specification#
NDArrays are defined by an ArrayExpression
(metamodel docs)
An ArrayExpression
has many of the default properties other LinkML expressions have – you can annotate your arrays like the rest of your data.
To define the array, an ArrayExpression
uses the following unique properties:
Shape
Shape can be specified numerically:
- maximum_number_dimensions: int | False | None#
Maximum (inclusive) number of dimensions.
When used with
dimensions
, to differentiate with being unset orNone
, needs to be set toFalse
explicitly to indicate an “infinite”[1] number of dimensions (see Complex Shaped Arrays)
- exact_number_dimensions: int | None#
An exact number of dimensions.
Equivalent to
minimum_number_dimensions
andmaximum_number_dimensions
being equalint
s
- dimensions: list[DimensionExpression]#
Parameterization of individual dimensions (see below)
Dimensions
Dimensions can be further parameterized by defining dimensions
with a list of DimensionExpression
s, which accept:
- maximum_cardinality: int | None#
The maximum size of this dimension (inclusive). If
None
, no maximum is set
Shape Forms#
The combinations of the different ArrayExpression
properties imply four NDArray forms:
Any Shape - Arrays without limits to their shape
Bounded Shape - Arrays with constraints on the number of dimensions without further parameterization
Parameterized Shape - Arrays with parameterized dimensions
Complex Shape - Arrays with both constraints on the number of dimensions and parameterized dimensions
Any#
An Any shaped array can take any shape. This is the simplest array form, indicating the mere presence of an array.
An any shaped array is specified with an empty array
dictionary:
AnyType:
attributes:
array:
range: AnyType
array: {}
Typed:
attributes:
array:
range: integer
array: {}
class AnyType(ConfiguredBaseModel):
array: Optional[AnyShapeArray] = Field(None)
class Typed(ConfiguredBaseModel):
array: Optional[AnyShapeArray[int]] = Field(None)
class AnyType(ConfiguredBaseModel):
array: Optional[NDArray] = Field(None)
class Typed(ConfiguredBaseModel):
array: Optional[NDArray[Any, int]] = Field(None)
The resulting pydantic models use a special AnyShapeArray
class injected by pydanticgen’s
template
system when using the List of List (LoL) representation
(see Representations).
Note
In the future, Any shape arrays might also be able to be specified with an explicit None
value, e.g.:
MyClass:
attributes:
data:
range: integer
array:
This is a technical limitation in SchemaView
- see linkml-model#189
and #1975
Bounded#
We have already seen so-called “Bounded” shaped arrays, which don’t add any additional parameterization beyond the number of their dimensions.
The maximum_
, minimum_
, and exact_number_dimensions
properties can be used in combination to indicate…
A minimum without a maximum, using the AnyShapeArray
model internally -
Show code cell source
sch = schemas / 'bounded_shape.yaml'
render_comparison(sch, 'MinDimensions')
MinDimensions:
attributes:
array:
range: integer
array:
minimum_number_dimensions: 2
class MinDimensions(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "bounded_shape_array"})
array: Optional[List[AnyShapeArray[int]]] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"minimum_number_dimensions": 2},
"domain_of": ["ExactDimensions", "MinDimensions", "MaxDimensions", "RangeDimensions"],
}
},
)
class MinDimensions(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "bounded_shape_array"})
array: Optional[NDArray[Shape["*, *, ..."], int]] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"minimum_number_dimensions": 2},
"domain_of": ["ExactDimensions", "MinDimensions", "MaxDimensions", "RangeDimensions"],
}
},
)
A maximum without a minimum -
Show code cell source
sch = schemas / 'bounded_shape.yaml'
render_comparison(sch, 'MaxDimensions')
MaxDimensions:
attributes:
array:
range: integer
array:
maximum_number_dimensions: 5
class MaxDimensions(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "bounded_shape_array"})
array: Optional[
Union[
List[int],
List[List[int]],
List[List[List[int]]],
List[List[List[List[int]]]],
List[List[List[List[List[int]]]]],
]
] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"maximum_number_dimensions": 5},
"domain_of": ["ExactDimensions", "MinDimensions", "MaxDimensions", "RangeDimensions"],
}
},
)
class MaxDimensions(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "bounded_shape_array"})
array: Optional[
Union[
NDArray[Shape["*"], int],
NDArray[Shape["*, *"], int],
NDArray[Shape["*, *, *"], int],
NDArray[Shape["*, *, *, *"], int],
NDArray[Shape["*, *, *, *, *"], int],
]
] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"maximum_number_dimensions": 5},
"domain_of": ["ExactDimensions", "MinDimensions", "MaxDimensions", "RangeDimensions"],
}
},
)
An exact number of dimensions -
Show code cell source
sch = schemas / 'bounded_shape.yaml'
render_comparison(sch, 'ExactDimensions')
ExactDimensions:
attributes:
array:
range: integer
array:
exact_number_dimensions: 3
class ExactDimensions(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "bounded_shape_array"})
array: Optional[List[List[List[int]]]] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"exact_number_dimensions": 3},
"domain_of": ["ExactDimensions", "MinDimensions", "MaxDimensions", "RangeDimensions"],
}
},
)
class ExactDimensions(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "bounded_shape_array"})
array: Optional[NDArray[Shape["*, *, *"], int]] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"exact_number_dimensions": 3},
"domain_of": ["ExactDimensions", "MinDimensions", "MaxDimensions", "RangeDimensions"],
}
},
)
And a range of dimensions -
Show code cell source
sch = schemas / 'bounded_shape.yaml'
render_comparison(sch, 'RangeDimensions')
RangeDimensions:
attributes:
array:
range: integer
array:
minimum_number_dimensions: 2
maximum_number_dimensions: 5
class RangeDimensions(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "bounded_shape_array"})
array: Optional[
Union[List[List[int]], List[List[List[int]]], List[List[List[List[int]]]], List[List[List[List[List[int]]]]]]
] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"maximum_number_dimensions": 5, "minimum_number_dimensions": 2},
"domain_of": ["ExactDimensions", "MinDimensions", "MaxDimensions", "RangeDimensions"],
}
},
)
class RangeDimensions(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "bounded_shape_array"})
array: Optional[
Union[
NDArray[Shape["*, *"], int],
NDArray[Shape["*, *, *"], int],
NDArray[Shape["*, *, *, *"], int],
NDArray[Shape["*, *, *, *, *"], int],
]
] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"maximum_number_dimensions": 5, "minimum_number_dimensions": 2},
"domain_of": ["ExactDimensions", "MinDimensions", "MaxDimensions", "RangeDimensions"],
}
},
)
Parameterized#
Dimensions can be further parameterized, giving them names and cardinality constraints.
Similarly to bounded arrays, the following demonstrate setting a single-dimensional array with cardinality constraints…
Minimum cardinality:
Show code cell source
sch = schemas / 'parameterized_shape.yaml'
render_comparison(sch, 'MinCard')
MinCard:
attributes:
array:
range: integer
array:
dimensions:
- alias: min_card
minimum_cardinality: 2
class MinCard(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})
array: Optional[conlist(min_length=2, item_type=int)] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"dimensions": [{"alias": "min_card", "minimum_cardinality": 2}]},
"domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
}
},
)
class MinCard(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})
array: Optional[NDArray[Shape["2-* min_card"], int]] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"dimensions": [{"alias": "min_card", "minimum_cardinality": 2}]},
"domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
}
},
)
Maximum cardinality:
Show code cell source
sch = schemas / 'parameterized_shape.yaml'
render_comparison(sch, 'MaxCard')
MaxCard:
attributes:
array:
range: integer
array:
dimensions:
- alias: max_card
maximum_cardinality: 5
class MaxCard(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})
array: Optional[conlist(max_length=5, item_type=int)] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"dimensions": [{"alias": "max_card", "maximum_cardinality": 5}]},
"domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
}
},
)
class MaxCard(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})
array: Optional[NDArray[Shape["*-5 max_card"], int]] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"dimensions": [{"alias": "max_card", "maximum_cardinality": 5}]},
"domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
}
},
)
Exact cardinality:
Show code cell source
sch = schemas / 'parameterized_shape.yaml'
render_comparison(sch, 'ExactCard')
ExactCard:
attributes:
array:
range: integer
array:
dimensions:
- alias: max_card
exact_cardinality: 3
class ExactCard(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})
array: Optional[conlist(min_length=3, max_length=3, item_type=int)] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"dimensions": [{"alias": "max_card", "exact_cardinality": 3}]},
"domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
}
},
)
class ExactCard(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})
array: Optional[NDArray[Shape["3 max_card"], int]] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"dimensions": [{"alias": "max_card", "exact_cardinality": 3}]},
"domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
}
},
)
Cardinality range:
Show code cell source
sch = schemas / 'parameterized_shape.yaml'
render_comparison(sch, 'RangeCard')
RangeCard:
attributes:
array:
range: integer
array:
dimensions:
- alias: min_card
minimum_cardinality: 2
maximum_cardinality: 5
class RangeCard(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})
array: Optional[conlist(min_length=2, max_length=5, item_type=int)] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"dimensions": [{"alias": "min_card", "maximum_cardinality": 5, "minimum_cardinality": 2}]},
"domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
}
},
)
class RangeCard(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})
array: Optional[NDArray[Shape["2-5 min_card"], int]] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {"dimensions": [{"alias": "min_card", "maximum_cardinality": 5, "minimum_cardinality": 2}]},
"domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
}
},
)
And they can be used together, for example one can specify
A four dimensional array such that
The first dimension has at least two items
The second dimension has at most five items
The third dimension has between two and five items
The fourth dimension has exactly six items
Show code cell source
sch = schemas / 'parameterized_shape.yaml'
render_comparison(sch, 'ParameterizedArray')
parameterized_mod = compile_module(sch)
ParameterizedArray = getattr(parameterized_mod, 'ParameterizedArray')
ParameterizedArray:
attributes:
array:
range: integer
array:
dimensions:
- alias: min_card
minimum_cardinality: 2
- alias: max_card
maximum_cardinality: 5
- alias: range_card
minimum_cardinality: 2
maximum_cardinality: 5
- alias: exact_card
exact_cardinality: 6
class ParameterizedArray(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})
array: Optional[
conlist(
min_length=2,
item_type=conlist(
max_length=5,
item_type=conlist(
min_length=2, max_length=5, item_type=conlist(min_length=6, max_length=6, item_type=int)
),
),
)
] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {
"dimensions": [
{"alias": "min_card", "minimum_cardinality": 2},
{"alias": "max_card", "maximum_cardinality": 5},
{"alias": "range_card", "maximum_cardinality": 5, "minimum_cardinality": 2},
{"alias": "exact_card", "exact_cardinality": 6},
]
},
"domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
}
},
)
class ParameterizedArray(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})
array: Optional[NDArray[Shape["2-* min_card, *-5 max_card, 2-5 range_card, 6 exact_card"], int]] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {
"dimensions": [
{"alias": "min_card", "minimum_cardinality": 2},
{"alias": "max_card", "maximum_cardinality": 5},
{"alias": "range_card", "maximum_cardinality": 5, "minimum_cardinality": 2},
{"alias": "exact_card", "exact_cardinality": 6},
]
},
"domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
}
},
)
Which validates each of the constraints separately:
array = np.arange(4*1*2*6,dtype=int).reshape((4,1,2,6))
# array = np.ones((4,1,2,6), dtype=int)
console.print(ParameterizedArray(array=array))
ParameterizedArray( array=[ [[[0, 1, 2, 3, 4, 5], [6, 7, 8, 9, 10, 11]]], [[[12, 13, 14, 15, 16, 17], [18, 19, 20, 21, 22, 23]]], [[[24, 25, 26, 27, 28, 29], [30, 31, 32, 33, 34, 35]]], [[[36, 37, 38, 39, 40, 41], [42, 43, 44, 45, 46, 47]]] ] )
Complex#
Complex NDArrays combine all three of the prior forms.
For example:
An array with between 5 and 7 dimensions such that…
The first dimension has at least two items
The second dimension has at most five items
The third dimension has between two and five items
The fourth dimension has exactly six items
Show code cell source
sch = schemas / 'complex_shape.yaml'
render_comparison(sch, 'ComplexRangeShapeArray')
ComplexRangeShapeArray:
attributes:
array:
range: integer
array:
minimum_number_dimensions: 5
maximum_number_dimensions: 7
dimensions:
- alias: max_card
maximum_cardinality: 5
- alias: min_card
minimum_cardinality: 2
- alias: range_card
minimum_cardinality: 2
maximum_cardinality: 5
- alias: exact_card
exact_cardinality: 6
class ComplexRangeShapeArray(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "complex_shape_array"})
array: Optional[
conlist(
max_length=5,
item_type=conlist(
min_length=2,
item_type=conlist(
min_length=2,
max_length=5,
item_type=conlist(
min_length=6, max_length=6, item_type=Union[List[int], List[List[int]], List[List[List[int]]]]
),
),
),
)
] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {
"dimensions": [
{"alias": "max_card", "maximum_cardinality": 5},
{"alias": "min_card", "minimum_cardinality": 2},
{"alias": "range_card", "maximum_cardinality": 5, "minimum_cardinality": 2},
{"alias": "exact_card", "exact_cardinality": 6},
],
"maximum_number_dimensions": 7,
"minimum_number_dimensions": 5,
},
"domain_of": [
"ComplexAnyShapeArray",
"ComplexMaxShapeArray",
"ComplexRangeShapeArray",
"ComplexExactShapeArray",
],
}
},
)
class ComplexRangeShapeArray(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "complex_shape_array"})
array: Optional[
Union[
NDArray[Shape["*-5 max_card, 2-* min_card, 2-5 range_card, 6 exact_card, *"], int],
NDArray[Shape["*-5 max_card, 2-* min_card, 2-5 range_card, 6 exact_card, *, *"], int],
NDArray[Shape["*-5 max_card, 2-* min_card, 2-5 range_card, 6 exact_card, *, *, *"], int],
]
] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {
"dimensions": [
{"alias": "max_card", "maximum_cardinality": 5},
{"alias": "min_card", "minimum_cardinality": 2},
{"alias": "range_card", "maximum_cardinality": 5, "minimum_cardinality": 2},
{"alias": "exact_card", "exact_cardinality": 6},
],
"maximum_number_dimensions": 7,
"minimum_number_dimensions": 5,
},
"domain_of": [
"ComplexAnyShapeArray",
"ComplexMaxShapeArray",
"ComplexRangeShapeArray",
"ComplexExactShapeArray",
],
}
},
)
The only place where the syntax of complex arrays differ is that minimum
and maximum_number_dimensions
are set at the length of the specified dimensions
list by default. In order to specify an array
with a fixed number of parameterized dimensions with an arbitrary number of additional dimensions,
set maximum_number_dimensions
to False
:
Show code cell source
sch = schemas / 'complex_shape.yaml'
render_comparison(sch, 'ComplexAnyShapeArray')
ComplexAnyShapeArray:
attributes:
array:
range: integer
array:
maximum_number_dimensions: false
dimensions:
- alias: max_card
maximum_cardinality: 5
class ComplexAnyShapeArray(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "complex_shape_array"})
array: Optional[conlist(max_length=5, item_type=Union[AnyShapeArray[int], int])] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {
"dimensions": [{"alias": "max_card", "maximum_cardinality": 5}],
"maximum_number_dimensions": False,
},
"domain_of": [
"ComplexAnyShapeArray",
"ComplexMaxShapeArray",
"ComplexRangeShapeArray",
"ComplexExactShapeArray",
],
}
},
)
class ComplexAnyShapeArray(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "complex_shape_array"})
array: Optional[NDArray[Shape["*-5 max_card, ..."], int]] = Field(
None,
json_schema_extra={
"linkml_meta": {
"alias": "array",
"array": {
"dimensions": [{"alias": "max_card", "maximum_cardinality": 5}],
"maximum_number_dimensions": False,
},
"domain_of": [
"ComplexAnyShapeArray",
"ComplexMaxShapeArray",
"ComplexRangeShapeArray",
"ComplexExactShapeArray",
],
}
},
)
Generators#
Support#
At release, only pydanticgen supports arrays, but arrays will be implemented gradually for the rest of the generators.
Representations#
Since arrays are unlike other data types in that they usually require some specialized libraries to handle them, and many formats don’t have a single canonical array type, generators may accommodate multiple array representations.
The basic representation supported by the pydantic generator is the “List of lists” style array representation. This can be used without any additional dependencies beyond pydantic.
Pydanticgen now also supports most common array libraries from a single annotation using numpydantic
-
a single array specification generates a single pydantic model, but the numpydantic numpydantic.NDArray
type abstracts the validation process for an extensible set of array libraries. Use whatever you want as an array,
why stop at numpy arrays - currently it also supports hdf5, zarr, video files, and also allows custom array
interfaces via subclassing. See the numpydantic docs for more.
See each generator’s documentation page for a summary of the array representations they support.
Implementation Guidance#
TODO
Implementation docs for arrays are forthcoming.
For now see ArrayRangeGenerator
and ListOfListsArray
as the reference implementation
See Also#
How-to: Multidimensional arrays - discussion about the problems with array specification in linked data
Tricky Choices - Some further discussions of decisions made in the specification
References#
ArrayExpression metamodel specification