Arrays#

New in version 1.8.0: from Jonny Saunders, Ryan Ly, and Chris Mungall #1887, linkml-model#181


We can divide data types into a few abstract forms (see “Recognizing Structural Forms”). Linked data tools regularly handle scalars, lists, tables, graphs, and trees, but less commonly handle multidimensional arrays. The LinkML metamodel has first-class support to specify them, and its generators are growing first-class support to make those specifications work with array formats and libraries that people actually use.

Types#

There are two types of array specification in LinkML:

  • NDArrays - “regular” dense multidimensional arrays with shape and dtype constraints

  • Coming Soon Labeled Arrays - arrays that use additional arrays as their indices (eg. a set of temperature measurements indexed by latitude and longitude).

NDArrays#

Hide code cell content
from linkml.generators import PydanticGenerator
from linkml_runtime.loaders.yaml_loader import YAMLLoader
from linkml_runtime.dumpers.yaml_dumper import YAMLDumper
from IPython.display import display, Markdown
from pathlib import Path
import numpy as np
from rich.console import Console
from rich.theme import Theme
from rich.style import Style
from rich.color import Color

theme = Theme({
    "repr.call": Style(color=Color.from_rgb(110,191,38), bold=True),
    "repr.attrib_name": Style(color="slate_blue1"),
    "repr.number": Style(color="deep_sky_blue1"),
    
})
console = Console(theme=theme)

schemas = Path('.').resolve().parent / '_includes' / 'arrays'

COMPARISON = """
:::::{{tab-set}}
::::{{tab-item}} LinkML
:::{{code-block}} yaml
{linkml}
:::
::::
::::{{tab-item}} pydantic - LoL
:::{{code-block}} python
{pydantic_lol}
:::
::::
::::{{tab-item}} numpydantic
:::{{admonition}} Coming Soon
Direct support for array libraries like numpy, hdf5, dask, zarr, and xarray with pydantic validation <3
::: 
::::
:::::
"""

def render_module(path):
    generator = PydanticGenerator(str(path), array_representations=['list'])
    module = generator.render()
    return module
    
def compile_module(path):
    generator = PydanticGenerator(str(path), array_representations=['list'])
    module = generator.compile_module()
    return module

def render_class(path, cls) -> str:
    module = render_module(path)
    cls = module.classes[cls]
    code = cls.render(black=True)
    return code

def render_comparison(path, cls, string=False) -> str:
    if not isinstance(cls, list):
        cls = [cls]
        
    path = str(path)
    sch = YAMLLoader().load_as_dict(path)
    class_strs = []
    pydantic_strs = []
    
    for a_cls in cls:
        class_def = sch['classes'][a_cls]
        class_def = {a_cls: class_def}
        class_strs.append(YAMLDumper().dumps(class_def))
        pydantic_strs.append(render_class(path, a_cls))
        
    class_str = "\n".join(class_strs)
    pydantic_str = "\n".join(pydantic_strs)
    md = COMPARISON.format(linkml=class_str, pydantic_lol=pydantic_str)
    if string:
        return md
    else:
        display(Markdown(md))
    

NDArrays are a slot-level feature - they augment the usual slot range syntax with an array property.

For example, A class with a data slot which consists of array of integers between 2 and 5 dimensions and its PydanticGenerator array representations look like:

Hide code cell source
sch = schemas / 'example.yaml'
render_comparison(sch, 'MyClass')
example_mod = compile_module(sch)
MyClass = getattr(example_mod, 'MyClass')
MyClass:
  attributes:
    data:
      range: integer
      array:
        minimum_number_dimensions: 3
        maximum_number_dimensions: 5
class MyClass(ConfiguredBaseModel):
    linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "example"})

    data: Optional[Union[List[List[List[int]]], List[List[List[List[int]]]], List[List[List[List[List[int]]]]]]] = (
        Field(
            None,
            json_schema_extra={
                "linkml_meta": {
                    "alias": "data",
                    "array": {"maximum_number_dimensions": 5, "minimum_number_dimensions": 3},
                    "domain_of": ["MyClass"],
                }
            },
        )
    )

Coming Soon

Direct support for array libraries like numpy, hdf5, dask, zarr, and xarray with pydantic validation <3

The generated pydantic model can validate all the constraints of the array, serialize it to JSON, and everything else that you’d expect from a pydantic model.

This model correctly validates:

model = MyClass(data=np.ones((5,4,3), dtype=int))
console.print(model)
MyClass(
    data=[
        [[1, 1, 1], [1, 1, 1], [1, 1, 1], [1, 1, 1]],
        [[1, 1, 1], [1, 1, 1], [1, 1, 1], [1, 1, 1]],
        [[1, 1, 1], [1, 1, 1], [1, 1, 1], [1, 1, 1]],
        [[1, 1, 1], [1, 1, 1], [1, 1, 1], [1, 1, 1]],
        [[1, 1, 1], [1, 1, 1], [1, 1, 1], [1, 1, 1]]
    ]
)

But an array with the wrong shape doesn’t:

try:
    MyClass(data=np.ones((1,)))
except Exception as e:
    console.print(e)
Hide code cell output
3 validation errors for MyClass
data.list[list[list[int]]].0
  Input should be a valid list [type=list_type, input_value=1.0, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].0
  Input should be a valid list [type=list_type, input_value=1.0, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].0
  Input should be a valid list [type=list_type, input_value=1.0, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type

Nor does an array with the wrong type

try:
    MyClass(data=np.random.rand(5,4,3))
except Exception as e:
    console.print(e)
Hide code cell output
180 validation errors for MyClass
data.list[list[list[int]]].0.0.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.42234513583392064, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].0.0.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.9981640966089224, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].0.0.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.09632932364502289, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].0.1.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.42081777591692904, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].0.1.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.5887311084240875, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].0.1.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.3385201082164039, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].0.2.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.6435850727612293, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].0.2.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.16461686522171493, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].0.2.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.3927389944829386, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].0.3.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.32706336401985225, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].0.3.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.565638108937007, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].0.3.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.6387940621663579, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].1.0.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.4087321129803253, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].1.0.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.5261610218219884, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].1.0.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.7762568903748635, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].1.1.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.38478470547415966, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].1.1.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.2322302584804432, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].1.1.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.059532643058274215, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].1.2.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.8735040489010808, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].1.2.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.09807075061654813, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].1.2.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.07074527512286577, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].1.3.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.7661475943816916, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].1.3.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.36025616097354807, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].1.3.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.8032431315384145, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].2.0.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.20773498794392165, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].2.0.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.8912371261036824, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].2.0.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.5524036456566535, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].2.1.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.3396462290701292, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].2.1.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.9365729412321798, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].2.1.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.19075518266606717, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].2.2.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.6002502925113803, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].2.2.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.07906354053760545, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].2.2.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.918351053759276, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].2.3.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.12932846306037848, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].2.3.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.9570938466466345, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].2.3.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.9313595127526528, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].3.0.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.8618129658923739, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].3.0.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.7214154633883629, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].3.0.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.5176409748603018, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].3.1.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.9060300848865271, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].3.1.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.29296842574562965, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].3.1.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.8991997907883464, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].3.2.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.021171414546333978, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].3.2.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.9436070537149975, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].3.2.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.3213290303624765, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].3.3.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.33460194706240765, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].3.3.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.8885681590782408, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].3.3.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.800206221466748, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].4.0.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.5102787372752916, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].4.0.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.49350987307401684, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].4.0.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.9747759628198037, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].4.1.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.019008716486376476, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].4.1.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.18241175067171844, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].4.1.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.31927609128421863, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].4.2.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.406477661326343, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].4.2.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.37342995998046336, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].4.2.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.9898332181903231, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].4.3.0
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.6627684885197693, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].4.3.1
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.11436811682659254, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[int]]].4.3.2
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, 
input_value=0.9912529439241704, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/int_from_float
data.list[list[list[list[int]]]].0.0.0
  Input should be a valid list [type=list_type, input_value=0.42234513583392064, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].0.0.1
  Input should be a valid list [type=list_type, input_value=0.9981640966089224, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].0.0.2
  Input should be a valid list [type=list_type, input_value=0.09632932364502289, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].0.1.0
  Input should be a valid list [type=list_type, input_value=0.42081777591692904, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].0.1.1
  Input should be a valid list [type=list_type, input_value=0.5887311084240875, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].0.1.2
  Input should be a valid list [type=list_type, input_value=0.3385201082164039, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].0.2.0
  Input should be a valid list [type=list_type, input_value=0.6435850727612293, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].0.2.1
  Input should be a valid list [type=list_type, input_value=0.16461686522171493, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].0.2.2
  Input should be a valid list [type=list_type, input_value=0.3927389944829386, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].0.3.0
  Input should be a valid list [type=list_type, input_value=0.32706336401985225, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].0.3.1
  Input should be a valid list [type=list_type, input_value=0.565638108937007, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].0.3.2
  Input should be a valid list [type=list_type, input_value=0.6387940621663579, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].1.0.0
  Input should be a valid list [type=list_type, input_value=0.4087321129803253, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].1.0.1
  Input should be a valid list [type=list_type, input_value=0.5261610218219884, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].1.0.2
  Input should be a valid list [type=list_type, input_value=0.7762568903748635, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].1.1.0
  Input should be a valid list [type=list_type, input_value=0.38478470547415966, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].1.1.1
  Input should be a valid list [type=list_type, input_value=0.2322302584804432, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].1.1.2
  Input should be a valid list [type=list_type, input_value=0.059532643058274215, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].1.2.0
  Input should be a valid list [type=list_type, input_value=0.8735040489010808, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].1.2.1
  Input should be a valid list [type=list_type, input_value=0.09807075061654813, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].1.2.2
  Input should be a valid list [type=list_type, input_value=0.07074527512286577, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].1.3.0
  Input should be a valid list [type=list_type, input_value=0.7661475943816916, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].1.3.1
  Input should be a valid list [type=list_type, input_value=0.36025616097354807, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].1.3.2
  Input should be a valid list [type=list_type, input_value=0.8032431315384145, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].2.0.0
  Input should be a valid list [type=list_type, input_value=0.20773498794392165, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].2.0.1
  Input should be a valid list [type=list_type, input_value=0.8912371261036824, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].2.0.2
  Input should be a valid list [type=list_type, input_value=0.5524036456566535, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].2.1.0
  Input should be a valid list [type=list_type, input_value=0.3396462290701292, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].2.1.1
  Input should be a valid list [type=list_type, input_value=0.9365729412321798, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].2.1.2
  Input should be a valid list [type=list_type, input_value=0.19075518266606717, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].2.2.0
  Input should be a valid list [type=list_type, input_value=0.6002502925113803, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].2.2.1
  Input should be a valid list [type=list_type, input_value=0.07906354053760545, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].2.2.2
  Input should be a valid list [type=list_type, input_value=0.918351053759276, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].2.3.0
  Input should be a valid list [type=list_type, input_value=0.12932846306037848, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].2.3.1
  Input should be a valid list [type=list_type, input_value=0.9570938466466345, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].2.3.2
  Input should be a valid list [type=list_type, input_value=0.9313595127526528, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].3.0.0
  Input should be a valid list [type=list_type, input_value=0.8618129658923739, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].3.0.1
  Input should be a valid list [type=list_type, input_value=0.7214154633883629, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].3.0.2
  Input should be a valid list [type=list_type, input_value=0.5176409748603018, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].3.1.0
  Input should be a valid list [type=list_type, input_value=0.9060300848865271, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].3.1.1
  Input should be a valid list [type=list_type, input_value=0.29296842574562965, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].3.1.2
  Input should be a valid list [type=list_type, input_value=0.8991997907883464, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].3.2.0
  Input should be a valid list [type=list_type, input_value=0.021171414546333978, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].3.2.1
  Input should be a valid list [type=list_type, input_value=0.9436070537149975, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].3.2.2
  Input should be a valid list [type=list_type, input_value=0.3213290303624765, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].3.3.0
  Input should be a valid list [type=list_type, input_value=0.33460194706240765, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].3.3.1
  Input should be a valid list [type=list_type, input_value=0.8885681590782408, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].3.3.2
  Input should be a valid list [type=list_type, input_value=0.800206221466748, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].4.0.0
  Input should be a valid list [type=list_type, input_value=0.5102787372752916, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].4.0.1
  Input should be a valid list [type=list_type, input_value=0.49350987307401684, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].4.0.2
  Input should be a valid list [type=list_type, input_value=0.9747759628198037, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].4.1.0
  Input should be a valid list [type=list_type, input_value=0.019008716486376476, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].4.1.1
  Input should be a valid list [type=list_type, input_value=0.18241175067171844, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].4.1.2
  Input should be a valid list [type=list_type, input_value=0.31927609128421863, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].4.2.0
  Input should be a valid list [type=list_type, input_value=0.406477661326343, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].4.2.1
  Input should be a valid list [type=list_type, input_value=0.37342995998046336, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].4.2.2
  Input should be a valid list [type=list_type, input_value=0.9898332181903231, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].4.3.0
  Input should be a valid list [type=list_type, input_value=0.6627684885197693, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].4.3.1
  Input should be a valid list [type=list_type, input_value=0.11436811682659254, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[int]]]].4.3.2
  Input should be a valid list [type=list_type, input_value=0.9912529439241704, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].0.0.0
  Input should be a valid list [type=list_type, input_value=0.42234513583392064, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].0.0.1
  Input should be a valid list [type=list_type, input_value=0.9981640966089224, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].0.0.2
  Input should be a valid list [type=list_type, input_value=0.09632932364502289, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].0.1.0
  Input should be a valid list [type=list_type, input_value=0.42081777591692904, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].0.1.1
  Input should be a valid list [type=list_type, input_value=0.5887311084240875, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].0.1.2
  Input should be a valid list [type=list_type, input_value=0.3385201082164039, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].0.2.0
  Input should be a valid list [type=list_type, input_value=0.6435850727612293, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].0.2.1
  Input should be a valid list [type=list_type, input_value=0.16461686522171493, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].0.2.2
  Input should be a valid list [type=list_type, input_value=0.3927389944829386, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].0.3.0
  Input should be a valid list [type=list_type, input_value=0.32706336401985225, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].0.3.1
  Input should be a valid list [type=list_type, input_value=0.565638108937007, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].0.3.2
  Input should be a valid list [type=list_type, input_value=0.6387940621663579, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].1.0.0
  Input should be a valid list [type=list_type, input_value=0.4087321129803253, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].1.0.1
  Input should be a valid list [type=list_type, input_value=0.5261610218219884, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].1.0.2
  Input should be a valid list [type=list_type, input_value=0.7762568903748635, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].1.1.0
  Input should be a valid list [type=list_type, input_value=0.38478470547415966, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].1.1.1
  Input should be a valid list [type=list_type, input_value=0.2322302584804432, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].1.1.2
  Input should be a valid list [type=list_type, input_value=0.059532643058274215, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].1.2.0
  Input should be a valid list [type=list_type, input_value=0.8735040489010808, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].1.2.1
  Input should be a valid list [type=list_type, input_value=0.09807075061654813, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].1.2.2
  Input should be a valid list [type=list_type, input_value=0.07074527512286577, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].1.3.0
  Input should be a valid list [type=list_type, input_value=0.7661475943816916, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].1.3.1
  Input should be a valid list [type=list_type, input_value=0.36025616097354807, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].1.3.2
  Input should be a valid list [type=list_type, input_value=0.8032431315384145, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].2.0.0
  Input should be a valid list [type=list_type, input_value=0.20773498794392165, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].2.0.1
  Input should be a valid list [type=list_type, input_value=0.8912371261036824, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].2.0.2
  Input should be a valid list [type=list_type, input_value=0.5524036456566535, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].2.1.0
  Input should be a valid list [type=list_type, input_value=0.3396462290701292, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].2.1.1
  Input should be a valid list [type=list_type, input_value=0.9365729412321798, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].2.1.2
  Input should be a valid list [type=list_type, input_value=0.19075518266606717, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].2.2.0
  Input should be a valid list [type=list_type, input_value=0.6002502925113803, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].2.2.1
  Input should be a valid list [type=list_type, input_value=0.07906354053760545, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].2.2.2
  Input should be a valid list [type=list_type, input_value=0.918351053759276, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].2.3.0
  Input should be a valid list [type=list_type, input_value=0.12932846306037848, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].2.3.1
  Input should be a valid list [type=list_type, input_value=0.9570938466466345, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].2.3.2
  Input should be a valid list [type=list_type, input_value=0.9313595127526528, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].3.0.0
  Input should be a valid list [type=list_type, input_value=0.8618129658923739, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].3.0.1
  Input should be a valid list [type=list_type, input_value=0.7214154633883629, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].3.0.2
  Input should be a valid list [type=list_type, input_value=0.5176409748603018, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].3.1.0
  Input should be a valid list [type=list_type, input_value=0.9060300848865271, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].3.1.1
  Input should be a valid list [type=list_type, input_value=0.29296842574562965, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].3.1.2
  Input should be a valid list [type=list_type, input_value=0.8991997907883464, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].3.2.0
  Input should be a valid list [type=list_type, input_value=0.021171414546333978, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].3.2.1
  Input should be a valid list [type=list_type, input_value=0.9436070537149975, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].3.2.2
  Input should be a valid list [type=list_type, input_value=0.3213290303624765, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].3.3.0
  Input should be a valid list [type=list_type, input_value=0.33460194706240765, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].3.3.1
  Input should be a valid list [type=list_type, input_value=0.8885681590782408, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].3.3.2
  Input should be a valid list [type=list_type, input_value=0.800206221466748, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].4.0.0
  Input should be a valid list [type=list_type, input_value=0.5102787372752916, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].4.0.1
  Input should be a valid list [type=list_type, input_value=0.49350987307401684, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].4.0.2
  Input should be a valid list [type=list_type, input_value=0.9747759628198037, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].4.1.0
  Input should be a valid list [type=list_type, input_value=0.019008716486376476, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].4.1.1
  Input should be a valid list [type=list_type, input_value=0.18241175067171844, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].4.1.2
  Input should be a valid list [type=list_type, input_value=0.31927609128421863, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].4.2.0
  Input should be a valid list [type=list_type, input_value=0.406477661326343, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].4.2.1
  Input should be a valid list [type=list_type, input_value=0.37342995998046336, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].4.2.2
  Input should be a valid list [type=list_type, input_value=0.9898332181903231, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].4.3.0
  Input should be a valid list [type=list_type, input_value=0.6627684885197693, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].4.3.1
  Input should be a valid list [type=list_type, input_value=0.11436811682659254, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type
data.list[list[list[list[list[int]]]]].4.3.2
  Input should be a valid list [type=list_type, input_value=0.9912529439241704, input_type=float64]
    For further information visit https://errors.pydantic.dev/2.8/v/list_type

Specification#

NDArrays are defined by an ArrayExpression (metamodel docs)

An ArrayExpression has many of the default properties other LinkML expressions have – you can annotate your arrays like the rest of your data. To define the array, an ArrayExpression uses the following unique properties:

Shape

Shape can be specified numerically:

maximum_number_dimensions: int | False | None#

Maximum (inclusive) number of dimensions.

When used with dimensions, to differentiate with being unset or None, needs to be set to False explicitly to indicate an “infinite”[1] number of dimensions (see Complex Shaped Arrays)

minimum_number_dimensions: int | None#

Minimum (inclusive) number of dimensions.

exact_number_dimensions: int | None#

An exact number of dimensions.

Equivalent to minimum_number_dimensions and maximum_number_dimensions being equal ints

dimensions: list[DimensionExpression]#

Parameterization of individual dimensions (see below)

Dimensions

Dimensions can be further parameterized by defining dimensions with a list of DimensionExpressions, which accept:

alias: str | None#

The name of the dimension

maximum_cardinality: int | None#

The maximum size of this dimension (inclusive). If None, no maximum is set

minimum_cardinality: int | None#

The minimum size of this dimension (inclusive). If None, no minimum is set

exact_cardinality: int | None#

The exact size of this dimension. Equivalent to minimum_cardinality and maximum_cardinality being equal ints

Shape Forms#

The combinations of the different ArrayExpression properties imply four NDArray forms:

  • Any Shape - Arrays without limits to their shape

  • Bounded Shape - Arrays with constraints on the number of dimensions without further parameterization

  • Parameterized Shape - Arrays with parameterized dimensions

  • Complex Shape - Arrays with both constraints on the number of dimensions and parameterized dimensions

Any#

An Any shaped array can take any shape. This is the simplest array form, indicating the mere presence of an array.

An any shaped array is specified with an empty array dictionary:

AnyType:
  attributes:
    array:
      range: AnyType
      array: {}

Typed:
  attributes:
    array:
      range: integer
      array: {}
class AnyType(ConfiguredBaseModel):
    array: Optional[AnyShapeArray] = Field(None)

class Typed(ConfiguredBaseModel):
    array: Optional[AnyShapeArray[int]] = Field(None)

Coming Soon

Direct support for array libraries like numpy, hdf5, dask, zarr, and xarray with pydantic validation <3

The resulting pydantic models use a special AnyShapeArray class injected by pydanticgen’s template system when using the List of List (LoL) representation (see Representations).

Note

In the future, Any shape arrays might also be able to be specified with an explicit None value, e.g.:

MyClass:
  attributes:
    data:
      range: integer
      array:

This is a technical limitation in SchemaView - see linkml-model#189 and #1975

Bounded#

We have already seen so-called “Bounded” shaped arrays, which don’t add any additional parameterization beyond the number of their dimensions.

The maximum_, minimum_, and exact_number_dimensions properties can be used in combination to indicate…

A minimum without a maximum, using the AnyShapeArray model internally -

Hide code cell source
sch = schemas / 'bounded_shape.yaml'
render_comparison(sch, 'MinDimensions')
MinDimensions:
  attributes:
    array:
      range: integer
      array:
        minimum_number_dimensions: 2
class MinDimensions(ConfiguredBaseModel):
    linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "bounded_shape_array"})

    array: Optional[List[AnyShapeArray[int]]] = Field(
        None,
        json_schema_extra={
            "linkml_meta": {
                "alias": "array",
                "array": {"minimum_number_dimensions": 2},
                "domain_of": ["ExactDimensions", "MinDimensions", "MaxDimensions", "RangeDimensions"],
            }
        },
    )

Coming Soon

Direct support for array libraries like numpy, hdf5, dask, zarr, and xarray with pydantic validation <3

A maximum without a minimum -

Hide code cell source
sch = schemas / 'bounded_shape.yaml'
render_comparison(sch, 'MaxDimensions')
MaxDimensions:
  attributes:
    array:
      range: integer
      array:
        maximum_number_dimensions: 5
class MaxDimensions(ConfiguredBaseModel):
    linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "bounded_shape_array"})

    array: Optional[
        Union[
            List[int],
            List[List[int]],
            List[List[List[int]]],
            List[List[List[List[int]]]],
            List[List[List[List[List[int]]]]],
        ]
    ] = Field(
        None,
        json_schema_extra={
            "linkml_meta": {
                "alias": "array",
                "array": {"maximum_number_dimensions": 5},
                "domain_of": ["ExactDimensions", "MinDimensions", "MaxDimensions", "RangeDimensions"],
            }
        },
    )

Coming Soon

Direct support for array libraries like numpy, hdf5, dask, zarr, and xarray with pydantic validation <3

An exact number of dimensions -

Hide code cell source
sch = schemas / 'bounded_shape.yaml'
render_comparison(sch, 'ExactDimensions')
ExactDimensions:
  attributes:
    array:
      range: integer
      array:
        exact_number_dimensions: 3
class ExactDimensions(ConfiguredBaseModel):
    linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "bounded_shape_array"})

    array: Optional[List[List[List[int]]]] = Field(
        None,
        json_schema_extra={
            "linkml_meta": {
                "alias": "array",
                "array": {"exact_number_dimensions": 3},
                "domain_of": ["ExactDimensions", "MinDimensions", "MaxDimensions", "RangeDimensions"],
            }
        },
    )

Coming Soon

Direct support for array libraries like numpy, hdf5, dask, zarr, and xarray with pydantic validation <3

And a range of dimensions -

Hide code cell source
sch = schemas / 'bounded_shape.yaml'
render_comparison(sch, 'RangeDimensions')
RangeDimensions:
  attributes:
    array:
      range: integer
      array:
        minimum_number_dimensions: 2
        maximum_number_dimensions: 5
class RangeDimensions(ConfiguredBaseModel):
    linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "bounded_shape_array"})

    array: Optional[
        Union[List[List[int]], List[List[List[int]]], List[List[List[List[int]]]], List[List[List[List[List[int]]]]]]
    ] = Field(
        None,
        json_schema_extra={
            "linkml_meta": {
                "alias": "array",
                "array": {"maximum_number_dimensions": 5, "minimum_number_dimensions": 2},
                "domain_of": ["ExactDimensions", "MinDimensions", "MaxDimensions", "RangeDimensions"],
            }
        },
    )

Coming Soon

Direct support for array libraries like numpy, hdf5, dask, zarr, and xarray with pydantic validation <3

Parameterized#

Dimensions can be further parameterized, giving them names and cardinality constraints.

Similarly to bounded arrays, the following demonstrate setting a single-dimensional array with cardinality constraints…

Minimum cardinality:

Hide code cell source
sch = schemas / 'parameterized_shape.yaml'
render_comparison(sch, 'MinCard')
MinCard:
  attributes:
    array:
      range: integer
      array:
        dimensions:
        - alias: min_card
          minimum_cardinality: 2
class MinCard(ConfiguredBaseModel):
    linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})

    array: Optional[conlist(min_length=2, item_type=int)] = Field(
        None,
        json_schema_extra={
            "linkml_meta": {
                "alias": "array",
                "array": {"dimensions": [{"alias": "min_card", "minimum_cardinality": 2}]},
                "domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
            }
        },
    )

Coming Soon

Direct support for array libraries like numpy, hdf5, dask, zarr, and xarray with pydantic validation <3

Maximum cardinality:

Hide code cell source
sch = schemas / 'parameterized_shape.yaml'
render_comparison(sch, 'MaxCard')
MaxCard:
  attributes:
    array:
      range: integer
      array:
        dimensions:
        - alias: max_card
          maximum_cardinality: 5
class MaxCard(ConfiguredBaseModel):
    linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})

    array: Optional[conlist(max_length=5, item_type=int)] = Field(
        None,
        json_schema_extra={
            "linkml_meta": {
                "alias": "array",
                "array": {"dimensions": [{"alias": "max_card", "maximum_cardinality": 5}]},
                "domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
            }
        },
    )

Coming Soon

Direct support for array libraries like numpy, hdf5, dask, zarr, and xarray with pydantic validation <3

Exact cardinality:

Hide code cell source
sch = schemas / 'parameterized_shape.yaml'
render_comparison(sch, 'ExactCard')
ExactCard:
  attributes:
    array:
      range: integer
      array:
        dimensions:
        - alias: max_card
          exact_cardinality: 3
class ExactCard(ConfiguredBaseModel):
    linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})

    array: Optional[conlist(min_length=3, max_length=3, item_type=int)] = Field(
        None,
        json_schema_extra={
            "linkml_meta": {
                "alias": "array",
                "array": {"dimensions": [{"alias": "max_card", "exact_cardinality": 3}]},
                "domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
            }
        },
    )

Coming Soon

Direct support for array libraries like numpy, hdf5, dask, zarr, and xarray with pydantic validation <3

Cardinality range:

Hide code cell source
sch = schemas / 'parameterized_shape.yaml'
render_comparison(sch, 'RangeCard')
RangeCard:
  attributes:
    array:
      range: integer
      array:
        dimensions:
        - alias: min_card
          minimum_cardinality: 2
          maximum_cardinality: 5
class RangeCard(ConfiguredBaseModel):
    linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})

    array: Optional[conlist(min_length=2, max_length=5, item_type=int)] = Field(
        None,
        json_schema_extra={
            "linkml_meta": {
                "alias": "array",
                "array": {"dimensions": [{"alias": "min_card", "maximum_cardinality": 5, "minimum_cardinality": 2}]},
                "domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
            }
        },
    )

Coming Soon

Direct support for array libraries like numpy, hdf5, dask, zarr, and xarray with pydantic validation <3

And they can be used together, for example one can specify

A four dimensional array such that

  • The first dimension has at least two items

  • The second dimension has at most five items

  • The third dimension has between two and five items

  • The fourth dimension has exactly six items

Hide code cell source
sch = schemas / 'parameterized_shape.yaml'
render_comparison(sch, 'ParameterizedArray')
parameterized_mod = compile_module(sch)
ParameterizedArray = getattr(parameterized_mod, 'ParameterizedArray')
ParameterizedArray:
  attributes:
    array:
      range: integer
      array:
        dimensions:
        - alias: min_card
          minimum_cardinality: 2
        - alias: max_card
          maximum_cardinality: 5
        - alias: range_card
          minimum_cardinality: 2
          maximum_cardinality: 5
        - alias: exact_card
          exact_cardinality: 6
class ParameterizedArray(ConfiguredBaseModel):
    linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "parameterized_shape_array"})

    array: Optional[
        conlist(
            min_length=2,
            item_type=conlist(
                max_length=5,
                item_type=conlist(
                    min_length=2, max_length=5, item_type=conlist(min_length=6, max_length=6, item_type=int)
                ),
            ),
        )
    ] = Field(
        None,
        json_schema_extra={
            "linkml_meta": {
                "alias": "array",
                "array": {
                    "dimensions": [
                        {"alias": "min_card", "minimum_cardinality": 2},
                        {"alias": "max_card", "maximum_cardinality": 5},
                        {"alias": "range_card", "maximum_cardinality": 5, "minimum_cardinality": 2},
                        {"alias": "exact_card", "exact_cardinality": 6},
                    ]
                },
                "domain_of": ["MinCard", "MaxCard", "ExactCard", "RangeCard", "ParameterizedArray"],
            }
        },
    )

Coming Soon

Direct support for array libraries like numpy, hdf5, dask, zarr, and xarray with pydantic validation <3

Which validates each of the constraints separately:

array = np.arange(4*1*2*6,dtype=int).reshape((4,1,2,6))
# array = np.ones((4,1,2,6), dtype=int)

console.print(ParameterizedArray(array=array))
ParameterizedArray(
    array=[
        [[[0, 1, 2, 3, 4, 5], [6, 7, 8, 9, 10, 11]]],
        [[[12, 13, 14, 15, 16, 17], [18, 19, 20, 21, 22, 23]]],
        [[[24, 25, 26, 27, 28, 29], [30, 31, 32, 33, 34, 35]]],
        [[[36, 37, 38, 39, 40, 41], [42, 43, 44, 45, 46, 47]]]
    ]
)

Complex#

Complex NDArrays combine all three of the prior forms.

For example:

An array with between 5 and 7 dimensions such that…

  • The first dimension has at least two items

  • The second dimension has at most five items

  • The third dimension has between two and five items

  • The fourth dimension has exactly six items

Hide code cell source
sch = schemas / 'complex_shape.yaml'
render_comparison(sch, 'ComplexRangeShapeArray')
ComplexRangeShapeArray:
  attributes:
    array:
      range: integer
      array:
        minimum_number_dimensions: 5
        maximum_number_dimensions: 7
        dimensions:
        - alias: max_card
          maximum_cardinality: 5
        - alias: min_card
          minimum_cardinality: 2
        - alias: range_card
          minimum_cardinality: 2
          maximum_cardinality: 5
        - alias: exact_card
          exact_cardinality: 6
class ComplexRangeShapeArray(ConfiguredBaseModel):
    linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "complex_shape_array"})

    array: Optional[
        conlist(
            max_length=5,
            item_type=conlist(
                min_length=2,
                item_type=conlist(
                    min_length=2,
                    max_length=5,
                    item_type=conlist(
                        min_length=6, max_length=6, item_type=Union[List[int], List[List[int]], List[List[List[int]]]]
                    ),
                ),
            ),
        )
    ] = Field(
        None,
        json_schema_extra={
            "linkml_meta": {
                "alias": "array",
                "array": {
                    "dimensions": [
                        {"alias": "max_card", "maximum_cardinality": 5},
                        {"alias": "min_card", "minimum_cardinality": 2},
                        {"alias": "range_card", "maximum_cardinality": 5, "minimum_cardinality": 2},
                        {"alias": "exact_card", "exact_cardinality": 6},
                    ],
                    "maximum_number_dimensions": 7,
                    "minimum_number_dimensions": 5,
                },
                "domain_of": [
                    "ComplexAnyShapeArray",
                    "ComplexMaxShapeArray",
                    "ComplexRangeShapeArray",
                    "ComplexExactShapeArray",
                ],
            }
        },
    )

Coming Soon

Direct support for array libraries like numpy, hdf5, dask, zarr, and xarray with pydantic validation <3

The only place where the syntax of complex arrays differ is that minimum and maximum_number_dimensions are set at the length of the specified dimensions list by default. In order to specify an array with a fixed number of parameterized dimensions with an arbitrary number of additional dimensions, set maximum_number_dimensions to False :

Hide code cell source
sch = schemas / 'complex_shape.yaml'
render_comparison(sch, 'ComplexAnyShapeArray')
ComplexAnyShapeArray:
  attributes:
    array:
      range: integer
      array:
        maximum_number_dimensions: false
        dimensions:
        - alias: max_card
          maximum_cardinality: 5
class ComplexAnyShapeArray(ConfiguredBaseModel):
    linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({"from_schema": "complex_shape_array"})

    array: Optional[conlist(max_length=5, item_type=Union[AnyShapeArray[int], int])] = Field(
        None,
        json_schema_extra={
            "linkml_meta": {
                "alias": "array",
                "array": {
                    "dimensions": [{"alias": "max_card", "maximum_cardinality": 5}],
                    "maximum_number_dimensions": False,
                },
                "domain_of": [
                    "ComplexAnyShapeArray",
                    "ComplexMaxShapeArray",
                    "ComplexRangeShapeArray",
                    "ComplexExactShapeArray",
                ],
            }
        },
    )

Coming Soon

Direct support for array libraries like numpy, hdf5, dask, zarr, and xarray with pydantic validation <3

Generators#

Support#

At release, only pydanticgen supports arrays, but arrays will be implemented gradually for the rest of the generators.

generator

representation

anyshape

bounded

parameterized

complex

pydantic

List of Lists

Y

Y

Y

Y

pydantic

Numpydantic

X

X

X

X

… (the rest of em)

Representations#

Since arrays are unlike other data types in that they usually require some specialized libraries to handle them, and many formats don’t have a single canonical array type, generators may accommodate multiple array representations.

The pydantic generator currently supports a “List of lists” style array representation to be able to use arrays without any additional external dependencies beyond pydantic.

See each generator’s documentation page for a summary of the array representations they support.

Numpy And Friends

We are in the process of implementing an additional pydantic array representation that can directly work with numpy-style arrays, including lazy loading and passthrough array routines for several major array libraries as a standalone package. See numpydantic

Implementation Guidance#

TODO

Implementation docs for arrays are forthcoming.

For now see ArrayRangeGenerator and ListOfListsArray as the reference implementation

See Also#

References#