About

LinkML-Store is an early effort to provide a unifying storage layer over multiple different backends, unified via LinkML schemas.

The overall goals are to provide:

  • Make it easier to work with data in different forms (tabular, JSON, columnar, RDF)

  • Expressive validation at scale, including full referential integrity validation

  • Ability to mix and match different backends (e.g. DuckDB, MongoDB, Solr, ChromaDB, HDF5)

  • Composability of different search indexes, including LLM textual embeddings

  • LAMP-like stack for LinkML

Installation

At this stage we recommend installing all extras:

pip install "linkml-store[all]"

Minimal implementation:

pip install "linkml-store"

For developers working on the linkml-store codebase, we recommend checking out the repo, and then installing all packages via the make command:

git clone <URL>
cd linkml-store
make install

Quickstart

See the tutorials

Data Model

  • A Client provides a top-level interface over one or more databases.

  • A Database consists of one or more possibly heterogeneous collections.

  • A Collection is a queryable set of objects of a similar type.

  • A Indexer creates indexes over collections to enable efficient searching.

Adapters

The current backends are:

  • DuckDB (supports read and write)

  • MongoDB (supports read and write)

  • Solr (write not yet supported)

  • ChromaDB (pre-alpha)

  • HDF5 (pre-alpha)

Indexing

This frameworks also allows composable indexes. Currently two indexers are supported:

  • SimpleIndexer Simple native trigram method

  • LLMIndexer LLM text embedding

Metadata and Configuration

  • ClientConfig provides a structure for configuring the client