Schema#

Class for managing, building, and caching built schemas.

The nwb.core and hdmf-common schema are statically built and stored in this repository, but to make it feasible to use arbitrary schema, eg. those stored inside of an NWB file, we need a bit of infrastructure for generating and caching pydantic models on the fly.

Relationship to other modules: * adapters manage the conversion from NWB schema language to linkML. * generators create models like pydantic models from the linkML schema * providers then use adapters and generators to provide models from generated schema!

Providers create a set of directories with namespaces and versions, so eg. for the linkML and pydantic providers:

cache_dir
  - linkml
    - nwb_core
      - v0_2_0
        - namespace.yaml
        - nwb.core.file.yaml
        - ...
      - v0_2_1
        - namespace.yaml
        - ...
    - my_schema
      - v0_1_0
        - ...
  - pydantic
    - nwb_core
      - v0_2_0
        - namespace.py
        - ...
      - v0_2_1
        - namespace.py
        - ...

class Provider(path: Path | None = None, allow_repo: bool = True, verbose: bool = True)#

Metaclass for different kind of providers!

Parameters:

path (pathlib.Path) – Override the temporary directory configured by the environment-wide Config object as the base directory that the subclasses provide to.
verbose (bool) – If True, print things like progress bars to stdout :)

config#

Configuration for the directories used by this provider, unless overridden by path

Type:: Config

allow_repo#

Allow the pathfinder to return the installed repository/package, useful to enforce building into temporary directories, decoupling finding a path during loading vs. building. Building into the repo is still possible if both namespace and version are provided (ie. the path is fully qualified) and config’s path is the repository path.

Type:: bool

cache_dir#

The main cache directory under which the other providers will store the things they provide

Type:: pathlib.Path

PROVIDES: str#

PROVIDES_CLASS: P = None#

abstract property path: Path#: Base path for this kind of provider

abstract build(*args: Any)#: Whatever needs to be done to build this thing, if applicable

abstract get(*args: Any) → Any#

Get a cached item.

Optionally, try any build it if it’s possible to do so

namespace_path(namespace: str, version: str | None = None, allow_repo: bool | None = None) → Path#

Get the location for a given namespace of this type.

Note that we don’t check for existence, because this method should also be used when generating schema — this is the canonical location

Parameters:

namespace (str) – Namespace to get!
version (str) – Optional, version of namespace. If None, either get the most recent version built, or if namespace is core or hdmf-common, use the modules provided with the package. We do not use the most recent version, but the most recently generated version because it’s assumed that’s the one you want if you’re just gesturally reaching for one.
allow_repo (bool) – Optional - override instance-level allow_repo attr

property available_versions: Dict[str, List[str]]#: Dictionary mapping a namespace to a list of built versions

class LinkMLSchemaBuild#

Build result from LinkMLProvider.build()

version: str#

namespace: Path#

name: str#

result: BuildResult | None#

class LinkMLProvider(path: Path | None = None, allow_repo: bool = True, verbose: bool = True)#

Provider for conversions from nwb schema language to linkML.

By default, generate and manage a nest of temporary cache directories (as configured by Config) for each version of a given namespace.

Like other Provider classes, this model is not a singleton but behaves a bit like one in that when instantiated without arguments it is stateless (except for configuration by environment-level variables). So we don’t use @classmethod s here, but instantiating the class should remain cheap.

Namespaces can be built from:

namespace .yaml files: build_from_yaml()
dictionaries, as are usually packaged in nwb files: build_from_dicts()

All of which feed into…

NamespacesAdapter used throughout the rest of nwb_linkml - build()

After a namespace is built, it can be accessed using LinkMLProvider.get(), which can also be consumed by other providers, so a given namespace and version should only need to be built once.

Note

At the moment there is no checking (eg. by comparing hashes) of different sources that purport to be a given version of a namespace. When ambiguous, the class prefers to build sets of namespaces together and use the most recently built ones since there is no formal system for linking versions of namespaced schemas in nwb schema language.

Examples

>>> provider = LinkMLProvider()
>>> # Simplest case, get the core nwb schema from the default NWB core repo
>>> core = provider.get('core')
>>> # Get a specific version of the core schema
>>> core_other_version = provider.get('core', '2.2.0')
>>> # Build a custom schema and then get it
>>> # provider.build_from_yaml('myschema.yaml')
>>> # my_schema = provider.get('myschema')

PROVIDES: str = 'linkml'#

PROVIDES_CLASS#: alias of SchemaDefinition

property path: Path#: Base path for this kind of provider

build_from_yaml(path: Path, **kwargs)#

Build a namespace’s schema

Parameters:

path (pathlib.Path) – Path to the namespace .yaml
kwargs – passed to build()

build_from_dicts(schemas: Dict[str, dict], **kwargs) → Dict[str | SchemaDefinitionName, LinkMLSchemaBuild]#

Build from schema dictionaries, eg. as come from nwb files

Parameters:: schemas (dict) – A dictionary of {'schema_name': {:schema_definition}}. The “namespace” schema should have the key namespace, which is used to infer version and schema name. Post-load maps should have already been applied

build(ns_adapter: NamespacesAdapter, versions: dict | None = None, dump: bool = True, force: bool = False) → Dict[str | SchemaDefinitionName, LinkMLSchemaBuild]#

Parameters:

namespaces (NamespacesAdapter) – Adapter (populated with any necessary imported namespaces) to build
versions (dict) – Dict of specific versions to use for cross-namespace imports. as {'namespace': 'version'} If none is provided, use the most recent version available.
dump (bool) – If True (default), dump generated schema to YAML. otherwise just return
force (bool) – If False (default), don’t build schema that already exist. If True , clear directory and rebuild

Returns:

Dict[str, LinkMLSchemaBuild]. For normal builds, LinkMLSchemaBuild.result will be populated with results of the build. If force == False and the schema already exist, it will be None

get(namespace: str, version: str | None = None) → SchemaView#

Get a schema view over the namespace.

If a matching path for the namespace and version exists in the path, then return the SchemaView over that namespace.

Otherwise, try and find a source using our providers.git.DEFAULT_REPOS.

If none is found, then you need to build and cache the (probably custom) schema first with build()

class PydanticProvider(path: Path | None = None, verbose: bool = True)#

Provider for pydantic models built from linkml-style nwb schema (ie. as provided by LinkMLProvider)

PROVIDES: str = 'pydantic'#

property path: Path#: Base path for this kind of provider

Notes

We currently infer namespace and version from the path when namespace is a Path, which is a patently Bad Thing To Do. This is a temporary measure until we decide on a permanent means by which we want to cache built artifacts <3. Hierarchies of folders is not the target design.

Parameters:

namespace (Union[str, pathlib.Path]) – If a string, use a LinkMLProvider to get the converted schema. If a path, assume we have been given an explicit namespace.yaml from a converted NWB -> LinkML schema to load from.
out_file (Optional[Path]) – Optionally override the output file. If None, generate from namespace and version
version (Optional[str]) – The version of the schema to build, if present. Works similarly to version in LinkMLProvider. Ignored if namespace is a Path.
versions (Optional[dict]) – An explicit mapping of namespaces and versions to use when building the combined pydantic namespace.py file. Since NWB doesn’t have an explicit version dependency system between schema, there is intrinsic ambiguity between which version of which schema should be used when imported from another. This mapping allows those ambiguities to be resolved. See NWBPydanticGenerator ‘s versions argument for more information.
split (bool) – If False (default), generate a single namespace.py file, otherwise generate a python file for each schema in the namespace in addition to a namespace.py that imports from them
dump (bool) – If True (default), dump the model to the cache, otherwise just return the serialized string of built pydantic model
force (bool) – If False (default), don’t build the model if it already exists, if True , delete and rebuild any model
**kwargs – Passed to NWBPydanticGenerator

Returns:

The built model file as returned from NWBPydanticGenerator.serialize()

Return type:

str

classmethod module_name(namespace: str, version: str) → str#

import_module(namespace: str, version: str | None = None) → module#

Import a module within the temporary directory from its namespace and version

In most cases, you’re looking for PydanticProvider.get(), this method is made available in case you don’t want to accidentally build something or invoke the rest of the provisioning system.

Parameters:

namespace (str) – Name of namespace
version (Optional[str]) – Version to import, if None, try and get the most recently built version.

Returns:

types.ModuleType

get(namespace: str, version: str | None = None, allow_repo: bool | None = None) → module#

Get the imported module for a given namespace and version.

A given namespace will be stored in sys.modules as nwb_linkml.models.{namespace}, so first check if there is any already-imported module, and return that if so.

Then we check in the temporary directory for an already-built namespace.py file

Otherwise we pass arguments to PydanticProvider.build() and attempt to build them before returning.

Notes

The imported modules shadow the “actual” nwb_linkml.models module as would be imported from the usual location within the package directory. This is intentional, as models can then be used as if they were integrated parts of the package, and also so the active version of a namespace can be cleanly accessed (ie. without from nwb_linkml.models.core import v2_2_0 as core ). Accordingly, we assume that people will only be using a single version of NWB in a given Python session.

Parameters:

namespace (str) – Name of namespace to import. Must have either been previously built with PydanticProvider.build() or a matching namespace/version combo must be available to the LinkMLProvider
version (Optional[str]) – Version to import. If None, get the most recently build module
allow_repo (bool) – Allow getting modules provided within nwb_linkml.models.pydantic

Returns:

The imported types.ModuleType object that has all the built classes at the root level.

get_class(namespace: str, class_: str, version: str | None = None) → Type[BaseModel]#

Get a class from a given namespace and version!

Parameters:

namespace (str) – Name of a namespace that has been previously built and cached, otherwise we will attempt to build it from the providers.git.DEFAULT_REPOS
class (str) – Name of class to retrieve
version (Optional[str]) – Optional version of the schema to retrieve from

Returns:

pydantic.BaseModel

class EctopicModelFinder(path: Path, *args, **kwargs)#

A meta path finder that allows the import machinery to find a model package even if it might be outside the actual nwb_linkml namespace, as occurs when building split models in a temporary directory.

References

MODEL_STEM = 'nwb_linkml.models.pydantic'#

find_spec(fullname, path, target=None)#

class SchemaProvider(versions: Dict[str, str] | None = None, **kwargs)#

Class to manage building and caching linkml and pydantic models generated from nwb schema language. Combines LinkMLProvider and PydanticProvider

Behaves like a singleton without needing to be one - since we’re working off caches on disk that are indexed by hash in most “normal” conditions you should be able to use this anywhere, though no file-level locks are present to ensure consistency.

Store each generated schema in a directory structure indexed by schema namespace name and version

Parameters:

versions (dict) – Dictionary like {'namespace': 'v1.0.0'} used to specify that this provider should always return models from a specific version of a namespace (unless explicitly requested otherwise in a call to get() ).
**kwargs – passed to superclass __init__ (see Provider )

build_from_yaml(path: Path, **kwargs)#: Alias for LinkMLProvider.build_from_yaml() that also builds a pydantic model

build_from_dicts(schemas: Dict[str, dict], **kwargs) → Dict[str | SchemaDefinitionName, LinkMLSchemaBuild]#: Alias for LinkMLProvider.build_from_dicts() that also builds a pydantic model

property path: Path#: Base path for this kind of provider

build(ns_adapter: NamespacesAdapter, verbose: bool = True, linkml_kwargs: dict | None = None, pydantic_kwargs: dict | None = None, **kwargs) → Dict[str, str]#

Build a namespace, storing its linkML and pydantic models.

Parameters:

ns_adapter –
verbose (bool) – If True (default), show progress bars
linkml_kwargs (Optional[dict]) – Dictionary of kwargs optionally passed to LinkMLProvider.build()
pydantic_kwargs (Optional[dict]) – Dictionary of kwargs optionally passed to PydanticProvider.build()
**kwargs – Common options added to both linkml_kwargs and pydantic_kwargs

Returns:

Dict[str,str] mapping namespaces to built pydantic sources

get(namespace: str, version: str | None = None) → module#

Get a built pydantic model for a given namespace and version.

Wrapper around PydanticProvider.get()

get_class(namespace: str, class_: str, version: str | None = None) → Type[BaseModel]#

Get a pydantic model class from a given namespace and version!