Purpose#

If pynwb already exists, why nwb_linkml?

Two kinds of reasons:

  • using NWB as a test case for a larger infrastructure project, and

  • potentially improving the state of NWB itself.

A Stepping Stone…#

In the (word on how and why we are focusing on NWB as part of a larger project)

Interoperable Schema Language#

We want to make NWB a seed format in an interoperable, peer-to-peer graph of research data

NWB is written with its own schema language (And see the next section for more information). It seems to have been created primarily because other schema languages at the time couldn’t easily handle array specifications with fine-grained control of numerical format and shape.

The schema language is now relatively stable and does what it’s designed to do, but it being a domain-specific language rather than a general one makes it very difficult to use NWB data alongside other formats.

nwb_linkml translates NWB to linkml, a schema language for declaring Linked Data schema. Linked Data schema consist of semantic triplets, rather than an object hierarchy, and can make use of controlled vocabularies to reuse terms and classes from other schemas and ontologies.

Storage Format Flexibility#

We want to use NWB in lots of different ways

NWB as a format is designed with the intention for use with multiple storage backends, but patterns and features of HDF5 have made their way into the schema and the schema language, making direct translation to other storage systems difficult. This is a problem for practical usage of NWB data, since HDF5 files don’t lend themselves to querying across many files - eg. to find datasets that have some common piece of metadata, one would have to download them all in full first. Having a whole hierarchy of data in a single file is convenient in some ways, but this also makes them difficult to share or split between computers which is a common need when collecting data across multiple instruments and computers.

NWB, currently, lends itself towards being an archival format — where data is converted as a last step before publishing — rather than a experimental or computational format that can be used as a convenient container of heterogeneous data during collection and analysis.

The LinkML team has also made a large number of generators to convert LinkML schema to different formats, including JSON Schema, GraphQL, SPARQL, SQL/SQLAlchemy, and pydantic.

Since we have to use LinkML in a somewhat nonstandard way to accommodate NWB’s arrays, references, and naming conventions, these generators won’t be immediately available for use, but with some minor modification we should be able to get NWB out of HDF5 files and into other formats.

Zero-code Schema Extensions#

We want every researcher and every tool to have their own schemas.

pynwb makes use of NWB Schema internally, but schema extensions require a decent amount of adjoining code to use. The underlying hdmf library is relatively complex, and so to use a schema extension one must also program the python classes or mappings to python class attributes needed to use them, configuration for getter and setter methods, i/o routines, etc. Since schema extensions are relatively hard to make, to accomodate heterogeneous data NWB uses DynamicTables, which can be given arbitrary new columns.

The loose coupling between schema and code has a few impacts:

  • Many labs end up with their own independent software library for converting their data into NWB

  • Interoperability and meta-analysis suffer because terms are defined ad-hoc and with little discoverability.

  • Linking and versioning schema is hard, as the schema language doesn’t support it, and the code needs to be kept in-sync with the schema

  • It’s hard for tool-builders to implement direct export to NWB while maintaining flexibility in their libraries

Instead by making all models directly generated from schema, and by making use of pydantic and other validation and metaprogramming tools, we want to make it possible for every experiment to have its own schema extension. We want to make experimental data part of the normal social process of sharing results — translation: we want to be able to put our work in conversation with other related work!

Pythonic API#

We want NWB to be as simple to use as a python dataclass.

We think there is room for improvement in NWB’s API:

From the ndx-miniscope extension:

The extension code is intended to be used like this:

from pynwb import NWBFile, NWBHDF5IO
from pynwb.image import ImageSeries
from natsort import natsorted

from ndx_miniscope.utils import (
    add_miniscope_device,
    get_starting_frames,
    get_timestamps,
    read_miniscope_config,
    read_notes,
)

nwbfile = NWBFile(...)

# Load the miscroscope settings
miniscope_folder_path = "C6-J588_Disc5/15_03_28/Miniscope/"
miniscope_metadata = read_miniscope_config(folder_path=miniscope_folder_path)
# Create the Miniscope device with the microscope metadata and add it to NWB
add_miniscope_device(nwbfile=nwbfile, device_metadata=miniscope_metadata)

# Load the behavioral camera settings
behavcam_folder_path = "C6-J588_Disc5/15_03_28/BehavCam_2/"
behavcam_metadata = read_miniscope_config(folder_path=behavcam_folder_path)
# Create the Miniscope device with the behavioral camera metadata and add it to NWB
add_miniscope_device(nwbfile=nwbfile, device_metadata=behavcam_metadata)

save_path = os.path.join(folder_path, "test_out.nwb")
with NWBHDF5IO(save_path, "w") as io:
    io.write(nwbfile)

That uses these underlying functions to handle validation, coercion, and add to the NWB file:

def add_miniscope_device(nwbfile: NWBFile, device_metadata: dict) -> NWBFile:
    """
    Adds a Miniscope device based on provided metadata.
    Can be used to add device for the microscope and the behavioral camera.

    Parameters
    ----------
    nwbfile : NWBFile
        The nwbfile to add the Miniscope device to.
    device_metadata: dict
        The metadata for the device to be added.

    Returns
    -------
    NWBFile
        The NWBFile passed as an input with the Miniscope added.

    """
    device_metadata_copy = deepcopy(device_metadata)
    assert "name" in device_metadata_copy, "'name' is missing from metadata."
    device_name = device_metadata_copy["name"]
    if device_name in nwbfile.devices:
        return nwbfile

    roi = device_metadata_copy.pop("ROI", None)
    if roi:
        device_metadata_copy.update(ROI=[roi["height"], roi["width"]])

    device = Miniscope(**device_metadata_copy)
    nwbfile.add_device(device)

    return nwbfile

def add_miniscope_image_series(
    nwbfile: NWBFile,
    metadata: dict,
    timestamps: np.ndarray,
    image_series_index: int = 0,
    external_files: Optional[List[str]] = None,
    starting_frames: Optional[List[int]] = None,
) -> NWBFile:
    """
    Adds an ImageSeries with a linked Miniscope device based on provided metadata.
    The metadata for the device to be linked should be stored in metadata["Behavior]["Device"].

    Parameters
    ----------
    nwbfile : NWBFile
        The nwbfile to add the image series to.
    metadata: DeepDict
        The metadata storing the necessary metadata for creating the image series and linking it to the appropriate device.
    timestamps : np.ndarray
        The timestamps for the behavior movie source.
    image_series_index : int, optional
        The metadata for ImageSeries is a list of the different image series to add.
        Specify which element of the list with this parameter.
    external_files : List[str], optional
        List of external files associated with the ImageSeries.
    starting_frames :  List[int], optional
        List of starting frames for each external file.

    Returns
    -------
    NWBFile
        The NWBFile passed as an input with the ImageSeries added.

    """
    assert "Behavior" in metadata, "The metadata for ImageSeries and Device should be stored in 'Behavior'."
    assert (
        "ImageSeries" in metadata["Behavior"]
    ), "The metadata for ImageSeries should be stored in metadata['Behavior']['ImageSeries']."
    assert (
        "Device" in metadata["Behavior"]
    ), "The metadata for Device should be stored in metadata['Behavior']['Device']."
    image_series_kwargs = deepcopy(metadata["Behavior"]["ImageSeries"][image_series_index])
    image_series_name = image_series_kwargs["name"]

    if image_series_name in nwbfile.acquisition:
        return nwbfile

    # Add linked device to ImageSeries
    device_metadata = metadata["Behavior"]["Device"][image_series_index]
    device_name = device_metadata["name"]
    if device_name not in nwbfile.devices:
        add_miniscope_device(nwbfile=nwbfile, device_metadata=device_metadata)
    device = nwbfile.get_device(name=device_name)
    image_series_kwargs.update(device=device)

    assert external_files, "'external_files' must be specified."
    if starting_frames is None and len(external_files) == 1:
        starting_frames = [0]
    assert len(starting_frames) == len(
        external_files
    ), "The number of external files must match the length of 'starting_frame'."
    image_series_kwargs.update(
        format="external",
        external_file=external_files,
        starting_frame=starting_frames,
        timestamps=H5DataIO(timestamps, compression=True),
    )

    image_series = ImageSeries(**image_series_kwargs)
    nwbfile.add_acquisition(image_series)

An example of how we want nwb_linkml to work.

There are no additional underlying classes or functions to be written, since the pydantic models are directly generated from the schema extension, and to and from methods are generic for different types of input data (json files, videos). Tool developers can distribute NWB schema that map 1:1 to their output formats, decreasing the need for conversion code.

from pathlib import Path
from nwb_linkml.models.miniscope import Miniscope
from nwb_linkml.models.core import ImageSeries, NWBFile

# Load data for miniscope and videos
miniscope = Miniscope.from_json('config.json')
videos = []
for video_path in Path('./my_data/').glob('*.avi'):
    video = ImageSeries.from_video(video_path)
    video.device = miniscope
    videos.append(video)
    
# add to file
file = NWBFile.from_hdf('my_data.nwb')
file.devices['my_miniscope'] = miniscope
file.acquisition['my_videos'] = videos
file.save()