zarr_metadata.typed_json

zarr_metadata.typed_json

JSON checked against a TypedDict: the public door.

Every Zarr document and configuration this package describes is a TypedDict -- ZarrV3ArrayMetadataJSON, GzipCodecConfiguration -- and check makes any of them executable. It type-checks a JSON value against the TypedDict and gives back a value of it, or None, and every problem, each located in the value:

import json

from zarr_metadata import ZarrV3ArrayMetadataJSON
from zarr_metadata.typed_json import check

document, problems = check(json.loads(raw), ZarrV3ArrayMetadataJSON)
for problem in problems:
    print(problem.loc, problem.kind, problem.message)

A TypedDict reads as the typing spec defines it, however its module writes annotations:

• A key is required when its Required or NotRequired qualifier says so, and otherwise when the total of the class that declared it does. The runtime's __required_keys__ cannot see a qualifier written as a string, as from __future__ import annotations writes every one; typeddict_keys reads the annotations evaluated, and can.
• A key the TypedDict does not declare is what closed, extra_items or, when the class says neither, its bases make it: in a closed TypedDict it is reported, as unknown_key, and left out, and the value still comes back; with extra_items= it is checked as that type; in an open one it is kept.
• Each annotation is evaluated in the module of the class that wrote it, as the spec has it, where get_type_hints would read what a subclass inherited in the subclass's module.
• Required, NotRequired and ReadOnly are peeled wherever they are written; a NewType reads as the type it names, a type alias as the type it stands for, and a TypedDict or alias that holds itself as deep as the value goes.
• A number's type may carry bounds, in annotated-types' vocabulary as pydantic reads it: Gt, Ge, Lt, Le and Interval, at any depth, so tuple[Annotated[int, Ge(1)], ...] bounds each element. A value out of them is a problem, invalid_value, whose message says what the type admits and whose ctx holds the bounds. Annotated may also carry a note, a string or a Doc; any other metadata is a TypeError, since a constraint check does not read would be one it does not hold a value to.
• A union of TypedDicts that each require a key as a Literal of values of their own is read by the branch that key names, and its problems are that branch's. Otherwise a value is read by the first branch it fits with no problem -- among TypedDicts, the one declaring the most of its keys -- and failing that, reported by the branch with the fewest problems.

What comes back is a value of the TypedDict: arrays as tuples, and each object a new dict of the keys its type admits. Problems are values, not exceptions: ValidationProblem(loc, message, kind), with kind one of missing_key, invalid_type, invalid_value, unknown_key and invalid_json, so a caller that tolerates a key the TypedDict does not declare can tell it from a wrong value. Each carries its message's data, as pydantic's errors and zod's issues do: input, the JSON the value holds at loc, and ctx, what was expected there -- a type's bounds, or the values of a Literal.

json_schema writes what check reads as a JSON Schema, draft 2020-12, for a validator in another language, or an editor: the JSON Schema of the values check finds no problem with. A TypedDict is an object of its keys, closed or not as it says; a bound is JSON Schema's keyword for it, Interval(ge=0, le=9) a minimum and a maximum; a TypedDict or a type alias is written once, in $defs, under its name. JSON Schema takes a number with no fraction, 1.0, for an integer, where check wants 1:

from zarr_metadata.typed_json import json_schema

json_schema(GzipCodecConfiguration)
# {'$schema': 'https://json-schema.org/draft/2020-12/schema',
#  'type': 'object',
#  'properties': {'level': {'type': 'integer', 'minimum': 0, 'maximum': 9}},
#  'required': ['level'], 'additionalProperties': False}

check reads the shapes JSON takes and no others -- int, float for any number, bool, str, None, JSONValue, a Literal, tuple[T, ...] and tuple[T1, T2], a union, a TypedDict, Mapping[str, V], a NewType and a type alias -- and a TypedDict holding anything else is a TypeError naming the member, down to the TypedDict that holds it. So is one the spec itself refuses, such as a key both Required and NotRequired, and a generic TypedDict, since no runtime records what its type arguments bind in its bases' keys. A string annotation resolves only in its module, so a TypedDict written inside a function cannot name a class of that function. typing.TypedDict on Python 3.11 does not record a class's bases, so there a subclass evaluates what it inherits in its own module; typing_extensions.TypedDict records them on every version.

JSONSchema module-attribute

JSONSchema: TypeAlias = dict[str, JSONValue]

A JSON Schema, as the JSON object it is: arrays as lists, as validators take them.

JSONValue module-attribute

JSONValue = TypeAliasType(
    "JSONValue",
    int
    | float
    | bool
    | str
    | list["JSONValue"]
    | tuple["JSONValue", ...]
    | Mapping[str, "JSONValue"]
    | None,
)

A recursive type alias for JSON-encodable values.

Defined via TypeAliasType (rather than a plain TypeAlias) so the self-reference is a named recursion point that pydantic can resolve when building a TypeAdapter; a bare recursive TypeAlias raises PydanticUserError/RecursionError at validation time.

Loc module-attribute

Loc: TypeAlias = tuple[str | int, ...]

Where in a document a value sits: the keys and indices down to it.

ProblemKind module-attribute

ProblemKind = Literal[
    "missing_key",
    "invalid_type",
    "invalid_value",
    "invalid_json",
    "unknown_key",
]

Machine-readable classification of a ValidationProblem.

• missing_key: a required key (document key or store key) is absent.
• invalid_type: a value has the wrong structural type (e.g. a string where a mapping is required, a non-JSON-serializable object).
• invalid_value: a value has an acceptable type but an invalid content (e.g. zarr_format: 2 in a v3 document, order: "Q").
• invalid_json: bytes that do not decode as JSON.
• unknown_key: a key an object's type does not declare, where the type says it is closed, as a closed TypedDict does. Whether a Zarr configuration is closed is rarely said (zarr-developers/zarr-specs#270 has been open since 2023), and many readers refuse such a key. It gets a kind of its own so that a caller who tolerates it can tell it from a wrong value, and so that it never masks the other findings about the same object.

__all__ module-attribute

__all__ = [
    "JSONSchema",
    "JSONValue",
    "Loc",
    "ProblemKind",
    "ValidationProblem",
    "check",
    "json_schema",
]

ValidationProblem dataclass

A single problem found in a value: where it is, what is wrong, what kind of wrong, and the data the message is made of.

loc is the path from the root of what was judged to the offending value, e.g. ("codecs", 0, "name") in a document, and an empty loc refers to that root. kind classifies the failure mode for programmatic dispatch; message is the human-readable description.

input and ctx are what the message says, as data, as pydantic's ErrorDetails and zod's issues carry theirs. input is the JSON found at loc -- 12, for a gzip level of 12 -- and UNSET where nothing is there, as zod has it for a key that is missing (pydantic gives the object missing it), or where what is there is not JSON, which the message shows. It is the object the caller handed in, as pydantic's is, not a copy: a caller that changes its document afterwards changes what input shows. ctx is what was expected, where that is more than a type:

• gt, ge, lt and le: the bounds the value's type carries, as pydantic names them -- {"ge": 0, "le": 9} for a gzip level, whose type is Annotated[int, Interval(ge=0, le=9)] -- or a rule says.
• expected: the values of a closed set, as zod's values holds them, in the order the message lists them -- a Literal's, node_type's, zarr_format's.

Neither takes part in equality or the repr: a problem is the same problem when it is found at the same place and says the same thing. Every function that returns or raises problems fills input from the value its caller handed it, so a rule says only where a problem is.

Source code in src/zarr_metadata/_json.py

@dataclass(frozen=True, slots=True)
class ValidationProblem:
    """A single problem found in a value: where it is, what is wrong, what kind of wrong, and the data the message is made of.

    `loc` is the path from the root of what was judged to the offending
    value, e.g. `("codecs", 0, "name")` in a document, and an empty `loc`
    refers to that root.
    `kind` classifies the failure mode for programmatic dispatch; `message`
    is the human-readable description.

    `input` and `ctx` are what the message says, as data, as pydantic's
    `ErrorDetails` and zod's issues carry theirs. `input` is the JSON found
    at `loc` -- `12`, for a gzip `level` of 12 -- and `UNSET` where nothing
    is there, as zod has it for a key that is missing (pydantic gives the
    object missing it), or where what is there is not JSON, which the
    message shows. It is the object the caller handed in, as pydantic's
    is, not a copy: a caller that changes its document afterwards changes
    what `input` shows. `ctx` is what was expected, where that is more
    than a type:

    - `gt`, `ge`, `lt` and `le`: the bounds the value's type carries, as
      pydantic names them -- `{"ge": 0, "le": 9}` for a gzip `level`,
      whose type is `Annotated[int, Interval(ge=0, le=9)]` -- or a rule
      says.
    - `expected`: the values of a closed set, as zod's `values` holds
      them, in the order the message lists them -- a `Literal`'s,
      `node_type`'s, `zarr_format`'s.

    Neither takes part in equality or the repr: a problem is the same
    problem when it is found at the same place and says the same thing.
    Every function that returns or raises problems fills `input` from the
    value its caller handed it, so a rule says only where a problem is.
    """

    loc: tuple[str | int, ...]
    message: str
    kind: ProblemKind
    input: JSONValue | UNSET = dataclasses.field(
        default=UNSET, kw_only=True, compare=False, repr=False
    )
    ctx: Mapping[str, JSONValue] = dataclasses.field(
        default_factory=_no_ctx, kw_only=True, compare=False, repr=False
    )

    def __post_init__(self) -> None:
        # The runtime half of the annotations: a rule written without a type
        # checker, as an extension's may be, fails where it builds a problem
        # rather than reporting one at a location that is not one.
        loc = cast("object", self.loc)
        if not isinstance(loc, tuple) or not all(
            isinstance(part, str) or (isinstance(part, int) and not isinstance(part, bool))
            for part in cast("tuple[object, ...]", loc)
        ):
            msg = f"a ValidationProblem's loc is a tuple of keys and indices, got {loc!r}"
            raise TypeError(msg)
        message = cast("object", self.message)
        if not isinstance(message, str):
            msg = f"a ValidationProblem's message is a string, got {message!r}"
            raise TypeError(msg)
        kind = cast("object", self.kind)
        if not isinstance(kind, str) or kind not in get_args(ProblemKind):
            msg = f"a ValidationProblem's kind is one of {get_args(ProblemKind)!r}, got {kind!r}"
            raise TypeError(msg)
        ctx = cast("object", self.ctx)
        if isinstance(ctx, _Ctx):
            # A problem's own, checked when it was made: what `replace`
            # hands a copy.
            return
        if (
            not isinstance(ctx, Mapping)
            or not all(isinstance(key, str) for key in cast("Mapping[object, object]", ctx))
            or not is_json(dict(cast("Mapping[str, object]", ctx)))
        ):
            msg = f"a ValidationProblem's ctx is an object of JSON values, got {ctx!r}"
            raise TypeError(msg)
        # Held as a view of a copy of its own at every level, arrays as
        # tuples, so a raised error, a finished report, cannot be edited
        # through it, nor through what was handed in.
        held = cast(
            "dict[str, JSONValue]",
            copied(cast("JSONValue", arrays_to_tuples(dict(cast("Mapping[str, object]", ctx))))),
        )
        object.__setattr__(self, "ctx", _Ctx(held))

    def __str__(self) -> str:
        location = ".".join(str(part) for part in self.loc) if self.loc else "<root>"
        return f"{location}: {self.message}"

    def __reduce__(
        self,
    ) -> tuple[Callable[..., ValidationProblem], tuple[object, ...]]:
        # Pickled and copied as its constructor called again: the view
        # `ctx` is held as does not pickle, and the dict it views does.
        return (_problem, (self.loc, self.message, self.kind, self.input, dict(self.ctx)))

ctx class-attribute instance-attribute

ctx: Mapping[str, JSONValue] = dataclasses.field(
    default_factory=_no_ctx,
    kw_only=True,
    compare=False,
    repr=False,
)

input class-attribute instance-attribute

input: JSONValue | UNSET = dataclasses.field(
    default=UNSET, kw_only=True, compare=False, repr=False
)

kind instance-attribute

kind: ProblemKind

loc instance-attribute

loc: tuple[str | int, ...]

message instance-attribute

message: str

__init__

__init__(
    loc: tuple[str | int, ...],
    message: str,
    kind: ProblemKind,
    *,
    input: JSONValue | UNSET = UNSET,
    ctx: Mapping[str, JSONValue] = _no_ctx(),
) -> None

__post_init__

__post_init__() -> None

Source code in src/zarr_metadata/_json.py

def __post_init__(self) -> None:
    # The runtime half of the annotations: a rule written without a type
    # checker, as an extension's may be, fails where it builds a problem
    # rather than reporting one at a location that is not one.
    loc = cast("object", self.loc)
    if not isinstance(loc, tuple) or not all(
        isinstance(part, str) or (isinstance(part, int) and not isinstance(part, bool))
        for part in cast("tuple[object, ...]", loc)
    ):
        msg = f"a ValidationProblem's loc is a tuple of keys and indices, got {loc!r}"
        raise TypeError(msg)
    message = cast("object", self.message)
    if not isinstance(message, str):
        msg = f"a ValidationProblem's message is a string, got {message!r}"
        raise TypeError(msg)
    kind = cast("object", self.kind)
    if not isinstance(kind, str) or kind not in get_args(ProblemKind):
        msg = f"a ValidationProblem's kind is one of {get_args(ProblemKind)!r}, got {kind!r}"
        raise TypeError(msg)
    ctx = cast("object", self.ctx)
    if isinstance(ctx, _Ctx):
        # A problem's own, checked when it was made: what `replace`
        # hands a copy.
        return
    if (
        not isinstance(ctx, Mapping)
        or not all(isinstance(key, str) for key in cast("Mapping[object, object]", ctx))
        or not is_json(dict(cast("Mapping[str, object]", ctx)))
    ):
        msg = f"a ValidationProblem's ctx is an object of JSON values, got {ctx!r}"
        raise TypeError(msg)
    # Held as a view of a copy of its own at every level, arrays as
    # tuples, so a raised error, a finished report, cannot be edited
    # through it, nor through what was handed in.
    held = cast(
        "dict[str, JSONValue]",
        copied(cast("JSONValue", arrays_to_tuples(dict(cast("Mapping[str, object]", ctx))))),
    )
    object.__setattr__(self, "ctx", _Ctx(held))

__reduce__

__reduce__() -> tuple[
    Callable[..., ValidationProblem], tuple[object, ...]
]

Source code in src/zarr_metadata/_json.py

def __reduce__(
    self,
) -> tuple[Callable[..., ValidationProblem], tuple[object, ...]]:
    # Pickled and copied as its constructor called again: the view
    # `ctx` is held as does not pickle, and the dict it views does.
    return (_problem, (self.loc, self.message, self.kind, self.input, dict(self.ctx)))

__str__

__str__() -> str

Source code in src/zarr_metadata/_json.py

def __str__(self) -> str:
    location = ".".join(str(part) for part in self.loc) if self.loc else "<root>"
    return f"{location}: {self.message}"

check

check(
    value: object, shape: type[T], loc: Loc = ()
) -> tuple[T | None, tuple[ValidationProblem, ...]]

value type-checked as shape, a TypedDict: a value of it or None, and every problem.

value is refined to JSON first -- arrays as tuples, string keys, finite floats -- and then checked member by member, each problem located under loc. What comes back holds what shape admits and nothing else: a key a closed TypedDict does not declare is reported, as unknown_key, and left out, and the value still comes back. Anything else wrong and it does not. TypeError for a shape that is not a TypedDict, or holds something no parser reads.

Source code in src/zarr_metadata/_typed_json.py

def check(
    value: object, shape: type[T], loc: Loc = ()
) -> tuple[T | None, tuple[ValidationProblem, ...]]:
    """`value` type-checked as `shape`, a TypedDict: a value of it or None, and every problem.

    `value` is refined to JSON first -- arrays as tuples, string keys,
    finite floats -- and then checked member by member, each problem
    located under `loc`. What comes back holds what `shape` admits and
    nothing else: a key a closed TypedDict does not declare is reported,
    as `unknown_key`, and left out, and the value still comes back.
    Anything else wrong and it does not. `TypeError` for a `shape` that is
    not a TypedDict, or holds something no parser reads.
    """
    if not is_typeddict(shape):
        msg = f"{shape!r} is not a TypedDict"
        raise TypeError(msg)
    refined, problems = refine_json(value, loc)
    if len(problems) != 0:
        return None, with_input(problems, value, loc)
    typed, found = _checker(shape)(refined, loc)
    readable = all(problem.kind == "unknown_key" for problem in found)
    return (cast("T", typed) if readable else None), with_input(found, value, loc)

json_schema

json_schema(shape: type) -> JSONSchema

The JSON Schema of the JSON check finds no problem with as shape, a TypedDict.

Draft 2020-12, as a JSON object: arrays as lists, and $schema first. A TypedDict is an object of its keys, those it requires, and what any other key may hold -- nothing, in a closed one; a bound is the keyword JSON Schema has for it, Ge(0) a minimum; a Doc is the description, which is all that says one, as zod writes only what .describe() said: a docstring is written for Python's readers; a Literal is its values; a union is anyOf its branches. A TypedDict or type alias is written once in $defs, under its name, and referred to wherever it occurs, but for shape itself, which is written in place unless it holds itself.

One difference is JSON Schema's own: it takes a number with no fraction, 1.0, for an integer, where check wants 1. TypeError for a shape that is not a TypedDict, or holds something no parser reads, as check raises it.

Source code in src/zarr_metadata/_typed_json.py

def json_schema(shape: type) -> JSONSchema:
    """The JSON Schema of the JSON `check` finds no problem with as `shape`, a TypedDict.

    Draft 2020-12, as a JSON object: arrays as lists, and `$schema`
    first. A TypedDict is an object of its keys, those it requires, and
    what any other key may hold -- nothing, in a closed one; a bound is
    the keyword JSON Schema has for it, `Ge(0)` a `minimum`; a `Doc` is
    the `description`, which is all that says one, as zod writes only
    what `.describe()` said: a docstring is written for Python's readers;
    a `Literal` is its values; a union is `anyOf` its branches. A
    TypedDict or type alias is written once in `$defs`, under its name,
    and referred to wherever it occurs, but for `shape` itself, which is
    written in place unless it holds itself.

    One difference is JSON Schema's own: it takes a number with no
    fraction, `1.0`, for an integer, where `check` wants `1`. `TypeError`
    for a `shape` that is not a TypedDict, or holds something no parser
    reads, as `check` raises it.
    """
    if not is_typeddict(shape):
        msg = f"{shape!r} is not a TypedDict"
        raise TypeError(msg)
    _checker(shape)
    schemas = Schemas()
    return schemas.document(schemas.of(shape))