Skip to content

Comment on Loading Pydantic models from JSON without running out of memoryparent

Comments

That's an interesting idea. It's possible there's a Dict[str,Any] in there. And yeah, my assumption was that it tried everything recursively, but I just wasn't seeing that, and my LLM council said that it did not. But I'll check for a Dict[str,Any]. Unfortunately, I don't have a minimal example, but making one should be my next step.

One thing to watch out for while you debug is that the default 'smart' mode for union discrimination can be very unintuitive. As you can see in this example, an int vs a string can cause a different model to be chosen two layers up even though both are valid. You may have perfectly valid uses of Dict within your model that are being chosen in error because they result in less type coercion. left_to_right mode (or ideally discriminated unions if your data has easy discriminators) will be much more consistent.

    >>> class A(BaseModel):
    >>>     a: int
    >>> class B(BaseModel):
    >>>     b: A
    >>> class C(BaseModel):
    >>>     c: B | Dict[str, Any]

    >>> C.model_validate({'c':{'b':{'a':1}}})
    
    C(c=B(b=A(a=1)))

    >>> C.model_validate({'c':{'b':{'a':"1"}}})

    C(c={'b': {'a': '1'}})

    >>> class C(BaseModel):
    >>>     c: B | Dict[str, Any] = Field(union_mode='left_to_right')
    
    >>> C.model_validate({'c':{'b':{'a':"1"}}})

    C(c=B(b=A(a=1)))
AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.