https://flyte.org logo
Join Slack
Powered by
<@U01DYLVUNJE> could you please take a look at <ht...
# contribute
c
@broad-monitor-993 could you please take a look at https://github.com/unionai-oss/pandera/pull/2448? Not necessarily a full review, mainly that the direction is acceptable. I added a couple of ADRs that explain the motivation (TL;DR support static analysis by 
ty
)
b
will take a look! would you mind rebasing onto main? I think the test failure is fixed in 
main
c
Sure! I was about to do that to add a signed off message
b
the change makes a lot of sense, I’ve been thinking of this feature for a while. One naming discussion first though: with pandera supporting 
xarray
and very soon tensordict, I think 
pandera.typing.Field
would be more future-proof. Just thinking through an alternative… but could we do something akin to Pydantic and use 
Annotated
? pandera--2448.org.readthedocs.build/en/2448/dataframe_models.html#… • 
T
required column, non-null by default • 
T | None
required, nullable • 
Annotated[T, pa.Field(required=False)]
optional column
even though it’s more verbose, I like the 
Annotated
solution somehow
c
• Not opposed to 
Field
• As a matter of fact, my two references for the implementation were SQLAlchemy and pydantic, so the current impl supposedly combines the best of both worlds. I can change it to 
Annotated
.
b
let’s go with 
Annotated
for now. If you’re still feeling the pain we can revisit adding 
pandera.typing.Field
👍 1
I think the 
required
kwarg needs to be added to 
pa.Field
c
I updated the nullability vs presence annotations as you requested. However, the main motivation for the PR, which is to stop 
ty
from emitting false positives cannot be addressed without a 
pandera.typing.Field
as a descriptor layer. How about:
Copy code
from typing import Annotated
  from pandera.typing import Field as TypedField
  import pandera.polars as pa

  class Schema(pa.DataFrameModel):
      values: Annotated[TypedField[pl.List], pa.Field()]
      nullable: Annotated[TypedField[int | None], pa.Field()]
      optional: Annotated[TypedField[int], pa.Field(required=False)]
The responsibilities stay separate: • 
pandera.typing.Field[T]
tells 
ty
that class-level access is 
str
. • 
Annotated[...]
remains the standard metadata carrier. • Runtime 
pa.Field(...)
still handles checks, aliases, 
required
, etc. • Pandera unwraps 
TypedField[T]
back to 
T
when building the schema.
b
that’s fair… if we’re going to have 
pandera.typing.Field
, how about something more concise (going back to your original idea):
Copy code
from pandera.typing import FieldType
import pandera.polars as pa

class Schema(pa.DataFrameModel):
      values: FieldType[pl.List, pa.Field()]
      nullable_field: FieldType[int | None, pa.Field()]
      optional_field: FieldType[int] | None
then for runtime checks they can just do:
Copy code
field: FieldType[T] = pa.Field(...)
c
PR updated
b
looks like some tests are failing
c
I'll take care of those + missing test coverage. In the meantime - what do you say about the ADRs? I'd like to change their status to approved or remove them from the PR if this is not something you want to start collecting in the repo.
b
that’s okay, let’s throw them in github.com/unionai-oss/pandera/…/specs
c
new subfolder?
b
it already exists, see the link
c
Yes I meant a new subfolder under the linked one. ADRs aren't exactly specs though, they explain the "why" more than the "what"
b
I see… yeah I think it’s fine as is in the PR
👍 1
c
Hi Niels, I rebased my PR on top of your mypy bump, and everything is green except one job that fails due to what seems to be spark flakiness (
ConnectionRefusedError: [Errno 61] Connection refused
). What can I do about that?
b
that’s okay, I can take care of that in a follow-up PR
just to clarify on this:
is Annotated no longer supported? This would be a breaking change and that wouldn’t be a good thing 😞
c
I think it's just the best practices. • Retained 
Annotated
compatibility for existing dtype metadata and field declarations, while making 
FieldType
the primary descriptor contract.
👍 1
b
it would be nice to show at least one example of the 
Annotated
syntax (and preserve mention of it in the documentation)
c
Sure, will add
Copy code
You can also embed field metadata with {data}`typing.Annotated`:

```{code-cell} python
from typing import Annotated


class AnnotatedSchema(pa.DataFrameModel):
    value: Annotated[int, pa.Field(gt=0, description="A positive value")]


AnnotatedSchema.to_schema().columns["value"].description
This syntax is valid at runtime and may be convenient when you prefer standard typing constructs. Its trade-off is that 
Annotated
preserves the underlying 
int
annotation for static analysis; it does not provide the class-level 
str
descriptor contract. If model fields are not passed to string-taking APIs, this limitation may not matter. If 
ty
is part of your tooling, use 
FieldType
instead.```
2 Views
Open in Slack
PreviousNext
Powered by