https://flyte.org logo
#ask-the-community
Title
# ask-the-community
l

Laura Lin

11/29/2023, 12:52 AM
I know resource based overrides work with
map_task
but the newer feature of overriding
container_image
doesn't seem to work with map_tasks. Here's a minimal example where its still using the default container image and not the overwritten one. It does work with regular tasks, just not map_task?
Copy code
@task
def my_mappable_task(a: int) -> typing.Optional[str]:
    return str(a)

@workflow
def my_map_wf(x: typing.List[int]) -> typing.List[typing.Optional[str]]:
    return map_task(
        my_mappable_task,
        concurrency=10,
        min_success_ratio=0.75,
    )(a=x).with_overrides(container_image="random:image")
cc @Kevin Su since he added the original feature https://github.com/flyteorg/flytekit/commit/eafcc820303367749e63edc62190b9153fd6be5e
we are running 1.9.1 flyte stack if that matters
g

Gopal Vashishtha

11/29/2023, 6:28 AM
k

Kevin Su

11/29/2023, 6:23 PM
Thank you for opening an issue. @Laura Lin I’ll create a PR later today
g

Gopal Vashishtha

11/30/2023, 4:35 AM
Thanks for giving this your attention, Kevin!
k

Kevin Su

11/30/2023, 8:27 AM
g

Gopal Vashishtha

12/01/2023, 12:34 AM
Looking forward to using this feature as soon as it gets to a release, thanks for the quick turnaround Kevin!
k

Kevin Su

12/01/2023, 12:43 AM
@Gopal Vashishtha could you help me test it on your side?
Copy code
pip install git+<https://github.com/flyteorg/flytekit.git@7a30eb575e38a75a6311ab534ff8806304a1dd93>
g

Gopal Vashishtha

12/01/2023, 1:26 AM
Just tested, yes it works!
k

Kevin Su

12/01/2023, 1:40 AM
nice, thank you. will merge it soon
g

Gopal Vashishtha

12/01/2023, 11:09 PM
Saw the change went in, that's awesome! Is it possible to remove the upper bound on
fsspec
that you introduced in this PR? We're getting a dependency conflict with another library that we are unable to resolve
k

Kevin Su

12/01/2023, 11:16 PM
which version of fsspec you are using
IIRC, there are some issues in fsspec 2023.10+, maybe they already fix it in the latest version. anyway, we can definitely make it less restrictive. like something !fsspec==2023.10.
g

Gopal Vashishtha

12/01/2023, 11:28 PM
we're actually not using fsspec at all
we're using s3fs which pins to 2023.10.0: https://github.com/fsspec/s3fs/blob/main/requirements.txt#L2