https://flyte.org logo
#ask-the-community
Title
# ask-the-community
p

Peter Klingelhofer

03/19/2024, 2:54 PM
Good morning, I'm attempting to use
map_task
or
@dynamic
to run tasks in parallel depending on the results of a database query (one task for every row that is returned). However, the pods spun up for both the
map_task
and just looping over the entries in a dynamic workflow use an outdated Docker image (the first one we ever used, 0.0.1). We specified this image in the pod_template and that's worked fine (it spins up with the right Docker image, whatever is in the pod_template). How can we get dynamic / mapped tasks to use a specified pod_template? To be clear,
kpi_wf
is a task with a specified pod_template (which it has been using properly in our other use-cases), but it seems to be ignoring the pod_template's container_image for this parallelization attempt (I saw this PR which made me think this should be supported: https://github.com/flyteorg/flytekit/pull/2088):
Copy code
@dynamic(pod_template=pod_template)
@logger.catch
def kpi_backfill() -> str:
    entries = fetch_backfill_entries()
    start_times, end_times, site_ids, _ = zip(*entries)
    mapped_kpi_init = map_task(
        kpi_init.
        concurrency=10,
        min_success_ratio=0.6,
    )

    mapped_kpi_init(start_timestamp=list(start_times), end_timestamp=list(end_times), site_id=list(site_ids))

    return result

# for context, kpi_init declaration:
@task(pod_template=pod_template, requests=Resources(cpu="2", mem="63Gi"), limits=Resources(cpu="2", mem="63Gi"))
def kpi_init(start_timestamp: datetime, end_timestamp: datetime, site_id: str):
k

Ketan (kumare3)

03/19/2024, 4:36 PM
the PR is slightly different. What it means is that the mapped task (the underlying task of map_task) can have pod_templates. So you can create bespoke pods in a map job. But, if i understand correctly you are saying that map_task ignores the pod_template when using dynamic task?
p

Peter Klingelhofer

03/19/2024, 4:40 PM
The task has a pod_template which defines various things like a Persistent Volume Container as well as Resources. The container that is spun up uses the custom defined PVC as well as the specific Resource overrides I've provided for the task, it's just that it's using the
0.0.1
tag instead of the tag that is defined in the pod_template. What's interesting is the dynamic task itself that is spun up when I run the workflow uses the correct image (so I know the pod template is working correctly there, it's been solid), but it's just as we use the mapped task the tag is ignored and it uses
0.0.1
instead... 🤔
Okay, figured it out! It seems that tasks in the dynamic workflow pull from whatever secret is stored on K8s, whereas non-dynamic tasks use whatever I have in the .env file locally for registration, so updating the secret in K8 fixed this (the secret itself was stale and was still on
0.0.1
, it just hadn't needed updating because the tasks I had been registering weren't using it anyway). Thanks @Ketan (kumare3), the context you provided and follow-up questions helped me get to the bottom of this.