https://flyte.org logo
Join Slack
Powered by
New RunLLM response generated: >>>*# Question* Hi...
# runllm-flyte-trial
a
New RunLLM response generated: >>>*# Question* Hi Folks! Hope you are doing well. I hoped in because I have a quick Q in regards to a demo I'm putting together for my team. Whenever I use any map task, I've been getting an error: currentAttempt done. Last Error: UNKNOWN::Outputs not generated by task execution here's the code I've been working with. I've blocked out the actual functionality since the error seems to persist no matter what is done. the code was working entirely until the process task was changed to a map task. Due to enterprise constraints we're running flyte "the hard way" with onprem kube, but wondering if anyone has had a similar bug and knows what they did to resolve!
Copy code
pt = PodTemplate(
        pod_spec=V1PodSpec(
            containers=[
                V1Container(
                    name="primary",
                    volume_mounts=[
                        V1VolumeMount(
                            name="wip",
                            mount_path="/mnt/WIP/",
                            read_only = True,
                        ),
                        V1VolumeMount(
                            name="models",
                            mount_path="/models/",
                            read_only = True,
                        ),
                        V1VolumeMount(
                            name="files",
                            mount_path="/mnt/files/",
                            read_only = False,
                        )
                    ],
                    security_context=V1PodSecurityContext(run_as_user= 0,),
                ),
            ],
            volumes=[
                V1Volume(
                    name="wip",
                    host_path= V1HostPathVolumeSource("/mnt/wip/"),
                ),
                V1Volume(
                    name="models",
                    host_path= V1HostPathVolumeSource("/models/"),
                
                ),
                  V1Volume(
                    name="files",
                    host_path= V1HostPathVolumeSource("/mnt/files/"),
                
                )
            ],
        ))

@task(container_image=img)
def collectData(batch : str) -> list[str]:
    #extracts a list of paths to seek

@task(pod_template=pt,
    requests=Resources(
        mem="2G",
    ),container_image=haifluimg)
def process(plate : str) -> np.ndarray:
    #applies ML ops to images and generates an array of data, writes to a disk

@task(container_image=img)
def coalesce(arrays : list[np.ndarray]) -> np.ndarray:
    return numpy.concatenate( arrays, axis=0 )


@workflow
def mlPipeline(batch: str = "98723") -> np.ndarray:
    plates = collectData(batch = batch)
    arrays = map_task(process)(plate=plates)
    return coalesce(arrays = arrays)
# Answer
>>The error "UNKNOWN::Outputs not generated by task execution" typically indicates that the task was reported as successfully completed, but FlytePropeller could not find the outputs file. This issue can arise due to several reasons, including misconfigurations in the environment or issues with the task execution itself.
Here are a few steps to troubleshoot and resolve this issue: 1. Check Task Logs: Ensure that the task logs do not show any errors or exceptions. You can use 
kubectl logs
to check the logs of the pods running the tasks. 2. Verify Output Paths: Ensure that the output paths are correctly configured and accessible. Sometimes, the issue might be due to incorrect paths or permissions. 3. Use Experimental Map Task: If you are not already using it, try using the 
experimental.map_task
as it might have better handling for certain edge cases. 4. Check Blobstore Configuration: Ensure that the blobstore (e.g., S3, GCS) where Flyte writes data is correctly configured and accessible. Verify that the output files are being written to the expected location. 5. Pod Security Context: Ensure that the security context and permissions are correctly set for the pods to write to the specified volumes. Here is a simplified version of your code using `experimental.map_task`:
Copy code
from flytekit import experimental, task, workflow, Resources
from kubernetes.client import V1PodSpec, V1Container, V1VolumeMount, V1Volume, V1HostPathVolumeSource, V1PodSecurityContext

pt = V1PodSpec(
    containers=[
        V1Container(
            name="primary",
            volume_mounts=[
                V1VolumeMount(name="wip", mount_path="/mnt/WIP/", read_only=True),
                V1VolumeMount(name="models", mount_path="/models/", read_only=True),
                V1VolumeMount(name="files", mount_path="/mnt/files/", read_only=False)
            ],
            security_context=V1PodSecurityContext(run_as_user=0),
        ),
    ],
    volumes=[
        V1Volume(name="wip", host_path=V1HostPathVolumeSource("/mnt/wip/")),
        V1Volume(name="models", host_path=V1HostPathVolumeSource("/models/")),
        V1Volume(name="files", host_path=V1HostPathVolumeSource("/mnt/files/"))
    ],
)

@task(container_image="img")
def collectData(batch: str) -> list[str]:
    # extracts a list of paths to seek
    pass

@task(pod_template=pt, requests=Resources(mem="2G"), container_image="haifluimg")
def process(plate: str) -> np.ndarray:
    # applies ML ops to images and generates an array of data, writes to a disk
    pass

@task(container_image="img")
def coalesce(arrays: list[np.ndarray]) -> np.ndarray:
    return np.concatenate(arrays, axis=0)

@workflow
def mlPipeline(batch: str = "98723") -> np.ndarray:
    plates = collectData(batch=batch)
    arrays = experimental.map_task(process)(plate=plates)
    return coalesce(arrays=arrays)
>>If the issue persists, please provide more details about the error logs or any additional context that might help in diagnosing the problem.
Would you like more detailed information on any specific part of this process? Sources: - Flyte GitHub Issue #4379 - Flyte GitHub Issue #5180 - Flyte GitHub Issue #5349 - Flyte Discuss Post Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1721851932.052019 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
2 Views
Open in Slack
PreviousNext
Powered by