https://flyte.org logo
#ask-the-community
Title
# ask-the-community
c

Clemente Cuevas

12/27/2023, 9:07 PM
Hi, I'm trying to set up what I hoped would be a simple test for a flyte workflow locally (in the sandbox and using a local run) but has been causing me some trouble. I moved my setup to this git repository: https://github.com/cuevasclemente/flyte-test My dynamic workflow first populates a new bucket in minio, then tries to go through each subpath in the bucket to look for objects (using the minio client and the minio service that comes with the sandbox) . I'm using a dynamic workflow and tasks. It's possible this is just a really bad pattern that I'm implementing here and there might be a much more correct way of doing this. I'm running into two different problems: 1. If i run the workflow locally, it hangs indefinitely (I left it running for hours) with the workflow receiving the list of objects in the bucket. The top level list of objects is passed back as a list of strings to the workflow and there are only 10 55 strings in that list. I get no error message since the local run just seems to take a very long time to get the strings back from the task (is 55 strings in a list too much to pass back as the results from a task?) 2. When I register the workflow (I need to use a custom image to include the minio requirement), I get no real error here either, I just get the following trace:
Copy code
[1/1] currentAttempt done. Last Error: USER::z-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=minio/20231227/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ccb01873733b3028b82c25e1b9fdcf8db8faeef0f45281047d154a4cf15cb7f9', 'amz-sdk-invocation-id': b'9c141759-d882-43c0-9e8a-30aa729c7e5f', 'amz-sdk-request': b'attempt=1'}> -  Line:163
parsers.py: 2023-12-27 03:37:25 - DEBUG - Response headers: HTTPHeaderDict({'accept-ranges': 'bytes', 'content-length': '0', 'content-security-policy': 'block-all-mixed-content', 'content-type': 'binary/octet-stream', 'etag': '"d41d8cd98f00b204e9800998ecf8427e"', 'last-modified': 'Wed, 27 Dec 2023 03:37:12 GMT', 'server': 'MinIO', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'vary': 'Accept-Encoding', 'x-amz-request-id': '17A4930D803A5853', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'date': 'Wed, 27 Dec 2023 03:37:25 GMT'}) -  Line:240
parsers.py: 2023-12-27 03:37:25 - DEBUG - Response body:
<StreamingBody at 0xffff8dc0d100 for ClientResponse at 0xffff8e8865d0> -  Line:241
hooks.py: 2023-12-27 03:37:25 - DEBUG - Event needs-retry.s3.GetObject: calling handler <aiobotocore.retryhandler.AioRetryHandler object at 0xffff8e8302d0> -  Line:63
retryhandler.py: 2023-12-27 03:37:25 - DEBUG - No retry needed. -  Line:111
hooks.py: 2023-12-27 03:37:25 - DEBUG - Event needs-retry.s3.GetObject: calling handler <bound method AioS3RegionRedirectorv2.redirect_from_error of <aiobotocore.utils.AioS3RegionRedirectorv2 object at 0xffff8e831150>> -  Line:63
test_dynamic_workflow.py: 2023-12-27 03:37:25 - DEBUG - Filling filesystem with data -  Line:102
test_dynamic_workflow.py: 2023-12-27 03:37:25 - DEBUG - Done filling filesystem -  Line:104
test_dynamic_workflow.py: 2023-12-27 03:37:25 - DEBUG - going into paginate bucket -  Line:109
Getting <s3://my-s3-bucket/flytesnacks/development/MHDQUGBBWOFSZ65ELNSYA5GZVU======/fastb4c5db4025521baba89c4623e426d902.tar.gz|s3://my-s3-bucket/flytesnacks/development/MHDQUGBBWOFSZ65ELNSYA5GZVU======/fastb4c5db4025521baba89c4623e426d902.tar.gz> to /root/
Not sure what is happening in this case but it looks similar to what happened to @Denis Shvetsov here: https://flyte-org.slack.com/archives/CP2HDHKE1/p1700229165926579?thread_ts=1700149252.292709&cid=CP2HDHKE1. I don't see anything different looking at the pod logs v. the Flyte UI logs. Any help would be appreciated! Thanks!
Here's the stacktrace after running the app locally for forty minutes (in case that's helpful)
I was able to get this to work locally, the fundamental problem with this code was that I was using a
dynamic
workflow when I really needed to use an
eager
workflow in order to represent what is essentially a recursive workflow. Now that I've got it to work locally, I'll try to get it to figure out how to implement it in a dynamic workflow since it should just be a recursive implementation.
s

Samhita Alla

12/28/2023, 5:03 AM
good to know that you got it working with eager workflow. in a dynamic workflow, the task outputs are promises, which means they aren't immediately materialized in the workflow. you need to send the outputs of a task to a task again to materialize the values.
c

Clemente Cuevas

12/28/2023, 5:23 AM
I also got it working with a dynamic workflow just now. I think I just had a lot of challenges in just understanding what is or isn't allowed in certain contexts. Like, I found that I couldn't evaluate a
task
inside a
conditional
expression (inside the
conditional.if
or
conditional.elif
, which is maybe obvious but the error that you get for doing this is very unclear and will end up just treating the task output as
None
.
s

Samhita Alla

12/28/2023, 5:27 AM
hmm, understood. would you mind creating a docs issue? [flyte-docs]
c

Clemente Cuevas

12/28/2023, 5:55 AM
Sure thing. I'll try to see if there are more things that I ran into that could be helpful to make issues around.
k

Ketan (kumare3)

12/28/2023, 5:07 PM
Sorry for the trouble @Clemente Cuevas we would Love to help folks understand this better
4 Views