I got: ```{"asctime": "2023-05-16 14:16:33,972", "name": "flytekit", "levelname": "ERROR", "message"...

salmon-refrigerator-32115

05/16/2023, 9:25 PM

I got:

Copy code

{
  "asctime": "2023-05-16 14:16:33,972",
  "name": "flytekit",
  "levelname": "ERROR",
  "message": "Error from command '['aws', 's3', 'cp', '--acl', 'bucket-owner-full-control', '/var/folders/xn/j7gcmr5j12b7jy0nm2kfykhm0000gp/T/flytex7wgfw1z/control_plane_metadata/local_flytekit/2461a0641858e66c16dc8214fffdb4b7', '<s3://my-s3-bucket/data/c36474e9b9ca42edb35ba64e2d17bab1/2461a0641858e66c16dc8214fffdb4b7']'>:\nb'upload failed: ../../../../var/folders/xn/j7gcmr5j12b7jy0nm2kfykhm0000gp/T/flytex7wgfw1z/control_plane_metadata/local_flytekit/2461a0641858e66c16dc8214fffdb4b7 to <s3://my-s3-bucket/data/c36474e9b9ca42edb35ba64e2d17bab1/2461a0641858e66c16dc8214fffdb4b7> An error occurred (AccessDenied) when calling the PutObject operation: Access Denied\\n'\n"
}

glamorous-carpet-83516

05/16/2023, 9:26 PM

which version of flytekit are you using? IIRC, there are some issues with uploading file in flytekit 1.4

salmon-refrigerator-32115

05/16/2023, 9:30 PM

@glamorous-carpet-83516, we are using flytekit==1.2.4 CC: @delightful-lion-57360

glamorous-carpet-83516

05/16/2023, 9:31 PM

only this workflow failed? or all other workflows failed as well.

salmon-refrigerator-32115

05/16/2023, 9:32 PM

Only this workflow is having a input param of type dict. All others have simple str or int as input, therefore no need to upload as files.

glamorous-carpet-83516

05/16/2023, 11:40 PM

just test it. this is a bug in flytekit v1.2.4, but there is another issue in flytekit v1.6. will create a pr shortly

tall-lock-23197

05/17/2023, 4:38 AM

@glamorous-carpet-83516, I believe I'm looking at the same error as well. I tested it on flytekit 1.5.0 https://flyte-org.slack.com/archives/CP2HDHKE1/p1684297314863949?thread_ts=1683861857.343639&cid=CP2HDHKE1

glamorous-carpet-83516

05/17/2023, 6:35 PM

I fixes it here, but you can’t use default input for now since

type_engine

doesn’t use dataproxy to upload data. this is another issue, we’ll fix it.

❤️ 1

salmon-refrigerator-32115

05/17/2023, 7:37 PM

@glamorous-carpet-83516 @tall-lock-23197, thank you very much!

salmon-refrigerator-32115

05/30/2023, 8:58 PM

Hi @glamorous-carpet-83516, Which flyte version has this feature working? I need this feature. Therefore I will re-test if you’ve got it released. Thanks!

glamorous-carpet-83516

05/30/2023, 10:00 PM

flytekit==1.6.2

salmon-refrigerator-32115

05/30/2023, 10:02 PM

Thanks @glamorous-carpet-83516, Right now I am using:

Copy code

flytekit==1.2.4
flytekitplugins-snowflake==1.2.4
flytekitplugins-spark==1.2.4
flytekitplugins-polars==1.2.4
pandas~=1.5.0
...

salmon-refrigerator-32115

05/30/2023, 10:03 PM

Do I need to upgrade the version of flytekitplugins to the same as flytekit version? And do I need to change the pandas version?

glamorous-carpet-83516

05/30/2023, 10:08 PM

1.2.4 is a bit old, I suggest upgrade them as well.

glamorous-carpet-83516

05/30/2023, 10:09 PM

but I think it should work as well if you don’t upgrade them.

salmon-refrigerator-32115

05/30/2023, 10:10 PM

Thanks Kevin.

glamorous-carpet-83516

05/30/2023, 10:11 PM

and make sure to use flyteidl==1.5.8. The latest flytekit is not compatible with latest flyteidl (1.5.9)

salmon-refrigerator-32115

05/30/2023, 10:17 PM

Thanks. @glamorous-carpet-83516, The reason I chose version 1.2.4 of flyte is because I had to build a custom image with spark. So I used this as part of my dependency req txt files. https://github.com/flyteorg/flytesnacks/blob/master/cookbook/integrations/kubernetes/k8s_spark/requirements.txt

salmon-refrigerator-32115

05/30/2023, 10:20 PM

I realize if I upgrade flyte version, I need to figure out the upgraded k8s_spark requirements.txt. Otherwise, I cannot upgrade flyte along without breaking the dependencies in my k8s_spark/requirements.txt

salmon-refrigerator-32115

05/30/2023, 10:21 PM

Could you point me to where the latest k8s_spark req txt file is?

glamorous-carpet-83516

05/30/2023, 10:24 PM

Try to use imageSpec, it’s a new feature in flytekit. you don’t need to build an image for spark anymore. https://github.com/flyteorg/flytekit/pull/1616

glamorous-carpet-83516

05/30/2023, 10:25 PM

Copy code

spark_image = ImageSpec(registry="<docker_id>", packages=["pandas", "numpy"])

glamorous-carpet-83516

05/30/2023, 10:25 PM

https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-spark/requirements.txt

salmon-refrigerator-32115

05/30/2023, 10:45 PM

@glamorous-carpet-83516, I need to upgrade the flyte remote server to also have flytekit upgraded to v1.6.2, not only locally, is that correct?

salmon-refrigerator-32115

05/30/2023, 10:57 PM

@glamorous-carpet-83516, just to clarify regarding not needing to build an image for spark. I need to build a custom image because I have custom dependencies. I do pyflyte run --remote --image with my custom image. And that’s why I built custom image with spark. Do you think I can remove the spark from my custom image now since I can use ImageSpec to run spark tasks?

glamorous-carpet-83516

05/30/2023, 11:12 PM

yes, you can use imageSpec for the spark task. You can still install custom dependencies in ImageSpec, but you don’t need to install spark dependencies yourself. imageSpec will use spark-base-image, and install you custom packages on top of it

glamorous-carpet-83516

05/30/2023, 11:12 PM

https://flyte--988.org.readthedocs.build/projects/cookbook/en/988/auto/core/containerization/image_spec_example.html

glamorous-carpet-83516

05/30/2023, 11:13 PM

here is a doc in flytesnacks, we’ll merge the doc soon

salmon-refrigerator-32115

05/30/2023, 11:17 PM

Thanks @glamorous-carpet-83516. You told me to upgrade to flytekit 1.6.2 for a bug fix. Does it mean I need to upgrade flytekit in the flyte remote server also to 1.6.2, not only locally?

glamorous-carpet-83516

05/30/2023, 11:35 PM

flyte remote server

you mean flyteadmin, flytepropeller?

glamorous-carpet-83516

05/30/2023, 11:35 PM

those deployments don’t need to be upgraded

salmon-refrigerator-32115

05/30/2023, 11:37 PM

You mean flytekit, flyteidl, etc. are not installed on the flyter remote server?

salmon-refrigerator-32115

05/30/2023, 11:37 PM

I am not the flyte remote server administrator, therefore I am not aware of that.

salmon-refrigerator-32115

05/30/2023, 11:38 PM

@glamorous-carpet-83516

glamorous-carpet-83516

05/30/2023, 11:42 PM

flytekit is only used in the task pods and local (for local execution). The remote server is written by golang, which is in a different repo.

salmon-refrigerator-32115

05/31/2023, 7:21 PM

Thanks @glamorous-carpet-83516. I have custom dependencies that is private to my organization and is in a private repository (in one of two repos, e.g. https://maven.homebox.com/repository/max-pypi-snapshots, & https://maven.homebox.com/repository/max-pypi-releases). How do I add them in the ImageSpec?

glamorous-carpet-83516

05/31/2023, 7:58 PM

ImageSpec(packages=[“max-pypi”, ….])

salmon-refrigerator-32115

06/01/2023, 5:15 PM

@glamorous-carpet-83516, how will ImageSpec know where to find packages from max-pypi? Where do I specify the repo url https://maven.homebox.com/repository?

glamorous-carpet-83516

06/01/2023, 6:10 PM

sorry, if you are trying to download it from others pypi, it’s not supported. I’ll add it to imageSpec

151 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.