https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
f

Frank Shen

05/16/2023, 9:25 PM
I got:
Copy code
{
  "asctime": "2023-05-16 14:16:33,972",
  "name": "flytekit",
  "levelname": "ERROR",
  "message": "Error from command '['aws', 's3', 'cp', '--acl', 'bucket-owner-full-control', '/var/folders/xn/j7gcmr5j12b7jy0nm2kfykhm0000gp/T/flytex7wgfw1z/control_plane_metadata/local_flytekit/2461a0641858e66c16dc8214fffdb4b7', '<s3://my-s3-bucket/data/c36474e9b9ca42edb35ba64e2d17bab1/2461a0641858e66c16dc8214fffdb4b7']'>:\nb'upload failed: ../../../../var/folders/xn/j7gcmr5j12b7jy0nm2kfykhm0000gp/T/flytex7wgfw1z/control_plane_metadata/local_flytekit/2461a0641858e66c16dc8214fffdb4b7 to <s3://my-s3-bucket/data/c36474e9b9ca42edb35ba64e2d17bab1/2461a0641858e66c16dc8214fffdb4b7> An error occurred (AccessDenied) when calling the PutObject operation: Access Denied\\n'\n"
}
k

Kevin Su

05/16/2023, 9:26 PM
which version of flytekit are you using? IIRC, there are some issues with uploading file in flytekit 1.4
f

Frank Shen

05/16/2023, 9:30 PM
@Kevin Su, we are using flytekit==1.2.4 CC: @karthikraj
k

Kevin Su

05/16/2023, 9:31 PM
only this workflow failed? or all other workflows failed as well.
f

Frank Shen

05/16/2023, 9:32 PM
Only this workflow is having a input param of type dict. All others have simple str or int as input, therefore no need to upload as files.
k

Kevin Su

05/16/2023, 11:40 PM
just test it. this is a bug in flytekit v1.2.4, but there is another issue in flytekit v1.6. will create a pr shortly
s

Samhita Alla

05/17/2023, 4:38 AM
@Kevin Su, I believe I'm looking at the same error as well. I tested it on flytekit 1.5.0 https://flyte-org.slack.com/archives/CP2HDHKE1/p1684297314863949?thread_ts=1683861857.343639&amp;cid=CP2HDHKE1
k

Kevin Su

05/17/2023, 6:35 PM
I fixes it here, but you can’t use default input for now since 
type_engine
doesn’t use dataproxy to upload data. this is another issue, we’ll fix it.
f

Frank Shen

05/17/2023, 7:37 PM
@Kevin Su @Samhita Alla, thank you very much!
Hi @Kevin Su, Which flyte version has this feature working? I need this feature. Therefore I will re-test if you’ve got it released. Thanks!
k

Kevin Su

05/30/2023, 10:00 PM
flytekit==1.6.2
f

Frank Shen

05/30/2023, 10:02 PM
Thanks @Kevin Su, Right now I am using:
Copy code
flytekit==1.2.4
flytekitplugins-snowflake==1.2.4
flytekitplugins-spark==1.2.4
flytekitplugins-polars==1.2.4
pandas~=1.5.0
...
Do I need to upgrade the version of flytekitplugins to the same as flytekit version? And do I need to change the pandas version?
k

Kevin Su

05/30/2023, 10:08 PM
1.2.4 is a bit old, I suggest upgrade them as well.
but I think it should work as well if you don’t upgrade them.
f

Frank Shen

05/30/2023, 10:10 PM
Thanks Kevin.
k

Kevin Su

05/30/2023, 10:11 PM
and make sure to use flyteidl==1.5.8. The latest flytekit is not compatible with latest flyteidl (1.5.9)
f

Frank Shen

05/30/2023, 10:17 PM
Thanks. @Kevin Su, The reason I chose version 1.2.4 of flyte is because I had to build a custom image with spark. So I used this as part of my dependency req txt files. https://github.com/flyteorg/flytesnacks/blob/master/cookbook/integrations/kubernetes/k8s_spark/requirements.txt
I realize if I upgrade flyte version, I need to figure out the upgraded k8s_spark requirements.txt. Otherwise, I cannot upgrade flyte along without breaking the dependencies in my k8s_spark/requirements.txt
Could you point me to where the latest k8s_spark req txt file is?
k

Kevin Su

05/30/2023, 10:24 PM
Try to use imageSpec, it’s a new feature in flytekit. you don’t need to build an image for spark anymore. https://github.com/flyteorg/flytekit/pull/1616
Copy code
spark_image = ImageSpec(registry="<docker_id>", packages=["pandas", "numpy"])
f

Frank Shen

05/30/2023, 10:45 PM
@Kevin Su, I need to upgrade the flyte remote server to also have flytekit upgraded to v1.6.2, not only locally, is that correct?
@Kevin Su, just to clarify regarding not needing to build an image for spark. I need to build a custom image because I have custom dependencies. I do pyflyte run --remote --image with my custom image. And that’s why I built custom image with spark. Do you think I can remove the spark from my custom image now since I can use ImageSpec to run spark tasks?
k

Kevin Su

05/30/2023, 11:12 PM
yes, you can use imageSpec for the spark task. You can still install custom dependencies in ImageSpec, but you don’t need to install spark dependencies yourself. imageSpec will use spark-base-image, and install you custom packages on top of it
here is a doc in flytesnacks, we’ll merge the doc soon
f

Frank Shen

05/30/2023, 11:17 PM
Thanks @Kevin Su. You told me to upgrade to flytekit 1.6.2 for a bug fix. Does it mean I need to upgrade flytekit in the flyte remote server also to 1.6.2, not only locally?
k

Kevin Su

05/30/2023, 11:35 PM
flyte remote server
you mean flyteadmin, flytepropeller?
those deployments don’t need to be upgraded
f

Frank Shen

05/30/2023, 11:37 PM
You mean flytekit, flyteidl, etc. are not installed on the flyter remote server?
I am not the flyte remote server administrator, therefore I am not aware of that.
@Kevin Su
k

Kevin Su

05/30/2023, 11:42 PM
flytekit is only used in the task pods and local (for local execution). The remote server is written by golang, which is in a different repo.
f

Frank Shen

05/31/2023, 7:21 PM
Thanks @Kevin Su. I have custom dependencies that is private to my organization and is in a private repository (in one of two repos, e.g. https://maven.homebox.com/repository/max-pypi-snapshots, & https://maven.homebox.com/repository/max-pypi-releases). How do I add them in the ImageSpec?
k

Kevin Su

05/31/2023, 7:58 PM
ImageSpec(packages=[“max-pypi”, ….])
f

Frank Shen

06/01/2023, 5:15 PM
@Kevin Su, how will ImageSpec know where to find packages from max-pypi? Where do I specify the repo url https://maven.homebox.com/repository?
k

Kevin Su

06/01/2023, 6:10 PM
sorry, if you are trying to download it from others pypi, it’s not supported. I’ll add it to imageSpec
Powered by