https://flyte.org
Join Slack
so absolutely dying trying to get this deployed in EKS -- namely on minio charts are effectively the...
# flyte-support
h
so absolutely dying trying to get this deployed in EKS -- namely on minio charts are effectively the same so I am not sure why minio all the sudden just doesnt work
Copy code
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmp030l19_7/script_mode.tar.gz to <http://localhost:30084/my-s3-bucket/a/b/PJPRTCSJDCALQRYI44IFNWNQ5M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230315%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230315T152958Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=9b449916e2b9ca077d45255d05f86fec463c3e999e14c2c38a0075928906afa0> (recursive=False).

Original exception: HTTPConnectionPool(host='localhost', port=30084): Max retries exceeded with url: /my-s3-bucket/a/b/PJPRTCSJDCALQRYI44IFNWNQ5M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230315%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230315T152958Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=9b449916e2b9ca077d45255d05f86fec463c3e999e14c2c38a0075928906afa0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f205d87edc0>: Failed to establish a new connection: [Errno 111] Connection refused'))
noticed it was using a signed url override (not sure why it's doing that all of the sudden) and pretty much anything I change/update in the charts just doesnt apply, for example:
Copy code
storage:
      # -- Sets the storage type. Supported values are sandbox, s3, gcs and custom.
      type: sandbox 
      # -- bucketName defines the storage bucket flyte will use. Required for all types except for sandbox.
      bucketName: my-s3-bucket 
      # -- settings for storage type s3
      signedUrl:
        stowConfigOverride:
          endpoint: <http://minio.flyte.svc.cluster.local:9000>
anyone have any thoughts?
f
@hundreds-toothbrush-11241 hopefully that was pun intended - please do not die
But on an important note - on EKS are you using minio? Why not s3
If you are using minio you will Have to make it available For flytekit
h
so setting up s3 as the backend also proved very problematic
f
error log says it’s trying to connect at localhost:30084
that’s exclusively a local sandbox thing
h
sure and i can't seem to update the 
stowConfigOverride
to address that
f
are you copying config from flyte-sandbox?
you want your storage endpoint to be set to the k8s address
h
@freezing-airport-6809 im not sure why, as a backend meta store for flyte, minio would be an issue if self hosted interm -- pointing at that or at s3 should be the same except config which is kinda the point of minio acting as a drop in interface from my understanding
f
and you can drop the stow config override unless you want to port-forward and use signed URLs.
h
would love to drop the stow config stuff, for some reason it's signed off the bat and i dont know why and as mentioned, i cant seem to update it
f
hmm how are you deploying,
h
if you look at the override above, i am trying to set it to the k8s fqdn
that chart
f
for a simple setup
stow config override is only needed for generating signed URLs for pyflyte
h
k so would really prefer not to change charts atm as the team uses a local version of the 
flyte/flyte
chart that works great and we're setting this up for enterprise and would like the ability to have more configuration options
as for the stow config, are you saying that running something such as 
pyflyte run
is what's signing?
for testing purposes i've been doing 
kubectl port-forward
to the 
flyte-contour-envoy
service and running 
pyflyte run
f
right ok
can you paste the values file you are using?
h
ya one sec
ok values yaml is too long How about we pivot really quickly, maybe it's easier to set it up via s3 for the backend (and I would prefer that anyways) my understanding is that:
Copy code
default-env-vars:
              - FLYTE_AWS_ENDPOINT:
              - FLYTE_AWS_ACCESS_KEY_ID:
              - FLYTE_AWS_SECRET_ACCESS_KEY:
need to be set is that pulled from the workflow/task? I don't see a secret mount option in the k8s plugin
f
that’s mostly only for minio. you can just use IRSA for allowing your tasks to talk to s3
h
my workflows/tasks communicate fine with s3 when i do 
pyflyte run
my understanding is the code is packaged and stored ... somewhere i think im confused as I can't seem to get that storage to work with either minio or s3
are you saying that service account should also be setup for the flyte deployment at large and not specific to the task/workflows?
f
yes the flyte control plane also needs access to a bucket to store metadata
h
right -- and is that not configurable through the chart?
f
it should be
if you can upload the values file, folks can take a look and help
h
values.yaml
values.yaml
so minio is off in favor of s3 in the above
Copy code
remoteData:
        remoteData:
          region: "us-gov-west-1"
          scheme: "local"
          signedUrls:
            durationMinutes: 3
had no comments behind 
scheme
but it looks sus -- is that what needs to be swapped out?
f
why are there 2 remoteData?
h
f
ah ok
ok that’s right
see here for EKS specific values for the child flyte-core chart: https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values-eks.yaml
h
kk will try that
thanks
187 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by