https://flyte.org logo
#ask-the-community
Title
# ask-the-community
j

justin hallquist

03/15/2023, 3:35 PM
so absolutely dying trying to get this deployed in EKS -- namely on minio charts are effectively the same so I am not sure why minio all the sudden just doesnt work
Copy code
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmp030l19_7/script_mode.tar.gz to <http://localhost:30084/my-s3-bucket/a/b/PJPRTCSJDCALQRYI44IFNWNQ5M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230315%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230315T152958Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=9b449916e2b9ca077d45255d05f86fec463c3e999e14c2c38a0075928906afa0> (recursive=False).

Original exception: HTTPConnectionPool(host='localhost', port=30084): Max retries exceeded with url: /my-s3-bucket/a/b/PJPRTCSJDCALQRYI44IFNWNQ5M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230315%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230315T152958Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=9b449916e2b9ca077d45255d05f86fec463c3e999e14c2c38a0075928906afa0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f205d87edc0>: Failed to establish a new connection: [Errno 111] Connection refused'))
noticed it was using a signed url override (not sure why it's doing that all of the sudden) and pretty much anything I change/update in the charts just doesnt apply, for example:
Copy code
storage:
      # -- Sets the storage type. Supported values are sandbox, s3, gcs and custom.
      type: sandbox 
      # -- bucketName defines the storage bucket flyte will use. Required for all types except for sandbox.
      bucketName: my-s3-bucket 
      # -- settings for storage type s3
      signedUrl:
        stowConfigOverride:
          endpoint: <http://minio.flyte.svc.cluster.local:9000>
anyone have any thoughts?
k

Ketan (kumare3)

03/15/2023, 3:44 PM
@justin hallquist hopefully that was pun intended - please do not die
But on an important note - on EKS are you using minio? Why not s3
If you are using minio you will Have to make it available For flytekit
j

justin hallquist

03/15/2023, 3:46 PM
so setting up s3 as the backend also proved very problematic
j

jeev

03/15/2023, 3:46 PM
error log says it’s trying to connect at localhost:30084
that’s exclusively a local sandbox thing
j

justin hallquist

03/15/2023, 3:47 PM
sure and i can't seem to update the
stowConfigOverride
to address that
j

jeev

03/15/2023, 3:47 PM
are you copying config from flyte-sandbox?
you want your storage endpoint to be set to the k8s address
j

justin hallquist

03/15/2023, 3:48 PM
@Ketan (kumare3) im not sure why, as a backend meta store for flyte, minio would be an issue if self hosted interm -- pointing at that or at s3 should be the same except config which is kinda the point of minio acting as a drop in interface from my understanding
j

jeev

03/15/2023, 3:48 PM
and you can drop the stow config override unless you want to port-forward and use signed URLs.
j

justin hallquist

03/15/2023, 3:49 PM
would love to drop the stow config stuff, for some reason it's signed off the bat and i dont know why and as mentioned, i cant seem to update it
j

jeev

03/15/2023, 3:49 PM
hmm how are you deploying,
j

justin hallquist

03/15/2023, 3:49 PM
if you look at the override above, i am trying to set it to the k8s fqdn
that chart
for a simple setup
stow config override is only needed for generating signed URLs for pyflyte
j

justin hallquist

03/15/2023, 3:57 PM
k so would really prefer not to change charts atm as the team uses a local version of the
flyte/flyte
chart that works great and we're setting this up for enterprise and would like the ability to have more configuration options
as for the stow config, are you saying that running something such as
pyflyte run
is what's signing?
for testing purposes i've been doing
kubectl port-forward
to the
flyte-contour-envoy
service and running
pyflyte run
j

jeev

03/15/2023, 4:07 PM
right ok
can you paste the values file you are using?
j

justin hallquist

03/15/2023, 4:07 PM
ya one sec
ok values yaml is too long How about we pivot really quickly, maybe it's easier to set it up via s3 for the backend (and I would prefer that anyways) my understanding is that:
Copy code
default-env-vars:
              - FLYTE_AWS_ENDPOINT:
              - FLYTE_AWS_ACCESS_KEY_ID:
              - FLYTE_AWS_SECRET_ACCESS_KEY:
need to be set is that pulled from the workflow/task? I don't see a secret mount option in the k8s plugin
j

jeev

03/15/2023, 4:18 PM
that’s mostly only for minio. you can just use IRSA for allowing your tasks to talk to s3
j

justin hallquist

03/15/2023, 4:20 PM
my workflows/tasks communicate fine with s3 when i do
pyflyte run
my understanding is the code is packaged and stored ... somewhere i think im confused as I can't seem to get that storage to work with either minio or s3
are you saying that service account should also be setup for the flyte deployment at large and not specific to the task/workflows?
j

jeev

03/15/2023, 4:24 PM
yes the flyte control plane also needs access to a bucket to store metadata
j

justin hallquist

03/15/2023, 4:24 PM
right -- and is that not configurable through the chart?
j

jeev

03/15/2023, 4:24 PM
it should be
if you can upload the values file, folks can take a look and help
j

justin hallquist

03/15/2023, 4:34 PM
values.yaml
so minio is off in favor of s3 in the above
Copy code
remoteData:
        remoteData:
          region: "us-gov-west-1"
          scheme: "local"
          signedUrls:
            durationMinutes: 3
had no comments behind
scheme
but it looks sus -- is that what needs to be swapped out?
j

jeev

03/15/2023, 4:56 PM
why are there 2 remoteData?
j

justin hallquist

03/15/2023, 4:56 PM
j

jeev

03/15/2023, 4:57 PM
ah ok
ok that’s right
see here for EKS specific values for the child flyte-core chart: https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values-eks.yaml
j

justin hallquist

03/15/2023, 5:02 PM
kk will try that
thanks
5 Views