Hi all I m testing some work locally via `flytectl demo` and Flyte #flyte-support

Hi all - I’m testing some work locally via `flytec...

hallowed-rain-91612

03/09/2023, 9:52 PM

Hi all - I’m testing some work locally via

flytectl demo

and it’d be really helpful to pull in some files from S3. Is this possible? I’m getting 403 forbidden errors when I run a pipeline that interns such a file.

glamorous-carpet-83516

03/09/2023, 9:55 PM

yes, you can. Update the default env in flyte propeller config map.

Copy code

k8s.yaml: |
    plugins:
      k8s:
        default-cpus: 100m
        default-env-vars:
        - FLYTE_AWS_ENDPOINT: <http://minio.flyte.svc.cluster.local:9000>
        - FLYTE_AWS_ACCESS_KEY_ID: minio
        - FLYTE_AWS_SECRET_ACCESS_KEY: miniostorage
        default-memory: 200Mi

flytekit will read these access key and id to before uploading the data

hallowed-rain-91612

03/09/2023, 9:56 PM

Cool thanks - do I modify this somewhere in

~/.flyte/config/

glamorous-carpet-83516

03/09/2023, 9:57 PM

no, it’s in the Kubernetes.

Copy code

kubectl edit cm flyte-propeller-config

hallowed-rain-91612

03/09/2023, 9:58 PM

Copy code

$ kubectl get configmap --all-namespaces
NAMESPACE                 NAME                                             DATA   AGE
kube-system               extension-apiserver-authentication               6      47m
kube-system               cluster-dns                                      2      47m
flyte                     flyte-sandbox-cluster-resource-templates         1      47m
flyte                     flyte-sandbox-config                             5      47m
flyte                     flyte-sandbox-docker-registry-config             1      47m
flyte                     flyte-sandbox-extra-cluster-resource-templates   0      47m
flyte                     flyte-sandbox-extra-config                       0      47m
flyte                     flyte-sandbox-proxy-config                       1      47m
flyte                     kubernetes-dashboard-settings                    0      47m
kube-system               local-path-config                                4      47m
default                   kube-root-ca.crt                                 1      46m
kube-system               kube-root-ca.crt                                 1      46m
kube-public               kube-root-ca.crt                                 1      46m
kube-node-lease           kube-root-ca.crt                                 1      46m
flyte                     kube-root-ca.crt                                 1      46m
kube-system               coredns                                          2      47m
flytesnacks-development   kube-root-ca.crt                                 1      46m
flytesnacks-staging       kube-root-ca.crt                                 1      46m
flytesnacks-production    kube-root-ca.crt                                 1      46m

hallowed-rain-91612

03/09/2023, 9:59 PM

Doesn’t look like I have that locally?

glamorous-carpet-83516

03/09/2023, 10:00 PM

sorry, I forgot we are using single binary in the demo cluster. it should be in the flyte-sandbox-config

glamorous-carpet-83516

03/09/2023, 10:00 PM

kubectl edit cm flyte-sandbox-config

hallowed-rain-91612

03/09/2023, 10:00 PM

no problem at all - thank you for helping!

hallowed-rain-91612

03/09/2023, 10:00 PM

looking now

hallowed-rain-91612

03/09/2023, 10:01 PM

here’s what I see that looks relevant:

Copy code

003-storage.yaml: |
    propeller:
      rawoutput-prefix: <s3://my-s3-bucket/data>
    storage:
      type: stow
      stow:
        kind: s3
        config:
          region: us-east-1
          disable_ssl: true
          v2_signing: true
          endpoint: <http://localhost:30002>
          auth_type: accesskey
          access_key_id: minio
          secret_key: miniostorage
      container: my-s3-bucket
  010-inline-config.yaml: |
    plugins:
      k8s:
        default-env-vars:
        - FLYTE_AWS_ENDPOINT: <http://flyte-sandbox-minio.flyte:9000>
        - FLYTE_AWS_ACCESS_KEY_ID: minio
        - FLYTE_AWS_SECRET_ACCESS_KEY: miniostorage

hallowed-rain-91612

03/09/2023, 10:02 PM

Maybe I need to modify

storage

too?

hallowed-rain-91612

03/09/2023, 10:03 PM

ah - there was more, looks like I already had the aws keys you suggested i add

glamorous-carpet-83516

03/09/2023, 10:04 PM

if you want flyteadmin to save some metadata in your s3. you could update storage config. if you only want to read the data from your own s3 bucket instead minio, you only need to update default-env-vars

glamorous-carpet-83516

03/09/2023, 10:05 PM

should change to your aws s3 access key and id

Copy code

FLYTE_AWS_ACCESS_KEY_ID: minio
FLYTE_AWS_SECRET_ACCESS_KEY: miniostorage

hallowed-rain-91612

03/09/2023, 10:05 PM

yeah, this is all I need:

Copy code

if you only want to read the data from your own s3 bucket instead minio, you only need to update default-env-vars

hallowed-rain-91612

03/09/2023, 10:05 PM

looks like they were already set to

minio

and

miniostorage

hallowed-rain-91612

03/09/2023, 10:07 PM

Here’s the actual log from the k8s job that’s failing to pull:

Copy code

{
  "asctime": "2023-03-09 22:04:39,613",
  "name": "flytekit",
  "levelname": "ERROR",
  "message": "Exception when trying to execute ['aws', '--endpoint-url', '<http://flyte-sandbox-minio.flyte:9000>', 's3', 'cp', 's3://$BUCKET_NAME/$FILENAME', '/tmp/flytexbbrupf4/local_flytekit/576657f4df3e38f46453a5c45b8eb0c0/$FILENAME'], reason: Called process exited with error code: 1.  Stderr dump:\n\nb'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\\n'"
}

152 Views

Open in Slack

Previous Next