George Horrell
03/09/2023, 9:52 PMflytectl demo
and it’d be really helpful to pull in some files from S3. Is this possible? I’m getting 403 forbidden errors when I run a pipeline that interns such a file.Kevin Su
03/09/2023, 9:55 PMk8s.yaml: |
plugins:
k8s:
default-cpus: 100m
default-env-vars:
- FLYTE_AWS_ENDPOINT: <http://minio.flyte.svc.cluster.local:9000>
- FLYTE_AWS_ACCESS_KEY_ID: minio
- FLYTE_AWS_SECRET_ACCESS_KEY: miniostorage
default-memory: 200Mi
flytekit will read these access key and id to before uploading the dataGeorge Horrell
03/09/2023, 9:56 PM~/.flyte/config/
?Kevin Su
03/09/2023, 9:57 PMkubectl edit cm flyte-propeller-config
George Horrell
03/09/2023, 9:58 PM$ kubectl get configmap --all-namespaces
NAMESPACE NAME DATA AGE
kube-system extension-apiserver-authentication 6 47m
kube-system cluster-dns 2 47m
flyte flyte-sandbox-cluster-resource-templates 1 47m
flyte flyte-sandbox-config 5 47m
flyte flyte-sandbox-docker-registry-config 1 47m
flyte flyte-sandbox-extra-cluster-resource-templates 0 47m
flyte flyte-sandbox-extra-config 0 47m
flyte flyte-sandbox-proxy-config 1 47m
flyte kubernetes-dashboard-settings 0 47m
kube-system local-path-config 4 47m
default kube-root-ca.crt 1 46m
kube-system kube-root-ca.crt 1 46m
kube-public kube-root-ca.crt 1 46m
kube-node-lease kube-root-ca.crt 1 46m
flyte kube-root-ca.crt 1 46m
kube-system coredns 2 47m
flytesnacks-development kube-root-ca.crt 1 46m
flytesnacks-staging kube-root-ca.crt 1 46m
flytesnacks-production kube-root-ca.crt 1 46m
Kevin Su
03/09/2023, 10:00 PMGeorge Horrell
03/09/2023, 10:00 PM003-storage.yaml: |
propeller:
rawoutput-prefix: <s3://my-s3-bucket/data>
storage:
type: stow
stow:
kind: s3
config:
region: us-east-1
disable_ssl: true
v2_signing: true
endpoint: <http://localhost:30002>
auth_type: accesskey
access_key_id: minio
secret_key: miniostorage
container: my-s3-bucket
010-inline-config.yaml: |
plugins:
k8s:
default-env-vars:
- FLYTE_AWS_ENDPOINT: <http://flyte-sandbox-minio.flyte:9000>
- FLYTE_AWS_ACCESS_KEY_ID: minio
- FLYTE_AWS_SECRET_ACCESS_KEY: miniostorage
storage
too?Kevin Su
03/09/2023, 10:04 PMFLYTE_AWS_ACCESS_KEY_ID: minio
FLYTE_AWS_SECRET_ACCESS_KEY: miniostorage
George Horrell
03/09/2023, 10:05 PMif you only want to read the data from your own s3 bucket instead minio, you only need to update default-env-vars
minio
and miniostorage
{
"asctime": "2023-03-09 22:04:39,613",
"name": "flytekit",
"levelname": "ERROR",
"message": "Exception when trying to execute ['aws', '--endpoint-url', '<http://flyte-sandbox-minio.flyte:9000>', 's3', 'cp', 's3://$BUCKET_NAME/$FILENAME', '/tmp/flytexbbrupf4/local_flytekit/576657f4df3e38f46453a5c45b8eb0c0/$FILENAME'], reason: Called process exited with error code: 1. Stderr dump:\n\nb'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\\n'"
}