Fhuad Balogun
03/06/2023, 4:26 AMSamhita Alla
Fhuad Balogun
03/06/2023, 6:32 AM"""Flyte Remote"""
from flytekit.remote import FlyteRemote # type: ignore
from flytekit.configuration import ( # type: ignore
Config,
SerializationSettings,
ImageConfig,
PlatformConfig,
S3Config,
DataConfig,
)
IMAGE_STR = "docker_user/repo_name:tag"
image_config = ImageConfig.auto(img_name=IMAGE_STR)
s3_config = S3Config(
endpoint="localhost:8089", # I used the kubernetes endpoint,
retries=3,
access_key_id="AKIA0123458DKF", # fake id
secret_access_key="0dhjheedh38ahdjaleurebad938ud", # fake key
)
remote = FlyteRemote(config=Config(
platform=PlatformConfig(
endpoint= "localhost:8089", # cluster dns
insecure=True,
insecure_skip_verify=True,
),
data_config=DataConfig(
s3=s3_config
),
))
Samhita Alla
Fhuad Balogun
03/06/2023, 7:01 AMSamhita Alla
Fhuad Balogun
03/06/2023, 7:35 AMSamhita Alla
Fhuad Balogun
03/06/2023, 8:54 AM<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>:
. Should I create a role that has the policy to manage any resource in the cluster and also be able to PassRole
?Samhita Alla
What type of ARN role do I pass to this key:That should do.. Should I create a role that has the policy to manage any resource in the cluster and also be able to<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>:
?PassRole
Fhuad Balogun
03/06/2023, 9:00 AMSamhita Alla
Fhuad Balogun
03/06/2023, 9:05 AMSamhita Alla
Fhuad Balogun
03/06/2023, 10:48 AM<USER_DATA_BUCKET_NAME>
? It's in the template eks-starter.yaml
. What will be the value of this?Samhita Alla
Fhuad Balogun
03/06/2023, 10:55 AMmetadataContainer
.Samhita Alla
metaDataContainer
is to store Flyte metadata and userDataContainer
is to store the actual data.Fhuad Balogun
03/06/2023, 11:03 AMcaused by: path:metadata/propeller/flexs-app-development-f72e5a548840e46ef8f1/n0/data/0/futures.pb: getItem, getting the object: Forbidden: Forbidden
metadata/propeller/flexs-app-development-f72e5a548840e46ef8f1/n0/data/0/outputs.pb
and not metadata/propeller/flexs-app-development-f72e5a548840e46ef8f1/n0/data/0/fututes.pb
. Is there a mistake somewhere?Samhita Alla
Ketan (kumare3)
Yee
caused by
error?Fhuad Balogun
03/07/2023, 6:26 PMWorkflow[flexs-app:development:flexs_app.workflows.train_wf] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to check existence of futures file: [User] Failed to do HEAD on futures file., caused by: path:metadata/propeller/flexs-app-development-faa05ff0ba4174952b2f/n0/data/0/futures.pb: getItem, getting the object: Forbidden: Forbidden
status code: 403, request id: ZCJRYJ394HERSEE, host id: dyfcQi1d00vsYsCm
Yee
Fhuad Balogun
03/08/2023, 1:39 PMflyte
namespace:
1. flyte-backend-flyte-binary-cluster-resource-templates.yaml
Name: flyte-backend-flyte-binary-cluster-resource-templates
Namespace: flyte
Labels: <http://app.kubernetes.io/instance=flyte-backend|app.kubernetes.io/instance=flyte-backend>
<http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
<http://app.kubernetes.io/name=flyte-binary|app.kubernetes.io/name=flyte-binary>
<http://app.kubernetes.io/version=1.16.0|app.kubernetes.io/version=1.16.0>
<http://helm.sh/chart=flyte-binary-v1.3.0|helm.sh/chart=flyte-binary-v1.3.0>
Annotations: <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte-backend
<http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
Data
====
namespace.yaml:
----
apiVersion: v1
kind: Namespace
metadata:
name: '{{ namespace }}'
BinaryData
====
Events: <none>
2. flyte-backend-flyte-binary-config.yaml
Name: flyte-backend-flyte-binary-config
Namespace: flyte
Labels: <http://app.kubernetes.io/instance=flyte-backend|app.kubernetes.io/instance=flyte-backend>
<http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
<http://app.kubernetes.io/name=flyte-binary|app.kubernetes.io/name=flyte-binary>
<http://app.kubernetes.io/version=1.16.0|app.kubernetes.io/version=1.16.0>
<http://helm.sh/chart=flyte-binary-v1.3.0|helm.sh/chart=flyte-binary-v1.3.0>
Annotations: <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte-backend
<http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
Data
====
001-plugins.yaml:
----
tasks:
task-plugins:
enabled-plugins:
- container
- sidecar
- K8S-ARRAY
default-for-task-types:
- container: container
- container_array: K8S-ARRAY
plugins:
logs:
kubernetes-enabled: false
cloudwatch-enabled: false
stackdriver-enabled: false
k8s-array:
logs:
config:
kubernetes-enabled: false
cloudwatch-enabled: false
stackdriver-enabled: false
002-database.yaml:
----
database:
postgres:
username: dbname
passwordPath: /var/run/secrets/flyte/db-pass
host: aurora-rds-host-dns
port: 5432
dbname: flyteadmin
options: "sslmode=disable"
003-storage.yaml:
----
propeller:
rawoutput-prefix: <s3://bucket_name/data>
storage:
type: stow
stow:
kind: s3
config:
region: aws-region
disable_ssl: false
v2_signing: false
auth_type: iam
container: bucket_name
010-inline-config.yaml:
----
plugins:
k8s:
default-env-vars:
- AWS_METADATA_SERVICE_TIMEOUT: 5
- AWS_METADATA_SERVICE_NUM_ATTEMPTS: 20
inject-finalizer: true
storage:
cache:
max_size_mbs: 100
target_gc_percent: 100
000-core.yaml:
----
admin:
endpoint: localhost:8089
insecure: true
catalog-cache:
endpoint: localhost:8081
insecure: true
type: datacatalog
cluster_resources:
standaloneDeployment: false
templatePath: /etc/flyte/cluster-resource-templates
logger:
show-source: true
level: 1
propeller:
create-flyteworkflow-crd: true
webhook:
certDir: /var/run/flyte/certs
localCert: true
secretName: flyte-backend-flyte-binary-webhook-secret
serviceName: flyte-backend-flyte-binary-webhook
servicePort: 443
BinaryData
====
Events: <none>
3. kube-root-ca.crt.yaml:
Name: kube-root-ca.crt
Namespace: flyte
Labels: <none>
Annotations: <http://kubernetes.io/description|kubernetes.io/description>:
Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubern...
Data
====
ca.crt:
----
-----BEGIN CERTIFICATE-----
redacted
-----END CERTIFICATE-----
BinaryData
====
Events: <none>
NB: sensitive info has been redacted as advised. Thank you @YeeYee
storage
sections. to answer your question about Flyteremote though, it doesn’t take much configuration. you just need a configuration file like
$ cat ~/.flyte/dev.yaml
admin:
endpoint: dns:///flytedev.company.net
authType: Pkce
insecure: false
and in python you can do
from flytekit.remote.remote import FlyteRemote
from flytekit.configuration import Config
r = FlyteRemote(
Config.auto(config_file="/Users/ytong/.flyte/dev.yaml"),
default_project="flytesnacks",
default_domain="development",
)
aws sso login
or something)Fhuad Balogun
03/09/2023, 8:09 AM@task(requests=Resources(cpu="100m", mem="100Mi"), limits=Resources(cpu="200m", mem="500Mi"))
def get_scores(seqs: List[str]) -> np.ndarray:
landscape = Landscape("landscape")
scores = landscape.get_fitness(seqs=seqs)
return scores
One thing I have been looking at. Could it be that there is a mistake in the naming of the file by the propeller
? It is named outputs.pb
in S3 but seem to be looking for futures.pb
?
Are they meant to both be created in the folder or just one is and the mistake is due to the name giving to the file?Yee
@dynamic
task.Fhuad Balogun
03/10/2023, 2:16 PM<https://web.archive.org/web/20220926215904/https://docs.flyte.org/en/latest/deployment/aws/manual.html>
I got from this github repo: <https://github.com/alexifm/flyte-eks-deployment>
.
The propeller issue no longer come up.
Thank you guys once again and have a great weekend.David Espejo (he/him)
03/10/2023, 3:26 PMYee