Sandra Youssef
Sandra Youssef
Samhita Alla
conda-forge
packages for unionml, flytekit, and flyteidl.
• @Robin Kahlow worked on a cookie cutter template that simplifies the creation and execution of flyte workflows. Do check out the repository!
• @Babis Kiosidis worked on performance improvements as he was seeing elevated RESOURCE_EXCHAUSTED
when writing various files from flytepropeller to GCS. It’ll hopefully be merged very soon into flytepropeller!
We hope to have many more of your contributions, Sugato, Robin, and Babis! All contributors of the month should be available on flyte.org.Sandra Youssef
Ailin Yu
07/26/2022, 12:39 AMflyte start --config=flyte_local_k3d.yaml
, (after having run make compile
already) for example when using k3d. Even though my .k3d/kubeconfig-flyte.yaml
config file works with k3d directly, when I try to run the above command I always get an error:
{"json":{"src":"start.go:113"},"level":"error","msg":"Failed to create controller manager. error building Kubernetes Clientset: Error building kubeconfig: error loading config file \"/Users/ailinyu/.k3d/kubeconfig-flyte.yaml\": no kind \"Simple\" is registered for version \"<http://k3d.io/v1alpha3\|k3d.io/v1alpha3\>" in scheme \"pkg/runtime/scheme.go:100\"","ts":"2022-07-25T17:17:20-07:00"} {"json":{"src":"start.go:164"},"level":"panic","msg":"Failed to start Propeller, err: error building Kubernetes Clientset: Error building kubeconfig: error loading config file \"/Users/ailinyu/.k3d/kubeconfig-flyte.yaml\": no kind \"Simple\" is registered for version \"<http://k3d.io/v1alpha3\|k3d.io/v1alpha3\>" in scheme \"pkg/runtime/scheme.go:100\"","ts":"2022-07-25T17:17:20-07:00"}
Am I using this incorrectly or do I need to have something else in my k3d yaml config perhaps?Sören Brunk
07/26/2022, 6:22 AMIf your task gets preempted, Flyte will retry your task on a non-spot (regular) instance. This retry will not count towards a retry that a user sets.Still I'd like to configure Flyte to try one more time on a spot instance. There's a interruptible-failure-threshold ("number of failures for a node to be still considered interruptible"). Is this the right config to tweak that behavior?
jeev
Sandra Youssef
Bernhard Stadlbauer
07/27/2022, 10:26 AMMatheus Moreno
07/27/2022, 5:45 PM--force
flag on flytectl register files
working? When I use it, the CLI logs "AlreadyExists", which makes it look like it didn't actually rewrite anything. Is this normal? If so, shouldn't the logs print another message, such as "Overwritten" or something?
I'm using FlyteCTL 0.5.6, by the way.SeungTaeKim
07/28/2022, 1:36 AMTom Szumowski
07/29/2022, 12:39 PMplugins:
k8s:
resource-tolerations:
- <http://nvidia.com/gpu:|nvidia.com/gpu:>
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoSchedule"
I saw this thread which mentioned updating the propellor configmap. Is that the way to go? Or is there a way to update plugins via the flyte config?
Thanks!Matheus Moreno
07/29/2022, 6:23 PMTraceback (most recent call last):
...
File "/home/flyte/.venv/lib/python3.8/site-packages/flytekit/types/structured/basic_dfs.py", line 63, in decode
ctx.file_access.get_data(path, local_dir, is_multipart=True)
File "/home/flyte/.venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 427, in get_data
raise FlyteAssertion(
Message:
Failed to get data from to /tmp/flyteltak17mx/local_flytekit/85e6844a8f8283f8841994437be73626 (recursive=True).
Original exception: cannot copy tree '': not a directory
User error.
No path for the resulting query file. Could it be that there was an error executing the task? Flyte says the task succeeded, but I can't check on the GCP console (I don't have the necessary permissions).Tao He
08/02/2022, 3:51 AMhorovod.spark.run
to execute the distributed training function" means that flyte will launch num_proc
spark workers on num_proc
flyte worker? Or it just launches them on one spark worker?Tao He
08/02/2022, 3:52 AMNiels Bantilan
08/02/2022, 12:42 PMKetan (kumare3)
Robert Everson
08/02/2022, 10:55 PMTao He
08/04/2022, 7:32 AMflytectl register examples
doesn't register such a project as well.
May I know where can I find that? Or it is just because the flytectl's doc is out-of-date?Ketan (kumare3)
Nada Saiyed
08/05/2022, 12:41 AMtask_config
as a HPOJob
but how do i specify which training_task
it’s supposed to execute? and also where should i specify the parameters like HyperparameterTuningJobConfig
, and ParameterRanges
?Sujith Samuel
08/05/2022, 5:25 AMSujith Samuel
08/05/2022, 5:25 AMflyte-secret-auth
and populate it with clientSecret
.
# If enabled is false, it's up to the user to create flyte-secret-auth
as described in
# https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#oauth2-authorization-server
enabled: true
clientSecret: "CO2n3hovN0J78FqxxOVtjOtkHH5fPL9C"
clientId: "flytepropeller"
-- FlyteAdmin server configuration
adminServer:
# Refer to the server config.
server:
httpPort: 8088
grpcPort: 8089
security:
# -- Controls whether to serve requests over SSL/TLS.
secure: false
# -- Controls whether to enforce authentication. Follow the guide in https://docs.flyte.org/ on how to setup authentication.
useAuth: true
allowCors: true
allowedOrigins:
# Accepting all domains for Sandbox installation
- "*"
allowedHeaders:
- "Content-Type"
- "flyte-authorization"
# Refer to the full structure for documentation.
flyteadmin:
roleNameKey: "iam.amazonaws.com/role"
profilerPort: 10254
metricsScope: "flyte:"
metadataStoragePrefix:
- "metadata"
- "admin"
eventVersion: 2
testing:
host: http://flyteadmin
# -- Authentication configuration
auth:
authorizedUris:
# This should point at your public http Uri.
- https://flytedeployment url
# This will be used by internal services in the same namespace as flyteadmin
- http://flyteadmin:80
# This will be used by internal services in the same cluster but different namespaces
- http://flyteadmin.nmlp.svc.cluster.local:80
# Controls app authentication config
appAuth:
thirdPartyConfig:
flyteClient:
clientId: flytectl
redirectUri: https://<flyte deployment url>/callback
scopes:
- offline
- all
# Controls user authentication
userAuth:
openId:
baseUrl: https://<keycloak production realm>/realms/nsdmlp
scopes:
- profile
- openid
clientId: flytepropeller
I have also edited the flyte-admin-secrets to have the correct client secret
apiVersion: v1
data:
claim_symmetric_key: cWlBYzlYWHdLN3lnaksrWUJGdStFUlRYK0RDdlk4SjVjZFJtaXBTcDBhdw==
cookie_block_key: ejZPdkhrZ1crWXdib21JZHdVZ05IOGJESVp0OE5KWnNZT285KzIyRVM1dw==
cookie_hash_key: Q093TUY2RTdOMW5MeFZ4Rnk1dGNzZGN5NU5aeTNWN2JTMXRPTjBLUGhQQ2JWZ3hGby9XQkVRdi84Yjk4ZEIyeEV3Zm5KYURDVzFkSjBuSGZrbS8zYVE=
oidc_client_secret: CO2n3hovN0J78FqxxOVtjOtkHH5fPL9C
With all the above setting, when i try to login to the flyte console via the ingress, it gives me the below message in the flyteadmin pod logs
{"json":{},"level":"error","msg":"Failed to retrieve tokens from request, redirecting to login handler. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_idt], caused by: http: named cookie not present","ts":"2022-08-04T183411Z"}
{"json":{},"level":"error","msg":"Error when exchanging code oauth2: cannot fetch token: 401 Unauthorized\nResponse: {"error":"unauthorized_client","error_description":"Invalid client secret"}","ts":"2022-08-04T183412Z"}
Logs from 8/5/2022, 120239 AM
And the chrome errors out with 403 access denied page.
I have tried secret and id with and without quotes(just in case if helm was acting crazy) but it still gives the same error.
Please assist.
Thanks,
SujithEdgar Trujillo
08/05/2022, 2:44 PM--fast flag will take the code from your local machine and provide it for execution without having to build the container and push it. The --force flag allows overriding your previously created package.
How does flyte decide what to push up and how does the updated code get packaged into the container without a rebuild?Alex Pozimenko
08/05/2022, 5:30 PMAlex Pozimenko
08/05/2022, 11:17 PMexecution = flyte_remote.fetch_execution(name = execution_id)
print (execution.node_executions)
is execution.node_executions
expected to be populated? If not, how do I get the node execution details? I'm looking for level of details similar to returned by
flytectl get execution execid --details --nodeID n0 -o yaml
Oliver Nguyen
08/08/2022, 5:10 AMAbdullah Mobeen
08/08/2022, 1:53 PMRahul Mehta
08/08/2022, 5:56 PM--name
is an available flag in pyflyte run
, but couldn't find the equivalent for flytectl create execution
. Mostly looking for how a user could surface a more informative name in the UI vs the randomly-generated execution IDvarsha Parthasarathy
08/08/2022, 11:50 PM