I registered my `ContainerTasks` with `pyflyte register -p myproj -d devel flyte/my_workflow.py` so ...
f

Felix Ruess

over 2 years ago
I registered my
ContainerTasks
with
pyflyte register -p myproj -d devel flyte/my_workflow.py
so far and it worked fine. Now I added one Python task, but that one fails with
Pod failed. No message received from kubernetes.

[a9lcdpfc5ldtrh9tlqf8-fl6pvdgq-0] terminated with exit code (1). Reason [Error]. Message: 
Traceback (most recent call last):
  File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 496, in fast_execute_task_cmd
    _download_distribution(additional_distribution, dest_dir)
  File "/usr/local/lib/python3.8/site-packages/flytekit/tools/fast_registration.py", line 107, in download_distribution
    raise ValueError("Destination path is required to download distribution and it should be a directory")
ValueError: Destination path is required to download distribution and it should be a directory
.
I don't get what is wrong here... I can see that the file is in the package..
Hello all! I’m deploying flyte to aws eks. I followed the instructions in <the docs>, and everything...
e

Ekku Jokinen

about 3 years ago
Hello all! I’m deploying flyte to aws eks. I followed the instructions in the docs, and everything seems to be set up. However, the ALB ingress controller is not giving me an address.
$ kubectl -n flyte get ingress
NAME              CLASS    HOSTS   ADDRESS   PORTS   AGE
flyte-core        <none>   *                 80      41m
flyte-core-grpc   <none>   *                 80      41m
This issue was mentioned in the troubleshooting section of the docs, and it suggested running
$ kubectl describe ingress -n flyte
Name:             flyte-core
Labels:           <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
Namespace:        flyte
Address:          
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /*               ssl-redirect:use-annotation (<error: endpoints "ssl-redirect" not found>)
              
...

Annotations:  <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>:
                {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
              <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: arn:aws:acm:us-east-1:752578504353:certificate/7a7065cb-1ffc-418e-8070-bc36fbaff7cb
              <http://alb.ingress.kubernetes.io/group.name|alb.ingress.kubernetes.io/group.name>: flyte
              <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: [{"HTTP": 80}, {"HTTPS":443}]
              <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
              <http://alb.ingress.kubernetes.io/tags|alb.ingress.kubernetes.io/tags>: service_instance=production
              <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: alb
              <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
              <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
              <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
Events:
  Type     Reason             Age                   From     Message
  ----     ------             ----                  ----     -------
  Warning  FailedDeployModel  3m33s (x20 over 42m)  ingress  Failed deploy model due to InvalidParameter: 1 validation error(s) found.
- minimum field value of 1, CreateTargetGroupInput.Port.


Name:             flyte-core-grpc
Labels:           <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
Namespace:        flyte
Address:          
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------

  ...

Annotations:  <http://alb.ingress.kubernetes.io/actions.ssl-redirect|alb.ingress.kubernetes.io/actions.ssl-redirect>:
               {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}

...

Events:
  Type     Reason             Age                   From     Message
  ----     ------             ----                  ----     -------
  Warning  FailedDeployModel  3m36s (x19 over 42m)  ingress  Failed deploy model due to InvalidParameter: 1 validation error(s) found.
- minimum field value of 1, CreateTargetGroupInput.Port.
Could someone point me to the right direction in debugging this? I checked the security groups of EKS cluster and RDS, they were the same. Thanks in advance. Best, Ekku, CTO & Co-Founder @ inven.ai
👋 1
We have a conditional branch that takes the output of two tasks, and, whether they are true or false...
f

Fredrik Lyford

almost 3 years ago
We have a conditional branch that takes the output of two tasks, and, whether they are true or false, takes the output of two different tasks to process in a final task. Once in a while, the inputs from the boolean checks are being bound to the branch instead of the other tasks (once or twice out of a couple hundred). This makes flyte’s input converter fail. Does anyone else have the case where the variable being conditionally evaluated is not the same variable being passed into the nested tasks? Happy to share more logging as needed. Running on GKE.
Traceback (most recent call last):

      File "/root/.varner_dp/venv/3.7.1/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/root/.varner_dp/venv/3.7.1/lib/python3.9/site-packages/flytekit/core/base_task.py", line 472, in dispatch_execute
        native_inputs = TypeEngine.literal_map_to_kwargs(exec_ctx, input_literal_map, self.python_interface.inputs)
      File "/root/.varner_dp/venv/3.7.1/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 800, in literal_map_to_kwargs
        return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}
      File "/root/.varner_dp/venv/3.7.1/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 800, in <dictcomp>
        return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}

Message:

    'n2.o0'

SYSTEM ERROR! Contact platform administrators.