https://flyte.org logo
Join Slack
Powered by
Hi Team! I have a flyte-binary deployment on a kub...
# flyte-v1-support
h
Hi Team! I have a flyte-binary deployment on a kubernetes cluster and I frequently run into 
Timeout in node
errors. I try to set
Copy code
node-active-deadline: 72h
node-execution-deadline: 72h
in propeller, but it does not have an effect. Moreover different nodes (tasks) have different deadlines. I'll post a screenshot from what I see when I describe a 
<http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com>
CRD, and will also share parts of the helm values used to deploy the instance.
These are the node deadlines from 
kubectl describe <http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com> <ID>
Copy code
nameOverride: ""
fullnameOverride: ""

flyte-core-components:
  admin:
    authType: Pkce
    disabled: false
    disableScheduler: false
    disableClusterResourceManager: false
    seedProjects:
      - project1
  propeller:
#    is this the right place to set deadlines?
    node-config:
      default-deadlines:
        node-active-deadline: 72h
        node-execution-deadline: 72h
        workflow-active-deadline: 72h
    disabled: false
    disableWebhook: false
  dataCatalog:
    disabled: false

configuration:
  auth:
    enabled: true
    ...
  inline:
    plugins:
      k8s:
        inject-finalizer: true
    domains:
      - id: staging
        name: staging
  database:
    ...
  storage:
    ...
  logging:
    ...
  co-pilot:
    ...
  connectorService:
    ...
  propeller:
#  this is not the right place, right?
    node-config:
      default-deadlines:
        node-active-deadline: 72h
        node-execution-deadline: 72h
        workflow-active-deadline: 72h
    createCRDs: true

deployment:
  image:
    repository: cr.flyte.org/flyteorg/flyte-binary
    tag: latest
    pullPolicy: IfNotPresent
  resources:
    ...
  podSecurityContext:
    enabled: false
    runAsUser: 65534
    runAsGroup: 65534
    fsGroup: 65534

  # CloudSQL Proxy sidecar container
  sidecars:
    - name: cloud-sql-proxy
      ...

enabled_plugins:
  tasks:
    task-plugins:
      enabled-plugins:
        - container
        - sidecar
        - k8s-array
        - connector-service
        - echo
      default-for-task-types:
        container: container
        sidecar: sidecar
        container_array: k8s-array
And these are parts that I use to overwrite the helm values.
I am not setting any timeouts anywhere and I do not understand why my nodes have different timouts, like:
Copy code
n0:
      Active Deadline:     10m0s
      Execution Deadline:  5m0s
      Id:                  n0
and
Copy code
n1:
      Active Deadline:     1h0m0s
      Execution Deadline:  30m0s
      Id:                  n1
I also cannot find references to these values 
1h0m0s
, 
30m0s
, 
10m0s
, 
5m0s
in flyte code 🤔
Sorry for the input overload, hope that helps to figure out what is not working here, though thanks
My bad... I did set timeouts! But somehow my tasks stayed in a queue for an hour and were not scheduled 🤔 Is it possible to make sure that flyte propeller immediately tries to schedule all of them if enough resources are there on the cluster?
a
I am not sure if you can force the flyte propeller to start the task immediately when the resource is available, the flyte team can answer that. But one thing to add is the retry option in your task like below:
Copy code
@task(
    timeout=timedelta(seconds=10),
    retries=1,
)
Have you set retries for your task if it fails? https://www.union.ai/docs/v1/flyte/user-guide/core-concepts/tasks/task-parameters/
👍 1
h
Setting 
cache_serialize
to false solved the issue.
a
Could you paste your input and output of your task definition. like below:
Copy code
@fl.task(cache=True)
def cached_data_processing_task(df: pandas.DataFrame) -> pandas.DataFrame:
    time.sleep(1)
    return df \* 2
This might have solved the timeout problem because the there is no reservation lock but would like to see why the task were not able to be cached. You might also have to see if perf takes a hit. https://www.union.ai/docs/v1/flyte/user-guide/core-concepts/caching/#how-does-caching-of-offloaded-objects-work
14 Views
Open in Slack
PreviousNext
Powered by