• seunggs

    seunggs

    2 weeks ago
    I’m trying to execute a simple test workflow and getting this error:
    Request failed with status code 500 failed to create workflow in propeller namespaces <project-name> not found
    . The project already shows up in the flyte console and I was able to package/register the workflow, but I can’t seem to execute it?
    seunggs
    Ketan (kumare3)
    +1
    60 replies
    Copy to Clipboard
  • James Evers

    James Evers

    1 week ago
    hey everyone, i'm trying to run ray on the flyte sandbox deployment, and am running into the following error. it looks like its basically an OOM error, but i'm not sure how to increase the default size of
    /dev/shm
    . what's the best approach to bumping this default in the sandbox deployment?
    /usr/local/lib/python3.8/dist-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
      "class": algorithms.Blowfish,
    2022-09-14 18:10:43,190	WARNING services.py:1882 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67108864 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=0.10gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.
    2022-09-14 18:10:46,611	INFO worker.py:1509 -- Started a local Ray instance. View the dashboard at [1m[32m<http://127.0.0.1:8265> [39m[22m
    [2022-09-14 18:10:57,864 E 1 1] <http://core_worker.cc:149|core_worker.cc:149>: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory
    James Evers
    Ketan (kumare3)
    +1
    19 replies
    Copy to Clipboard
  • Ailin Yu

    Ailin Yu

    1 week ago
    Hello! I am interested in using the published Grafana dashboards to monitor Flyte. But even though my Flyte instance has these pod annotations:
    <http://prometheus.io/path=/stats/prometheus|prometheus.io/path=/stats/prometheus>
    <http://prometheus.io/port=15020|prometheus.io/port=15020>
    <http://prometheus.io/scrape=true|prometheus.io/scrape=true>
    And I’ve confirmed prometheus is scraping the pods with those annotations, I’m not seeing any Flyte provided metrics, so the dashboards are not populating. Is there some config I need to enable in flytepropeller in order for metrics to be emitted?
    Ailin Yu
    Ketan (kumare3)
    +1
    8 replies
    Copy to Clipboard
  • seunggs

    seunggs

    1 week ago
    Hi, I’m trying to execute a workflow in EKS and my simple add_one task is failing with this error but not sure how to debug it:
  • seunggs

    seunggs

    1 week ago
    [4/4] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
    [a29fz6979nbn4dxsr7km-n0-3] terminated with exit code (1). Reason [Error]. Message: 
    Traceback (most recent call last):
      File "/opt/venv/bin/pyflyte-execute", line 8, in <module>
        sys.exit(execute_task_cmd())
      File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 470, in execute_task_cmd
        _execute_task(
      File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 348, in _execute_task
        _handle_annotated_task(ctx, _task_def, inputs, output_prefix)
      File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 291, in _handle_annotated_task
        _dispatch_execute(ctx, task_def, inputs, output_prefix)
      File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 80, in _dispatch_execute
        logger.debug(f"Starting _dispatch_execute for {task_def.name}")
    AttributeError: 'function' object has no attribute 'name'
    seunggs
    Eduardo Apolinario (eapolinario)
    37 replies
    Copy to Clipboard
  • Justin Tyberg

    Justin Tyberg

    1 week ago
    I’m having trouble accessing flyteadmin from flytectl that is running in a pod on a GKE cluster. • flyteadmin running in
    flyte
    namespace • flytectl running in
    foo
    namespace If I run flytectl from within the pod in
    foo
    namespace, and hit the external endpoint (through the ingress), it works
    bin/flytectl get projects \
      --admin.endpoint dns:///EXTERNAL_FQDN:443 \
      --admin.authType ClientSecret \
      --admin.clientId flytepropeller \
      --admin.clientSecretLocation /etc/secrets/client_secret
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
     ----- ------ ----------------- 
    | ID  | NAME | DESCRIPTION     |
     ----- ------ ----------------- 
    | dpp | dpp  | dpp description |
     ----- ------ ----------------- 
    1 rows
    However, if I try to hit the internal grpc endpoint, I get nada. No output.
    bin/flytectl get projects \
    >   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.insecure true \
    >   --admin.clientSecretLocation /etc/secrets/client_secret
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
    
    echo $?
    0
    🤔
    Justin Tyberg
    Samhita Alla
    +2
    86 replies
    Copy to Clipboard
  • seunggs

    seunggs

    1 week ago
    A quick question regarding flyte
    /workflows
    REST API endpoint - I see there’s an archive option in the dashboard but I don’t see a PUT endpoint for
    /workflows
    - is this expected or am I missing something?
    seunggs
    Samhita Alla
    3 replies
    Copy to Clipboard
  • l

    Leong Shing Chew

    1 week ago
    Hi, I have a question with regards to cluster resource templates, subsequent update to the templates(addition, update and removal) would be reflected in the k8s resources after the refresh interval right? Just want to confirm that deletion of the templates under "cluster resource manager" component would trigger the deletion of resources in the existing namespaces(project/domain).
    l
    p
    +1
    4 replies
    Copy to Clipboard
  • seunggs

    seunggs

    1 week ago
    Name:                flyte-executor
    Namespace:           myproject-development
    Labels:              <http://app.kubernetes.io/managed-by=pulumi|app.kubernetes.io/managed-by=pulumi>
    Annotations:         <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::xxx:role/flyte-user-role
    Image pull secrets:  gcr-json-key
    ...