Hi All, I’m trying to run a large workflow on our GKE cluster with many thousands of parallel tasks. Is there a recommended Flyte config to make sure Flyte is maximizing our compute resources? For example helm chart settings that might allow more tasks to be scheduled in parallel More information about my current situation in thread

Currently our GCP account has the ability to support about 2500-3000 tasks in parallel based on the CPU/MEM requirements of these Flyte tasks Initially the workflow started and FlytePropeller scheduled and ran about 500 tasks in parallel. I figured it would ramp up over time as new nodes were provisioned However after about 30 minutes the concurrent pods dropped to around 150 tasks and it’s been that way ever since (over 60 minutes) We have about 40 nodes running currently and they are only using about 25% of CPU/Mem, so not sure why FlytePropeller isn’t scheduling more tasks I also checked the CPU/Mem quota in the namespace and it’s well under the limit as well

Hey folks we recently upgraded copilot and tasks are failing with

[1/1] currentAttempt done. Last Error: USER::[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[flyte-copilotdownloader] terminated with exit code (1). Reason [Error]. Message: 
  --storage.cache.max_size_mbs int             Maximum size of the cache where the Blob store data is cached in-memory. If not specified or set to 0,  cache is not used
      --storage.cache.target_gc_percent int        Sets the garbage collection target percentage.

I took a look at the config in the docs and looks like we need to add these. Is there anything else I’m missing and will these default values work for our deployment?

co-pilot:
  cpu: 500m
  default-input-path: /var/flyte/inputs
  default-output-path: /var/flyte/outputs
  image: <http://cr.flyte.org/flyteorg/flytecopilot:v0.0.15|cr.flyte.org/flyteorg/flytecopilot:v0.0.15>
  input-vol-name: flyte-inputs
  memory: 128Mi
  name: flyte-copilot-
  output-vol-name: flyte-outputs
  start-timeout: 1m40s
  storage: ""

Hi all! Can you please advise me an example with hello world workflow on Flyte where at least two tasks executed on their own containers? With single python workflow code on flyte/workflow/example.py and such task option:

@task(container_image="<http://registry.name.com/project/image:tag|registry.name.com/project/image:tag>")

I am trying to build this example by myself on my local Flyte sandbox, but getting error messages:

ModuleNotFoundError: No module named 'flyte'

My workflow from example.py

@task(container_image="<http://registry.name.com/project/image:tag|registry.name.com/project/image:tag>")
def some_data_generation() -> PythonPickledFile:
    with open(BASE_FILE_PATH) as file:
        some_descriptors = json.load(file)

    some_set = generate_some_set(some_descriptors[0])
    with open(PICKLE_PATH, 'wb') as handle:
        pickle.dump(some_set, handle)
        return PICKLE_PATH

@task(container_image="<http://registry.name.com/project/image2:tag|registry.name.com/project/image2:tag>")
def load_pickle_dump(dump_file_path: PythonPickledFile) -> set:
    with open(dump_file_path, 'rb') as handle:
        return pickle.load(handle)

@workflow
def my_wf() -> set:
    dump_file_path = some_data_generation()
    return load_pickle_dump(dump_file_path=dump_file_path)

if __name__ == "__main__":
    a = my_wf()
    print(f"Running my_wf() {a}")

What was localised: 1. Workflow works fine without containers (without @task(container_image= ) 2. Images works fine if workflow contain only one task with Docker image where included Flyte Workflow folder with this file (example.py) 3. Problem appears if i build Docker Image for first task without Flyte workflow files inside (but with initial data for first files to skip downloading). 4. I am sure - i can skip second task at all from this test case, problem should appear. Root cause - i have Docker image without Flute workflow (example.py) but it seems that this code is required inside the container to be executed. I do not understand how can i split example.py between two tasks if it should be executed actually outside the tasks (because this is a workflow, if should contain tasks inside it according to example) Error:

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[j3nl8bafcr-n0-0] terminated with exit code (1). Reason [Error]. Message: 
thon3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 460, in execute_task_cmd
    _execute_task(
  File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
    return wrapped(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 327, in _execute_task
    _task_def = resolver_obj.load_task(loader_args=resolver_args)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/core/python_auto_container.py", line 189, in load_task
    task_module = importlib.import_module(task_module)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'flyte'

As i see - it caused by Docker Container without a Flyte workflow. But if i will place workflow in first task - how it should be correctly splitted between two tasks? Should not include code for first task inside the second task image/container? Am i on correct way at all? Can you please advise a simple short workflow example on Python with two tasks on separate containers?

Do I understand the slack notification documentation correctly that this is relying on a managed AWS service that flyte sends an email to and which then forwards to slack? 🙂 (Meaning will not work on GCP currently?)

Hi Flyers, Join us tomorrow! @Jake Neyer from Striveworks will be our Community Sync guest speaker, presenting Orchestrating MLOps With Flyte. Tuesday April 5, 9am PT Zoom Link: https://www.addevent.com/calendar/kE355955 See you all then! Flyte Team

Flyte Team, we have just updated to

v0.19.3

from an older version of Flyte. When we browse to Flyte Console, we get a new CORS error that we did not get before (see below). What configuration change can we make to fix it?

Access to XMLHttpRequest at '<https://avflyteadminhttp.pdx.l5.woven-planet.tech/api/v1/projects>' from origin '<https://avflyteconsole.pdx.l5.woven-planet.tech>' has been blocked by CORS policy: The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'. The credentials mode of requests initiated by the XMLHttpRequest is controlled by the withCredentials attribute.

(This ^^^^ is the error in my Chrome developer tools console output)

Flyte Team - we're still having difficult problems in between CORS with our gloo ingress, virtual services, etc. Is there any way we can just completely disable CORS for Flyte Admin / Flyte Console?

hi, after upgrading to the latest Flyte I no longer see any of the following metrics:

flyte-admin-executor-flytepropeller-execution.*

Have these been removed or renamed?

Hi all! Could you please help me - can i use PythonPickledFile between raw containers? Right now i am trying to use this data type with method described here https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/raw_container.html and getting

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

when trying to read them by pickle.load(file): inside

with open(f'{input_dir}/dump_file') as file:
        raw_data = pickle.load(file)

it is saved without errors by:

with open(f'{output_dir}/dump_file', 'wb') as handle:
        pickle.dump(raw_data, handle)

by previous stage raw container

stage1 = ContainerTask(
    name="stage1",
    output_data_dir="/var/outputs",
    outputs=kwtypes(generation=PythonPickledFile),
    image="<http://registry.project.com/flyte/stage1:0.0.1|registry.project.com/flyte/stage1:0.0.1>",
    command=[
        "python",
        "stage1.py",
        "/var/outputs",
    ],
)

stage2 = ContainerTask(
    name="stage2",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(dump_file=PythonPickledFile),
    outputs=kwtypes(stage2_out=str),
    image="<http://registry.project.com/flyte/stage2:0.0.1|registry.project.com/flyte/stage2:0.0.1>",
    command=[
        "python",
        "stage2.py",
        "/var/inputs",
        "/var/outputs",
    ],
)

@workflow
def my_wf() -> str:
    generation = stage1()
    return stage2(dump_file=generation)

Can i pass PythonPickledFile like that?

Hi, I am getting the following warning in a task Pod when I run a workflow, and I am not able to see the logs from the pods in the console, so I was wondering if that could be related?

{
  "asctime": "2022-04-05 13:58:10,552",
  "name": "flytekit",
  "levelname": "WARNING",
  "message": "No config file provided or invalid flyte config_file_path flytekit.config specified."
}

I am setting up authentication using OpenID Connect using Azure AD IdP. I have set up an app in the active directory and followed the OpenID Connect steps. I get stuck on the callback from the IdP seeing these logs in the

FlyteAdmin

pod. Could it be related to

oidc_client_secret

in the

flyte-admin-secrets

, or do anyone have an idea on what could cause this?

flyteadmin-7498449bc8-pl6rw flyteadmin {"json":{"src":"token.go:84"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T16:53:17Z"}
flyteadmin-7498449bc8-pl6rw flyteadmin {"json":{"src":"handlers.go:237"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T16:53:17Z"}
flyteadmin-7498449bc8-pl6rw flyteadmin {"json":{"src":"token.go:104"},"level":"debug","msg":"Could not retrieve id token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2022-04-05T16:53:17Z"}

Hey, I am setting up OIDC Authentication using Azure AD IdP. I followed the steps here. Right now when I hit the console I am prompted to authenticate against our IdP, then I get sent back to the callback URI,

https://<host>/callback

and get stuck in some redirect loop where the

FlyteAdmin

Pod keeps logging like below. Does anyone know what could cause this? Thanks 🙂

flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_idt] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_idt] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_at] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:71"},"level":"info","msg":"Could not detect existing cookie [flyte_rt]. Error: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie_manager.go:71"},"level":"info","msg":"Refresh token doesn't exist or failed to read it. Ignoring this error. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_rt], caused by: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_user_info] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"handlers.go:227"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"token.go:84"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"handlers.go:237"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"token.go:64"},"level":"debug","msg":"JWT parsing with claims failed failed to verify signature: failed to verify id token signature","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_at] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:71"},"level":"info","msg":"Could not detect existing cookie [flyte_rt]. Error: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}

Attention Airflow users, @Samhita Alla recently created a FlyteOperator in Airflow, and would love if some of you can upvote this operator - https://github.com/apache/airflow/pull/22646

We're updating our k8s cluster to use OIDC-enabled service accounts as much as possible, but we noticed the`flyte-copilotdownloader` init container seems to only be able to get IAM creds from the EC2 metadata service at the moment: https://github.com/flyteorg/flytecopilot/blob/master/cmd/root.go#L87. Is that right?

Can we use NFS instead of S3 in k8s cluster with Flyte?

set the channel topic: Next OSS Community Sync is on April 19th, 2021. Meeting Calendar: https://www.addevent.com/calendar/kE355955. Biweekly Minutes: https://bit.ly/37jvXgM. Email List: https://groups.google.com/a/flyte.org/g/users. Getting Started: https://flyte.org.

Hi Flyers, And here it is.. Strivework's @Jake Neyer presents

Orchestrating MLOps With Flyte▾

: Learn about their Chariot platform, how Flyte fits in, and why they chose Flyte. Thank you for contributing! Meeting Minutes See you all on April 19th! Flyte Team

Hi #general can we have multiple workflows inside one single workflow. Py file. If yes. Can you please share me the reference?

Any plans to have flytekit pushed into the anaconda repository as part of CI? I see anaconda has v0.26.1 where as pypi is at v0.31.0; I see that CI doesn't push to anaconda (https://github.com/flyteorg/flytekit/blob/master/.github/workflows/pythonpublish.yml). We have a solution for managing packages that are not pushed to anaconda - so I'm mostly just asking out of interest.

Can I get a maintainer to merge https://github.com/flyteorg/flytecopilot/pull/33 which is necessary to fix

make lint

CI for

flytecopilot

. Maybe also merge https://github.com/flyteorg/flytecopilot/pull/28 which looks fine. Then I can rebase my PR on top of these. If you could enable my GitHub user to trigger CI workflows that would also be great, my GH user is https://github.com/convexquad. @Haytham Abuelfutuh @Ketan (kumare3)

Hi, a few questions regarding permissions and configuration: • What is the preferred way to provide identities for task Pods? Would you specify IAM Role in Launch Plan, or provide an Service Account in the

flyte.config

? • Is it best practice to assign a SA/IAM role for each task, or reuse the same identity on a workflow basis? • What should be configured in

flyte.config

vs

~/.flyte/config.yaml

?

Hello! I am trying to use my private Docker Registry for node image, it is works in sandbox if executed ‘docker login’. But how it should be done on production cluster with Flyte? Tried to debug and found this line: https://github.com/flyteorg/flytepropeller/blob/6a9c1d4490c94e993b9a859d15c3b436c98af3c2/pkg/apis/flyteworkflow/v1alpha1/nodes.go#L119 as i understand - Propeller is able to read K8s secrets from this field. But how to pass this field via Python FlyteKit? Can i pass them using raw container definition?

What is the recommended way of structuring your source code if the workflow should reference Python files located directories relative the root path of the repository? I.e., let’s say

workflow.py

needs to access both data and train, do you still recommend this structure and reference everything in

workflow.py

relative root? Probably not explaining the idea that well, but interested in what you think is the best practice with respect to workflows and relative imports/access to other folders containing data/code important to your project.

├── Dockerfile
├── data
│   └── some_folder_with_data
├── train
│   ├── train.py
├── docker_build_and_tag.sh
├── flyte
│   ├── __init__.py
│   └── workflows
│       ├── __init__.py
│       └── workflow.py
├── flyte.config
└── requirements.txt

All, I am beyond excited to announce that Union.ai (the company I founded last year) just announced that we raised 10 million dollars to invest in Flyte community and open source. This would not have been possible without the support of all of you folks, and some who have been critical - @Guillaume Perchais / @Nelson Arapé / @Babis Kiosidis / @jeev / @Anmol / @Pradithya Aria Pura / @Miggy / @varsha Parthasarathy / @Viktor Barinov / @Grant McKenzie / @Stephen / @Maarten de Jong and many others. Apologize if I did not include you. I promise you this is the start. Stay tuned for massive updates to the user experience now and in this year to performance, reliability and focused features and especially UI. We are stoked to build the most differentiated complete orchestration platform powered by Kubernetes. https://techcrunch.com/2022/04/12/union-ai-raises-10m-to-simplify-ai-and-ml-workflow-orchestration/

Hi, everyone I'm new here and if this isn't the right place to ask, please, let me know I've been testing Flyte on a virtual machine. I've started with this tuturial: https://docs.flyte.org/en/latest/getting_started/deploy.html The first time was successful. Sandbox was created, run the project example from UI. But the second time, when I executed the command "flytectl sandbox start --source ." to start the sandbox, I choose to start the one that already exists from the previously execution and says "Flyte UI is available at http://localhost:30081/console", but I'm unable to connect.

@Prafulla Mahindrakar, mind helping @Alex Pozimenko with this? He's trying to get metrics on terminated tasks/executions annotated with the phase they ended as. We merged this PR https://github.com/flyteorg/flyteadmin/pull/386/files and he's trying with admin version 0.6.131 (should include this PR) but can't see any phases under:

flyte-admin-task.execution.manager-task.executions.terminated.counter

Hi everyone, im new here, Any notion of priority in flyte? I would like something that reorder workflow by priority for queued workflow and even better, running workflow.

hey, I cannot pull anymore this image

docker pull <http://cr.flyte.org/flyteorg/flyteconsole-release:v0.19.3|cr.flyte.org/flyteorg/flyteconsole-release:v0.19.3>

it doesn’t find it. Any idea why?

👋 Hey everybody, amazing project. Ive been learning and reading flyte's code. Is there any roadmap docs? so far I have only found https://github.com/orgs/flyteorg/projects/3/views/1 are there any recordings of the community syncs ? 🙂