flyte-org #flyte-github

GitHub

11/02/2023, 9:23 PM

#3130 [Plugin] Add dbt build to dbt plugin Issue created by brandon-segal The dbt plugin should have the ability to run dbt build statements to allow for users to run statements which would seed, snapshot, run, and test their projects in one command. This would be accomplished through an implementation of a dbtBuild Task which would have its own set of arguments and outputs. flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#2743 add: ` alb.ingress.kubernetes.io/target-type: 'ip'` in the ingress annotation of value-eks.yaml Issue created by brucearctor This breaks the https://docs.flyte.org/en/latest/deployment/aws/manual.html#connecting-to-flyte Now we will have to add:

<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: 'ip'

in the ingress annotation of value-eks.yaml @brucearctor @EngHabu RE: #2566 Originally posted by @jw0515 in #2566 (comment) flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#3163 [Docs] How to verify that the GRPC service of flyteadmin works as expected Issue created by samhita-alla Description Add a section to the troubleshooting guide: https://discuss.flyte.org/t/8152768/Hi-Community-is-there-any-simple-approach-to-verify-the-GRPC. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#3150 [BUG] Difficult to diagnose invalid sendgrid email failures Issue created by katrogan Describe the bug When using launch plan notifications with a sengrid emailer, emails can fail with difficult to diagnose issues, such as

Copy code

Sendgrid sent email {\"errors\":[{\"message\":\"The from email does not contain a valid address

Expected behavior It should be easy to pinpoint invalid notification emails from logs that report email failures Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#3139 [Housekeeping] Migrate protobuf stub generation to buf Issue created by eapolinario Describe the issue We currently rely on a series of old (and very-hard-to-update) code to generate the stubs for the different languages we support in flyteidl. Instead we should leverage buf and move over the step at which we generate the stubs, e.g. if we adopt buf remote plugins we don't even have to maintain generated stubs. What if we do not do this? It's nearly impossible to update the original script, this makes it hard for us to adopt newer technologies, for example grpc-gateway v2 (which includes the swagger/openapiv2 annotations). Related component(s) flyteidl Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#2797 [Plugin] MLflow integration Issue created by cosmicBboy The purpose of this task is to come up with a way of using mlflow in a flyte task. The light-weight integration would simply be to use the mlflow tracking API within a Flyte task to log metrics. Indeed, some of our users are already doing this, and it might make sense to document this in an "MLflow" section in the integration docs. However, there's an opportunity for a declarative API that would handle some subset of the mlflow logging functions, for example: 1. setting tracking uri 2. creating experiments 3. setting an experiment 4. handling start/end runs 5. automatic logging based on the ML framework This would be beneficial for end users so that they can follow what we think of as "best practices" for how to use MLFlow in Flyte, e.g. not having to think about how to parameterize and name experiments correctly. For example:

Copy code

pip install flytekitplugins-mlflow

API Proposal 1: Decorator Plugin Use the task decorator and/or workflow decorator pattern to create a more seamless experience. This would introduce a new plugin pattern in flytekit, which modifies the underlying function wrapped by

@task

and

@workflow

. Example

Copy code

import mlflow
import flytekitplugins.mlflow
from flytekit import task, dynamic


@dynamic
@flytekitplugins.mlflow.experiment(
    # args to <https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.create_experiment>
    name=None  # defaults to "{workflow_name}-{execution_id}" ?
    artifact_location=None,  # defaults to the flytekit location?
    tags=...,
)
def model_experiment(hyperparameter_grid: List[dict]):
    models = []
    data = ...
    for hyperparameters in hyperparameter_grid:
        models.append(train_model(hyperparameters=hyperparameters, data=data))
    ...


@task
@flytekitplugins.mlflow.run(
    # by default, this run will use the parent workflow's mlflow experiment config
    params="hyperparameters",  # log config parameters automatically
    autolog=True,  # enable autologging, could also be a dict of mlflow.autolog args: <https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.autolog>
)
def train_model(hyperparameters: dict, data: ...):
    model = MySklearnModel(**hyperparameters)
    ... # fit
    
    # without autolog=True, users can manually log here
    mlflow.log_metric("key", value)

    return model

API Proposal 2: extend
@task
and
@workflow
arguments Task config plugins don't really make sense for MLFlow experiment tracking/logging, since the

task_config

argument is typically used for task types that have specific backend resource requirements (e.g. Spark, Ray, MPI tasks) and is orthogonal to configuring experiments and logging metrics. Therefore, to support similar functionality to proposal 1, we could introduce additional arguments to the

@task

and

@workflow

decorators, e.g.

Copy code

import mlflow
from flytekitplugins.mlflow import RunConfig, ExperimentConfig
from flytekit import task, workflow

@dynamic(..., logging_config=ExperimentConfig(name=..., artifact_location=..., tags=...)
)
def model_experiment(hyperparameter_grid: List[dict]):
    models = []
    data = ...
    for hyperparameters in hyperparameter_grid:
        models.append(train_model(hyperparameters=hyperparameters, data=data))
    ...

@task(..., logging_config=RunConfig(experiment=..., params=..., autolog=...)
)
def train_model(hyperparameters: dict):
    model = MySklearnModel(**hyperparameters)
    ... # fit

    # without autolog=True, users can manually log here
    mlflow.log_metric("key", value)

    return model

flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#2795 [Core feature] [flytekit] Move big query writer to access storage layer directly Issue created by wild-endeavor Goal: What should the final outcome look like, ideally? The big query structured dataset reader currently reads from the storage client directly but the writer uses the higher level BigQuery client. This has additional cost implications. We should move the big query encoder to also use the storage layer directly. Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#2794 [Core feature] [flytekit] Run-time configuration for Structured Dataset Issue created by wild-endeavor Structured Dataset Configuration Goal The current encoders/decoders for Structured Dataset don't necessarily suit all use cases for all users at all times. Specifically behaviour around the run-time column strictness - for example, in the following scenario

Copy code

@task
def t1() -> Annotated[pd.DataFrame, kwtypes(a=int)]:
  ...

@task
def t2(df: Annotated[pd.DataFrame, kwtypes(a=float)]):
 ...

It's conceivable that sometimes you want this type conversion (from int to float) to happen and sometimes you do not. What is the correct way of allowing the user to control this behavior? Also where should this be enforced? Enforcing this at run-time is likely best to give as much flexibility to the plugin author as possible. Implementation ? Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#2790 [Docs] Docs on how to assign project-domain-specific permissions Issue created by samhita-alla Description We need docs on how to define a service account at the project-domain level and annotate it with an env-user-role. Reference Slack thread: https://discuss.flyte.org/t/2404214/yes-but-i-don-t-think-it-works-across-accounts-i-e-eks-is-de Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#2777 [FlyteCTL Feature] Add support for get workflow-meta,task-meta, launchplan-meta Issue created by pmahindrakar-oss Describe the feature/command for FlyteCTL There are commands to update launchplan-meta https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_launchplan-meta.html task-meta https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_task-meta.html workflow-meta https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_workflow-meta.html But we don't have corresponding get commands Provide a possible output or UX example flytectl get worflow-meta flytectl get task-meta flytectl get launchplan-meta If possible this can even be combined with existing flytectl get worflow flytectl get task flytectl get launchplan and merge the results Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#2771 [BUG] Flytekit is using `no-sign-request` when using `pyflyte-fast-execute` Issue created by stephen37 Describe the bug When using fast-registration, during the initialisation of the task, we copy the code from S3 to the container. Flytekit is using

pyflyte-fast-execute

in that case, the problem is that by default we add the argument

--no-sign-request

when using AWS CLI for S3 and that makes it impossible to download the content from private S3 buckets. Expected behavior Not being anonymous by default, we shouldn't add the argument

--no-sign-request

by default Additional context to reproduce 1. Run `pyflyte-fast-execute`with the needed arguments

Copy code

❯ pyflyte-fast-execute --additional-distribution s3://<private-bucket-name>/ea/<project-name>/<domain>/path_to.tar.gz --dest-dir /src

{"asctime": "2022-08-16 14:21:55,941", "name": "flytekit", "levelname": "ERROR", "message": "Error from command '['aws', '--no-sign-request', 's3', 'cp', 's3://<private-bucket-name>/ea/<project-name>/<domain>/path_to.tar.gz', '/src']':\nb'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\\n'\n"}
{"asctime": "2022-08-16 14:21:55,941", "name": "flytekit", "levelname": "ERROR", "message": "Exception when trying to execute ['aws', 's3', 'cp', '3://<private-bucket-name>/ea/<project-name>/<domain>/path_to.tar.gz', '/src'], reason: Called process exited with error code: 1.  Stderr dump:\n\nb'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\\n'"}

2. It can also be reproduced by using the AWS CLI commands directly When using

--no-sign-request

Copy code

❯ aws --no-sign-request s3 cp s3://<private-bucket-name>/ea/<project-name>/<domain>/path_to.tar.gz .

fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

When not using

--no-sign-request

Copy code

❯ aws s3 cp s3://<private-bucket-name>/ea/<project-name>/<domain>/path_to.tar.gz .

download: s3://<private-bucket-name>/ea/<project-name>/<domain>/path_to.tar.gz to ./fast14de81dc9a6b0f459a1ab49e1d871e01.tar.gz

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#3036 [Core feature] [Flytekit] Implement Multipart upload and download for flyteplugins-vaex Issue created by ryankarlos Motivation: Why do you think this is important? vaex plugin currently implemented flyteorg/flytekit#1230 only supports writing chunks to single file using

df.export(...)

as explained in the docs. We would ideally also like to support cases where if the dataframe is large, we can export the chunks in parallel to multiple parts using

df.export_many(...)

as demostrated in the vaex api docs https://vaex.readthedocs.io/en/docs/api.html#vaex.dataframe.DataFrameLocal.export_many Goal: What should the final outcome look like, ideally? if df chunksize is greater than some threshold (1M ?) than should serialise to blob using

df.export_many(...)

, otherwise default to current implementation of

df.export

https://github.com/flyteorg/flytekit/blob/8ae879eb379acf2e0b4923f1b0c855d01a1f14e5/plugins/flytekit-vaex/flytekitplugins/vaex/sd_transformers.py#L29-L38 When decoding use

df.open_many(...)

if dir has multiple files/parts (use glob pattern) or

df.open(...)

(as currently implemented) if just single part. https://github.com/flyteorg/flytekit/blob/8ae879eb379acf2e0b4923f1b0c855d01a1f14e5/plugins/flytekit-vaex/flytekitplugins/vaex/sd_transformers.py#L51-L57 Describe alternatives you've considered N/A Propose: Link/Inline OR Additional context See discussion in this PR flyteorg/flytekit#1230 (comment) Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#3028 [BUG] UI sends an invalid request when selecting my executions on an unauthenticated admin Issue created by pmahindrakar-oss Describe the bug UI has a new feature to filter executions by user and which is shown as checkbox on the project dashboard page. But this checkbox doesn't make sense when the UI is talking to non-Oauth enabled admin eg request : http://localhost:30080/api/v1/executions/flytesnacks/development?filters=eq(user,)&limit=25&sort_by.direction=DESCENDING&sort_by.key=created_at

Screenshot 2022-10-26 at 10 26 00 AM▾

Expected behavior UI Shouldnot have filter by my executions check box when talking to non-Oauth enabled admin Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#3003 [BUG] flytekitplugins-dbt not logging std output Issue created by roek5803 Describe the bug When running a DBT task with the flytekit DBT plugin we expect it to log the std output so we can see what is happening. Looking at the code, the current implementation collects the logs from std output but will only print it in case of an error. Expected behavior We would expect the library to print the std output of the DBT commands ideally with some controlling mechanism e.g. with a verbose flag. Additional context to reproduce

Copy code

from flytekit import workflow
from flytekitplugins.dbt.task import DBTRun, DBTRunInput

dbt_task = DBTRun(name="the_name_of_the_task")

@workflow
def my_wf() -> None:
    input = DBTRunInput(
            project_dir="dbt_project_dir",
            profiles_dir="dbt_project_dir/docker-context",
            profile="default",
            select=["some_model"]
    )
    
    dbt_task(input=input)

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#2998 [Core feature] Support launchplans in pyflyte run Issue created by eapolinario Motivation: Why do you think this is important? Currently we only support launching tasks and workflows in

pyflyte run

. We should add support launchplans as well. Goal: What should the final outcome look like, ideally? We should be able to use the same experience currently offered in

pyflyte run

to kickoff launchplans, i.e. users should be able to execute:

Copy code

> pyflyte run --remote path/to/my/file.py my_launchplan
Go to <http://flyte.example.com/console/projects/flytesnacks/domains/development/executions/f6e7113930e3043e79cc> to see execution in the console.
...

Describe alternatives you've considered launchplans are an indispensable entity type in the Flyte programming model, hence it should be supported in the CLIs. Propose: Link/Inline OR Additional context We should add launch plans to the list of entities handled by

pyflyte run

in here. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#2997 [Housekeeping] Makefile commands not working on windows 10 Issue created by jerempy Describe the issue On windows 10, in https://github.com/flyteorg/flyteidl and unable to use make commands for

make download_tooling

make generate

in order to compile proto buffs. Using wsl2 get this error:

Copy code

Makefile:17: warning: overriding recipe for target 'generate'
boilerplate/flyte/golang_test_targets/Makefile:13: warning: ignoring old recipe for target 'generate'
Makefile:30: warning: overriding recipe for target 'test_unit'
boilerplate/flyte/golang_test_targets/Makefile:38: warning: ignoring old recipe for target 'test_unit'
make: boilerplate/flyte/golang_test_targets/download_tooling.sh: Command not found
make: *** [boilerplate/flyte/golang_test_targets/Makefile:9: download_tooling] Error 127

Also tried running the make commands in windows shell wth wsl, in wsl shell directly, in git bash. - no results. What if we do not do this? Some windows developers might not be able to help extend or add proto buffs if needed Related component(s) flyteorg/flyteidl#331 #2911 <-- this might help my issue Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#2990 [BUG] Dashboard components overlap with each other when user goes back to dashboard after terminating a workflow Issue created by aliabbasjaffri Describe the bug I was running flytesnacks, and wanted to terminate a workflow. After i clicked terminate, and then went back to the dashboard using the

<-

button, the UI of some components appears to be overlapping with each other. Expected behavior The UI should remain consistent. After user refreshes the page, the UI gets consistent again. Additional context to reproduce No response Screenshots

Screenshot 2022-10-14 at 10 31 23▾

UI with overlapping components

Screenshot 2022-10-14 at 10 31 36▾

UI after i refreshed the page. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:23 PM

#2985 [Core feature] : Support disabling profiler Issue created by fredrickprashanth Motivation: Why do you think this is important? Currently pprof is enabled by default on all golang components flytepropeller, flyte-pod-webhook etc. Profiler should be default disabled. Please add an option in the config to enable/disable the profiler. It should be allowed to enable only via config option as it serves mainly developement. Goal: What should the final outcome look like, ideally? Golang pprof is not enabled by default on the flyte components. Describe alternatives you've considered There is no current alternative to disable profiler. Propose: Link/Inline OR Additional context Example Propeller profiler config Provide a config field to disable the profiler. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:24 PM

#2980 [BUG] Task returning NamedTuple fails with USER::Pod failed. No message received from kubernetes Issue created by tekumara Describe the bug A task with the following signature:

Copy code

from typing import Dict, List, NamedTuple
from flytekit import task, workflow

class OpenFlightsData(NamedTuple):
    routes: List[Dict[str, str]]
    airlines: Dict[str, str]
    airports: Dict[str, Dict[str, str]]


@task()
def extract_reference_data() -> OpenFlightsData:
   pass

Fails with

Copy code

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[fb5562eecbc2f40f9b0d-n0-0] terminated with exit code (137). Reason [Error]. Message: 

.

Expected behavior No failure Additional context to reproduce flytekit 1.2.1 flyte admin 1.1.46 Screenshots

Screen Shot 2022-10-13 at 10 36 58 pm▾

Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:24 PM

#2977 Failed to create a launch plan for FlyteWorkflow Issue created by pingsutw Motivation: Why do you think this is important? For now, we can only create a launch plan for WorkflowBase, and we'll get the following error if we try to create LP for FlyteWorkflow (The workflow fetched by flyte remote)

Copy code

File "/Users/.../lib/python3.7/site-packages/flytekit/core/launch_plan.py", line 143, in create
    native_types=workflow.python_interface.inputs,
AttributeError: 'NoneType' object has no attribute 'inputs'

Code snippet:

Copy code

from flytekit import LaunchPlan

flyte_workflow = remote.fetch_workflow(
    name="my_workflow", version="v1", project="flytesnacks", domain="development"
)
launch_plan = LaunchPlan.get_or_create(name="my_launch_plan", workflow=flyte_workflow)

Goal: What should the final outcome look like, ideally? Should be able to create a launch plan for FlyteWorkflow Describe alternatives you've considered NA Propose: Link/Inline OR Additional context https://flyte-org.slack.com/archives/CP2HDHKE1/p1665377001295529 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:24 PM

#2969 [FlyteCTL Feature] Add verbose logging for other components while using `--verbose` Issue created by DarthBenro008 Describe the feature/command for FlyteCTL As discussed in flyteorg/flytectl#359 (review) of flyteorg/flytectl#359 The

--verbose

flag of

flytectl sandbox start/teardown

and

flytectl demo start/teardown

only captures logs of

helm/k8s

commands, we would want to capture the logs of other components as well like

docker

k3s

, etc. The above mentioned PR already brings support to interfacing with

docker

I would be happy to send a PR for the same! :D Provide a possible output or UX example Kindly refer to flyteorg/flytectl#359 for more details (screenshots attached) Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:24 PM

#2942 A developer environment, for faster and efficient contributions Issue created by AnirudhDaya Feature request Hello maintainers. I would like to add gitpod to your repo to help beginners contribute. • Gitpod is an online IDE which can be launched from any GitHub page. Within seconds, Gitpod provides a fully working development environment, including a VS Code-powered IDE and a cloud-based Linux container explicitly configured for the project • Gitpod is highly contextual, such that it opens the IDE in the correct mode depending on the context: • If you are looking at a particular file of a certain commit on GitHub, starting a Gitpod workspace will check out the right version and open the file you’ve been looking at in the IDE. • Starting a Gitpod workspace from an issue will automatically create a branch and preconfigure the commit message. • Once you are in the IDE, you can interact with GitHub in various ways. Besides the obvious Git integration, you can do things like commenting inline in editors, approving and even merging PRs. Motivation I have seen many repos adopting gitpod to help new contributors in their open-source journey. It's hassle-free. You don't need to worry about dependencies not being present, since gitpod does all the heavy lifting. My contribution I can add a badge to your repo in the contribute.rst file similar to this one: <https://gitpod.io/#<your-repository-url>%7C Contribute with Gitpod

flyteorg/flyte

GitHub

11/02/2023, 9:24 PM

#2937 Workflow name changes based on the script name or package structure when registering Issue created by sanjaychouhan-adf Describe the feature/command for FlyteCTL Detail : https://flyte-org.slack.com/archives/CP2HDHKE1/p1664440196382279 Basically we should have option to not prepend the workflow or task names with the script file name and the package structure. Provide a possible output or UX example Workflow and task names on the UI should be just the function name and not the scriptname.functionname Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:24 PM

#2931 Issue with registering files to projects in flytectl Issue created by sakthi-tech4th After, packaging the files with pyflyte, I have been trying to register them to flytesnacks project in the flyteconsole. FlyteConsole is up and running in the Sandbox cluster. However, I'm getting the following error:

image▾

Any idea on the cause of this error? Is there a quick fix, i can work on? Thanks and Regards flyteorg/flyte

GitHub

11/02/2023, 9:24 PM

#2929 Flyte Console - ValueError: Generic Type typing.Union not supported currently in Flytekit Issue created by sakthi-tech4th Hi, I'm trying to launch a workflow in the Flyte Console (Sandbox Environment in a Windows Machine) and I'm getting the above mentioned error for a task that accept pandas DataFrames (2 data frames) and returns a DataFrame.

Copy code

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.

[adbrq7s7x2kbwln2k5df-n0-0] terminated with exit code (1). Reason [Error]. Message:  
a_worflows.py", line 211, in  
def process_data_frames(lc_df: pd.DataFrame=None, data_frame: pd.DataFrame=None)->pd.DataFrame:  
File "/usr/local/lib/python3.9/site-packages/flytekit/core/task.py", line 209, in task  
return wrapper(_task_function)  
File "/usr/local/lib/python3.9/site-packages/flytekit/core/task.py", line 193, in wrapper  
task_instance = TaskPlugins.find_pythontask_plugin(type(task_config))(  
File "/usr/local/lib/python3.9/site-packages/flytekit/core/tracker.py", line 30, in *call*  
o = super(InstanceTrackingMeta, cls).*call*(*args, **kwargs)  
File "/usr/local/lib/python3.9/site-packages/flytekit/core/python_function_task.py", line 118, in *init*  
super().*init*(  
File "/usr/local/lib/python3.9/site-packages/flytekit/core/python_auto_container.py", line 72, in *init*  
super().*init*(  
File "/usr/local/lib/python3.9/site-packages/flytekit/core/base_task.py", line 389, in *init*  
interface=transform_interface_to_typed_interface(interface),  
File "/usr/local/lib/python3.9/site-packages/flytekit/core/interface.py", line 210, in transform_interface_to_typed_interface  
inputs_map = transform_variable_map(interface.inputs, input_descriptions)  
File "/usr/local/lib/python3.9/site-packages/flytekit/core/interface.py", line 318, in transform_variable_map  
res[k] = transform_type(v, descriptions.get(k, k))  
File "/usr/local/lib/python3.9/site-packages/flytekit/core/interface.py", line 332, in transform_type  
return _interface_models.Variable(type=TypeEngine.to_literal_type(x), description=description)  
File "/usr/local/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 611, in to_literal_type  
transformer = cls.get_transformer(python_type)  
File "/usr/local/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 584, in get_transformer  
raise ValueError(f"Generic Type {python_type.*origin*} not supported currently in Flytekit.")  
ValueError: Generic Type typing.Union not supported currently in Flytekit.

the data frames has columns with strings, consequently the

dtypes

for these columns were

object

. Would it be possible, for some one to let me know what is the problem or if I'm missing something? Of if this because, I'm running it in a Windows machine? Thanks and Regards flyteorg/flyte

GitHub

11/02/2023, 9:24 PM

#2914 [UI Feature] Users should be able to create new schedules on the UI Issue created by kumare3 Motivation: Why do you think this is important? LaunchPlan creation and schedule creation should be supported in the UI Goal: What should the final outcome look like, ideally? Users can create new schedules for workflows from the UI Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:24 PM

#2913 [UI Feature] FlyteConsole launchForm should allow naming an execution Issue created by kumare3 Motivation: Why do you think this is important? Executions in Flyte can be named using this parameter Goal: What should the final outcome look like, ideally? Users are optionally able to enter the name for an execution Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:24 PM

#2910 [Core feature] Support for File Families/Files With Indices Indices Issue created by CalvinLeather Motivation: Why do you think this is important? In bioinformatics work especially, file families, or files with indices, are common. This occurs when multiple files with the same stem, but different suffixes, jointly contain a dataset. This occurs either because a single csv-like has a index allowing for efficient lookups (e.g., a .vcf file and a tabix-based .vcf index), or when a library stores a file in multiple parts, often because matrix/tabular data is stored seperately from metadata about rows and columns (for example, a plink .bed triple containing genetic information about a number of samples/people, where the .bed file stores the matrix of data, .fam stores information about samples in the data e.g. IDs of the people aka the columns, and the .bim stores informations about the sites/variants of DNA aka the rows. Providing better support for these files will enhance the UX for people writing bioinformatics-pipelines with Flyte. Goal: What should the final outcome look like, ideally? The following should be true: • When using this input pattern in a shell task, the various components of the family are all put into the same path. This is required, as, for example, vcftools expects the index of a .vcf file to have the same stem (i.e., to be next to it with the same filename, e.g.

~/dir/test.vcf

and

~/dir/test.vcf.tbi

). • A task that takes a file family as input should take 1 input value, not N. In the user's mental model of computation to be done, the file family of N files is one entity, not N. Allowing the inputs/outputs to map to this model will improve the UX. • Helper utilities to enforce presence of files in the family at runtime (e.g., if you create one of these objects, and you're missing one the components, you can have an error raised) • It should be easy to extend these classes with additional methods/transformers to allow easy read out of the file families into memory. Typically this is done with another library (e.g., vcf + .vcf.tbi is loaded with pyvcf, a python library backing C-based htslib, a high performance library for loading/manipulating vcf files and their indices). Describe alternatives you've considered • FlyteDirectory • Extend FlyteFile (suggested by Greg Gydush in slack https://flyte-org.slack.com/archives/CP2HDHKE1/p1663719669816229?thread_ts=1663692816.505609&cid=CP2HDHKE1) • Compose multiple FlyteFiles into a composite Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:24 PM

#2905 [Docs] Render flytekit.configuration.FastSerializationSettings as a clickable entity Issue created by SmritiSatyanV Description Here, render the class definition

FastSerializationSettings

as rendered in the flytekit repo. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

11/02/2023, 9:24 PM

#2908 [UI Feature] Set timestamps to local timezone in console Issue created by kpalen Motivation: Why do you think this is important? Folks checking on the status of workflows and tasks in the Flyte console sometimes have trouble with time zone conversion in their head. Goal: What should the final outcome look like, ideally? Users would like to set the timezone on their console to their local timezone instead of UTC. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte