#1388 [Housekeeping] Remove execution time task resource resolution from flyteadmin Issue created by katrogan Describe the Issue As part of #1361 we introduce task resource resolution in propeller, rather than flyteadmin. However to support backwards compatibility during the upgrade process, the existing code in flyteadmin needs to be cleaned up in a subsequent release. What if we do not do this? Convoluted task resource resolution at execution time within flyteadmin that is discarded. Related component flyteadmin flyteorg/flyte

#1369 [Core Feature] Capture and save documentation from flytekit entities Issue created by katrogan Motivation: Why do you think this is important? See #531 for a richer discussion on this feature. For this specific issue, we will want to capture user documentation, links to source code and commit version and pipe that through the Flyte system and make entity descriptions for tasks, workflows, launch plans and executions retrievable via a separate endpoint. Goal: What should the final outcome look like, ideally? It should be possible to add RST-formatted documentation inline with flytekit entity definitions or when launching executions from any means and have that information made retrievable by end users. Describe alternatives you've considered Again, see #531 [Optional] Propose: Link/Inline OR Additional context Design doc: https://docs.google.com/document/d/1wvCvM6pLQ17Zr0DcpS9es2vOFQsNeOaNN99ufGw8O1U/edit?usp=sharing flyteorg/flyte

#1354 [BUG] Flyteadmin shouldn't allow launching an execution in archived state Issue created by pmahindrakar-oss Describe the bug Each launch plan can be in one of two states. • Archived • Active Admin exposes an api to toggle between these two states. But in case a user launches an execution for a launchplan then this state flag is ignored. Expected behavior Flyteadmin create execution should throw an error in case the launch plan state is archived. [Optional] Additional context To Reproduce Steps to reproduce the behavior: 1. update the launch plan state to archived 2. create an execution for the same launchplan Screenshots If applicable, add screenshots to help explain your problem. flyteorg/flyte

#1338 [Flytectl Feature] support local registry for sandbox Issue created by tekumara Describe the feature/command for flytectl Any locally built images need to be created inside the sandbox. This means mounting the local directory at sandbox creation time, and entering the sandbox to build the image. The sandbox can only build images for the mounted directory (ie: repo) it was started with. To switch repos, a new sandbox needs to be created. Overall the process is a little unituitative and leads to scenarios like #1319 Provide a possible output or ux example It might be a better UX to run a local registry that both the sandbox and host have access to. Images can then be pushed to the local registry and be accessed from the sandbox without mounting local source directories. This would also make it easier to move between source repos. flyteorg/flyte

#1312 [BUG][flytekit] Default input argument not being considered in PythonInstanceTask Issue created by samhita-alla Describe the bug The default input argument is not being considered when calling the

PythonInstanceTask

. Observed this when building a task plugin. Task's `__init__()`:

def __init__(
        self,
        ...
        **kwargs,
    ):
        ...
        inputs.update({"runtime": (bool, False)})

        super(..., self).__init__(
            ...
            interface=Interface(inputs=inputs, outputs=outputs),
            **kwargs,
        )

Example:

task_object = ...(
        ...
        inputs=kwtypes(dataset=str),
        ...
    )

    @workflow
    def valid_wf(dataset: str = "...") -> None:
        task_object(dataset=dataset)

    valid_wf()

Error:

flytekit.common.exceptions.user.FlyteAssertion: Input was not specified for: runtime of type simple: BOOLEAN

Expected behavior The above code shouldn't be throwing an error. [Optional] Additional context To Reproduce Steps to reproduce the behavior: 1. Same as the above code flow. Screenshots If applicable, add screenshots to help explain your problem. flyteorg/flyte

#1300 [BUG] FlyteFile type parameter causes type checker failures Issue created by tekumara FlyteFile uses type parameters to specify metadata about the file type, eg:

return FlyteFile["jpg"](path=out_path)

This causes my type checker (pylance/pyright) to complain because

"jpg"

is not defined. And when a type parameter isn't supplied in a signature (ie: it's just FlyteFile) then the return type is partially unknown and inferred as

FlyteFile[Unknown]

NB: The same applies to

FlyteDirectory

. flyteorg/flyte

#1273 [Housekeeping] Upgrade to apiextensions.k8s.io/v1 CustomResourceDefinition in deployment resources (prepare for v1.22+) Issue created by brucearctor Describe the bug

Warning: <http://apiextensions.k8s.io/v1beta1|apiextensions.k8s.io/v1beta1> CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use <http://apiextensions.k8s.io/v1|apiextensions.k8s.io/v1> CustomResourceDefinition

is pointed out when running Kustomize deployment Expected behavior Current version of K8s is 1.21 -- we should upgrade that to refer to rbac.authorization.k8s.io/v1 by v.1.22+ release flyteorg/flyte

#1260 [Flytectl Feature] [flytekit-java] Flytectl support Flytekit-Java based workflows Issue created by brucearctor Motivation: Why do you think this is important? Flytectl is the to-be tool for interacting with Flyte. Flyte is to support tasks/workflows in multiple languages (notably python, which is well established and java/scala). FlyteCTL therefore should work with the various supported languages. Goal: What should the final outcome look like, ideally? Flytectl allows the same [or similar] registration, running, etc of workflows for Java/Scala [or future languages] as it does for Python Describe alternatives you've considered Using JFlyte ... * Caveats * The lack of support but ability to work comes from a slack conversation: https://flyte-org.slack.com/archives/C014PL3HUTF/p1626473952005700 ... Perhaps this is more about bundling JFlyte. I imagine once workflows are registered (regardless of language) that Flytectl should be able to trigger/run them -- but I haven't tested that, yet. Anyways, seems I should clarity the extent of what specifically is possible and what doesn't work level-of-effort, etc -- and perhaps this warrants a design doc [ @brucearctor willing to take on if that is the case ] rather than a github issue ( will also make sure to connect with @kanterov ). flyteorg/flyte

#1254 [Docs] Flytekit-Java Docs Issue created by brucearctor Describe the documentation update Flytekit-java docs don't exist nicely like flytekit and other parts of flyte. The link just goes to the github repo. It is likely early, but this would be to track what to use to generate docs, and highlight this is an [eventual ... perhaps not the priority, yet] nice-to-have. flyteorg/flyte

#1177 [Core Feature] Support DynamicWorkflow TaskResolver Issue created by EngHabu Motivation: Why do you think this is important? I would like to be able to dictate which secrets to load based on inputs. The only way I could think of doing that is something like this:

@dynamic
def my_task(secret_key: str) -> str:
   @task(secret_requests=[Secret(group="my-group", key=secret_key)])
   def inner_task() -> str:
      ...
   return inner_task()

Describe alternatives you've considered 1. Loading all keys for a secret. Doesn't work with some secret providers (particularly AWS Secret Manager) 2. This worked:

@task(secret_requests=[Secret(group="my-group", key="")])
def inner_task() -> str:
    ...

@dynamic
def my_task(secret_key: str) -> str:
   inner_task.secret_requests = [Secret(group="my-group", key=secret_key)]

   return inner_task()

flyteorg/flyte

#1169 [Housekeeping] Make NamedEntity requests consistent Issue created by katrogan Describe the Issue Resource Types aren't inherently bound to NamedEntityIdentifiers. As a result they are defined separately, like here. Resource types are sometimes embedded in list requests but sometimes not. We should moved the resource type to the NamedEntityIdentifier and make all list request references to NamedEntities consistent. What if we do not do this? Leads to a more confusing API Related component flyteidl flyteorg/flyte

#1149 [Core Feature] Allow tasks/config to specify max queue/wait time Issue created by EngHabu Motivation: Why do you think this is important? In cases when the underlying execution engine (AWS Batch, K8s, Spark, Hive, AWS EMR, GCP BigQuery... etc.) is having issues scheduling flyte workloads, sometimes the workload get stuck. While Flyte has a concept of timeout, it only measures the execution timeout overall. Which doesn't allow the users to express their tolerance for how much they can wait in a queue to get a task executing. Goal: What should the final outcome look like, ideally? Expose an additional

queue_timeout

flag that can be set at a global scope through configs, or at a task scope (ideally can also be on a project/domain/WF levels). And when flytepropeller detects that a task hasn't started executing for that period of time, it should just abort it. flyteorg/flyte

#1144 [Plugin][Flytekit] Support for TFRecord as loadable schema type Issue created by kumare3 Why would this plugin be helpful to the Flyte community Often times users want to process data using Spark, but data is passed to a Tensorflow training process. Parquet or other columnar structures are highly in-efficient for training. To solve this problem, the TF community has done some work. It would be wonderful, if we could perform this conversion automatically depending on the context. e.g. If the user accepts a TFRecord (data format) as a spark dataframe then we can convert, similarly if the User writes Spark dataframe, but somehow annotates it as TFRecord then we can auto-convert. Similarly, if the user reads the SparkDataframe into a process as TFRecords, we can do the conversion This library provides this trait https://github.com/tensorflow/ecosystem/tree/master/spark/spark-tensorflow-connector LinkedIn has further updated this library to make it better in some ways https://github.com/linkedin/spark-tfrecord Type of Plugin ☑︎ Python/Java interface only plugin ☐ Web Service (e.g. AWS Sagemaker, GCP DataFlow, Qubole etc...) ☐ Kubernetes Operator (e.g. TfOperator, SparkOperator, FlinkK8sOperator, etc...) ☐ Customized Plugin using native kubernetes constructs ☐ Other Can you help us with the implementation? ☐ Yes ☐ No flyteorg/flyte

#1132 [Flytectl Feature] Flag and config comparison with flyte-cli Issue created by pmahindrakar-oss Describe the feature/command for flytectl parse_proto Not yet supported list_task_names flytectl get task -p -d list_task_versions flytectl get task -p -d get_task flytectl get task -p -d --version launch_task flytectl get tasks -d -p --version --execFile <execution_spec.yaml> flytectl create execution --execFile <execution_spec.yaml> -p -d list_workflow_names flytectl get workflow -p -d list_workflow_versions flytectl get workflow -p -d get_workflow flytectl get workflow -p -d --version list_launch_plan_names flytectl get launchplan -p -d list_active_launch_plans flytectl get launchplan -p -d list_launch_plan_versions flytectl get launchplan -p -d --version get_launch_plan flytectl get launchplan -p -d --version get_active_launch_plan Only active launch plans are retruned always from the admin API update_launch_plan flytectl get launchplan -p -d --version execute_launch_plan flytectl get launchplan -d -p --version --execFile <execution_spec.yaml> flytectl create execution --execFile <execution_spec.yaml> -p -d watch_execution Not yet supported relaunch_execution flytectl create execution --relaunch -p flytectldemo -d development terminate_execution flytectl delete execution -d development -p flytesnacks list_executions flytectl get execution -p -d get_execution flytectl get execution -p flytesnacks -d development get_child_executions Not yet supported register_project flytectl create project --name flytesnacks --id flytesnacks --description "flytesnacks description" --labels app=flyte flytectl create project --file project.yaml Additionally supports adding labels list_projects flytectl get projects archive_project flytectl update project --archiveProject activate_project flytectl update project --activateProject register_files flytectl register files fast_register_files Supported

flytectl register files -p flytesnacks -d development -a flyte-package.tgz  -v v1-fast1

update_workflow_meta flytectl update workflow --activate -p -d flytectl update workflow --archive -p -d update_task_meta flytectl update task --activate -p -d flytectl update task --archive -p -d update_launch_plan_meta flytectl update launchplan --activate -p -d flytectl update launchplan --archive -p -d update_cluster_resource_attributes flytectl update cluster-resource-attribute --attrFile cra.yaml update_execution_queue_attributes flytectl update execution-queue-attribute --attrFile cra.yaml update_execution_cluster_label flytectl update execution-cluster-label --attrFile cra.yaml get_matching_attributes flytectl get task-resource-attribute -p -d flytectl get cluster-resource-attribute -p -d flytectl get execution-cluster-label -p -d flytectl get execution-queue-attribute -p -d flytectl get plugin-override -p -d list_matching_attributes flytectl get task-resource-attribute -p -d flytectl get cluster-resource-attribute -p -d flytectl get execution-cluster-label -p -d flytectl get execution-queue-attribute -p -d flytectl get plugin-override -p -d setup_config Not yet supported ** Linked tickets * Pagination support #1146 flyteorg/flyte

#1124 [Flytectl Feature] FlyteCli command porting to Flytectl Issue created by pmahindrakar-oss Describe the feature/command for flytectl The following lists the commands available in flyte-cli and status wrt porting over to flytectl. Note: I wasn't able to use table format so excuse the long listing. github is yet to support persisting the checkbox state in table markdowns https://stackoverflow.com/questions/66560651/checkbox-in-github-markdown-table-not-updating (https://github.github.com/gfm/#task-list-items-extension-) parse-proto ☑︎ Under-Review ☐ Port ☐ Do not port Status ☐ In-Progress ☐ Done ☐ NA list-task-names ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA list-task-versions ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA get-task ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA launch-task ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA list-workflow-names ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA list-workflow-versions ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA get-workflow ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA list-launch-plan-names ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA list-active-launch-plans ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA list-launch-plan-versions ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA get-launch-plan ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA get-active-launch-plan ☑︎ Under-Review ☐ Port ☐ Dont Port Status ☐ In-Progress ☐ Done ☐ NA update-launch-plan ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA execute-launch-plan ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA watch-execution ☑︎ Under-Review ☐ Port ☐ Dont Port Status ☐ In-Progress ☐ Done ☐ NA relaunch-execution ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA terminate-execution ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA list-executions ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA get-execution ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA get-child-executions ☑︎ Under-Review ☐ Port ☐ Dont Port Status ☐ In-Progress ☐ Done ☐ NA register-project ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA list-projects ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA archive-project ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA activate-project ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA register-files ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA fast-register-files ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☑︎ In-Progress ☐ Done ☐ NA update-workflow-meta ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA update-task-meta ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA update-launch-plan-meta ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA update-cluster-resource-attributes ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA update-execution-queue-attributes ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA update-execution-cluster-label ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA update-plugin-override ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA get-matching-attributes ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA list-matching-attributes ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☑︎ Done ☐ NA setup-config ☐ Under-Review ☑︎ Port ☐ Dont Port Status ☐ In-Progress ☐ Done ☐ NA Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Provide a possible output or ux example A clear and concise description of what you want to happen or achieve. flyteorg/flyte

#1123 [Core Feature] Flytekit client-side validation of pod specs Issue created by katrogan Motivation: Why do you think this is important? Before serializing a pod spec, we should validate it. Although we use the kube client library to generate pod spec constructs, we should explore using the api client to validate we can deserialize the pod spec before registering it. Goal: What should the final outcome look like, ideally? Faster turn-around for catching client-side pod spec validation issues. Describe alternatives you've considered The current solution of postponed debugging at execution time. [Optional] Propose: Link/Inline OR Additional context If you have ideas about the implementation please propose the change. If inline keep it short, if larger then you link to an external document. flyteorg/flyte

#1122 [Core Feature] Allow specifying k8sPod task targets using just yaml Issue created by katrogan Motivation: Why do you think this is important? The K8sPod target is a first-class task template target now. However, the only way to use it is to install the flytekit pod plugin which marshals a python-defined kubernetes pod spec to it. To utilize the K8sPod target in core/ flytekit we could allow users to specify a pod spec with yaml, which doesn't require installing the pod plugin's dependencies (in this case, the python kubernetes-client library). Goal: What should the final outcome look like, ideally? It should be possible to define a flyte task using valid k8s yaml without installing additional flytekit plugins. Describe alternatives you've considered Current implementation where only the pod plugin can serialize to the K8sPod target. [Optional] Propose: Link/Inline OR Additional context If you have ideas about the implementation please propose the change. If inline keep it short, if larger then you link to an external document. flyteorg/flyte

#1113 [Core Feature] Run Propeller K8sObjectMetadata adding through templating engine Issue created by wild-endeavor on <!date^1623288838^{date_short}|2021-06-10T01:33:58Z> Motivation: Why do you think this is important? Propeller today can inject default annotations and labels into the metdata of K8s objects it creates. * Goal * It might be useful for the values (maybe even the keys) to be passed through the templating system, so that a default label config of

"<http://corp.net/uniqueKey|corp.net/uniqueKey>": "{{ .nodeNameWithRetry }}"

would get filled in correctly. This is useful for metrics libraries potentially. Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered. flyteorg/flyte

#1063 [Housekeeping] [flyteadmin] Clean up emailer configuration Issue created by wild-endeavor Describe the Issue In adding support for external email services, configuration handling became ugly because it tries to handle both sources. See flyteorg/flyteadmin#199 What if we do not do this? Messier code to handle emailer configuration Related component flyteadmin flyteorg/flyte

#1033 [Housekeeping] [flytekit] Exception cleanup Issue created by wild-endeavor Describe the Issue During development of native-typing-flytekit, insufficient attention was paid to the selection of exception classes - generally we just raised

Exception

or at best

ValueError

in pretty much all cases. We should clean this up • Create a proper exception hierarchy for system errors (one already exists for user exceptions). • Potentially strip out the scoped exception business. See flyteorg/flytekit#543 for more background. What if we do not do this? Lack of polish. Related component Possibly related to #740. flyteorg/flyte

#1015 [flytekit-java] Support the rest of LaunchPlan fields Issue created by kanterov Why would this be helpful to the Flyte community There are several fields not accessible through `SdkLaunchPlan`: • annotations • labels • entity_metadata.notifications • security_context • quality_of_service • raw_output_data_config See: flyteidl/admin/launch_plan.proto flyteorg/flyte

#1014 [flytekit-java] Support for cache and cache_version Issue created by kanterov Why would this be helpful to the Flyte community Flyte provides the ability to cache the output of task executions in order to make subsequent executions faster. A well-behaved Flyte Task should generate deterministic output given the same inputs and task functionality. This is useful in situations where a user knows that many executions with the exact same inputs can occur. For example, your task may be periodically run on a schedule, run multiple times when debugging workflows, or commonly shared across different workflows but receive the same inputs. See documentation: https://docs.flyte.org/en/latest/howto/enable_and_use_memoization.html flyteorg/flyte

#1010 [flytekit-java] Support WorkflowMetadata and WorkflowMetadataDefaults Issue created by kanterov Why would this be helpful to the Flyte community From flyteidl:

// This is workflow layer metadata. These settings are only applicable to the workflow as a whole, and do not
// percolate down to child entities (like tasks) launched by the workflow.
message WorkflowMetadata {
    // Indicates the runtime priority of workflow executions. 
    QualityOfService quality_of_service = 1;

    // Failure Handling Strategy
    enum OnFailurePolicy {
        // FAIL_IMMEDIATELY instructs the system to fail as soon as a node fails in the workflow. It'll automatically
        // abort all currently running nodes and clean up resources before finally marking the workflow executions as
        // failed.
        FAIL_IMMEDIATELY = 0;

        // FAIL_AFTER_EXECUTABLE_NODES_COMPLETE instructs the system to make as much progress as it can. The system will
        // not alter the dependencies of the execution graph so any node that depend on the failed node will not be run.
        // Other nodes that will be executed to completion before cleaning up resources and marking the workflow
        // execution as failed.
        FAIL_AFTER_EXECUTABLE_NODES_COMPLETE = 1;
    }

    // Defines how the system should behave when a failure is detected in the workflow execution.
    OnFailurePolicy on_failure = 2;
}

// The difference between these settings and the WorkflowMetadata ones is that these are meant to be passed down to
// a workflow's underlying entities (like tasks). For instance, 'interruptible' has no meaning at the workflow layer, it
// is only relevant when a task executes. The settings here are the defaults that are passed to all nodes
// unless explicitly overridden at the node layer.
// If you are adding a setting that applies to both the Workflow itself, and everything underneath it, it should be
// added to both this object and the WorkflowMetadata object above.
message WorkflowMetadataDefaults {
    // Whether child nodes of the workflow are interruptible.
    bool interruptible = 1;
}

flyteorg/flyte

#1008 [flytekit-java] End-to-end tests Issue created by kanterov Why would this be helpful to the Flyte community We don't have OSS end-to-end tests. We can live for now with only testing workflows using container tasks. With us supporting more plugins, we should extend the scope of end-to-end tests. Something we have found useful before is using control plane classes from flytekit-python to execute flytekit-java workflows. flyteorg/flyte

#1006 [flytekit-java] Support non-runnable tasks Issue created by kanterov Why would this be helpful to the Flyte community Currently, all plugin tasks are supported by overriding

getType()

and

getCustom()

methods in

SdkRunnableTask

. All runnable tasks require users to implement

run()

method. Many plugins don't need tasks to be runnable, for instance, pseudo-code for Athena plugin.

abstract class AthenaTask extends SdkRunnableTask<Void, Void> {
  String getType() {
    return "athena";
  }

  abstract String getQuery();
  
  SdkStruct getCustom() { ... }

  Void run(Void inputs) {
    return null;
  }
}

class MyQueryTask extends AthenaTask {
  String getQuery() {
    return "SELECT 1";
  }
}

As you can see from this example, it's possible to hide

run()

method from users. However, SDK will see that the task is runnable, and will stage jars for it. That can be avoided if we introduce non-runnable tasks. We need to carefully iterate on the design of this feature, and see how to consistently fit it into existing testing support, and methods that work with runnable tasks. flyteorg/flyte

#1004 [flytekit-java] Add type parameter to SdkBindingData Issue created by kanterov Why would this be helpful to the Flyte community

SdkBindingData

should become

SdkBindingData<T>

, where

T

should reflect the corresponding Java or Scala type. This will allow surface type information only available in workflow graph as type information embedded into Java/Scala types. We should update all existing methods to preserve type information where it's possible, for instance:

class SdkBindingData {
  static SdkBindingData<String> literalOfString(String value);
}

flyteorg/flyte

#1003 [flytekit-java] Support dynamic workflows Issue created by kanterov Why would this plugin be helpful to the Flyte community Dynamic workflows allow to access input values during graph construction. In flytepropeller, they are implemented as tasks that produce workflow graphs, however, from the user perspective they are workflows. Presumably, the interface can look like this:

public abstract class SdkDynamicWorkflow<InputT> {
  public abstract void expand(SdkWorkflowBuilder builder, InputT input);
}

flyteorg/flyte

#1002 [flytekit-java] Support WorkflowNode Issue created by kanterov Why would this be helpful to the Flyte community WorkflowNode allows referencing sub-workflows or other launch plans. WorkflowNode should work with local execution as well. It should be possible to reference both

SdkWofkflow

,

SdkLaunchPlan

, and

SdkRemoteWorkflow

(to be added, similarly to

SdkRemoteTask

). flyteorg/flyte

#943 [Housekeeping] Supporting archival for launchplans and tasks Issue created by pmahindrakar-oss Describe the Issue Archiving an entity in flyte make it unavailable in flyteconsole in its current state. The UI filters and show only non-archived entities. Same goes for flytectl get commands for these entities Currently flyteadmin supports archival for projects and workflows. The other flyte entities of tasks and launchplans lack this feature and it would be useful to do the same for these entities. What if we do not do this? Short description of potential consequences Related component Either Specific / all flyteorg/flyte

#897 [BUG] Add retries in flytekit for network related errors Issue created by migueltol22 Describe the bug Intermittent network issues are to be expected. In the case of receiving a network related error response flytekit should retry the same request (e.g. serialization, executing workflow, listing workflows, etc). Example Error to retry on:

[2021-04-07T04:45:51Z] 	status = StatusCode.UNAVAILABLE
[2021-04-07T04:45:51Z] 	details = "upstream connect error or disconnect/reset before headers. reset reason: connection failure"
[2021-04-07T04:45:51Z] 	debug_error_string = "{"created":"@1617770750.462629552","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"upstream connect error or disconnect/reset before headers. reset reason: connection failure","grpc_status":14}"

Expected behavior On Network related error responses Flytekit should retry the request. This retry should follow a specified retry policy whether that be exponential back offs, incremental intervals, randomization or some other retry policy. [Optional] Additional context To Reproduce Steps to reproduce the behavior: 1. 2. Screenshots If applicable, add screenshots to help explain your problem. flyteorg/flyte