#3037 [Core feature] [Flytekit] Add support for HDF5 and Arrow in flyteplugins-vaex Issue created by ryankarlos Motivation: Why do you think this is important? Currently

flyteplugins-vaex

supports automatic serialization and deserialization of vaex dataframe between consecutive tasks using parquet flyteorg/flytekit#1230 It would be good to extend this to HDF5 and arrow for performance and interoperability, when data sets are too large to fit into memory https://vaex.readthedocs.io/en/latest/faq.html#What-is-the-optimal-file-format-to-use-with-vaex Goal: What should the final outcome look like, ideally? Register extra handlers

VaexDataFrameToHDF5EncodingHandler

and

VaexDataFrameToArrowEncodingHandler

, so users can use

Annotated

to update the default format:

@task
def t1(f: vaex.dataframe.DataFrameLocal) -> Annotated[StructuredDataset, HDF5]

@task
def t2(f: vaex.dataframe.DataFrameLocal) -> Annotated[StructuredDataset, Arrow]

Describe alternatives you've considered N/A Propose: Link/Inline OR Additional context See discussion thread here flyteorg/flytekit#1230 (comment) Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3049 [Core feature] Use nightly flytekit builds in functional tests Issue created by eapolinario Motivation: Why do you think this is important? The release cycle for new flytekit versions is extremely long, requiring a (manual) process. This makes the feedback loop for catching errors in build steps more error-prone than necessary. Goal: What should the final outcome look like, ideally? Invest in nightly flytekit builds and use them in functional tests. Describe alternatives you've considered We could have a manual weekly release as an alternative, but that would still leave us with a longer-than-necessary turnaround time. Propose: Link/Inline OR Additional context Generate nightly flytekit images and pipe them through functional tests. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3058 [Core feature] Provide a way to ignore portions of the signature for cache key calculation purposes Issue created by eapolinario Motivation: Why do you think this is important? It's useful to skip some inputs for cache key calculation purposes For example, let's say we have this task:

@task(cache=True, cache_version="v1")
def t(log_level: int, a: str) -> str:
  ...

According to the docs, one of the inputs to cache key calculation is the task signature, but in the case of this example, it'd be great if we could ignore

log_level

as part of the cache key calculation. Goal: What should the final outcome look like, ideally? We should be able to do something along the lines of:

@task(cache=True, cache_version="v1", ignore_input_vars=["log_level"])
def t(log_level: int, a: str) -> str:
    ...

This would essentially skip some of the parameters for cache key calculation purposes. Describe alternatives you've considered We have the ability to override the hashing mechanism used to translate python types into Flyte types, as described in https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task_cache.html#caching-of-non-flyte-offloaded-objects. One could use this idea and provide constant hashes for the arguments they want to ignore, for example:

def constant_function(x: int) -> str:
    return "const"

@task 
def t_produce_annotated_literals() -> Annotated[int, HashMethod(constant_function)]:
   log_level = ...
   return log_level

@task(cache=True, cache_version="v1")
def t(log_level: int, a: str) -> str:
   ...

@workflow
def wf() -> str:
  log_level = t_produce_annotated_literals()
  return t(log_level=log_level, a="some string")

Propose: Link/Inline OR Additional context Expose

ignore_input_vars

in the

@task

decorator and ensure the new interface is used during cache key calculation in both local and remote executions. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2310 [Core feature] [flytekit] Better OAuth2 login experience in jupyter Issue created by kumare3 Motivation: Why do you think this is important? Currently, the default experience of flytekit.Remote when working with an authenticated FlyteRemote endpoint is to open a new browser window to sign in, without the control coming back to the original notebook. This can cause a bad experience for the user. IpyAuth library as described here allows you to control the experience and makes it feel native to Notebook Goal: What should the final outcome look like, ideally? Users should return back to the notebook after the login is successful Describe alternatives you've considered NA Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2481 [Core feature] [flytekit] Cloudpickle and run a task Issue created by wild-endeavor Goal: What should the final outcome look like, ideally? Users should be able to write an

@task

decorated task in a Jupyter notebook, and be able to register and run it from the same Jupyter notebook. Details This is part of the data-scientist first story we are pursuing in H2 2022 - continuing to tie together as seamlessly as possible the local/back-end execution story in support of a data science first iteration cycle. This should probably happen through the FlyteRemote experience. We are thinking of using cloud pickle to pickle the task and ship it off to S3 much like how script mode works. Some questions to think about • Does cloud pickle actually work? What are the limitations and how can we detect/prevent the user from making them (for example, cloud pickle probably doesn't work if you're pickling a function that has an import statement inside it). We should at least list out all these limitations and make the user aware of them. • What will be the task command, specifically how will the task resolver bit work? Unlike in script-mode, which merely updated the code and relied on the same usual python-module/task-finding mechanisms as in a regular non-fast run, this will not only need to download the file of the pickled code from S3, it will also need to unpickle it as part of the resolver process. There's already a cloudpickle resolver, can we use that? Playing around with cloudpickle Testing notes Cursory testing was done by going back and forth between a jupyter notebook with one virtual environment and running code in PyCharm with another. The version of Python has to match (from docs). The version of cloudpickle also needs to match. Had issues between v2.0 and v2.1 of cloudpickle where something written by 2.1 was not readable by 2.0. Did not try the other way around, but probably best to not assume it will always work. We should just aim for minor version matching. If in the jupyter notebook I did

from dyn_task import t1

and then used

t1

inside a dynamic task, and I then pickled the dynamic task, both the jupyter notebook and the Pycharm instance were able to unpickle and run it. If I did

import dyn_task

and then did

dyn_task.t1

in the dynamic task, and then pickled it, it only worked in Jupyter, not in Pycharm. If you then add

cloudpickle.register_pickle_by_value(dyn_task)

then again both work. Resources • GH repo for cloudpickle. The serialize by value or reference discussion is relevant. • The background section of this stackoverflow thread is a good short read. Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2482 [Core feature] Get data from execution into jupyter Issue created by wild-endeavor Problem Statement Users can run tasks currently and have the Flyte memoization service (datacatalog) cache them. The issue I personally find is that it's not easy to then use that data afterwards in iterating. Let's say I can have local access to the appropriate S3 buckets. I run a complicated task or workflow to gather a bunch of data and that result is cached. That's great but now if I want to download that data locally and use it, how do I do that? One could do it, but it's not easy and it's certainly not documented. We think this is important because we think this exact scenario is common - run something in the cloud to get data, but then the resulting dataset is something that fits on a laptop and they want to play around with it locally. Goal No more than five keystrokes or clicks to get the above, assuming a Jupyter notebook is already open. Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2483 [Core feature] Mixed-mode execution Issue created by wild-endeavor Goal Workflows currently can be run locally in Python, or with

pyflyte run

. We would like to at least consider supporting mixed-mode execution, where a locally declared workflow has tasks that run remotely and tasks that run locally. This issue probably warrants a POC effort and a design doc. This possibly relates to the broader async effort in flytekit (no issue for that yet - will try to remember to link when that is available). Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

Hi all, I'm looking for a way to cancel a workflow from within a task, so it stops and goes into the

aborted

state, not

failed

. Is it possible to trigger that from flytekit somehow? Background is that based on a precondition like too few data points, I don't want to trigger a training task, and

aborted

would be perfect to communicate that it's not a failure, but still didn't run.

#2502 [BUG] [flytekit] Improve handling around StructuredDataset and other lossy types in Flyte remote Issue created by wild-endeavor Description If you

remote.fetch

a task or workflow or launch plan where one of the inputs is a StructuredDataset, and then try to execute it, flytekit will try to "guess" the interface for that structured dataset input and the type that it will come up with is the Python/flytekit

StructuredDataset

class. This is correct, but when we go and try to create the execution, we need to translate the dataframe from a pd.DataFrame or whatever instance into a StructuredDataset Literal. Since flytekit thinks the type annotation is a Python StructuredDataset, it will try to look it up in the list of formats/encoders it has and fail because it's not a real dataframe type. An example stack trace:

Traceback (most recent call last):
  File "/Users/nielsbantilan/miniconda3/envs/unionml/bin/unionml", line 33, in <module>
    sys.exit(load_entry_point('unionml', 'console_scripts', 'unionml')())
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "/Users/nielsbantilan/git/unionml/unionml/cli.py", line 99, in predict
    predictions = model.remote_predict(app_version, model_version, wait=True, **prediction_inputs)
  File "/Users/nielsbantilan/git/unionml/unionml/model.py", line 535, in remote_predict
    execution = self._remote.execute(
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/flytekit/remote/remote.py", line 796, in execute
    return self.execute_remote_wf(
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/flytekit/remote/remote.py", line 889, in execute_remote_wf
    return self.execute_remote_task_lp(
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/flytekit/remote/remote.py", line 862, in execute_remote_task_lp
    return self._execute(
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/flytekit/remote/remote.py", line 658, in _execute
    lit = TypeEngine.to_literal(ctx, v, hint, variable.type)
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 696, in to_literal
    lv = transformer.to_literal(ctx, python_val, python_type, expected)
  File "/Users/nielsbantilan/miniconda3/envs/unionml/lib/python3.9/site-packages/flytekit/types/structured/structured_dataset.py", line 486, in to_literal
    fmt = self.DEFAULT_FORMATS[python_type]
KeyError: <class 'flytekit.types.structured.structured_dataset.StructuredDataset'>

We need to improve the erroring/experience around this. Potential things include: • Asking the user to provide a StructuredDataset instance instead of the raw dataframe? • Continuing to error but informing the user to provide the type_hints map. Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2540 [Housekeeping] [flytekit] Streamline module loading code Issue created by wild-endeavor TL;DR With all the work behind us in

pyflyte package

,

run

, and

register

, along with the legacy

serialize

, the pyflyte code could probably use some cleanup. Details This ticket includes but shouldn't be construed as to be limited to the following: • Use the same module loading code between the

run

and

register

commands. Since

run

has to first

register

, and since it also loads when listing commands in a module, it makes sense to reuse the same code. • There's a lot helper code in the

flytekit/tools

folder, and kind organized poorly. Attempt to streamline this. • Figure out proper handling of the

--pkgs

switch that exists at the top level of

pyflyte

, in the

register

command. Currently it just raises an error. Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2588 [Docs] correction for a command: pyflyte run Issue created by drleowang In https://docs.flyte.org/en/latest/getting_started/index.html There is a correction for a command: pyflyte run example.py wf --n 500 --mean 42 --sigma 2 pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2 should be pyflyte run example.py:wf --n 500 --mean 42 --sigma 2 pyflyte run --remote example.py:wf --n 500 --mean 42 --sigma 2 Thanks Leo flyteorg/flyte

#3907 [BUG] Non-python tasks don't support passing secrets Issue created by zeryx Describe the bug Secret requests cannot be found for ContainerTask and other plugins like Elastic, which create task types that are not the vanilla python task. Container tasks ContainerTasks are powerful flyte constructs; they should be able to support providing secrets via the following

secret_requests=[
        Secret(
            group=SECRET_GROUP,
            key=SECRET_FLYTE_APP_SECRET,
            mount_requirement=Secret.MountType.FILE),
    Secret(
        group=SECRET_GROUP,
        key=SECRET_IM,
        mount_requirement=Secret.MountType.FILE)
    ])

When added to a ContainerTask decorator, the task registers and executes fine with pyflyte register - but the secrets are not available either via FILE (/var/...) nor as an ENV_VAR (/etc/secrets/...) Elastic Plugin Given a task configuration like:

@flytekit.task(
    ...
    task_config=Elastic(
        nnodes=2,
        nproc_per_node=8,
        rdzv_configs={"timeout": 1200, "join_timeout": 900}
    ),
    ...
)

An error like the following is raised:

Root Cause (first observed failure):
[0]:
  time      : 2023-08-14_15:54:34
  host      : f9944ebe99b184ee293a-n1-0-worker-0
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 254)
  error_file: /tmp/torchelastic_xc62kj37/f9944ebe99b184ee293a_78vmhc5o/attempt_0/0/error.json
  traceback : Traceback (most recent call last):
    File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
      return f(*args, **kwargs)
    File "/opt/conda/lib/python3.10/site-packages/flytekitplugins/kfpytorch/task.py", line 233, in spawn_helper
      return_val = fn(**kwargs)
    File "/root/fine_tuning/llm_fine_tuning.py", line 422, in train
      os.environ["WANDB_API_KEY"] = flytekit.current_context().secrets.get(
    File "/opt/conda/lib/python3.10/site-packages/flytekit/core/context_manager.py", line 365, in get
      raise ValueError(
  ValueError: Unable to find secret for key wandb_api_key-5t1ZwJ in group arn:aws:secretsmanager:us-east-2:590375264460:secret: in Env Var:_FSEC_ARN:AWS:SECRETSMANAGER:US-EAST-2:590375264460:SECRET:_WANDB_API_KEY-5T1ZWJ and FilePath: /etc/secrets/arn:aws:secretsmanager:us-east-2:590375264460:secret:/wandb_api_key-5t1zwj
  
============================================================

User error.

Expected behavior The expected behavior is htat if a secret is mounted as a File, that the file will be avialable in /var/secrets/... If the secret is mounted as an ENV_VAR, it should be already added to the environment and directly accessable (essentially echo $MY_SECRET_VAR should return the value of MY_SECRET_VAR) Additional context to reproduce create a simple ContainerTask, have it

sleep 2h

as entrypoint Add secrets to it (see above example, make sure credentials are configured correctly) register the ContainerTask to flyte cluster / sandbox ssh into the container, check /etc/secrets / /var/secrets Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2852 [Core feature] Use fast_package in pyflyte run Issue created by pingsutw Motivation: Why do you think this is important? Currently, pyflyte run use register_script, and it has some limitations. such as 1. Failed to run the workflow on the remote cluster if workflow script imports other files because pyflyte run only uploads a single script to blob store. 2. It can't register a LaunchPlan. now, it will only register tasks and workflows. We can add a new flag

--pkg

in the

pyflyte run

, and use

fast_package

to register the entire package (directory) if people use this flag. Goal: What should the final outcome look like, ideally?

pyflyte run --remote --pkg example.py wf

Use

pkg

plag to fast register the entire package. Otherwise, register a single script by default. Describe alternatives you've considered Replace

register_script

with fast_package Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2881 [Core feature] Flytekit execution models should expose more fields Issue created by katrogan Motivation: Why do you think this is important? Currently the execution models expose a subset of the underlying execution fields. We should expose the abort_metadata in the closure and reference execution in the metadata Goal: What should the final outcome look like, ideally? Flytekit Execution models should expose fields from the underlying protobuf definition from executions fetched using flytekit remote Describe alternatives you've considered Current implementation Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2895 [BUG] Dataclass becomes empty after running ray remote function Issue created by pingsutw Describe the bug Slack thread Failed to serialize dataclass because the value became empty for some reason after running ray remote function Expected behavior Successfully serialize the dataclass Additional context to reproduce

from dataclasses import dataclass
from typing import cast
from dataclasses_json import dataclass_json, DataClassJsonMixin
import ray


@dataclass_json
@dataclass
class Datum(object):
    x: int
    y: str


@ray.remote
def ray_create_datum(x: int, y: str):
    return Datum(x=x, y=y)


d = Datum(x=1, y='hello')

print("value", cast(DataClassJsonMixin, d).to_json()) # value {"x": 1, "y": "hello"}
ray.get(ray_create_datum.remote(x=1, y='hello'))
print("value", cast(DataClassJsonMixin, d).to_json()) # value {}

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2977 Failed to create a launch plan for FlyteWorkflow Issue created by pingsutw Motivation: Why do you think this is important? For now, we can only create a launch plan for WorkflowBase, and we'll get the following error if we try to create LP for FlyteWorkflow (The workflow fetched by flyte remote)

File "/Users/.../lib/python3.7/site-packages/flytekit/core/launch_plan.py", line 143, in create
    native_types=workflow.python_interface.inputs,
AttributeError: 'NoneType' object has no attribute 'inputs'

Code snippet:

from flytekit import LaunchPlan

flyte_workflow = remote.fetch_workflow(
    name="my_workflow", version="v1", project="flytesnacks", domain="development"
)
launch_plan = LaunchPlan.get_or_create(name="my_launch_plan", workflow=flyte_workflow)

Goal: What should the final outcome look like, ideally? Should be able to create a launch plan for FlyteWorkflow Describe alternatives you've considered NA Propose: Link/Inline OR Additional context https://flyte-org.slack.com/archives/CP2HDHKE1/p1665377001295529 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2980 [BUG] Task returning NamedTuple fails with USER::Pod failed. No message received from kubernetes Issue created by tekumara Describe the bug A task with the following signature:

from typing import Dict, List, NamedTuple
from flytekit import task, workflow

class OpenFlightsData(NamedTuple):
    routes: List[Dict[str, str]]
    airlines: Dict[str, str]
    airports: Dict[str, Dict[str, str]]


@task()
def extract_reference_data() -> OpenFlightsData:
   pass

Fails with

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[fb5562eecbc2f40f9b0d-n0-0] terminated with exit code (137). Reason [Error]. Message: 

.

Expected behavior No failure Additional context to reproduce flytekit 1.2.1 flyte admin 1.1.46 Screenshots Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2998 [Core feature] Support launchplans in pyflyte run Issue created by eapolinario Motivation: Why do you think this is important? Currently we only support launching tasks and workflows in

pyflyte run

. We should add support launchplans as well. Goal: What should the final outcome look like, ideally? We should be able to use the same experience currently offered in

pyflyte run

to kickoff launchplans, i.e. users should be able to execute:

> pyflyte run --remote path/to/my/file.py my_launchplan
Go to <http://flyte.example.com/console/projects/flytesnacks/domains/development/executions/f6e7113930e3043e79cc> to see execution in the console.
...

Describe alternatives you've considered launchplans are an indispensable entity type in the Flyte programming model, hence it should be supported in the CLIs. Propose: Link/Inline OR Additional context We should add launch plans to the list of entities handled by

pyflyte run

in here. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3003 [BUG] flytekitplugins-dbt not logging std output Issue created by roek5803 Describe the bug When running a DBT task with the flytekit DBT plugin we expect it to log the std output so we can see what is happening. Looking at the code, the current implementation collects the logs from std output but will only print it in case of an error. Expected behavior We would expect the library to print the std output of the DBT commands ideally with some controlling mechanism e.g. with a verbose flag. Additional context to reproduce

from flytekit import workflow
from flytekitplugins.dbt.task import DBTRun, DBTRunInput

dbt_task = DBTRun(name="the_name_of_the_task")

@workflow
def my_wf() -> None:
    input = DBTRunInput(
            project_dir="dbt_project_dir",
            profiles_dir="dbt_project_dir/docker-context",
            profile="default",
            select=["some_model"]
    )
    
    dbt_task(input=input)

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

What is the correct way to get the unique Execution ID of a workflow within a task? And what is the correct way to determine if a task is running locally or remotely? - It seems the plugins do this slightly different e.g. the spark plugin vs the mlflow plugin?

#2794 [Core feature] [flytekit] Run-time configuration for Structured Dataset Issue created by wild-endeavor Structured Dataset Configuration Goal The current encoders/decoders for Structured Dataset don't necessarily suit all use cases for all users at all times. Specifically behaviour around the run-time column strictness - for example, in the following scenario

@task
def t1() -> Annotated[pd.DataFrame, kwtypes(a=int)]:
  ...

@task
def t2(df: Annotated[pd.DataFrame, kwtypes(a=float)]):
 ...

It's conceivable that sometimes you want this type conversion (from int to float) to happen and sometimes you do not. What is the correct way of allowing the user to control this behavior? Also where should this be enforced? Enforcing this at run-time is likely best to give as much flexibility to the plugin author as possible. Implementation ? Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2795 [Core feature] [flytekit] Move big query writer to access storage layer directly Issue created by wild-endeavor Goal: What should the final outcome look like, ideally? The big query structured dataset reader currently reads from the storage client directly but the writer uses the higher level BigQuery client. This has additional cost implications. We should move the big query encoder to also use the storage layer directly. Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3123 [Core feature] Add PodTemplate configuration to flytekit task definitions Issue created by hamersaw Motivation: Why do you think this is important? Many users require more complex k8s configuration within the Flyte ecosystem. This may include specifying task-specific tolerations, volume mounts, etc. Currently this is unsupported within flytekit, where the API favors ergonomics in abstracting away the complexities of k8s configuration from end-users. However, some use-cases require this complex configuration and striking a balance is very difficult. Goal: What should the final outcome look like, ideally? Users should be able to configure every aspect of the k8s Pod that Flyte creates without bloating the task definition. One solution is to add a PodTemplate configuration to every flytekit task definition. This would work similarly to the default PodTemplate scheme where this PodTemplate definition serves as the base in creating the k8s Pod. By default this will be empty and therefore not applied, but if it is specified it will serve as the base for Pod configuration. This API could look something like:

@task(
    request=Resources(),
    pod_template=V1PodTemplate{
        spec=V1PodSpec{
            tolerations=[
                V1Toleration{
                    key= "num-gpus"
                    operator= "Equal"
                    value= 1
                    effect= "NoSchedule"
                },
            ]
        }
    }
)

Describe alternatives you've considered Currently, proposals often include one-off configuration updates that solve a specific use-case but are not applicable in generalized terms. This approach was used in the k8s plugin and has proved unmaintainable in that every option requires code changes which may change between k8s versions. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3862 [BUG] Flytekit Contribution dependency doesn't work Issue created by Future-Outlier Describe the bug Hi, community I just follow the contribution guide to install the dependency to contribute. However, I encounter 2 environment erros and doesn't know how to solve it. Can anyone guide me ? I would love to contribute. I use Python 3.9.17 as my environment and simple just run.

conda create env -n flytekit  python=3.9
make setup
pip install -e .
pip install gsutil awscli

and run

make requirements

The below will appear the logs the first one:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorboard 2.8.0 requires google-auth-oauthlib<0.5,>=0.4.1, but you have google-auth-oauthlib 1.0.0 which is incompatible.
Successfully installed setuptools-68.0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: <https://pip.pypa.io/warnings/venv>
pip-compile --upgrade --verbose <http://doc-requirements.in|doc-requirements.in>
/root/miniconda3/envs/build/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Using pip-tools configuration defaults found in 'pyproject.toml'.
WARNING: the legacy dependency resolver is deprecated and will be removed in future versions of pip-tools. The default resolver will be changed to 'backtracking' in pip-tools 7.0.0. Specify --resolver=backtracking to silence this warning.

the second one

Using legacy resolver. Consider using backtracking resolver with `--resolver=backtracking`.
Could not find a version that matches grpcio!=1.55.0,<2.0,<2.0dev,<=1.49.1,<=1.51.3,>=1.24.3,>=1.32.0,>=1.33.2,>=1.47.0,>=1.50.0 (from -r <http://doc-requirements.in|doc-requirements.in> (line 4))
Tried: 0.13.0, 0.13.1, 0.14.0, 0.15.0, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.1.0, 1.1.3, 1.2.0, 1.2.1, 1.3.0, 1.3.5, 1.4.0, 1.6.0, 1.6.3, 1.7.0, 1.7.3, 1.8.1, 1.8.2, 1.8.3, 1.8.4, 1.8.6, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.11.0, 1.11.1, 1.12.0, 1.12.1, 1.13.0, 1.14.0, 1.14.1, 1.14.2, 1.15.0, 1.16.0, 1.16.1, 1.17.0, 1.17.1, 1.18.0, 1.19.0, 1.20.0, 1.20.1, 1.21.1, 1.22.0, 1.22.1, 1.23.0, 1.23.1, 1.24.0, 1.24.1, 1.24.3, 1.25.0, 1.26.0, 1.27.1, 1.27.2, 1.28.1, 1.29.0, 1.30.0, 1.31.0, 1.32.0, 1.33.1, 1.33.1, 1.33.1, 1.33.2, 1.33.2, 1.33.2, 1.34.0, 1.34.0, 1.34.0, 1.34.1, 1.34.1, 1.34.1, 1.35.0, 1.35.0, 1.35.0, 1.36.0, 1.36.0, 1.36.0, 1.36.1, 1.36.1, 1.36.1, 1.37.0, 1.37.0, 1.37.0, 1.37.1, 1.37.1, 1.37.1, 1.38.0, 1.38.0, 1.38.0, 1.38.1, 1.38.1, 1.38.1, 1.39.0, 1.39.0, 1.39.0, 1.40.0, 1.40.0, 1.40.0, 1.41.0, 1.41.0, 1.41.0, 1.41.1, 1.41.1, 1.41.1, 1.42.0, 1.42.0, 1.42.0, 1.43.0, 1.43.0, 1.43.0, 1.44.0, 1.44.0, 1.44.0, 1.45.0, 1.45.0, 1.45.0, 1.46.0, 1.46.0, 1.46.0, 1.46.1, 1.46.1, 1.46.1, 1.46.3, 1.46.3, 1.46.3, 1.46.5, 1.46.5, 1.46.5, 1.47.0, 1.47.0, 1.47.2, 1.47.2, 1.47.5, 1.47.5, 1.48.0, 1.48.0, 1.48.1, 1.48.1, 1.48.2, 1.48.2, 1.49.0, 1.49.0, 1.49.1, 1.49.1, 1.50.0, 1.50.0, 1.51.0, 1.51.0, 1.51.1, 1.51.1, 1.51.3, 1.51.3, 1.52.0, 1.52.0, 1.53.0, 1.53.0, 1.53.1, 1.53.1, 1.54.0, 1.54.0, 1.54.2, 1.54.2, 1.55.0, 1.55.0, 1.56.0, 1.56.0
Skipped pre-versions: 0.4.0a0, 0.4.0a1, 0.4.0a2, 0.4.0a3, 0.4.0a4, 0.4.0a5, 0.4.0a6, 0.4.0a7, 0.4.0a8, 0.4.0a13, 0.4.0a14, 0.5.0a0, 0.5.0a1, 0.5.0a2, 0.9.0a0, 0.9.0a1, 0.10.0a0, 0.11.0b0, 0.11.0b1, 0.12.0b0, 0.13.1rc1, 0.14.0rc1, 1.0.0rc1, 1.0.0rc2, 1.0.1rc1, 1.9.0rc1, 1.9.0rc2, 1.9.0rc3, 1.10.0rc2, 1.10.1rc1, 1.10.1rc2, 1.11.0rc1, 1.11.0rc2, 1.11.1rc1, 1.12.0rc1, 1.13.0rc1, 1.13.0rc2, 1.13.0rc3, 1.14.0rc1, 1.14.0rc2, 1.14.2rc1, 1.15.0rc1, 1.16.0rc1, 1.20.0rc1, 1.20.0rc2, 1.20.0rc3, 1.21.0rc1, 1.21.1rc1, 1.22.0rc1, 1.23.0rc1, 1.24.0rc1, 1.25.0rc1, 1.26.0rc1, 1.27.0rc1, 1.27.0rc2, 1.28.0rc1, 1.28.0rc2, 1.34.0rc1, 1.34.0rc1, 1.34.0rc1, 1.35.0rc1, 1.35.0rc1, 1.35.0rc1, 1.36.0rc1, 1.36.0rc1, 1.36.0rc1, 1.37.0rc1, 1.37.0rc1, 1.37.0rc1, 1.38.0rc1, 1.38.0rc1, 1.38.0rc1, 1.39.0rc1, 1.39.0rc1, 1.39.0rc1, 1.40.0rc1, 1.40.0rc1, 1.40.0rc1, 1.41.0rc2, 1.41.0rc2, 1.41.0rc2, 1.42.0rc1, 1.42.0rc1, 1.42.0rc1, 1.43.0rc1, 1.43.0rc1, 1.43.0rc1, 1.44.0rc1, 1.44.0rc1, 1.44.0rc1, 1.44.0rc2, 1.44.0rc2, 1.44.0rc2, 1.45.0rc1, 1.45.0rc1, 1.45.0rc1, 1.46.0rc1, 1.46.0rc1, 1.46.0rc1, 1.46.0rc2, 1.46.0rc2, 1.46.0rc2, 1.47.0rc1, 1.47.0rc1, 1.48.0rc1, 1.48.0rc1, 1.49.0rc1, 1.49.0rc1, 1.49.0rc3, 1.49.0rc3, 1.50.0rc1, 1.50.0rc1, 1.51.0rc1, 1.51.0rc1, 1.52.0rc1, 1.52.0rc1, 1.53.0rc2, 1.53.0rc2, 1.54.0rc1, 1.54.0rc1, 1.55.0rc1, 1.55.0rc1, 1.56.0rc2, 1.56.0rc2
There are incompatible versions in the resolved dependencies:
  grpcio<=1.49.1 (from -r <http://doc-requirements.in|doc-requirements.in> (line 4))
  grpcio<2.0dev,>=1.47.0 (from google-cloud-bigquery==3.11.3->-r <http://doc-requirements.in|doc-requirements.in> (line 27))
  grpcio<2.0,>=1.24.3 (from tensorflow==2.8.1->-r <http://doc-requirements.in|doc-requirements.in> (line 44))
  grpcio<=1.51.3,>=1.32.0 (from ray==2.5.1->-r <http://doc-requirements.in|doc-requirements.in> (line 47))
  grpcio<2.0dev,>=1.33.2 (from google-api-core[grpc]==2.11.1->-r <http://doc-requirements.in|doc-requirements.in> (line 17))
  grpcio!=1.55.0,<2.0,>=1.50.0 (from flytekit==0.0.0+develop->-r <http://doc-requirements.in|doc-requirements.in> (line 1))
make: *** [Makefile:65: doc-requirements.txt] Error 2

Expected behavior Build successfully. Additional context to reproduce No response Screenshots Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3765 [Core feature] Support configuring pip index url with the image spec feature Issue created by mucahitkantepe Motivation: Why do you think this is important? We have some private common libraries that we want to access in our Flyte workflows using image spec feature. To be able to use imageSpec feature that was highlighted with Flyte 1.6 release, we need image spec feature to support configuring pip index url. envd does support this, and it should be fairly small change. Goal: What should the final outcome look like, ideally?

image_spec=ImageSpec(
    base_image="some-base-image:latest
     packages=["a-private-library==1.6.0"
     pip_index="<https://private-pip-index/simple>"
)

and the generated build.envd file would look like this

# syntax=v1

def build():
...
    config.pip_index(url = "<https://private-pip-index/simple>")
...

Describe alternatives you've considered I think it's pretty straightforward. Propose: Link/Inline OR Additional context envd has built-in support for this, as shown in the main example in their readme https://github.com/tensorchord/envd#create-an-envd-environment Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3364 [Core feature] Use mashumaro to serialize/deserialize dataclass Issue created by pingsutw Motivation: Why do you think this is important? mashumaro has better performance than marshmallow we use now. we should be able to replace it with marshmallow in flytekit. Goal: What should the final outcome look like, ideally? 1. Reduce overhead while serialize/deserialize dataclass. 2. Should not break existing users Describe alternatives you've considered NA Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3166 [Housekeeping] Remove deprecated numpy aliases Issue created by eapolinario Describe the issue A few aliases defined in

numpy

were deprecated in v1.20.0 and finally removed in

v1.24.0

(check Deprecations in https://github.com/numpy/numpy/releases/tag/v1.20.0). We should remove the mentions to those deprecated aliases from the FlyteSchemaTransformer. What if we do not do this? Users who install

numpy>=1.24.0

will see this error in flytekit:

/tmp/tmp.pgqfAvNZOC/venv/lib/python3.10/site-packages/flytekit/types/schema/types.py:324: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.  (This may have returned Python scalars in past versions.
  _np.bool: SchemaType.SchemaColumn.SchemaColumnType.BOOLEAN,  # type: ignore
Traceback (most recent call last):
  File "/tmp/tmp.pgqfAvNZOC/venv/bin/pyflyte", line 5, in <module>
    from flytekit.clis.sdk_in_container.pyflyte import main
  File "/tmp/tmp.pgqfAvNZOC/venv/lib/python3.10/site-packages/flytekit/__init__.py", line 195, in <module>
    from flytekit.types import directory, file, numpy, schema
  File "/tmp/tmp.pgqfAvNZOC/venv/lib/python3.10/site-packages/flytekit/types/schema/__init__.py", line 1, in <module>
    from .types import (
  File "/tmp/tmp.pgqfAvNZOC/venv/lib/python3.10/site-packages/flytekit/types/schema/types.py", line 314, in <module>
    class FlyteSchemaTransformer(TypeTransformer[FlyteSchema]):
  File "/tmp/tmp.pgqfAvNZOC/venv/lib/python3.10/site-packages/flytekit/types/schema/types.py", line 324, in FlyteSchemaTransformer
    _np.bool: SchemaType.SchemaColumn.SchemaColumnType.BOOLEAN,  # type: ignore
  File "/tmp/tmp.pgqfAvNZOC/venv/lib/python3.10/site-packages/numpy/__init__.py", line 284, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'bool'. Did you mean: 'bool_'?

Related component(s) flytekit Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2999 [Core feature] Add support native python type in the condition task Issue created by pingsutw Motivation: Why do you think this is important? For now, The variable in the

_if

in the

condition

must be a promise, and can't use a native python type in the if statement.

@dynamic
def d1() -> bool:
    a = t1()
    return (
        conditional("train_estimator")
        .if_(a==True)
        .then(t2())
        .else_()
        .then(t2()))

@dynamic
def d1(a: bool) -> bool:
    return (
        conditional("train_estimator")
        .if_(a==True)  # <- failed because the a isn't a promise
        .then(t2())
        .else_()
        .then(t2()))

Goal: What should the final outcome look like, ideally? Both promise and native python types should be supported in the conditions Describe alternatives you've considered Users must create a task to convert it to a promise and use it in the conditions.

@task
def convert_to_promise(a: bool) -> bool:
    return a

Propose: Link/Inline OR Additional context https://flyte-org.slack.com/archives/CREL4QVAQ/p1666110935856779 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#4079 [Core feature] Sagemaker Agent Issue created by pingsutw Motivation: Why do you think this is important? This agent can used to submit the training or serving jot to AWS sagemaker. It allows people to easily run SageMaker training in a Flyte workflow. Currently, we have sagemaker backend plugin to submit the job, but the code it outdated, and doesn't work for now. We can write an agent in Python for Sagemaker, and it will be easy to maintain as well. Goal: What should the final outcome look like, ideally? We'll able to submit a training job by leveraging agent. Describe alternatives you've considered NA Propose: Link/Inline OR Additional context • Writing Agents in Python • flyteorg/flytekit#1797 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#4081 [Core feature] Rust Data Persistence Issue created by pingsutw Motivation: Why do you think this is important? Write a Rust plugin to read/write s3 files. It can replace current data persistence plugin in flytekit. Goal: What should the final outcome look like, ideally? The Rust plugin should offer better data read/write performance than the current one Describe alternatives you've considered Use existing data persistence plugin. Propose: Link/Inline OR Additional context https://docs.rs/object_store/latest/object_store/ Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte