#2499 [Docs] Improve Docs on how to login docker registry in flyte to download private image. Issue created by AlekhyaSasi Description Slack archives K8s doc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2497 [Docs] Update DOCS based on Doc Standards Issue created by AlekhyaSasi Description As per Doc standards (meeting), update the following: ☐ The

Code Example

is redundant and can be removed. [Example page (https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html#code-examples). ☐ Replace

caution

with

warning

. ☐ Indicate

Tip

is optional. ☐ Look into headers in the docs and come up with a common syntax for all the headers [H1 to H4]. ☐ Change all headers and sub-headers to Title case. ☐ Change the content to simple, active, and direct language. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2495 [Docs] Design doc doesn't lead to a page, instead asks for access request Issue created by SmritiSatyanV Description Similar to #2485 issue, the

design doc

in

Note

section of https://docs.flyte.org/projects/flytekit/en/latest/design/authoring.html should show a doc, not a page that leads to access request. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2491 [Housekeeping] Enable verbose logging for sandbox and demo environments Issue created by pmahindrakar-oss Describe the issue Currently its hard to debug errors in sandbox and demo environments if the installation fails during helm installation . Enable verbose logging for helm commands and also kubectl commands https://kubernetes.io/docs/reference/kubectl/cheatsheet/#kubectl-output-verbosity-and-debugging The default option should be with this flag off.

flytectl demo start --verbose
 flytectl sandbox start --verbose

Have similar flags during teardown of the environments

flytectl demo teardown --verbose
flytectl sandbox teardown --verbose

What if we do not do this? Longer debugging cycles Related component(s) flytectl,flyte Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2485 [Docs] Launch plans page links to inaccessible Google drawing Issue created by marc-union Description Steps: 1. Go to https://docs.flyte.org/en/latest/concepts/launchplans.html 2. In introduction, click link in "See here for an overview." What's expected: Browser shows diagram. Should this diagram be in the Flyte docs instead of a 3rd party external site? What happens instead: Browser shows Google's page for requesting access to an inaccessible Google drawing. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2483 [Core feature] Mixed-mode execution Issue created by wild-endeavor Goal Workflows currently can be run locally in Python, or with

pyflyte run

. We would like to at least consider supporting mixed-mode execution, where a locally declared workflow has tasks that run remotely and tasks that run locally. This issue probably warrants a POC effort and a design doc. This possibly relates to the broader async effort in flytekit (no issue for that yet - will try to remember to link when that is available). Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2481 [Core feature] [flytekit] Cloudpickle and run a task Issue created by wild-endeavor Goal: What should the final outcome look like, ideally? Users should be able to write an

@task

decorated task in a Jupyter notebook, and be able to register and run it from the same Jupyter notebook. Details This is part of the data-scientist first story we are pursuing in H2 2022 - continuing to tie together as seamlessly as possible the local/back-end execution story in support of a data science first iteration cycle. This should probably happen through the FlyteRemote experience. We are thinking of using cloud pickle to pickle the task and ship it off to S3 much like how script mode works. Some questions to think about • Does cloud pickle actually work? What are the limitations and how can we detect/prevent the user from making them (for example, cloud pickle probably doesn't work if you're pickling a function that has an import statement inside it). We should at least list out all these limitations and make the user aware of them. • What will be the task command, specifically how will the task resolver bit work? Unlike in script-mode, which merely updated the code and relied on the same usual python-module/task-finding mechanisms as in a regular non-fast run, this will not only need to download the file of the pickled code from S3, it will also need to unpickle it as part of the resolver process. There's already a cloudpickle resolver, can we use that? Playing around with cloudpickle Testing notes Cursory testing was done by going back and forth between a jupyter notebook with one virtual environment and running code in PyCharm with another. The version of Python has to match (from docs). The version of cloudpickle also needs to match. Had issues between v2.0 and v2.1 of cloudpickle where something written by 2.1 was not readable by 2.0. Did not try the other way around, but probably best to not assume it will always work. We should just aim for minor version matching. If in the jupyter notebook I did

from dyn_task import t1

and then used

t1

inside a dynamic task, and I then pickled the dynamic task, both the jupyter notebook and the Pycharm instance were able to unpickle and run it. If I did

import dyn_task

and then did

dyn_task.t1

in the dynamic task, and then pickled it, it only worked in Jupyter, not in Pycharm. If you then add

cloudpickle.register_pickle_by_value(dyn_task)

then again both work. Resources • GH repo for cloudpickle. The serialize by value or reference discussion is relevant. • The background section of this stackoverflow thread is a good short read. Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2473 [Docs] Include `bigquery.readsessions.create` permission requirement in StructuredDataSet Issue created by AlekhyaSasi Description To use this API, first enable it in the Cloud Console. You must also have the bigquery.readsessions.create permission on the project you are billing queries to. @AlekhyaSasi Slack archives Related Doc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2470 [Docs] Launch Plans vs Subworkflows Issue created by wild-endeavor Description User requested that we explain the difference between launch plans and subworkflows. He said he's seen users just kinda pick one or the other without a good understanding of why. We should improve the documentation to explain what the difference is, why they both exist, and include some criteria to choose between them as well as give performance tradeoffs. cc @marc-union Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2451 [Feature] Type Transformer for MatPlotLib Issue created by smadarab Describe the bug Type translator for matplotlib chart image doesn't exist. When i try to return a matplotlib image from task to workflow, the rurned datatype is saved as figure in local location but not into s3 bucket. Expected behavior output chart returned from task created using matplotlib should be saved into a output variable and then iunto s3 bucket. Additional context to reproduce from copyreg import pickle import pandas as pd from sklearn.datasets import load_digits from sklearn.linear_model import LogisticRegression import requests import io import numpy as np from sklearn.pipeline import Pipeline, FeatureUnion from sklearn.preprocessing import FunctionTransformer, MinMaxScaler from sklearn.impute import SimpleImputer imputer_num = SimpleImputer(missing_values=np.nan, strategy='mean') from sklearn.model_selection import train_test_split from sklearn.impute import SimpleImputer imputer_cat = SimpleImputer(strategy="most_frequent") from sklearn.ensemble import AdaBoostClassifier from sklearn.metrics import roc_auc_score from flytekit import task, workflow from joblib import dump from sklearn.preprocessing import OneHotEncoder from typing import Tuple from matplotlib import pyplot import pickle hi=None scale=None from sklearn.preprocessing import MinMaxScaler @task def get_dataset() -> pd.DataFrame: #return load_digits(as_frame=True).frame

#url = "<https://github.com/smadarab/flytelab/raw/main/census.csv>" # Make sure the url is the raw version of the file on GitHub
url="<https://github.com/smadarab/flytelab/raw/main/census> 2.csv"
download = requests.get(url).content
df = pd.read_csv(io.StringIO(download.decode('utf-8')),sep=',')
print("df is created",df.columns)
#df.dropna(inplace=True)
#df = df.reset_index()
print(df.columns)
return(df)

@task def plot_dataset(df:pd.DataFrame) -> pyplot: x = df['age'].value_counts().index y = df['age'].value_counts().values fig = pyplot.figure(figsize =(10, 7)) # Horizontal Bar Plot pyplot.bar(x,y) # Show Plot #pyplot.show() return(fig) #@task #def clean_dataset() -> pd.DataFrame: pass #@task #def transform_cat() -> pd.DataFrame: pass #@task #def transform_num() -> pd.DataFrame: pass @task def train_model(train: pd.DataFrame) -> Tuple[AdaBoostClassifier,OneHotEncoder,MinMaxScaler]: num_cols = ['age', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week'] cat_cols = ['workclass', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'native-country'] log_transform_cols = ['capital-loss', 'capital-gain'] def get_cat_cols(X): return X[cat_cols] def get_num_cols(X): return X[num_cols] def get_log_transform_cols(X): return X[log_transform_cols] def get_dummies(X): print('\n \n',type(X)) return pd.get_dummies(pd.DataFrame(X)) def cat_imputer(X): print(X.shape) return(imputer_cat.fit_transform(X)) #return X.apply(lambda col: imputer_cat.fit_transform(col)) def one_hot_encode(X): print("one hot encode") ohe = OneHotEncoder(handle_unknown = 'ignore') #print("DF:: ") ohe.fit(pd.DataFrame(X)) global hi hi=ohe #dump(ohe, 'onehot.joblib') return ohe.transform(pd.DataFrame(X)).toarray() def min_max_scaling(X): scaler = MinMaxScaler() scaler.fit(X) global scale scale=scaler return scaler.transform(pd.DataFrame(X)).tolist() log_transform_pipeline = Pipeline([ ('get_log_transform_cols', FunctionTransformer(get_log_transform_cols, validate=False)), ('imputer', SimpleImputer(strategy='mean')), ('log_transform', FunctionTransformer(np.log1p)) ])

num_cols_pipeline = Pipeline([
('get_num_cols', FunctionTransformer(get_num_cols, validate=False)),
('imputer', SimpleImputer(strategy='mean')),
('min_max_scaler', FunctionTransformer(min_max_scaling, validate=False))
])

cat_cols_pipeline = Pipeline([
('get_cat_cols', FunctionTransformer(get_cat_cols, validate=False)),
('imputer', SimpleImputer(strategy="most_frequent")),
#    ('get_dummies', FunctionTransformer(get_dummies, validate=False))
('one_hot_encode', FunctionTransformer(one_hot_encode, validate=False))
])       

steps_ = FeatureUnion([
('log_transform', log_transform_pipeline),
('num_cols', num_cols_pipeline),
('cat_cols', cat_cols_pipeline)
])
full_pipeline = Pipeline([('steps_', steps_)])
y = train['income'].map({'<=50K': 0, '>50K': 1})
X = full_pipeline.fit_transform(train)
model = AdaBoostClassifier(n_estimators=300)
X_train, X_test, y_train, y_test = train_test_split(X, y,shuffle=False)
model = model.fit(X_train, y_train) 
return model,hi,scale

#@task #def test_model() -> pd.DataFrame: pass #@workflow #def main1() -> pd.DataFrame: return get_dataset() @workflow def main() -> pyplot: return plot_dataset(df=get_dataset()) #@workflow #def main() -> Tuple[AdaBoostClassifier,OneHotEncoder,MinMaxScaler]: return train_model(train=get_dataset()) if name == "*main*": print(f"trained model: {main()}") Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2441 Allow user named tasks and workflows for PythonFunction* Issue created by dylanwilder Currently

PythonFunctionTask

and

PythonFunctionWorkflow

derive their names implicitly from the module and function where they are defined. While this is convenient, it makes refactoring difficult as names often need to be stable identifiers for downstream consumption. Proposal is to include and optional

name

parameter to the decorators which allows overriding this behavior. flyteorg/flyte

#2418 Add support for Spark tasks in Scala Issue created by MariaSL Motivation: Why do you think this is important? It's quite common to run Spark jobs in Scala. Adding support for Spark tasks in Scala will allow users to write and execute Flyte workflows in Scala with an ability to run Spark jobs. Goal: What should the final outcome look like, ideally? The outcome is similar to what can currently be done with pyspark: • The user is able to create a workflow with a spark task written in Scala; • The user is able to run the workflow with spark task in scala locally; • The user is able to run the workflow with spark task in scala remotely - once the spark task runs, a Spark cluster is spinned up. • The user is able to configure spark properties (SparkContext); Additionally • Flyte documentation (or flyte scala examples) is extended with an example of running Spark task in Scala and deploying a workflow. Describe alternatives you've considered None Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2417 [Housekeeping] Remove the usage of IAMRole and move towards using service account as part of SecurityContext Issue created by pmahindrakar-oss Describe the issue Flyte supports passing the authorization information for the user pods using K8ServiceAccount , IamRole as part of the SecurityContext/Deprecated AuthRole/Deprecated Auth fields. These can be passed by the users at various levels in this order of overrides • execution level • launchplan level • workflow execution config matchable attribute per project domain/ project domain worflow /project domain workflow launchplan level • application default config level as part of the values.yaml IamRole usage was added to flyte which relied on services like KIAM https://github.com/uswitch/kiam Flyte would add annotations to user pods and KIAM like services would make sure the authentication information is available to the PODS to talk to AWS services like S3. The KIAM solution had race condition issues where in if the user pod made a call before the authentication information was available to the pod then it would fail KIAM has been deprecated as AWS now supports IAM roles for Service Accounts which solves such races as it awaits for the authetication information to be ready before the POD itself can be declared ready and hence preventing such issues. With IAM roles for service account, now the users need to just create a service account and annotate it with right IAM role which has permissions for the pods to access the required AWS services. This can be done manually by the user or they can rely on cluster resource manager to create those for them using the IAM role provided in the values.yaml of flyte Other major cloud provider GCP uses similar concept through workload identity where they have mapping between google service account and kubernetes service accounts and have permissions being defined using this mapping https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity and hence avoiding usage of any external service to provide authentication info for the pods. We should move away from using IAM Role from UI and only allow service accounts to be overriden. There might be some cases like map tasks and some plugins which read such annotations but we should explicitly mention in the UX that they should use these roles only for the above purposes and for plain vanilla flow always use service account What if we do not do this? Confusing UX where users try to send IAM Role which don't get used on normal workflows as they are ignored and instead service account is used Related component(s) flyteadmin Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2414 [Docs] Improve documentation on project/domain configurations Issue created by AlekhyaSasi Description We need to improve documentation on what kind of project/domain level configurations can be changed at registration time, launch time, runtime etc. Pasting verbatim from @Ketan (kumare3) Slack thread: https://flyte-org.slack.com/archives/CP2HDHKE1/p1650846353153889?thread_ts=1650841210.830509&cid=CP2HDHKE1 -- There is a hierarchy for config: When you create a project domain, you can now configure defaults for a project/domain combination, things like • Service account + iam role • raw data output buckets (this is where data will be automatically stored) • max parallelism • couple others • default cpu/mem for tasks where it is not specified Then when you register you can override these things and add more, like • labels / annotations etc • on termination behavior • notifications and for runtime all of this can be overridden. Reference: https://docs.flyte.org/projects/flytekit/en/latest/design/control_plane.html#design-control-plane -- Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2388 [Docs] Port the Documentation Standards Guide to a Flytesnacks contributing page Issue created by cosmicBboy Description Move the content in https://docs.google.com/document/d/1rhEZqd3Ae8Y9HUxg7yF6LarHrfIHKnPyeke8aMSPgis/edit#heading=h.au3aq9ubs9hf Into a dedicated "Style Guide" page in the Community section of the docs: https://docs.flyte.org/en/latest/community/index.html# Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2387 [Docs] Flytesnacks examples have pyflyte run and FlyteRemote code snippets (with tests) Issue created by cosmicBboy Description To make UX better, have code snippets at the bottom of each example showing how to run it using

pyflyte run

and

FlyteRemote

Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2386 [Docs] Create github action workflow for automating flytekit plugin docs generation Issue created by cosmicBboy Description TODO: add description @AlekhyaSasi Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2385 [Core feature] Add sha to images Issue created by wild-endeavor WIP Motivation: Why do you think this is important? Users have requested to be able to specify the sha256 digest of an image in addition to the tag. This is something we've discussed in the past to strengthen reproducibility and as a security measure. User Experience Docker allows you to pull images like

$ docker pull ubuntu@sha256:82becede498899ec668628e7cb0ad87b6e1c371cb8a1e597d83a47fac21d6af3

When working in Flyte, users should be able to do the same. Questions Should users be able to specify both the tag and a sha for it? Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2343 [Docs] Search query in the github runner produced readthedocs webpage uses index from docs.flyte.org Issue created by pmahindrakar-oss Description Eg : https://flyte--2339.org.readthedocs.build/en/2339/ Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2331 [Core feature] Topic for registration events Issue created by sososohan Motivation: Why do you think this is important? Workflow lineage is stored to the FlyteAdmin DB at registration time, and can be queried through the API. This information is valuable for systems where workflows depend on external inputs - it allows verification that the correct inputs are put into place for a smooth run without the workflow itself needing to be executed and failing. Adding this info to a topic similarly to NodeExecutionEvents allows easy consumption of this data without having to query the API for all workflows at registration time. Goal: What should the final outcome look like, ideally? Topic similar to the existing WorkflowExecutionEvent topic, but populated as workflows get registered. Our direct need is the Workflow proto, but im sure there may be a need for other information as well. Describe alternatives you've considered Adding a hook to our workflow registration script to query the FlyteAdmin API post registration to get the Workflow proto for all workflows just registered. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2327 [BUG] ListNamedEntity breaks when passing a filter with sortKey of created_at Issue created by pmahindrakar-oss Describe the bug By default flytectl uses created_at as sort key when listing flyte entities When making ListNamedEntities with workflow resource type , the current api breaks with the following error

flytectl get workflow -p flytesnacks -d development                                  
Error: rpc error: code = Unknown desc = failed database operation with column "workflows.created_at" must appear in the GROUP BY clause or be used in an aggregate function

Similarly it breaks on passing a field selector too

flytectl get workflow -p flytesnacks -d development --filter.fieldSelector "state=0"
Error: rpc error: code = Unknown desc = failed database operation with column "workflows.created_at" must appear in the GROUP BY clause or be used in an aggregate function

It returns correctly in case of project, domain, name

flytectl get workflow -p flytesnacks -d development --filter.fieldSelector "state=0" --filter.sortBy "project"
 --------------- ------------- ------------------------------------------------------------- ------------- ------- 
| PROJECT (45)  | DOMAIN      | NAME                                                        | DESCRIPTION | STATE |
 --------------- ------------- ------------------------------------------------------------- ------------- ------- 
| flytesnacks   | development | core.containerization.use_secrets.my_secret_workflow        |             |       |
 --------------- ------------- ------------------------------------------------------------- ------------- ------- 
| flytesnacks   | development | core.control_flow.chain_tasks.chain_tasks_wf                |             |       |
 --------------- ------------- ------------------------------------------------------------- ------------- -------

Expected behavior Named Entities should be able to sorted by created_at Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2310 [Core feature] [flytekit] Better OAuth2 login experience in jupyter Issue created by kumare3 Motivation: Why do you think this is important? Currently, the default experience of flytekit.Remote when working with an authenticated FlyteRemote endpoint is to open a new browser window to sign in, without the control coming back to the original notebook. This can cause a bad experience for the user. IpyAuth library as described here allows you to control the experience and makes it feel native to Notebook Goal: What should the final outcome look like, ideally? Users should return back to the notebook after the login is successful Describe alternatives you've considered NA Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3073 [Core feature] add support for numpy.typing.NDArray generic Issue created by cosmicBboy Motivation: Why do you think this is important? Flytekit supports the

np.ndarray

type, but it doesn't support the

np.typing.NDArray

generic: https://numpy.org/devdocs/reference/typing.html#numpy.typing.NDArray This would enable flytekit users to express the actual datatypes of an ndarray. Goal: What should the final outcome look like, ideally? The user-facing API would be simple:

import numpy
import numpy.typing as npt


@task
def t1(input: npt.NDArray[np.float64]) -> npt.NDArray[np.int64]:
    ...

Describe alternatives you've considered Keeping the status quo, only supporting

np.ndarray

Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3049 [Core feature] Use nightly flytekit builds in functional tests Issue created by eapolinario Motivation: Why do you think this is important? The release cycle for new flytekit versions is extremely long, requiring a (manual) process. This makes the feedback loop for catching errors in build steps more error-prone than necessary. Goal: What should the final outcome look like, ideally? Invest in nightly flytekit builds and use them in functional tests. Describe alternatives you've considered We could have a manual weekly release as an alternative, but that would still leave us with a longer-than-necessary turnaround time. Propose: Link/Inline OR Additional context Generate nightly flytekit images and pipe them through functional tests. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3037 [Core feature] [Flytekit] Add support for HDF5 and Arrow in flyteplugins-vaex Issue created by ryankarlos Motivation: Why do you think this is important? Currently

flyteplugins-vaex

supports automatic serialization and deserialization of vaex dataframe between consecutive tasks using parquet flyteorg/flytekit#1230 It would be good to extend this to HDF5 and arrow for performance and interoperability, when data sets are too large to fit into memory https://vaex.readthedocs.io/en/latest/faq.html#What-is-the-optimal-file-format-to-use-with-vaex Goal: What should the final outcome look like, ideally? Register extra handlers

VaexDataFrameToHDF5EncodingHandler

and

VaexDataFrameToArrowEncodingHandler

, so users can use

Annotated

to update the default format:

@task
def t1(f: vaex.dataframe.DataFrameLocal) -> Annotated[StructuredDataset, HDF5]

@task
def t2(f: vaex.dataframe.DataFrameLocal) -> Annotated[StructuredDataset, Arrow]

Describe alternatives you've considered N/A Propose: Link/Inline OR Additional context See discussion thread here flyteorg/flytekit#1230 (comment) Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2289 [Core feature] Manage domain-wide defaults as matchable attributes Issue created by katrogan Motivation: Why do you think this is important? Currently matchable resources are customizable on a project-domain, project-domain-workflow or project-domain-workflow-launch-plan basis. Traditionally, per-domain defaults are set in the flyteadmin config, for example: task resources. But it could be desirable to update the domain default without restarting the application due to the configmap change. Goal: What should the final outcome look like, ideally? Matchable resource attributes should be configurable through the cli/api calls for domain-wide values too. Describe alternatives you've considered current impl Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2283 [Housekeeping] Remove ID from base model in flyteadmin Issue created by katrogan Describe the issue The ID field in the flyteadmin BaseModel embedded in every db model is problematic. Autoincrement presents a performance penalty, but furthermore, after #2282 the id won't even be strictly necessary. What we should do is completely eliminate the ID field (and probably re-examine the necessity of DeletedAt too) N.B. In the executions model, LaunchPlanID, WorkflowID, and TaskID all unofficially refer to this ID field used for joins in list calls. The right solution here might be to use composite foreign keys and introduce a one-time (admittedly, expensive) migration to move existing rows over. The SourceExecutionID and ParentNodeExecutionID are also unofficial foreign reference keys unused outside of tests. What if we do not do this? More confusing data model, performance penalties Related component(s) flyteadmin Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2270 [BUG] [flytekit] Can't enter parameter of type List[FlyteFile] in Flyte Console Issue created by zm8597 Describe the bug When we input blob paths into launch plan parameter of type List[FlyteFile] and then we run the launch plan. The launch plan gets list of NoneType objects as an input instead of defined blobs list. When run locally everything works fine. Expected behavior List of FlyteFiles/blobs is inputted to workflow instead of NoneType objects. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2272 [BUG] [flytekit] ShellTask example not working Issue created by myz540 Describe the bug Name: flytekit Version: 0.31.0 macOS Big Sur 11.6.2 Python 3.7.6 Following the example shown here, we encounter the following error when running locally.

==============================================


    set -ex
    echo "Hey there! Let's run some bash scripts using Flyte's ShellTask."
    echo "Showcasing Flyte's Shell Task." >> /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt
    if grep "Flyte" /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt
    then
        echo "Found it!" >> /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt
    else
        echo "Not found!"
    fi


==============================================

+ echo 'Hey there! Let'\''s run some bash scripts using Flyte'\''s ShellTask.'
Hey there! Let's run some bash scripts using Flyte's ShellTask.
+ echo 'Showcasing Flyte'\''s Shell Task.'
+ grep Flyte /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt
Showcasing Flyte's Shell Task.
+ echo 'Found it!'
Traceback (most recent call last):
  File "shell.py", line 98, in <module>
    print(f"Running wf() {wf()}")
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/workflow.py", line 237, in __call__
    return flyte_entity_call_handler(self, *args, **input_kwargs)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/promise.py", line 991, in flyte_entity_call_handler
    result = cast(LocallyExecutable, entity).local_execute(child_ctx, **kwargs)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/workflow.py", line 252, in local_execute
    function_outputs = self.execute(**kwargs)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/workflow.py", line 685, in execute
    return exception_scopes.user_entry_point(self._workflow_function)(**kwargs)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/exceptions/scopes.py", line 198, in user_entry_point
    return wrapped(*args, **kwargs)
  File "shell.py", line 91, in wf
    t1_out = t1(x=x)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/base_task.py", line 288, in __call__
    return flyte_entity_call_handler(self, *args, **kwargs)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/promise.py", line 984, in flyte_entity_call_handler
    return cast(LocallyExecutable, entity).local_execute(ctx, **kwargs)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/base_task.py", line 269, in local_execute
    outputs_literal_map = self.dispatch_execute(ctx, input_literal_map)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/base_task.py", line 520, in dispatch_execute
    raise TypeError(f"Output({k}) in task{self.name} received a tuple {v}, instead of {py_type}")
TypeError: Output(i) in tasktask_1 received a tuple (/tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt, /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt), instead of <class 'flytekit.types.file.file.FlyteFile'>

Task 1 is apparently returning a tuple of length 2, with the two items being identical (the same file). We expect a single output from this shell task. We have already investigated this issue briefly and believe the cause is here If we add print statements, we see that

self._output_locs

has length 1 as expected and we can see that

self._output_locs

contains:

[OutputLocation(var='i', var_type=<class 'flytekit.types.file.file.FlyteFile'>, location='{inputs.x}')]
<class 'flytekit.extras.tasks.shell.OutputLocation'>

However, the two if statements both resolve as

True

. I believe this a result of the metaprogramming being used and the metaclasses that

FlyteFile

and

FlyteDirectory

are inheriting from. I am new to the code base so it is not immediately obvious what the solution would be but using

issubclass()

is not sufficient for determining if the

var_type

is

FlyteFile

or

FlyteDirectory

. Perhaps a simple solution would be:

for v in self._output_locs:
            if v.var_type is FlyteFile:
                final_outputs.append(FlyteFile(outputs[v.var]))
            elif v.var_type is FlyteDirectory:
                final_outputs.append(FlyteDirectory(outputs[v.var]))

EDIT UPDATE: Discovered this issue occurs on python 3.7.6 as well as 3.8.0 but not in 3.8.5 3.8.0

Python 3.8.0 (default, Nov  6 2019, 15:49:01)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.1.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from flytekit.types.directory import FlyteDirectory
   ...:
   ...: from flytekit.types.file import FlyteFile

In [2]: issubclass(FlyteDirectory, FlyteFile)
Out[2]: True

In [3]: issubclass(FlyteFile, FlyteDirectory)
Out[3]: True

3.8.5

Python 3.8.5 (default, Sep  4 2020, 02:22:02)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.1.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from flytekit.types.directory import FlyteDirectory
   ...:
   ...: from flytekit.types.file import FlyteFile

In [2]: issubclass(FlyteDirectory, FlyteFile)
Out[2]: False

In [3]: issubclass(FlyteFile, FlyteDirectory)
Out[3]: False

EDIT UPDATE 2 Issue looks to have been raised and patched in 3.7.7 and 3.8.2 but restricts usage of

ShellTask

(in its current state) to those versions https://bugs.python.org/issue38878 Expected behavior The expected behavior is for the example

ShellTasks

and

workflow

to actually execute and run to completion Additional context to reproduce 1. Install python 3.7.6 2. Copy the example shell tasks and workflow into a file:

shell.py

3. Run locally using

python shell.py

Screenshots ``` ============================================== set -ex echo "Hey there! Let's run some bash scripts using Flyte's ShellTask." echo "Showcasing Flyte's Shell Task." >> /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt if grep "Flyte" /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt then echo "Found it!" >> /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt else echo "Not found!" fi ============================================== + echo 'Hey there! Let'\''s run some bash scripts using Flyte'\''s ShellTask.' Hey there! Let's run some bash scripts using Flyte's ShellTask. + echo 'Showcasing Flyte'\''s Shell Task.' + grep Flyte /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt Showcasing Flyte's Shell Task. + echo 'Found it!' Traceback (most recent call last): File "shell.py", line 98, in <module> print(f"Running wf() {wf()}") File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/workflow.py", line 237, in call return flyte_entity_call_handler(self, *args, **input_kwargs) File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/promise.py", line 991, in flyte_entity_call_handler result = cast(LocallyExecutable, entity).local_execute(child_ctx, **kwargs) File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/wo… flyteorg/flyte

#2269 [FlyteCTL Feature] Introduce flytectl context Issue created by katrogan Describe the feature/command for FlyteCTL It's a common scenario to use flytectl to interact with different Flyte back-ends, for example sandbox and a hosted deployment or staging and production deployments. However, the experience is frustrating and difficult to debug with multiple authenticated back-ends. Currently flytectl stores one token in keychain which leads to `302`s when trying to use flytectl with another flyte host/back-end. Fixing this requires end-users to understand the cause of the 302, delete the token from keychain and re-initiate their requests to go through the auth flow. Provide a possible output or UX example Like kubectx, it would be useful to introduce flytectl contexts (

flytectx

maybe?) to manage different authenticated sessions and possibly flytectl configs and make switching between different flyte deployment environments seamless. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte