#1 Pydata seattle 2023 Pull request opened by eapolinario This PR adds the necessary files for the pydata Seattle 2023 tutorial. TODO ☐ slide deck in pdf flyteorg/flyte-conference-talks ✅ All checks have passed 2/2 successful checks

18 new commits pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by eapolinario

<https://github.com/flyteorg/flyte-conference-talks/commit/161ab7855f338cea563f235425f4eda6338a4d16|161ab785>

- Initial version

<https://github.com/flyteorg/flyte-conference-talks/commit/51afb916786beb56abd402f07005bfdfaddf9f37|51afb916>

- Use python 3.10 and fix local sandbox setup

<https://github.com/flyteorg/flyte-conference-talks/commit/44fe53bb4de1d8d202f216dcdcd646193e80d6d1|44fe53bb>

- Add support for partials

<https://github.com/flyteorg/flyte-conference-talks/commit/28be9739ff138c9253c130f9178d38e3e569685a|28be9739>

- Fix bug in partial tasks

<https://github.com/flyteorg/flyte-conference-talks/commit/a01cb5f9c6868cc6ccc6b591b497abe4ed3233a2|a01cb5f9>

- Nit: typo

<https://github.com/flyteorg/flyte-conference-talks/commit/ffb06a2ba1ddc7453d9e5a2759b2321f604d904a|ffb06a2b>

- flytekit 1.5.0

<https://github.com/flyteorg/flyte-conference-talks/commit/8bdbdb329ed225215708cd9463f4199f04056524|8bdbdb32>

- use correct

get_best_model

<https://github.com/flyteorg/flyte-conference-talks/commit/2d670a170cc2f027ba75abc57f1ff51611f66d57|2d670a17>

- Update workshop.ipynb

<https://github.com/flyteorg/flyte-conference-talks/commit/86654c04dbe9834d198a03cdd2eb5b3df868f9af|86654c04>

- Include mentions to Seattle

<https://github.com/flyteorg/flyte-conference-talks/commit/311d473571cc286bdbc32ab1cff0c02aef96c919|311d4735>

- Clear cell outputs

<https://github.com/flyteorg/flyte-conference-talks/commit/1aea59bbbe277c3b437170acc0ec11916be4edb2|1aea59bb>

- Use docker buildx to build and push images

<https://github.com/flyteorg/flyte-conference-talks/commit/1eeec77e0df6c9e0fc8cb0adf4b3dee32ce57c36|1eeec77e>

- Push images to ghcr.io/flyteorg

<https://github.com/flyteorg/flyte-conference-talks/commit/27b9a02d71ff0ef31d53ad0647b9c5fc4e501fac|27b9a02d>

- Simplify local setup

<https://github.com/flyteorg/flyte-conference-talks/commit/b975fdf917c1cdb780fc64e0958db7b7609fcddf|b975fdf9>

- Move definition of version to the right place

<https://github.com/flyteorg/flyte-conference-talks/commit/11fdd7fa9b9a006a9a374d986924d65a480326a5|11fdd7fa>

- Simplify dockerfile

<https://github.com/flyteorg/flyte-conference-talks/commit/eeff1da78cc2781a1cd00ec74c37e40ce9983c52|eeff1da7>

- Separate calls to download_deck

<https://github.com/flyteorg/flyte-conference-talks/commit/8dd9a91176ea71dcae162545511aa0bdf40f0487|8dd9a911>

- make local-sandbox-setup

<https://github.com/flyteorg/flyte-conference-talks/commit/e2661f756b3bc326fbd2b5042f4e98a7294c8032|e2661f75>

- Merge pull request #1 from flyteorg/pydata-seattle-2023 flyteorg/flyte-conference-talks

#749 chore: [tlm] comprehensive node execution query Pull request opened by ursucarina TL;DR Adds a node execution query that also fetches tasks and task closures Type ☑︎ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Adds a node execution query that also fetches tasks and task closures Tracking Issue NA Follow-up issue NA flyteorg/flyteconsole Codecov: 49.47% of diff hit (target 66.40%) GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release ✅ 7 other checks have passed 7/11 successful checks

#2 Minor fixes to pydata seattle 2023 tutorial Pull request opened by eapolinario flyteorg/flyte-conference-talks ✅ All checks have passed 2/2 successful checks

3 new commits pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by eapolinario

<https://github.com/flyteorg/flyte-conference-talks/commit/e8919f50fb22928f471f4db55afe3f05c33a801e|e8919f50>

- Minor adjustments

<https://github.com/flyteorg/flyte-conference-talks/commit/b9713e27ed81216cd05576edf6be74b1e820f026|b9713e27>

- Split commands into their own entries in notebook

<https://github.com/flyteorg/flyte-conference-talks/commit/70fb2c024ac095c5ba83864d63b8fdff8864e967|70fb2c02>

- Merge pull request #2 from flyteorg/minor-fixes-to-pydata-seattle-2023-tutorial flyteorg/flyte-conference-talks

force-pushed the

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

branch from

70fb2c02

to

e8919f50

flyteorg/flyte-conference-talks

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by eapolinario

<https://github.com/flyteorg/flyte-conference-talks/commit/be303de4443d5133fe0bba00ad6059f136574cd3|be303de4>

- Remove unnecessary instruction flyteorg/flyte-conference-talks

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by eapolinario

<https://github.com/flyteorg/flyte-conference-talks/commit/5650b89af72dd778bd7453bba33cbb807e54a6ed|5650b89a>

- Add PyData Seattle 2023 slides flyteorg/flyte-conference-talks

#8 feat: update ADOPTERS.md Pull request opened by michaeltinsley flyteorg/community ✅ All checks have passed 2/2 successful checks

#3621 [Core Feature] Flytekit Remote get task status within workflow Issue created by rbisecke I have a complicated workflow with a conditional break that kicks off one of two subsequent sub-workflows. When I use the flytekit remote functionality, I'd like to check the status of a given task within that workflow. Given conditional branching, there's no easy way to get the status of a given task( especially if it's not in the parent workflow). From a glance, I want to find out if the task is done or not which is different from the node status. This may be possible but not documented in case this is a documentation issue instead of a functionality one. flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/community/tree/main|main>

by kumare3

<https://github.com/flyteorg/community/commit/2520178661dce8a97f3eccd9eaa0e286108dbd17|25201786>

- feat: update ADOPTERS.md (#8) flyteorg/community

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by samhita-alla

<https://github.com/flyteorg/flyte/commit/22d2586bdf45aa0e3fd952ba014961d544e9cb48|22d2586b>

- Update README (#3618) flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by eapolinario

<https://github.com/flyteorg/flyte-conference-talks/commit/62a55d7684fdc831b206bde4351ccfafd9df7d6c|62a55d76>

- Simplify local install instructions flyteorg/flyte-conference-talks

#1610 always a default image Pull request opened by wild-endeavor TL;DR If not, if i do

pyflyte register --image custom=blah:latest ...

and i have non-"custom" tasks, registration fails. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin flyteorg/flytekit ✅ All checks have passed 2/2 successful checks

Release - [Alpha] Image Spec fixes New release published by wild-endeavor What's Changed • Add ImageRender class for Flyte Decks by @peridotml in #1599 • Bump tensorflow from 2.10.0 to 2.11.1 in /plugins/flytekit-mlflow by @dependabot in #1586 • Handle batchable lists of length 0 by @eapolinario in #1600 • pyflyte run imperative workflow by @pingsutw in #1597 • Backfill should fill in the right input vars by @kumare3 in #1593 • Pyflyte prettified by @kumare3 in #1602 New Contributors • @peridotml made their first contribution in #1599 Full Changelog: v1.6.0b0...v1.6.0a1 flyteorg/flytekit

#3623 [BUG] Spark Task Driver UI Link is broken Issue created by mucahitkantepe Describe the bug The Spark Driver UI link is broken. It refers to a link like

<https://http/>....

This line of code adds the additional

https://

https://github.com/flyteorg/flyteplugins/blob/63e1e45de43c39d0bae7dfb973f0176a18228ecb/go/tasks/plugins/k8s/spark/spark.go#L411 Older versions of Spark Operator didn't have http section but now it does https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/5f2efd4ff97e7c0bfdb726a066118d3401576730/pkg/controller/sparkapplication/sparkui.go#L57 Expected behavior The link should not contain duplicated http section Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#347 Fix double http in the Spark Driver UI Link Pull request opened by mucahitkantepe Fixes flyteorg/flyte#3623 Spark operator code: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/5f2efd4ff97e7c0bfdb726a066118d3401576730/pkg/controller/sparkapplication/sparkui.go#L57 This change solves the problem by keeping Flyte compatible with older version of Spark Operator. Another, simpler implentation is to just pass whatever spark operator gives. Browsers will add https automatically? flyteorg/flyteplugins DCO: DCO ✅ 1 other check has passed 1/2 successful checks

#747 feat: runtime metrics UI Pull request opened by 4nalog TL;DR This PR adds logic for allowing runtime task metrics to me displayed in the timeline chart component To run: • setup the project by cloning • run

yarn install

and

yarn start

• navigate to a specific workflow and check out the timeline view Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description This feature: • add logic to fetch task level runtime metrics in the timeline view using the new workflow execution metrics api • destructures the data returned from the workflow execution metrics api to get values from the corresponding phases in detail based on the operation id • displays those as a chart on the timeline Tracking Issue fixes flyteorg/flyteconsole GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release DCO: DCO ✅ 5 other checks have passed 5/9 successful checks

#988 Image spec example Pull request opened by pingsutw Add an example for using imageSpec in the task preview: https://flyte--988.org.readthedocs.build/projects/cookbook/en/988/auto/core/containerization/image_spec_example.html flyteorg/flytesnacks GitHub Actions: Create a list of packages GitHub Actions: Docs Warnings ✅ 2 other checks have passed 2/4 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by eapolinario

<https://github.com/flyteorg/flyte-conference-talks/commit/3ab5cbfdcff108cf991156085ea5c4622c45ed66|3ab5cbfd>

- Final update to Seattle slides flyteorg/flyte-conference-talks

#3624 [BUG] specifying image in PodTemplate does not set the image used for a NotebookTask Issue created by blair-anson Describe the bug I have been using a PodTemplate to set the default docker image for a normal @task(), as per the link below. However the NotebookTask ignores the image setting in the PodTemplate, although it does apply other settings such as VolumeMount https://docs.flyte.org/en/latest/deployment/configuration/general.html#using-default-k8s-podtemplates Example of a PodTemplate that is generated by clusterResourceTemplates in the values.yaml

$ kubectl get podtemplate flyte-template -n test8-development -o yaml
apiVersion: v1
kind: PodTemplate
...
template:
...
  spec:
    containers:
    - image: <http://xxxxxx.dkr.ecr.us-west-2.amazonaws.com/my-own-image:my-tag|xxxxxx.dkr.ecr.us-west-2.amazonaws.com/my-own-image:my-tag>
      imagePullPolicy: IfNotPresent
      name: default
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
...

Example of the Pod that runs

$ kubectl get pod f05d6a5f90403454f88b-n0-0 -n test8-development -o yaml
apiVersion: v1
kind: Pod
...
    image: <http://cr.flyte.org/flyteorg/flytekit:py3.8-1.4.1|cr.flyte.org/flyteorg/flytekit:py3.8-1.4.1>
    imagePullPolicy: IfNotPresent
    name: f05d6a5f90403454f88b-n0-0
...
  containerStatuses:
  - containerID: <docker://adda36ed3823592a117e5b99f284accddee987e5d966e6b48b1c46eab7e98fa7>
    image: <http://cr.flyte.org/flyteorg/flytekit:py3.8-1.4.1|cr.flyte.org/flyteorg/flytekit:py3.8-1.4.1>
    imageID: <docker-pullable://cr.flyte.org/flyteorg/flytekit@sha256:dcc3c40249bb983eb96dc0024f1506f4e261a64f1c3a1f447f94750720cb8ae3>
    lastState: {}
    name: f05d6a5f90403454f88b-n0-0
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
...

Expected behavior The Pod processing the NotebookTask should use the docker image defined in the PodTemplate Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3625 Add azure ad auth setup doc Pull request opened by ByronHsu Tracking issue #3620 Describe your changes Add azure ad setup doc to Authenticating in Flyte Check all the applicable boxes ☑︎ I updated the documentation accordingly. ☐ All new and existing tests passed. ☐ All commits are signed-off. Screenshots Note to reviewers flyteorg/flyte ✅ All checks have passed 6/6 successful checks

#1611 Add http_proxy to client & Fix deviceflow Pull request opened by ByronHsu TL;DR 1. Add http_proxy to oauth2 client because in on-prem cluster use cases, the client might not have access to the external network 2. Remove

verification_url_complete

because that field not exists in azure ad and the oauth2 standard. 3. Concat the scopes by white space per the documentation Type ☑︎ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Tracking Issue flyteorg/flyte#3620 flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#398 Add oauth2 http proxy client Pull request opened by ByronHsu TL;DR Add http_proxy to oauth2 client because in on-prem cluster use cases, the client might not have access to the external network Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ flyteorg/flyte#3620 Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteidl Codecov: 46.66% of diff hit (target 76.16%) Codecov: 75.90% (-0.26%) compared to 21107f5 ✅ 11 other checks have passed 11/13 successful checks

#553 Add oauth http proxy for external server & Extract email from azure claim Pull request opened by ByronHsu TL;DR 1. Add oauth http proxy for external server 2. Extract email from azure claim Type ☑︎ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ flyteorg/flyte#3620 Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteadmin ✅ All checks have passed 2/2 successful checks

#1612 [Issue-3617] Enables FlyteFiles, FlyteDirectors, and StructuredDatasets inputs in papermill plugin Pull request opened by peridotml TL;DR This is a first draft for allowing the papermill plugin to access more complex inputs. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Right now the papermill plugin can't use

FlyteFile

or

StructuredDataset

because of a Papermill limitation that everything is json serializable. My strategy is mostly copying

record_outputs

. These are the steps 1. Reserializing the inputs and save to local disk 2. Add a helper function

read_input

where users must give the input type (maybe there is a smart fix) 3.

read_input

loads the protos and uses the type to convert it to the correct format Basically I am using flyte to serialize the inputs locally (see comments), then you just need to do this.

### Params
pi = (filled in by papermill)

### In a new cell
pi = read_input(pi, float)

Tracking Issue flyteorg/flyte#3617 Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#3626 [Core feature] Flytekit: Add support for execution name prefixes in FlyteRemote.execute() Issue created by tomasz-sodzawiczny Motivation: Why do you think this is important? Setting human-readable names might be quite for useful inspecting running executions; at the same time the behavior might be unexpected / dangerous if used naively: Running:

ex1 = flyte_remote.execute(my_task, {"input": 1}, execution_name="foo")
ex2 = flyte_remote.execute(my_task, {"input": 2}, execution_name="foo")

Creates only 1 execution (or 0, if execution named "foo" already existed). The second set of inputs is silently ignored, and the existing execution is returned. Goal: What should the final outcome look like, ideally? I would suggest a way to generate a name with a human-readable prefix (with behavior similar to e.g. Kubernetes's

generateName

):

ex1 = flyte_remote.execute(my_task, {"input": 1}, execution_name_prefix="foo-")
ex2 = flyte_remote.execute(my_task, {"input": 2}, execution_name_prefix="foo-")

ex1.id.name # Something like "foo-41e2c72fa869"
ex2.id.name # Something like "foo-7742ba115212"

Describe alternatives you've considered Propose: Link/Inline OR Additional context Note: I'm happy to create a PR to flytekit if the idea seems reasonable. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3627 [Core feature] Flytekit: Add support for ReferenceEntities in FlyteRemode.execute() Issue created by tomasz-sodzawiczny Motivation: Why do you think this is important? If I have a ReferenceEntity, e.g.

@reference_task(
    project="flytesnacks",
    domain="development",
    name="recipes.aaa.simple.join_strings",
    version="553018f39e519bdb2597b652639c30ce16b99c79",
)
def ref_t1(a: typing.List[str]) -> str:
    ...

I would like to be able to call it with FlyteRemote.execute:

flyte_remote.execute(ref_t1, dict(a=["foo", "bar"]))

Currently this code runs (because RemoteTask extends PythonTask and is treated as such), but ignores the version (and probably domain and project) - runs with the latest registered version of that task. Goal: What should the final outcome look like, ideally? Ideally I'd want to run any ReferenceEntity with

flyte_remote.execute(ref_entity, inputs=...)

And execute would respect all set values. For consistency with the other types, methods like

FlyteRemote.execute_reference_X()

would have to be introduced too. Describe alternatives you've considered This is a working work-around:

flyte_remote.execute(ref_entity, version=ref_entity.id.version, inputs=...)

(Not considering project/domain, but same thing should work). Propose: Link/Inline OR Additional context Note: I'm happy to create a PR to flytekit if this change is reasonable. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3628 [BUG] Flytekit: 'FlyteBranchNode' object has no attribute 'interface' Issue created by tomasz-sodzawiczny Describe the bug

FlyteRemote.sync_execution()

(and other methods that rely on it, e.g.

FlyteRemote.wait()

and

FlyteRemote.execute()

with wait=True) occasionally fails with

AttributeError: 'FlyteBranchNode' object has no attribute 'interface'

The offending line is remote.py#L1644 (I pasted full stack trace in the "Sreenshots" below). The error only happens while the task is running (and is some specific state). Seems to happen for us mostly when the tasks inside of the conditional take some time to schedule (e.g. tasks with resource requirements that trigger cluster scale-up). This pretty much renders

wait()

unusable for us, we had do our own wait wrapper that has a try..except around the

sync_execution

to handle this case. Expected behavior It shuold not raise? ;) Additional context to reproduce No response Screenshots Stack trace:

AttributeError                            Traceback (most recent call last)
Cell In[9], line 3
----> 3 flyte.sync_execution(ex, sync_nodes=True)

File [~/Library/Caches/pypoetry/virtualenvs/platform-QGRyQAM2-py3.10/lib/python3.10/site-packages/flytekit/remote/remote.py:1503](<https://file>+.<http://vscode-resource.vscode-cdn.net/Users/tomasz/projects/workflows/demos/~/Library/Caches/pypoetry/virtualenvs/platform-QGRyQAM2-py3.10/lib/python3.10/site-packages/flytekit/remote/remote.py:1503|vscode-resource.vscode-cdn.net/Users/tomasz/projects/workflows/demos/~/Library/Caches/pypoetry/virtualenvs/platform-QGRyQAM2-py3.10/lib/python3.10/site-packages/flytekit/remote/remote.py:1503>), in FlyteRemote.sync_execution(self, execution, entity_definition, sync_nodes)
   1501     node_execs = {}
   1502     for n in underlying_node_executions:
-> 1503         node_execs[n.id.node_id] = self.sync_node_execution(n, node_mapping)  # noqa
   1504     execution._node_executions = node_execs
   1505 return self._assign_inputs_and_outputs(execution, execution_data, node_interface)

File [~/Library/Caches/pypoetry/virtualenvs/platform-QGRyQAM2-py3.10/lib/python3.10/site-packages/flytekit/remote/remote.py:1645](<https://file>+.<http://vscode-resource.vscode-cdn.net/Users/tomasz/projects/workflows/demos/~/Library/Caches/pypoetry/virtualenvs/platform-QGRyQAM2-py3.10/lib/python3.10/site-packages/flytekit/remote/remote.py:1645|vscode-resource.vscode-cdn.net/Users/tomasz/projects/workflows/demos/~/Library/Caches/pypoetry/virtualenvs/platform-QGRyQAM2-py3.10/lib/python3.10/site-packages/flytekit/remote/remote.py:1645>), in FlyteRemote.sync_node_execution(self, execution, node_mapping)
   1639 # This is the plain ol' task execution case
   1640 else:
   1641     execution._task_executions = [
   1642         self.sync_task_execution(FlyteTaskExecution.promote_from_model(t))
   1643         for t in iterate_task_executions(self.client, execution.id)
   1644     ]
-> 1645     execution._interface = execution._node.flyte_entity.interface
   1647 self._assign_inputs_and_outputs(
   1648     execution,
   1649     node_execution_get_data_response,
   1650     execution.interface,
   1651 )
   1653 return execution

AttributeError: 'FlyteBranchNode' object has no attribute 'interface'

Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flytesnacks/commit/67c8c4166ed56a6915200fd392cb464455cf26cf|67c8c416>

- Update vault secret manager documentation (#986) flyteorg/flytesnacks