GitHub
04/26/2023, 6:26 AMGitHub
04/26/2023, 6:27 AM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by eapolinario
<https://github.com/flyteorg/flyte-conference-talks/commit/161ab7855f338cea563f235425f4eda6338a4d16|161ab785>
- Initial version
<https://github.com/flyteorg/flyte-conference-talks/commit/51afb916786beb56abd402f07005bfdfaddf9f37|51afb916>
- Use python 3.10 and fix local sandbox setup
<https://github.com/flyteorg/flyte-conference-talks/commit/44fe53bb4de1d8d202f216dcdcd646193e80d6d1|44fe53bb>
- Add support for partials
<https://github.com/flyteorg/flyte-conference-talks/commit/28be9739ff138c9253c130f9178d38e3e569685a|28be9739>
- Fix bug in partial tasks
<https://github.com/flyteorg/flyte-conference-talks/commit/a01cb5f9c6868cc6ccc6b591b497abe4ed3233a2|a01cb5f9>
- Nit: typo
<https://github.com/flyteorg/flyte-conference-talks/commit/ffb06a2ba1ddc7453d9e5a2759b2321f604d904a|ffb06a2b>
- flytekit 1.5.0
<https://github.com/flyteorg/flyte-conference-talks/commit/8bdbdb329ed225215708cd9463f4199f04056524|8bdbdb32>
- use correct get_best_model
<https://github.com/flyteorg/flyte-conference-talks/commit/2d670a170cc2f027ba75abc57f1ff51611f66d57|2d670a17>
- Update workshop.ipynb
<https://github.com/flyteorg/flyte-conference-talks/commit/86654c04dbe9834d198a03cdd2eb5b3df868f9af|86654c04>
- Include mentions to Seattle
<https://github.com/flyteorg/flyte-conference-talks/commit/311d473571cc286bdbc32ab1cff0c02aef96c919|311d4735>
- Clear cell outputs
<https://github.com/flyteorg/flyte-conference-talks/commit/1aea59bbbe277c3b437170acc0ec11916be4edb2|1aea59bb>
- Use docker buildx to build and push images
<https://github.com/flyteorg/flyte-conference-talks/commit/1eeec77e0df6c9e0fc8cb0adf4b3dee32ce57c36|1eeec77e>
- Push images to ghcr.io/flyteorg
<https://github.com/flyteorg/flyte-conference-talks/commit/27b9a02d71ff0ef31d53ad0647b9c5fc4e501fac|27b9a02d>
- Simplify local setup
<https://github.com/flyteorg/flyte-conference-talks/commit/b975fdf917c1cdb780fc64e0958db7b7609fcddf|b975fdf9>
- Move definition of version to the right place
<https://github.com/flyteorg/flyte-conference-talks/commit/11fdd7fa9b9a006a9a374d986924d65a480326a5|11fdd7fa>
- Simplify dockerfile
<https://github.com/flyteorg/flyte-conference-talks/commit/eeff1da78cc2781a1cd00ec74c37e40ce9983c52|eeff1da7>
- Separate calls to download_deck
<https://github.com/flyteorg/flyte-conference-talks/commit/8dd9a91176ea71dcae162545511aa0bdf40f0487|8dd9a911>
- make local-sandbox-setup
<https://github.com/flyteorg/flyte-conference-talks/commit/e2661f756b3bc326fbd2b5042f4e98a7294c8032|e2661f75>
- Merge pull request #1 from flyteorg/pydata-seattle-2023
flyteorg/flyte-conference-talksGitHub
04/26/2023, 6:34 AMGitHub
04/26/2023, 7:06 AMGitHub
04/26/2023, 7:06 AM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by eapolinario
<https://github.com/flyteorg/flyte-conference-talks/commit/e8919f50fb22928f471f4db55afe3f05c33a801e|e8919f50>
- Minor adjustments
<https://github.com/flyteorg/flyte-conference-talks/commit/b9713e27ed81216cd05576edf6be74b1e820f026|b9713e27>
- Split commands into their own entries in notebook
<https://github.com/flyteorg/flyte-conference-talks/commit/70fb2c024ac095c5ba83864d63b8fdff8864e967|70fb2c02>
- Merge pull request #2 from flyteorg/minor-fixes-to-pydata-seattle-2023-tutorial
flyteorg/flyte-conference-talksGitHub
04/26/2023, 7:13 AM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
branch from 70fb2c02
to e8919f50
flyteorg/flyte-conference-talksGitHub
04/26/2023, 7:25 AM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by eapolinario
<https://github.com/flyteorg/flyte-conference-talks/commit/be303de4443d5133fe0bba00ad6059f136574cd3|be303de4>
- Remove unnecessary instruction
flyteorg/flyte-conference-talksGitHub
04/26/2023, 7:35 AM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by eapolinario
<https://github.com/flyteorg/flyte-conference-talks/commit/5650b89af72dd778bd7453bba33cbb807e54a6ed|5650b89a>
- Add PyData Seattle 2023 slides
flyteorg/flyte-conference-talksGitHub
04/26/2023, 8:11 AMGitHub
04/26/2023, 12:28 PMGitHub
04/26/2023, 2:07 PM<https://github.com/flyteorg/community/tree/main|main>
by kumare3
<https://github.com/flyteorg/community/commit/2520178661dce8a97f3eccd9eaa0e286108dbd17|25201786>
- feat: update ADOPTERS.md (#8)
flyteorg/communityGitHub
04/26/2023, 5:10 PM<https://github.com/flyteorg/flyte/tree/master|master>
by samhita-alla
<https://github.com/flyteorg/flyte/commit/22d2586bdf45aa0e3fd952ba014961d544e9cb48|22d2586b>
- Update README (#3618)
flyteorg/flyteGitHub
04/26/2023, 6:18 PM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by eapolinario
<https://github.com/flyteorg/flyte-conference-talks/commit/62a55d7684fdc831b206bde4351ccfafd9df7d6c|62a55d76>
- Simplify local install instructions
flyteorg/flyte-conference-talksGitHub
04/26/2023, 6:41 PMpyflyte register --image custom=blah:latest ...
and i have non-"custom" tasks, registration fails.
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
flyteorg/flytekit
✅ All checks have passed
2/2 successful checksGitHub
04/26/2023, 8:06 PMGitHub
04/26/2023, 9:36 PM<https://http/>....
image▾
https://
https://github.com/flyteorg/flyteplugins/blob/63e1e45de43c39d0bae7dfb973f0176a18228ecb/go/tasks/plugins/k8s/spark/spark.go#L411
Older versions of Spark Operator didn't have http section but now it does https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/5f2efd4ff97e7c0bfdb726a066118d3401576730/pkg/controller/sparkapplication/sparkui.go#L57
Expected behavior
The link should not contain duplicated http section
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/26/2023, 9:42 PMGitHub
04/26/2023, 9:44 PMyarn install
and yarn start
• navigate to a specific workflow and check out the timeline view
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
This feature:
• add logic to fetch task level runtime metrics in the timeline view using the new workflow execution metrics api
• destructures the data returned from the workflow execution metrics api to get values from the corresponding phases in detail based on the operation id
• displays those as a chart on the timeline
Tracking Issue
fixes
flyteorg/flyteconsole
GitHub Actions: Build & Push FlyteConsole Image
GitHub Actions: Get Release Tag
GitHub Actions: Generate Release
DCO: DCO
✅ 5 other checks have passed
5/9 successful checksGitHub
04/26/2023, 9:47 PMimage▾
GitHub
04/27/2023, 4:50 AM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by eapolinario
<https://github.com/flyteorg/flyte-conference-talks/commit/3ab5cbfdcff108cf991156085ea5c4622c45ed66|3ab5cbfd>
- Final update to Seattle slides
flyteorg/flyte-conference-talksGitHub
04/27/2023, 6:26 AM$ kubectl get podtemplate flyte-template -n test8-development -o yaml
apiVersion: v1
kind: PodTemplate
...
template:
...
spec:
containers:
- image: <http://xxxxxx.dkr.ecr.us-west-2.amazonaws.com/my-own-image:my-tag|xxxxxx.dkr.ecr.us-west-2.amazonaws.com/my-own-image:my-tag>
imagePullPolicy: IfNotPresent
name: default
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
...
Example of the Pod that runs
$ kubectl get pod f05d6a5f90403454f88b-n0-0 -n test8-development -o yaml
apiVersion: v1
kind: Pod
...
image: <http://cr.flyte.org/flyteorg/flytekit:py3.8-1.4.1|cr.flyte.org/flyteorg/flytekit:py3.8-1.4.1>
imagePullPolicy: IfNotPresent
name: f05d6a5f90403454f88b-n0-0
...
containerStatuses:
- containerID: <docker://adda36ed3823592a117e5b99f284accddee987e5d966e6b48b1c46eab7e98fa7>
image: <http://cr.flyte.org/flyteorg/flytekit:py3.8-1.4.1|cr.flyte.org/flyteorg/flytekit:py3.8-1.4.1>
imageID: <docker-pullable://cr.flyte.org/flyteorg/flytekit@sha256:dcc3c40249bb983eb96dc0024f1506f4e261a64f1c3a1f447f94750720cb8ae3>
lastState: {}
name: f05d6a5f90403454f88b-n0-0
ready: false
restartCount: 0
started: false
state:
terminated:
...
Expected behavior
The Pod processing the NotebookTask should use the docker image defined in the PodTemplate
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/27/2023, 8:05 AMGitHub
04/27/2023, 8:17 AMverification_url_complete
because that field not exists in azure ad and the oauth2 standard.
3. Concat the scopes by white space per the documentation
Type
☑︎ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☐ Smoke tested
☑︎ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Tracking Issue
flyteorg/flyte#3620
flyteorg/flytekit
✅ All checks have passed
30/30 successful checksGitHub
04/27/2023, 8:22 AMGitHub
04/27/2023, 8:27 AMGitHub
04/27/2023, 7:08 PMFlyteFile
or StructuredDataset
because of a Papermill limitation that everything is json serializable.
My strategy is mostly copying record_outputs
. These are the steps
1. Reserializing the inputs and save to local disk
2. Add a helper function read_input
where users must give the input type (maybe there is a smart fix)
3. read_input
loads the protos and uses the type to convert it to the correct format
Basically I am using flyte to serialize the inputs locally (see comments), then you just need to do this.
### Params
pi = (filled in by papermill)
### In a new cell
pi = read_input(pi, float)
Screenshot 2023-05-02 at 1 42 43 PM▾
GitHub
04/27/2023, 9:41 PMex1 = flyte_remote.execute(my_task, {"input": 1}, execution_name="foo")
ex2 = flyte_remote.execute(my_task, {"input": 2}, execution_name="foo")
Creates only 1 execution (or 0, if execution named "foo" already existed). The second set of inputs is silently ignored, and the existing execution is returned.
Goal: What should the final outcome look like, ideally?
I would suggest a way to generate a name with a human-readable prefix (with behavior similar to e.g. Kubernetes's generateName
):
ex1 = flyte_remote.execute(my_task, {"input": 1}, execution_name_prefix="foo-")
ex2 = flyte_remote.execute(my_task, {"input": 2}, execution_name_prefix="foo-")
ex1.id.name # Something like "foo-41e2c72fa869"
ex2.id.name # Something like "foo-7742ba115212"
Describe alternatives you've considered
Propose: Link/Inline OR Additional context
Note: I'm happy to create a PR to flytekit if the idea seems reasonable.
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/27/2023, 10:00 PM@reference_task(
project="flytesnacks",
domain="development",
name="recipes.aaa.simple.join_strings",
version="553018f39e519bdb2597b652639c30ce16b99c79",
)
def ref_t1(a: typing.List[str]) -> str:
...
I would like to be able to call it with FlyteRemote.execute:
flyte_remote.execute(ref_t1, dict(a=["foo", "bar"]))
Currently this code runs (because RemoteTask extends PythonTask and is treated as such), but ignores the version (and probably domain and project) - runs with the latest registered version of that task.
Goal: What should the final outcome look like, ideally?
Ideally I'd want to run any ReferenceEntity with
flyte_remote.execute(ref_entity, inputs=...)
And execute would respect all set values.
For consistency with the other types, methods like FlyteRemote.execute_reference_X()
would have to be introduced too.
Describe alternatives you've considered
This is a working work-around:
flyte_remote.execute(ref_entity, version=ref_entity.id.version, inputs=...)
(Not considering project/domain, but same thing should work).
Propose: Link/Inline OR Additional context
Note: I'm happy to create a PR to flytekit if this change is reasonable.
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/28/2023, 9:01 AMFlyteRemote.sync_execution()
(and other methods that rely on it, e.g. FlyteRemote.wait()
and FlyteRemote.execute()
with wait=True) occasionally fails with
AttributeError: 'FlyteBranchNode' object has no attribute 'interface'
The offending line is remote.py#L1644 (I pasted full stack trace in the "Sreenshots" below).
The error only happens while the task is running (and is some specific state). Seems to happen for us mostly when the tasks inside of the conditional take some time to schedule (e.g. tasks with resource requirements that trigger cluster scale-up).
This pretty much renders wait()
unusable for us, we had do our own wait wrapper that has a try..except around the sync_execution
to handle this case.
Expected behavior
It shuold not raise? ;)
Additional context to reproduce
No response
Screenshots
Stack trace:
AttributeError Traceback (most recent call last)
Cell In[9], line 3
----> 3 flyte.sync_execution(ex, sync_nodes=True)
File [~/Library/Caches/pypoetry/virtualenvs/platform-QGRyQAM2-py3.10/lib/python3.10/site-packages/flytekit/remote/remote.py:1503](<https://file>+.<http://vscode-resource.vscode-cdn.net/Users/tomasz/projects/workflows/demos/~/Library/Caches/pypoetry/virtualenvs/platform-QGRyQAM2-py3.10/lib/python3.10/site-packages/flytekit/remote/remote.py:1503|vscode-resource.vscode-cdn.net/Users/tomasz/projects/workflows/demos/~/Library/Caches/pypoetry/virtualenvs/platform-QGRyQAM2-py3.10/lib/python3.10/site-packages/flytekit/remote/remote.py:1503>), in FlyteRemote.sync_execution(self, execution, entity_definition, sync_nodes)
1501 node_execs = {}
1502 for n in underlying_node_executions:
-> 1503 node_execs[n.id.node_id] = self.sync_node_execution(n, node_mapping) # noqa
1504 execution._node_executions = node_execs
1505 return self._assign_inputs_and_outputs(execution, execution_data, node_interface)
File [~/Library/Caches/pypoetry/virtualenvs/platform-QGRyQAM2-py3.10/lib/python3.10/site-packages/flytekit/remote/remote.py:1645](<https://file>+.<http://vscode-resource.vscode-cdn.net/Users/tomasz/projects/workflows/demos/~/Library/Caches/pypoetry/virtualenvs/platform-QGRyQAM2-py3.10/lib/python3.10/site-packages/flytekit/remote/remote.py:1645|vscode-resource.vscode-cdn.net/Users/tomasz/projects/workflows/demos/~/Library/Caches/pypoetry/virtualenvs/platform-QGRyQAM2-py3.10/lib/python3.10/site-packages/flytekit/remote/remote.py:1645>), in FlyteRemote.sync_node_execution(self, execution, node_mapping)
1639 # This is the plain ol' task execution case
1640 else:
1641 execution._task_executions = [
1642 self.sync_task_execution(FlyteTaskExecution.promote_from_model(t))
1643 for t in iterate_task_executions(self.client, execution.id)
1644 ]
-> 1645 execution._interface = execution._node.flyte_entity.interface
1647 self._assign_inputs_and_outputs(
1648 execution,
1649 node_execution_get_data_response,
1650 execution.interface,
1651 )
1653 return execution
AttributeError: 'FlyteBranchNode' object has no attribute 'interface'
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/28/2023, 3:12 PM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flytesnacks/commit/67c8c4166ed56a6915200fd392cb464455cf26cf|67c8c416>
- Update vault secret manager documentation (#986)
flyteorg/flytesnacks