GitHub
06/12/2023, 3:16 PMGitHub
06/12/2023, 3:31 PMfrom flytekit import task, workflow
from flytekit.image_spec import ImageSpec
new_flytekit = "git+<https://github.com/flyteorg/flytekit@8c7c43ad310ec74b4e8c06c2b4604b6cb9692f86>"
image_spec = ImageSpec(base_image="ubuntu20.04", packages=[new_flytekit], cuda="11.2.2", cudnn="8",
registry="pingsutw", apt_packages=["git"], python_version="3.9",)
Type
☐ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☐ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
^^^
Tracking Issue
NA
Follow-up issue
NA
flyteorg/flytekit
✅ All checks have passed
30/30 successful checksGitHub
06/12/2023, 3:32 PMpyflyte register
.
https://gist.github.com/zeryx/a3074a2aade6a7263c3d796d08891741
However, when you register and execute with pyflyte run
, pyflyte run --copy-all
, or flytekit.remote.register_workflow
the ImageSpec build step is skipped, and the container_image argument failsover to use the base image.
Expected behavior
The expected behavior is regardless of the mechanism used to register the workflow onto a flyte cluster, if an imagespec image is required for any tasks; those ImageSpec steps should be built via envd and deployed to the provided registry.
If that fails for whatever reason, the registration itself should fail; and return a useful error indicating that an ImageSpec build process failed for some reason; rather than registering the task with the default flyte base image.
If one is using flytekit as a library and triggering register_workflow() or register_task() and an imagespec image is provided as a container_image
argument; that ImageSpec image should attempt to build said image, or validate that the existing image already exists.
Additional context to reproduce
• pyflyte init, replace your workflow/example.py with https://gist.github.com/zeryx/a3074a2aade6a7263c3d796d08891741
• call pyflyte run --copy-all workflows/example.py
• verify that envd is not being interacted with, and no images are written to registry
Screenshots
image▾
pyflyte --config ~/.uctl/config.yaml run --remote --copy-all workflows/example.py wf
2023-06-12 11:23:46,062144 WARNING {"asctime": "2023-06-12 11:23:46,062", "name": "flytekit.cli", "levelname": "WARNING", ignore.py:51
"message": "Could not determine ignored files due to:\nb'fatal: not a git repository
(or any of the parent directories): .git\\n'\nNot applying any filters"}
Opening in existing browser session.
libva error: /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so init failed
libva error: /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so init failed
127.0.0.1 - - [12/Jun/2023 11:24:13] "GET /callback?code=WnyxL2kywRTHw3tSSMLVO3T-38OLUwpQbOr5_8i_snY&state=feEPffKlUICJ63Y8bCuEtXBtgcU3kY2Oi1sQNrme5amT17xMnhgNOA HTTP/1.1" 200 -
Go to <https://demo.hosted.unionai.cloud/console/projects/flytesnacks/domains/development/executions/f8a710d7a26b74e77a9e> to see execution in the console.
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
06/12/2023, 3:42 PMGitHub
06/12/2023, 4:01 PMGitHub
06/12/2023, 4:01 PMGitHub
06/12/2023, 4:25 PMFLYTE_NAVIGATION.items[]
. Currently, when a user exports FLYTE_NAVIGATION
and provides values for custom navigation using items[]
we render a dropdown with those links.
Example from documentation:
{
color:"white", // optional - default NavBar text color, if not provided uses Flyte colors
background:"black", // optional - default NavBar background color, if not provided uses Flyte colors
console:"FlyteConsole" // optional - name of the default navigation, if not provided uses "Console"
items:[ // required - if no dropdown needed provide an empty array
{title:"Remote", url:"<https://remote.site/>"},
{title:"Dashboard", url:"+/projects/flytesnacks/executions?domain=development&duration=all"},
{title:"Information", url:"/information"}
]
}
1. Add the ability to have dropdown links open in a new tab (current behavior redirects/adds to history) via some optional property (eg, items[{title:"foo", url:"bar", target:"_blank"}]
)
2. Do not include BASE_URL
as a default in the dropdown or at least gives users the choice where it appears in the list. This could work a couple of ways (eg, we could just rely on users clicking the logo to return to console
from some external app or we could even add some sort of logic when creating that list to ensure that console
is present but users could choose which position in the list console
appears.
Goal: What should the final outcome look like, ideally?
1. Users can choose to have certain links in the dropdown open in new tabs.
2. Users can choose where console
appears in that list or if it appears at all.
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
06/12/2023, 5:25 PM<https://github.com/flyteorg/flyte/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flyte/commit/1ee95de9fab19af1e058ab02137a68101bf53c38|1ee95de9>
- Add agent to helm chart and single binary (#3454)
flyteorg/flyteGitHub
06/12/2023, 5:25 PMGitHub
06/12/2023, 5:47 PMGitHub
06/12/2023, 6:12 PMpyflyte run
involving ImageSpec
rebuild python and apt dependencies from scratch instead of leveraging cache.
Expected behavior
When an `ImageSpec`definition is not updated, image rebuilds should reuse cached layers where possible to speed up build times.
Additional context to reproduce
1. Create a script, wf.py
with the following contents:
from flytekit import task, workflow, ImageSpec
image = ImageSpec(
packages=["smart-open[gcs]"],
apt_packages=["git"],
)
@task(container_image=image)
def my_task() -> str:
return "hello, world!"
@workflow
def my_wf() -> str:
return my_task()
2. Run the following command multiple times:
pyflyte run --remote wf.py my_wf
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
06/12/2023, 7:05 PMFlyteRemote.execute
, the user must know the workflow interface and provide workflow inputs as a Python dictionary when launching a workflow. This works, but often involves the user going back and forth between the FlyteConsole and their Jupyter Notebook to ensure all inputs are provided and are of the correct type.
To address this, I created a version of FlyteRemote that leverages Pydantic to dynamically create workflow interfaces. This works by:
1. Querying Flyte for workflows and their input/output interface
2. Converting the interface to a Pydantic model and populating the docstring
3. Returning this model with a execute
method that enables launching the workflow on a remote cluster
flyteorg/flytekit
✅ All checks have passed
30/30 successful checksGitHub
06/12/2023, 11:12 PMGitHub
06/13/2023, 4:54 AM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by ByronHsu
<https://github.com/flyteorg/flytesnacks/commit/4b031a565c3320d80edc23098f38be15986f29b0|4b031a56>
- Update tasks_and_workflows.md (#997)
flyteorg/flytesnacksGitHub
06/13/2023, 9:43 AMGitHub
06/13/2023, 10:43 AMGitHub
06/13/2023, 1:48 PMGitHub
06/13/2023, 2:23 PMuint
and int64
(I have no proof though, but https://github.com/flyteorg/flyteadmin/blob/master/pkg/repositories/models/resource.go#L17 might indicate that because resources_id_seq
is bigint
), but we have seen different sequence types in our database, some of them being integer
while some of them being bigint
. For example the newer sequence description_entities_id_seq
is bigint
while older things like node_execution_events_id_seq
is integer
that is obviously too short.
Also the sequences are not cycle which means they will not wrap around when reaching max value. This is somewhat problematic especially for those integer
typed sequences. We have scheduled data retention job to clean up database so old entries are deleted, but since the sequence number is not reusable, we can hit max value easily when it is an integer
.
This gives a quick overview of what we have in database (note that some of the types have been manually altered: executions_id_seq
, node_execution_events_id_seq
)
sequencename | data_type | min_value | max_value | cycle
-----------------------------------------+-----------+-----------+---------------------+-------
projects_id_seq | integer | 1 | 2147483647 | f
description_entities_id_seq | bigint | 1 | 9223372036854775807 | f
tasks_id_seq | integer | 1 | 2147483647 | f
signals_id_seq | bigint | 1 | 9223372036854775807 | f
workflows_id_seq | integer | 1 | 2147483647 | f
launch_plans_id_seq | integer | 1 | 2147483647 | f
executions_id_seq | bigint | 1 | 9223372036854775807 | f
execution_events_id_seq | integer | 1 | 2147483647 | f
node_executions_id_seq | integer | 1 | 2147483647 | f
node_execution_events_id_seq | bigint | 1 | 9223372036854775807 | f
task_executions_id_seq | integer | 1 | 2147483647 | f
named_entity_metadata_id_seq | integer | 1 | 2147483647 | f
named_entity_metadata_resource_type_seq | integer | 1 | 2147483647 | f
resources_id_seq | bigint | 1 | 9223372036854775807 | f
schedulable_entities_id_seq | integer | 1 | 2147483647 | f
schedule_entities_snapshots_id_seq | integer | 1 | 2147483647 | f
Expected behavior
All sequences are typed as bigint
, and it would be nice to make them cycle (although a quick search didn't find how gorm could support that).
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
06/13/2023, 2:59 PMGitHub
06/13/2023, 3:44 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/c3b087623a15615a21966fa1641b55a960d22ea2|c3b08762>
- pyflyte run remote file (#1670)
flyteorg/flytekitGitHub
06/13/2023, 4:07 PMGitHub
06/13/2023, 4:27 PM<https://github.com/flyteorg/flytepropeller/tree/master|master>
by EngHabu
<https://github.com/flyteorg/flytepropeller/commit/89dfd5b3bb694d9a19108aa4c8a6d10a18fd3a32|89dfd5b3>
- Use GetExecutionData instead (#573)
flyteorg/flytepropellerGitHub
06/13/2023, 4:57 PM<https://github.com/flyteorg/flyte/tree/master|master>
by jeevb
<https://github.com/flyteorg/flyte/commit/47fea41ce2f6783217a49f2191f442eda16ce646|47fea41c>
- enable actions for RC branches (#3778)
flyteorg/flyteGitHub
06/13/2023, 5:10 PMGitHub
06/13/2023, 5:12 PMtimestamp >= "2016-11-29T23:00:00Z"
timestamp <= "2016-11-29T23:30:00Z"
This PR adds the pod start and finish time in this format to the variables available for templating logging links.
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☑︎ Code documentation added: flyteorg/flytesnacks#998
☑︎ Any pending items have an associated Issue
Complete description
NA
Tracking Issue
NA
Follow-up issue
NA
flyteorg/flyteplugins
✅ All checks have passed
7/7 successful checksGitHub
06/13/2023, 5:21 PMGitHub
06/13/2023, 6:20 PMflyte-sandbox-http
-> flyte-binary-http
Check all the applicable boxes
☐ I updated the documentation accordingly.
☐ All new and existing tests passed.
☐ All commits are signed-off.
Screenshots
Note to reviewers
flyteorg/flyte
GitHub Actions: build-and-push-sandbox-bundled-image
GitHub Actions: sandbox-bundled-functional-tests
✅ 10 other checks have passed
10/12 successful checksGitHub
06/13/2023, 7:01 PMGitHub
06/13/2023, 8:32 PM<https://github.com/flyteorg/flyte/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flyte/commit/1ae254f8683699b68ecddc89d775fc5d39cc3d84|1ae254f8>
- Update agent endpoint in the single binary (#3779)
flyteorg/flyteGitHub
06/13/2023, 8:34 PM<https://github.com/flyteorg/flyteplugins/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flyteplugins/commit/53e63bc0291fcc1a1f58a0adb8e963b2710a0777|53e63bc0>
- Feat: Add pod start and finish time in RFC3339 time format to logging link templating variables #minor (#360)
flyteorg/flyteplugins