flyte-org #flyte-github

GitHub

05/10/2023, 4:35 AM

#402 Rename user_identity to execution_identity Pull request opened by ByronHsu TL;DR Rename user_identity to execution_identity Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Tracking Issue flyteorg/flyte#3566 flyteorg/flyteidl ✅ All checks have passed 13/13 successful checks

GitHub

05/10/2023, 6:44 AM

#1629 Fallback to python task if worker is zero for pytorch Pull request opened by ByronHsu TL;DR Fallback to python task if worker is zero for pytorch Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Tracking Issue flyteorg/flyteplugins#348 flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-modin) GitHub Actions: build-plugins (3.8, flytekit-mlflow) GitHub Actions: build-plugins (3.8, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.8, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.8, flytekit-kf-mpi) GitHub Actions: build-plugins (3.8, flytekit-k8s-pod) GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-envd) GitHub Actions: build-plugins (3.8, flytekit-duckdb) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-dask) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: lint GitHub Actions: build (windows-latest, 3.11) GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.11) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: docs GitHub Actions: Docs Warnings ✅ 1 other check has passed 1/30 successful checks

GitHub

05/10/2023, 8:43 AM

#3659 [BUG] Flyte-binary Helm misconfigures flyte-binary pod config when auth.externalAuthServer: true, and auth.internal.clientId is undefined in values.yaml Issue created by PudgyPigeon Related issues #3660 #3661 Describe the bug When enabling external authorization servers for the Flyte-binary deployment, if

auth.externalAuthServer = true

in the values.yaml file, and if

auth.internal.clientId

is undefined (defaults to

flytepropeller

), the flyte-binary pod will query the external authorization server with the

flytepropeller

parameter as the clientId. This results in failed auth to the server when scheduling workflows/tasks/notebook tasks through pyflyte or the web console. Workflows and tasks will show up as

UNKNOWN

in the web console, and the flyte-binary pod will continuously make requests to the auth server - too many retries(? Have counted potentially 60 within a span of a minute to auth0,). Specifically, within the flyte-binary pod: if you exec into the

flyte

container in the pod at path:

/etc/flyte/config.d/000-core.yaml

or a similar path, the relevant parameters within the admin field aren't set correctly. Or alternatively, add more documentation to clearly outline this behaviour so future users are aware of how these parameters are set - specifically within the context of integrating Auth0. Tldr; Granted, this could be classified as a documentation issue, but perhaps the

internal

key could be renamed to something along the lines of

propeller

dataplane

(not sure if this is the correct terminology) in order to clarify what is being configured with internal. Expected behavior When

auth.enableAuthServer: false

, it might be a good idea to make

internal.clientId

be a required parameter. In addition to that, it might help to indicate to users that this will then correspond to

000-core.yaml

and will be used by the scheduler for workflow pods. The current documentation implies that this only needs to be set for internal use and makes no mention of it being linked to external. One alternative is if

auth.enableAuthserver: false

, the Helm chart template could derive the values for

000-core.yaml

admin.clientId

from a different field in the values.yaml - thirdpartyconfig.flyteclient is one suggestion. Relevant path in the helm chart is

templates/configmap.yaml

Additional context to reproduce

Copy code

## values.yaml
  auth: 
    enabled: true
    enableAuthServer: false
    internal:
#### Note that clientId is not set - will default to flytepropeller which is unwanted behaviour
      clientSecret: <client secret here>  #
      clientSecretHash: <client hash>
    oidc:
      baseUrl: <external auth server>
      clientId: <clientId>
      clientSecret: <client secret>
    flyteClient:
      clientId: <clientId>
      redirectUri: <http://localhost:53593/callback>
      scopes:
        - read:client_grants
      audience: <audience of your api>
    authorizedUris:
      - put the necessary uris here

Copy code

## in flyte-binary pod in K8S - /etc/flyte/config.d/000-core.yaml
admin:
  clientId: flytepropeller ### This would ideally be the clientId for auth server. Can be set with internal.clientId but unclear as to how to affect this specific parameter with current docs
  endpoint: localhost:8089
  insecure: true
catalog-cache:  
  endpoint: localhost:8081
  insecure: true
  type: datacatalog
cluster_resources:
  standaloneDeployment: false
  templatePath: /etc/flyte/cluster-resource-templates
logger:
  show-source: true
  level: 1
propeller:
  create-flyteworkflow-crd: true
webhook:
  certDir: /var/run/flyte/certs
  localCert: true
  secretName: flyte-backend-flyte-binary-webhook-secret
  serviceName: flyte-backend-flyte-binary-webhook
  servicePort: 443

Copy code

## /etc/flyte/config.d/004-auth.yaml

data: 
  auth:
    userAuth:
      openId:
        clientId: AUTH0_CLIENT_ID
        baseUrl: AUTH0_BASE_URL
        scopes:
          - profile
          - openid
          - offline_access
    appAuth:  
      authServerType: External
      externalAuthServer:
        baseUrl: AUTH0_BASE_URL
        metadataUrl: .well-known/openid-configuration
        allowedAudience: AUTH0_AUDIENCE
      thirdPartyConfig:
        flyteClient:
          clientId: AUTH0_CLIENT_ID
          redirectUri: <http://localhost:53593/callback>
          audience: AUTH0_AUDIENCE
          scopes:
            - read:client_grants
    authorizedUris:
      - <http://flyteadmin:80>
      - <http://flyteadmin.flyte.svc.cluster.local:80>
      - <http://flyteadmin.mlops-services.svc.cluster.local:80>
      - <http://HELM_NAME-flyte-binary.flyte.svc.cluster.local:80>
      - <http://HELM_NAME-flyte-binary.flyte.svc:80>
      - <http://HELM_NAME-flyte-binary.flyte:80>
      - <http://HELM_NAME-flyte-binary:80>
      - <http://localhost:8089>
      -  other URIs can go here as well
  server: 
    security:
      secure: false
      useAuth: true

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

05/10/2023, 9:08 AM

#3660 [BUG] Necessary configuration values not exposed in Helm for working Auth0 - Flyte-Binary deployment Issue created by PudgyPigeon Related issues #3661 #3659 Describe the bug With the current Flyte-binary Helm chart, the values.yaml file and chart templates do not allow the end-user to manually configure the settings for

auth.appAuth.authServerType: External

and

auth.appAuth.exernalAuthServer.baseUrl

metadataUrl

allowedAudience

- during a

helm install

helm upgrade --values values.yaml

command. These settings are necessary for a working auth0 integration. We may potentially be misguided about the procedure, but we found that we had to retrieve the configmap.yaml with kubectl; parse some fields with

yq

and `sed`; and then reapply with

kubectl

and then do

kubectl rollout restart

in order to get auth working on our EKS cluster with this flyte-binary chart. It would be highly appreciated if these values were exposed within the helm/template/values flows for easier end-user configuration. Expected behavior Flyte-binary exposes these configuration values:

Copy code

appAuth:  
      authServerType: External
      externalAuthServer:
        baseUrl: AUTH0_BASE_URL
        metadataUrl: .well-known/openid-configuration
        allowedAudience: AUTH0_AUDIENCE
      thirdPartyConfig:
        flyteClient:
          clientId: AUTH0_CLIENT_ID
          redirectUri: <http://localhost:53593/callback>
          audience: AUTH0_AUDIENCE
          scopes:
            - read:client_grants

within

values.yaml

as well as the

templates/configmap.yaml # 004-auth.yaml

Addendum: would also be nice to expose
server.security.secure
as well for easy deployments Necessary to set these values for working auth0 deployment at the moment. Additional context to reproduce Apply this values.yaml block to helm deployment

Copy code

auth: 
    enabled: true
    enableAuthServer: false
    internal:
      clientId: <>
      clientSecret: <>
      clientSecretHash: <>
    oidc:
      baseUrl: <>
      clientId: <>
      clientSecret: <> 
    flyteClient:
      clientId: <>
      redirectUri: <http://localhost:53593/callback>
      scopes:
        - read:client_grants
      audience: <>
    authorizedUris:
      - authorizedURIs go here

Results in failed deployment when using Auth0 The following block is the Rrequired 004-auth.yaml to pipe into flyte-binary in order to make auth0 work. Filling in the provided Helm values without changing the template results in failed deployment with CrashLoopBackOff due to misconfiguration.

Copy code

data: |
  auth:
    userAuth:
      openId:
        clientId: AUTH0_CLIENT_ID
        baseUrl: AUTH0_BASE_URL
        scopes:
          - profile
          - openid
          - offline_access
    appAuth:  
      authServerType: External
      externalAuthServer:
        baseUrl: AUTH0_BASE_URL
        metadataUrl: .well-known/openid-configuration
        allowedAudience: AUTH0_AUDIENCE
      thirdPartyConfig:
        flyteClient:
          clientId: AUTH0_CLIENT_ID
          redirectUri: <http://localhost:53593/callback>
          audience: AUTH0_AUDIENCE
          scopes:
            - read:client_grants
    authorizedUris:
      - AUTH0_AUDIENCE
      - LOADBALANCER_URL
      - AUTH0_BASE_URL
      - <http://LOADBALANCER_GRPC_URL.svc.aiq-index.com>
      - <https://LOADBALANCER_GRPC_URL.svc.aiq-index.com>
      - <http://flyteadmin:80>
      - <http://flyteadmin.flyte.svc.cluster.local:80>
      - <http://flyteadmin.mlops-services.svc.cluster.local:80>
      - <http://HELM_NAME-flyte-binary.flyte.svc.cluster.local:80>
      - <http://HELM_NAME-flyte-binary.flyte.svc:80>
      - <http://HELM_NAME-flyte-binary.flyte:80>
      - <http://HELM_NAME-flyte-binary:80>
      - <http://localhost:8089>
  server: 
    security:
      secure: false
      useAuth: true

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

05/10/2023, 9:36 AM

#3661 [BUG] Flytekit/Pyflyte not configured to handle Auth0 ClientCredentials Flow/Audience Parameter Issue created by PudgyPigeon Related issues #3660 #3659 Describe the bug Pyflyte/Flytekit does not pass in the appropriate

audience

parameter for the ClientSecret/ClientCredentials authorization flow w/ Auth0. The

ClientConfig

PlatformConfig

ClientCredentialsAuthenticator

classes, the

get_authenticator

function, and the

token_client.get_token

method are not currently configured to accept and pass around the required parameter, and the final get_token method never receives it as an input arg, nor does it pass it into the POST request when requesting a token. This results in a failed auth request as Auth0 requires an audience parameter in the body of the request. Expected behavior Allow the end-user to set the "audience" parameter either in

<flyteconsoleurl>/config/v1/flyte_client

or the

config.yaml

file, and have the pyflyte/flytekit package use this in its auth request for Auth0 integration. Will link fork with changes soon Additional context to reproduce Use an Auth0 external authorization server and attempt to run a

pyflyte run

command with the following config values:

Copy code

###004-auth.yaml configmap
data: |
  auth:
    userAuth:
      openId:
        clientId: AUTH0_CLIENT_ID
        baseUrl: AUTH0_BASE_URL
        scopes:
          - profile
          - openid
          - offline_access
    appAuth:  
      authServerType: External
      externalAuthServer:
        baseUrl: AUTH0_BASE_URL
        metadataUrl: .well-known/openid-configuration
        allowedAudience: AUTH0_AUDIENCE
      thirdPartyConfig:
        flyteClient:
          clientId: AUTH0_CLIENT_ID
          redirectUri: <http://localhost:53593/callback>
          audience: AUTH0_AUDIENCE
          scopes:
            - read:client_grants
    authorizedUris:
      - uris
  server: 
    security:
      secure: false
      useAuth: true

Copy code

config.yaml
admin:
  # For GRPC Endpoints you might want to use dns:///flyte.example.com
  #endpoint: dns:///localhost:8089
  endpoint: dns:///<endpointhere>
  authType: ClientSecret
  clientId: <>
  clientSecretLocation: <>
  scopes: 
  - read:client_grants
  #- offline    --------------------- set offline and all if desired, but not strictly necessary
  #- all
  audience: <> --------------------- deriving it from here is nice, but not strictly necessary as config/v1/flyte_client exists

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

05/10/2023, 11:54 AM

#3663 [BUG] Sending cloud events from flyteadmin not documented properly for GCP and results in errors Issue created by fg91 Describe the bug How the cloud events integration of flyteadmin can be configured is documented here. For GCP, for instance, it says:

Copy code

cloud_events.yaml: |
           cloudEvents:
             enable: true
             gcp:
               region: us-east-2
             eventsPublisher:
               eventTypes:
               - all # or node, task, workflow
               topicName: my-topic
             type: gcp

When including such a config in the helm values file, the following error is raised in flyteadmin:

Copy code

{"json":{},"level":"fatal","msg":"caught panic: project id is required [goroutine 1 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\<http://ngithub.com/flyteorg/flyteadmin/pkg/rpc/adminservice.NewAdminServer.func1()|ngithub.com/flyteorg/flyteadmin/pkg/rpc/adminservice.NewAdminServer.func1()>\n\t/go/src/github.com/flyteorg/flyteadmin/pkg/rpc/adminservice/base.go:74 +0x88\npanic({0x2273960, 0xc0013f1dd0})\n\t/usr/local/go/src/runtime/panic.go:838 +0x207\<http://ngithub.com/flyteorg/flyteadmin/pkg/async/cloudevent.NewCloudEventsPublisher(|ngithub.com/flyteorg/flyteadmin/pkg/async/cloudevent.NewCloudEventsPublisher(>{0x2c04440, 0xc000128000}, {0x1, {0xc00051b4e8, 0x3}, {{0xc00051b470, 0xb}}, {{0x0, 0x0}}, {{0x0, ...}, ...}, ...}, ...)\n\t/go/src/github.com/flyteorg/flyteadmin/pkg/async/cloudevent/factory.go:61 +0x905\<http://ngithub.com/flyteorg/flyteadmin/pkg/rpc/adminservice.NewAdminServer({0x2c04440|ngithub.com/flyteorg/flyteadmin/pkg/rpc/adminservice.NewAdminServer({0x2c04440>?, 0xc000128000}, 0xc0006a3b80, {0x2c0cb60, 0xc00058c6c0}, {0x0, 0x0}, {0x0, 0x0}, 0xc000b563c0, ...)\n\t/go/src/github.com/flyteorg/flyteadmin/pkg/rpc/adminservice/base.go:103 +0x845\<http://ngithub.com/flyteorg/flyteadmin/pkg/|ngithub.com/flyteorg/flyteadmin/pkg/>

When setting the

projectId

instead of the region as done here, this error disappears. However, the resulting yaml file in the flyteadmin configmap still contains aws config:

Copy code

❯ k -n flyte get configmaps flyte-admin-base-config -o yaml
apiVersion: v1
data:
  cloud_events.yaml: "cloudEvents: \n  aws:\n    region: us-east-2\n  enable: true\n
    \ eventsPublisher:\n    eventTypes:\n    - all\n    topicName: <my-topic>\n
    \ gcp:\n    projectId: <my-project>\n  type: gcp\n"

Despite the unintended aws config, the published events do make it to GCP pub/sub and can be pulled in the cloud console. However, even though the messages can successfully be pulled from the pub sub topic/subscription, flyteadmin always shows error logs:

Copy code

{"json":{"exec_id":"b862f22a535442bda2e4","node":"n0"},"level":"error","msg":"Failed to publish a message with key [flyteidl.admin.TaskExecutionEventRequest] and message [Context Attributes,\n  specversion: 1.0\n  type: com.flyte.resource.flyteidl.admin.TaskExecutionEventRequest\n   ... ] and error: context canceled","ts":"2023-05-10T11:46:42Z"}
{"json":{"exec_id":"b862f22a535442bda2e4","node":"n0"},"level":"error","msg":"Failed to send message [event:\u003ctask_id:\u003cresource_type:TASK ... ] with error: context canceled","ts":"2023-05-10T11:46:42Z"}

Expected behavior • The documentation should not say to add the region for GCP but the projedId. • When configuring GCP, the resulting configmap should not contain AWS config. • Flyteadmin should not show errors even though the published events make it to pub/sub and can be pulled from there. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

05/10/2023, 2:02 PM

#3664 [Docs] Logging links `messageFormat` makes flytepropeller fail Issue created by fg91 Description From the docs explaining how to configure logging:

Copy code

task_logs:
  plugins:
    logs:
      templates:
        - displayName: <name-to-show>
          templateUris:
            - ...
          messageFormat: "json" # "unknown" | "csv" | "json"

However, if one sets a

messageFormat

flytepropeller

crashes with this error message:

Copy code

Use "flyte-propeller [command] --help" for more information about a command.

time="2023-05-10T13:44:33Z" level=error msg="\n\n\n1 error(s) decoding:\n\n* cannot parse 'templates[0].messageFormat' as int: strconv.ParseInt: parsing \"json\": invalid syntax"

Without

messageFormat

everything works. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

05/10/2023, 2:26 PM

1 new commit pushed to

<https://github.com/flyteorg/flyteidl/tree/master|master>

by EngHabu

<https://github.com/flyteorg/flyteidl/commit/5e89823156a3dd616f57d8c9ac7495bf15b8ede6|5e898231>

- Rename user_identity to execution_identity (#402) flyteorg/flyteidl

GitHub

05/10/2023, 2:55 PM

#3665 [BUG] Flytekit 1.5.0 uses far more RAM than flytekit 1.4.2 Issue created by ossareh Describe the bug I've documented this in slack: https://flyte-org.slack.com/archives/CREL4QVAQ/p1683729924011689 Duplicated here: For the same set of inputs with the only difference between: • flytekit version • schema change (container tag + k8s resource requests) Flytekit 1.5.0 uses FAR more RAM. Based on the changelog my best guess is that s3fs is using memory where the older mechanism in 1.4.2 was using disk. This seems like a bad default when considering the cost basis of each resource. Expected behavior Minimal to no difference in memory pressure for the same workloads when switching between flytekit versions. Additional context to reproduce Fetch same data (in our case ~12Gb) using flytekit 1.4.2 and flytekit 1.5.0 and observe system memory usage. Screenshots Flytekit 1.4.2

Screenshot 2023-05-10 at 9 33 29 AM▾

Flytekit 1.5.0

Screenshot 2023-05-10 at 9 33 18 AM▾

Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

05/10/2023, 3:05 PM

Release - v1.5.5 New release published by flyte-bot Changelog • 5e89823 Rename user_identity to execution_identity (#402) flyteorg/flyteidl

GitHub

05/10/2023, 4:40 PM

#1628 Add audience Pull request opened by wild-endeavor on <!date^1683684155^{date_short}|2023-05-10T02:02:35Z> TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit Codecov: 0.00% of diff hit (target 71.22%) Codecov: 71.21% (-0.01%) compared to e44b802 ✅ 28 other checks have passed 28/30 successful checks

GitHub

05/10/2023, 6:26 PM

#3667 [BUG] ShellTasks return promises when called inside a task Issue created by brian-odonovan Unsure if this is a bug but the expected behavior and functionality feels counterintuitive. If I write a simple ShellTask the produces a FLyteFile I am unable to invoke that ShellTask in a task and do any computation or even return the FlyteFile given that it is a promise:

Copy code

from flytekit import kwtypes, task
from flytekit.extras.tasks.shell import OutputLocation, ShellTask
from flytekit.types.file import FlyteFile


st = ShellTask(
    name="shell_task",
    debug=True,
    script="""
    touch {inputs.filename}
    """,
    inputs=kwtypes(filename=str),
    output_locs=[OutputLocation(var="outfile", var_type=FlyteFile, location="{inputs.filename}")],
)


@task 
def t1_task( ) -> FlyteFile:
    outfile = st(filename='test.txt')

    return outfile


if __name__ == "__main__":
   
    print(f"let's run a shelltask and return a FlyteFile")
    outfile = t1_task()

flyteorg/flyte

GitHub

05/10/2023, 8:21 PM

#560 Add migration to turn `parent_id` column into `bigint` only if necess… Pull request opened by wild-endeavor Please see #561 instead. This was closed because the branch name was the same as the tag we wanted to create. flyteorg/flyteadmin GitHub Actions: Goreleaser / Goreleaser Codecov: 58.48% (-0.38%) compared to eb695b1 Codecov: 12.06% of diff hit (target 58.85%) DCO: DCO ✅ 9 other checks have passed 9/13 successful checks

GitHub

05/10/2023, 8:26 PM

#754 fix: content panel captures focus Pull request opened by ursucarina TL;DR Prevents the content panel from capturing focus away from other modals Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description added disableEnforceFocus to DetailsPanel Tracking Issue NA flyteorg/flyteconsole ✅ All checks have passed 2/2 successful checks

GitHub

05/10/2023, 9:06 PM

#736 chore: progress Pull request opened by ursucarina on <!date^1680107874^{date_short}|2023-03-29T16:37:54Z> Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteconsole GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release GitHub Actions: unit_tests_with_coverage ✅ 5 other checks have passed 5/9 successful checks

GitHub

05/10/2023, 9:07 PM

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by ursucarina

<https://github.com/flyteorg/flyteconsole/commit/5caa0abc12acad8e12acd58f039c97ff0cd9e3e7|5caa0abc>

- fix: content panel captures focus (#754) flyteorg/flyteconsole

GitHub

05/10/2023, 9:39 PM

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by jsonporter

<https://github.com/flyteorg/flyteconsole/commit/29babaa4d9a814f130b055da8bdc2ebdd101a667|29babaa4>

- feat: runtime metrics UI (#747) flyteorg/flyteconsole

GitHub

05/10/2023, 9:54 PM

Release - v1.7.0 New release published by flyte-bot 1.7.0 (2023-05-10) Bug Fixes • content panel captures focus (#754) (5caa0ab) Features • runtime metrics UI (#747) (29babaa) flyteorg/flyteconsole

GitHub

05/10/2023, 10:03 PM

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by ursucarina

<https://github.com/flyteorg/flyteconsole/commit/a7d7e1c2c351fc6d99106f9a2b2a1d0c64f639e2|a7d7e1c2>

- chore: remove disable_auth from docs (#753) flyteorg/flyteconsole

GitHub

05/10/2023, 10:25 PM

#755 fix: missing dependency Pull request opened by ursucarina TL;DR Fixes improperly added dependency Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Traverse was added to the workspace package.json instead of the console package Tracking Issue NA flyteorg/flyteconsole GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release ✅ 6 other checks have passed 6/9 successful checks

GitHub

05/10/2023, 10:29 PM

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by ursucarina

<https://github.com/flyteorg/flyteconsole/commit/d985dfd158b513f057dc2ae7a9f20a387d716f7f|d985dfd1>

- fix: missing dependency (#755) flyteorg/flyteconsole

GitHub

05/10/2023, 11:07 PM

#3668 V1.5 patch 1 Pull request opened by wild-endeavor Tracking issue Describe your changes Check all the applicable boxes ☐ I updated the documentation accordingly. ☐ All new and existing tests passed. ☐ All commits are signed-off. Screenshots Note to reviewers flyteorg/flyte ✅ All checks have passed 2/2 successful checks

GitHub

05/11/2023, 3:50 AM

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by peridotml

<https://github.com/flyteorg/flytekit/commit/26d1f290ab9e800acfd9af08595a82c3b9f962b6|26d1f290>

- [Issue-3617] Enables FlyteFiles, FlyteDirectors, and StructuredDatasets inputs in papermill plugin (#1612) flyteorg/flytekit

GitHub

05/11/2023, 4:32 AM

#1630 Multi arch imageSpec Pull request opened by pingsutw TL;DR Blocked by tensorchord/envd#1582 (review) Add a new property (platform) in imageSpec to build different arch image. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue NA Follow-up issue NA flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-kf-mpi) GitHub Actions: build-plugins (3.8, flytekit-k8s-pod) GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-envd) GitHub Actions: build-plugins (3.8, flytekit-duckdb) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-dask) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: lint GitHub Actions: build (windows-latest, 3.11) GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.11) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: Docs Warnings GitHub Actions: docs ✅ 2 other checks have passed 2/27 successful checks

GitHub

05/11/2023, 4:54 AM

#561 V1.1.88.1 patch Pull request opened by wild-endeavor TL;DR This is a PR to pull in the column fixes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteadmin GitHub Actions: Lint / Run Lint GitHub Actions: Check Go Generate / Go Generate GitHub Actions: Docker Build Images / Build Docker Image GitHub Actions: Unit Tests / Run Unit Test DCO: DCO ✅ 1 other check has passed 1/6 successful checks

GitHub

05/11/2023, 7:32 AM

#1631 Improve task decorator type hints with overload Pull request opened by ringohoffman TL;DR Functions decorated by

flytekit.task

are not hinted as

PythonFunctionTask

as a result of its return typing, so there are spurious type errors when attempting to register such tasks on

FlyteRemote

objects. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Here you can see an explanation of the problem and the fix:

Copy code

import flytekit
import flytekit.configuration as flyte_config
import flytekit.remote


@flytekit.task
def my_task1() -> int:
    return 0


@flytekit.task()
def my_task2() -> int:
    return 0


@flytekit.task(task_config=5)
def my_task3() -> int:
    return 0


my_task4 = flytekit.task(lambda: 0)

remote = flytekit.remote.FlyteRemote(...)  # type: ignore
serialization_settings = flyte_config.SerializationSettings(...)  # type: ignore

# before
reveal_type(my_task1)  # Type of "my_task1" is "((...) -> Any) | PythonFunctionTask"
remote.register_task(my_task1, serialization_settings)  # error: Argument of type "((...) -> Any) | PythonFunctionTask" cannot be assigned to parameter "entity" of type "PythonTask" in function "register_task"

reveal_type(my_task2)  # Type of "my_task2" is "() -> int"
remote.register_task(my_task2, serialization_settings)  # error: Argument of type "() -> int" cannot be assigned to parameter "entity" of type "PythonTask" in function "register_task"

reveal_type(my_task3)  # Type of "my_task3" is "() -> int"
remote.register_task(my_task3, serialization_settings)  # error: Argument of type "() -> int" cannot be assigned to parameter "entity" of type "PythonTask" in function "register_task"

reveal_type(my_task4)  # Type of "my_task4" is "((...) -> Any) | PythonFunctionTask"
remote.register_task(my_task4, serialization_settings)  # Argument of type "((...) -> Any) | PythonFunctionTask" cannot be assigned to parameter "entity" of type "PythonTask" in function "register_task"


# after
reveal_type(my_task1)  # Type of "my_task1" is "PythonFunctionTask"
remote.register_task(my_task1, serialization_settings)  # OK

reveal_type(my_task2)  # Type of "my_task2" is "PythonFunctionTask[T@task]"
remote.register_task(my_task2, serialization_settings)  # OK

reveal_type(my_task3)  # Type of "my_task3" is "PythonFunctionTask[int]"
remote.register_task(my_task3, serialization_settings)  # OK

reveal_type(my_task4)  # Type of "my_task4" is "PythonFunctionTask"
remote.register_task(my_task4, serialization_settings)  # OK

I accomplished this fix by using

typing.overload

to hint a different return type based on whether

_task_function

is passed to

flytekit.task

or not. If nothing is passed as in the case of

my_task2

my_task3

, we hint the return type as

Callable[[Callable[..., Any]], PythonFunctionTask[T]]

... that is a callable that accepts a single argument that is any function and returns a

PythonFunctionTask

. If something is passed for

_task_function

as in

my_task1

my_task4

, we hint the return type as

PythonFunctionTask[T]

. Tracking Issue N/A Follow-up issue N/A flyteorg/flytekit Codecov: 71.20% (-0.02%) compared to e44b802 Codecov: 11.11% of diff hit (target 71.22%) ✅ 28 other checks have passed 28/30 successful checks

GitHub

05/11/2023, 7:43 AM

#3669 Fix indentation in flyte-binary helm chart template Pull request opened by jeevb flyteorg/flyte GitHub Actions: build-and-push-sandbox-bundled-image ✅ 11 other checks have passed 11/12 successful checks

GitHub

05/11/2023, 8:11 AM

#1632 Allow annotated FlyteFile as task input argument Pull request opened by AdrianoKF TL;DR This PR allows annotated FlyteFiles to be used as input arguments of a task, e.g., when calculating a custom hash using

Annotated[FlyteFile, HashMethod(...)]

. See the discussion on Slack for additional details. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Using an annotated FlyteFile type as an input to a task was previously impossible due to an exception being raised in

FlyteFilePathTransformer.to_python_value

. This MR applies the fix previously used in

FlyteFilePathTransformer.to_literal

to permit using annotated FlyteFiles as either inputs or outputs of a task (see MR #1544), by unwrapping the origin type of the annotated type. Tracking Issue flyteorg/flyte#3424 Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 2/2 successful checks

GitHub

05/11/2023, 9:28 AM

#560 DO NOT MERGE: checking github ci Pull request opened by hamersaw on <!date^1683796903^{date_short}|2023-05-11T09:21:43Z> Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytepropeller GitHub Actions: End2End Test / End to End tests GitHub Actions: Integration Test / Integration tests GitHub Actions: Analyze (go) ✅ 6 other checks have passed 6/9 successful checks

GitHub

05/11/2023, 1:57 PM

#3671 [Core feature] Support more k8s auth providers Issue created by honnix Motivation: Why do you think this is important? Currently only native k8s token based auth is supported, in both flyteadmin and various plugins. For example in flyteadmin, in array plugin, and in ray plugin. It would be great to support for example GKE, and EKS (I assume it provides auth similarly as GKE). Goal: What should the final outcome look like, ideally? Cloud providers' native k8s auth mechanism can be used to avoid using static token that is less secure. Describe alternatives you've considered Keep using static token and cert. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte