Release - Flyte v1.3.0-b9 milestone release New release published by flyte-bot v1.3.0-b9 CI changes. flyteorg/flyte

Deployment to github-pages by flyte-bot flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyteadmin/tree/master|master>

by katrogan

<https://github.com/flyteorg/flyteadmin/commit/d6bc85d7e8295b14d57b4a13681bbeab31dbf2e3|d6bc85d7>

- Add user name to UserInfoResponse header (#507) flyteorg/flyteadmin

#66 Build multi-arch docker images Pull request opened by prabhu43 Signed-off-by: Prabhu Jayakumar j.prabhu91@gmail.com Fixes flyteorg/flyte#2935 flyteorg/flytetools ✅ All checks have passed 2/2 successful checks

#2935 [Housekeeping] Produce multi-arch images in CI/CD pipelines Issue created by hajapy Describe the issue Update the build pipelines to also produce arm64 images. At a glance, I think these workflows would need to be updated: https://github.com/flyteorg/flytetools/blob/master/.github/workflows/docker_build.yml#L34 https://github.com/flyteorg/flytetools/blob/master/.github/workflows/publish.yml#L36 https://github.com/flyteorg/flyte/blob/master/.github/workflows/create_release.yml I would suggest to look into using the official docker actions if possible: https://github.com/docker/setup-buildx-action https://github.com/docker/build-push-action What if we do not do this? Unable to use official flyte images on arm64 instances. Users would need to build their own images of all the flyte control plane components. Related component(s) No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flyte/commit/162c8dbe28c53d8b92368f0da55a7d7e879003fb|162c8dbe>

- Add conditional around the admin secrets (#3213) flyteorg/flyte

#3141 [BUG] Don't require flyte-admin-secrets for standalone cluster resource sync Issue created by andrewwdye on <!date^1670461457^{date_short}|2022-12-08T01:04:17Z> Describe the bug 18be8cc#diff-b87b59c495047ee004f0743b6970f8bbd98c7e26cf38c9e916927eb0c4edbdd1R42 added a default mount for

flyte-admin-secrets

in support of multi-cluster resource sync. These secrets should not be required when running cluster resource sync in standalone mode. Expected behavior

flyte-admin-secrets

are not required when running cluster resource sync in standalone mode Additional context to reproduce Deploy cluster resource sync with

standaloneDeployment: true

and without secrets defined. Container will fail with

Unable to attach or mount volumes: unmounted volumes=[flyte-admin-secrets]

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

Release - v1.1.68 New release published by flyte-bot Changelog • cb1224b Publish multi-architecture image (#510) flyteorg/flyteadmin

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytekit/commit/1311ea47242e4f32a9dc0769b6a51554ab41ee80|1311ea47>

- Read structured dataset from a folder (#1406) flyteorg/flytekit

Release - v1.1.69 New release published by flyte-bot Changelog • d6bc85d Add user name to UserInfoResponse header (#507) flyteorg/flyteadmin

#515 Add TensorFlow to loader.go Pull request opened by pingsutw Signed-off-by: Kevin Su pingsutw@gmail.com TL;DR Add

TensorFlow

to loader.go, so we can load

TensorFlow

plugin while starting the propeller. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue https://flyte-org.slack.com/archives/C045P7GBGBU/p1673314641468229 Follow-up issue NA flyteorg/flytepropeller

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by samhita-alla

<https://github.com/flyteorg/flytesnacks/commit/b61d89d8e5c4b4e7ce376d3fc8f3a97e76af0cf4|b61d89d8>

- Fix walkthrough command for pima diabetes classification example (#940) flyteorg/flytesnacks

Release - flytesnacks v0.3.160 New release published by flyte-bot Changelog • b61d89d Fix walkthrough command for pima diabetes classification example (#940) flyteorg/flytesnacks

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by honnix

<https://github.com/flyteorg/flytekit-java/commit/c5640872e4c605929a7163cdc611821282ab525b|c5640872>

- Update maven central URL (#163) flyteorg/flytekit-java

#3224 [UI Feature] launch LaunchPlan directly from LaunchPlan list/details Issue created by flixr Motivation: Why do you think this is important? Took us a while to find that we could launch a LaunchPlan from the UI via workflows -> launch -> select launchplan. Instead I expected to be able to launch a LaunchPlan from where it is displayed. Goal: What should the final outcome look like, ideally? Add a launch button to LauchPlans directly, making this a shortcut and more discoverable. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flytepropeller/commit/8cbb05a0fd43feae35f3611042769c6a7b181728|8cbb05a0>

- Add TensorFlow to loader.go (#515) flyteorg/flytepropeller

Release - v1.1.61 New release published by flyte-bot Changelog • 8cbb05a Add TensorFlow to loader.go (#515) flyteorg/flytepropeller

#665 Bug: Archived launchplans are not currently being filtered in the launch plan list Issue created by jsonporter Users have requested support for archiving launch plans. Acceptance Criteria: • Users can archive LP from search UI • Archived LPs are excluded from LP search UI • Filter text is 'Include Archived' • Filter is inclusive (like task/workflow toggle implemented in 661). • Archived LP are not included in drop-down on launch form UI unless that LP is the current selected LP (eg, relaunching an execution) flyteorg/flyteconsole

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flytesnacks/commit/9e28039d4b8ddf5c5f951859740af05c4bf6e26f|9e28039d>

- Document IgnoreOutputs exception (#938) flyteorg/flytesnacks

#3225 [Core feature] Raw claims should be passed along as header metadata Issue created by katrogan Motivation: Why do you think this is important? It would be useful for the flyteadmin authorization server /me endpoint handler to return raw claims from the OIDC provider in addition to the standard user info Goal: What should the final outcome look like, ideally? Raw claims should be propagated from the flyteadmin authorization server as metadata headers Describe alternatives you've considered N/A Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#357 Add raw claims to user info response Pull request opened by katrogan on <!date^1673379265^{date_short}|2023-01-10T19:34:25Z> Signed-off-by: Katrina Rogan katroganGH@gmail.com TL;DR Return all claims from OIDC user info response Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description NA Tracking Issue flyteorg/flyte#3225 Follow-up issue NA flyteorg/flyteidl ✅ All checks have passed 13/13 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteplugins/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyteplugins/commit/109224c2a0e65782fee53336b46cbe4bd0d2d189|109224c2>

- added raw-container to registered task types (#305) flyteorg/flyteplugins

Release - v1.0.29 New release published by flyte-bot Changelog • 109224c added raw-container to registered task types (#305) flyteorg/flyteplugins

1 new commit pushed to

<https://github.com/flyteorg/flyteidl/tree/master|master>

by katrogan

<https://github.com/flyteorg/flyteidl/commit/9fbac98b2d173fe1b30f18ac0487ff35b9627e3b|9fbac98b>

- Add raw claims to user info response (#357) flyteorg/flyteidl

Release - v1.3.3 New release published by flyte-bot Changelog • 9fbac98 Add raw claims to user info response (#357) flyteorg/flyteidl

#511 Forward all claims in userinfo response Pull request opened by katrogan Signed-off-by: Katrina Rogan katroganGH@gmail.com TL;DR Return all claims from OIDC user info server response as custom metadata headers. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Complete description NA Tracking Issue fixes flyteorg/flyte#3225 Follow-up issue NA flyteorg/flyteadmin GitHub Actions: Unit Tests / Run Unit Test GitHub Actions: Lint / Run Lint GitHub Actions: Docker Build Images / Build Docker Image GitHub Actions: Check Go Generate / Go Generate ✅ 2 other checks have passed 2/6 successful checks

#3226 [Core feature] map_task should be able to handle a partitioned StructuredDataset Issue created by cosmicBboy Motivation: Why do you think this is important? As a data practitioner, I should be able to apply a

map_task

to a partitioned StructuredDataset automatically so that I can process the partitions in an embarrassingly parallel fashion without too much extra code. Goal: What should the final outcome look like, ideally? Suppose we have a task that produces a

StructuredDataset

@task
def make_df() -> StructuredDataset:
    df = pd.DataFrame.from_records([
        {
            "id": i,
            "partition": (i % 10) + 1,
            "name": "".join(
                random.choices(string.ascii_uppercase + string.digits, k=10)
            )
        }
        for i in range(1000)
    ])
    return StructuredDataset(dataframe=df, partition_col=["partition"])

Ideally, I should be able to do something like this:

@task
def process_df(dataset: StructuredDataset) -> StructuredDataset:
    df = structured_dataset.open(pd.DataFrame).read_partition()  # read the partition
    ... # do stuff

@task
def use_processed_df(dataset: List[StructuredDataset]) -> ...:
    ...

@workflow
def wf() -> StructuredDataset:
    structured_dataset = make_df()
    # where structured_dataset.partitions is a list of unpartitioned StructuredDatasets
    results: List[StructuredDataset] = map_task(process_df)(dataset=structured_dataset.partitions)
    return use_processed_df(dataset=results)

Note that in this example code a few magical things are happening: 1. we pass in

structured_dataset.partitions

into the map task, which indicates that we want to apply

process_df

to each of the partitions defined in

make_df

2. The fact that

map_task(process_df)

returns a

StructuredDataset

implies that using map tasks with structured datasets does an implicit reduction, i.e. the outputs of

map_task(process_df)

are written to the same blob store prefix. Ideally the solution enables processing of

StructuredDataset

without having to manually handle reading in of partitions in the map task, and automatically reduces the results into a

StructuredDataset

without having to explicitly write a coalense/reduction task. Describe alternatives you've considered Users would have to roll their own way of processing partitions of a structured dataset using dynamic tasks. Propose: Link/Inline OR Additional context Slack context: https://flyte-org.slack.com/archives/CP2HDHKE1/p1673380243923279 Related to #3219 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyteadmin/tree/master|master>

by katrogan

<https://github.com/flyteorg/flyteadmin/commit/1ccd59c249e2305185bd7e5cd9340c4f60c6e8bd|1ccd59c2>

- Forward all claims in userinfo response (#511) flyteorg/flyteadmin

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flytekit/commit/581a5c66b6dec1d105f8f655918c405665ceebf6|581a5c66>

- Update default config to work out-of-the-box with flytectl demo (#1384) flyteorg/flytekit

#3116 Update Administrator docs Pull request opened by wild-endeavor Deployment Documentation Restructure and update some deployment docs. Main changes: • Remove all the cloud specific stuff. • Point users to a newly created GH Discussion topic for submitting articles on deployment experiences. • Use the new

flyte-binary

Helm chart. • Remove a lot of the sandbox stuff since the new

flytectl demo

environment is architected differently. • Change instructions to use

flytectl demo

instead of

flytectl sandbox

• Remove

ideal_flow.rst

- this is incomplete. We should eventually offer some basic gh workflow examples rather than just talking about it. Structure Changes Bundling all Flyte configuration under the deployment moniker felt a bit of a stretch. I think renaming it to a broader administrator's guide makes more sense. • Currently the generated component configs are missing, but this will end up changing with the partial mono-repo work anyways. Deployment Updates One of the things we want to do is change the deployment journey. We want to make sure users are led comfortably through the various stages that one might expect of something as complex as Flyte. Basically the steps of the journey should be 1. flytectl demo sandbox 2. a simple cloud (eks/gke) based deployment with nothing tricky - just helm install. users will have to port-forward to see anything. 3. a production ready deployment (ingress, auth, etc) (think stable enough for most companies). 4. a scalable multicluster setup (lots of deployments will never need this level). The middle steps will replace the aws/gcp guides that we have today. These have been completely deleted. From there we should link to a revamped version of the

ideal_flow.rst

file I think. #2993 flyteorg/flyte ✅ All checks have passed 11/11 successful checks