GitHub
01/10/2023, 12:26 AMGitHub
01/10/2023, 12:27 AMGitHub
01/10/2023, 12:38 AM<https://github.com/flyteorg/flyteadmin/tree/master|master>
by katrogan
<https://github.com/flyteorg/flyteadmin/commit/d6bc85d7e8295b14d57b4a13681bbeab31dbf2e3|d6bc85d7>
- Add user name to UserInfoResponse header (#507)
flyteorg/flyteadminGitHub
01/10/2023, 12:42 AMGitHub
01/10/2023, 12:43 AMGitHub
01/10/2023, 12:45 AM<https://github.com/flyteorg/flyte/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flyte/commit/162c8dbe28c53d8b92368f0da55a7d7e879003fb|162c8dbe>
- Add conditional around the admin secrets (#3213)
flyteorg/flyteGitHub
01/10/2023, 12:45 AMflyte-admin-secrets
in support of multi-cluster resource sync. These secrets should not be required when running cluster resource sync in standalone mode.
Expected behavior
flyte-admin-secrets
are not required when running cluster resource sync in standalone mode
Additional context to reproduce
Deploy cluster resource sync with standaloneDeployment: true
and without secrets defined. Container will fail with
Unable to attach or mount volumes: unmounted volumes=[flyte-admin-secrets]
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
01/10/2023, 12:47 AMGitHub
01/10/2023, 1:22 AM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/1311ea47242e4f32a9dc0769b6a51554ab41ee80|1311ea47>
- Read structured dataset from a folder (#1406)
flyteorg/flytekitGitHub
01/10/2023, 1:34 AMGitHub
01/10/2023, 1:47 AMTensorFlow
to loader.go, so we can load TensorFlow
plugin while starting the propeller.
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☐ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
^^^
Tracking Issue
https://flyte-org.slack.com/archives/C045P7GBGBU/p1673314641468229
Follow-up issue
NA
flyteorg/flytepropellerGitHub
01/10/2023, 6:19 AM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by samhita-alla
<https://github.com/flyteorg/flytesnacks/commit/b61d89d8e5c4b4e7ce376d3fc8f3a97e76af0cf4|b61d89d8>
- Fix walkthrough command for pima diabetes classification example (#940)
flyteorg/flytesnacksGitHub
01/10/2023, 6:56 AMGitHub
01/10/2023, 9:54 AM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by honnix
<https://github.com/flyteorg/flytekit-java/commit/c5640872e4c605929a7163cdc611821282ab525b|c5640872>
- Update maven central URL (#163)
flyteorg/flytekit-javaGitHub
01/10/2023, 4:53 PMGitHub
01/10/2023, 5:00 PM<https://github.com/flyteorg/flytepropeller/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flytepropeller/commit/8cbb05a0fd43feae35f3611042769c6a7b181728|8cbb05a0>
- Add TensorFlow to loader.go (#515)
flyteorg/flytepropellerGitHub
01/10/2023, 5:56 PMGitHub
01/10/2023, 6:12 PMGitHub
01/10/2023, 7:18 PM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by cosmicBboy
<https://github.com/flyteorg/flytesnacks/commit/9e28039d4b8ddf5c5f951859740af05c4bf6e26f|9e28039d>
- Document IgnoreOutputs exception (#938)
flyteorg/flytesnacksGitHub
01/10/2023, 9:23 PMGitHub
01/10/2023, 9:24 PMGitHub
01/10/2023, 9:39 PM<https://github.com/flyteorg/flyteplugins/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flyteplugins/commit/109224c2a0e65782fee53336b46cbe4bd0d2d189|109224c2>
- added raw-container to registered task types (#305)
flyteorg/flytepluginsGitHub
01/10/2023, 9:41 PMGitHub
01/10/2023, 9:47 PM<https://github.com/flyteorg/flyteidl/tree/master|master>
by katrogan
<https://github.com/flyteorg/flyteidl/commit/9fbac98b2d173fe1b30f18ac0487ff35b9627e3b|9fbac98b>
- Add raw claims to user info response (#357)
flyteorg/flyteidlGitHub
01/10/2023, 9:54 PMGitHub
01/10/2023, 10:08 PMGitHub
01/10/2023, 10:09 PMmap_task
to a partitioned StructuredDataset automatically so that I can process the partitions in an embarrassingly parallel fashion without too much extra code.
Goal: What should the final outcome look like, ideally?
Suppose we have a task that produces a StructuredDataset
@task
def make_df() -> StructuredDataset:
df = pd.DataFrame.from_records([
{
"id": i,
"partition": (i % 10) + 1,
"name": "".join(
random.choices(string.ascii_uppercase + string.digits, k=10)
)
}
for i in range(1000)
])
return StructuredDataset(dataframe=df, partition_col=["partition"])
Ideally, I should be able to do something like this:
@task
def process_df(dataset: StructuredDataset) -> StructuredDataset:
df = structured_dataset.open(pd.DataFrame).read_partition() # read the partition
... # do stuff
@task
def use_processed_df(dataset: List[StructuredDataset]) -> ...:
...
@workflow
def wf() -> StructuredDataset:
structured_dataset = make_df()
# where structured_dataset.partitions is a list of unpartitioned StructuredDatasets
results: List[StructuredDataset] = map_task(process_df)(dataset=structured_dataset.partitions)
return use_processed_df(dataset=results)
Note that in this example code a few magical things are happening:
1. we pass in structured_dataset.partitions
into the map task, which indicates that we want to apply process_df
to each of the partitions defined in make_df
2. The fact that map_task(process_df)
returns a StructuredDataset
implies that using map tasks with structured datasets does an implicit reduction, i.e. the outputs of map_task(process_df)
are written to the same blob store prefix.
Ideally the solution enables processing of StructuredDataset
without having to manually handle reading in of partitions in the map task, and automatically reduces the results into a StructuredDataset
without having to explicitly write a coalense/reduction task.
Describe alternatives you've considered
Users would have to roll their own way of processing partitions of a structured dataset using dynamic tasks.
Propose: Link/Inline OR Additional context
Slack context: https://flyte-org.slack.com/archives/CP2HDHKE1/p1673380243923279
Related to #3219
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
01/10/2023, 11:00 PM<https://github.com/flyteorg/flyteadmin/tree/master|master>
by katrogan
<https://github.com/flyteorg/flyteadmin/commit/1ccd59c249e2305185bd7e5cd9340c4f60c6e8bd|1ccd59c2>
- Forward all claims in userinfo response (#511)
flyteorg/flyteadminGitHub
01/10/2023, 11:26 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flytekit/commit/581a5c66b6dec1d105f8f655918c405665ceebf6|581a5c66>
- Update default config to work out-of-the-box with flytectl demo (#1384)
flyteorg/flytekitGitHub
01/10/2023, 11:36 PMflyte-binary
Helm chart.
• Remove a lot of the sandbox stuff since the new flytectl demo
environment is architected differently.
• Change instructions to use flytectl demo
instead of flytectl sandbox
• Remove ideal_flow.rst
- this is incomplete. We should eventually offer some basic gh workflow examples rather than just talking about it.
Structure Changes
Bundling all Flyte configuration under the deployment moniker felt a bit of a stretch. I think renaming it to a broader administrator's guide makes more sense.
• Currently the generated component configs are missing, but this will end up changing with the partial mono-repo work anyways.
Deployment Updates
One of the things we want to do is change the deployment journey. We want to make sure users are led comfortably through the various stages that one might expect of something as complex as Flyte.
Basically the steps of the journey should be
1. flytectl demo sandbox
2. a simple cloud (eks/gke) based deployment with nothing tricky - just helm install. users will have to port-forward to see anything.
3. a production ready deployment (ingress, auth, etc) (think stable enough for most companies).
4. a scalable multicluster setup (lots of deployments will never need this level).
The middle steps will replace the aws/gcp guides that we have today. These have been completely deleted.
From there we should link to a revamped version of the ideal_flow.rst
file I think.
#2993
flyteorg/flyte
✅ All checks have passed
11/11 successful checks