Release - flytesnacks v0.3.167 New release published by flyte-bot Changelog • 2903a95 Getting started cleanup (#958) flyteorg/flytesnacks

#1509 Propagate ca_cert_file_path and verify settings to auth client Pull request opened by goyalankit TL;DR Propagate ca_cert_file_path and verify settings to auth client Flytekit throws

CERTIFICATE_VERIFY_FAILED

on auth even if user has specified the certificate path or has set

insecureSkipVerify

to

False

. This change adds the

verify

to requests library according to user configured settings. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (windows-latest, 3.7, -spark2) GitHub Actions: build (windows-latest, 3.7) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.7, -spark2) GitHub Actions: build (ubuntu-latest, 3.7) GitHub Actions: docs GitHub Actions: lint GitHub Actions: build-plugins (3.10, flytekit-vaex) GitHub Actions: build-plugins (3.10, flytekit-sqlalchemy) GitHub Actions: build-plugins (3.10, flytekit-spark) GitHub Actions: build-plugins (3.10, flytekit-snowflake) GitHub Actions: build-plugins (3.10, flytekit-ray) GitHub Actions: build-plugins (3.10, flytekit-polars) GitHub Actions: build-plugins (3.10, flytekit-papermill) GitHub Actions: build-plugins (3.10, flytekit-pandera) GitHub Actions: build-plugins (3.10, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.10, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.10, flytekit-kf-mpi) GitHub Actions: build-plugins (3.10, flytekit-k8s-pod) GitHub Actions: build-plugins (3.10, flytekit-hive) GitHub Actions: build-plugins (3.10, flytekit-dolt) GitHub Actions: build-plugins (3.10, flytekit-deck-standard) GitHub Actions: build-plugins (3.10, flytekit-dbt) GitHub Actions: build-plugins (3.10, flytekit-data-fsspec) GitHub Actions: build-plugins (3.10, flytekit-dask) GitHub Actions: build-plugins (3.10, flytekit-bigquery) ✅ No checks have passed 0/30 successful checks

Release - v1.1.64 New release published by flyte-bot Changelog • 95a4791 Inline input data for execution events (#521) flyteorg/flytepropeller

1 new commit pushed to

<https://github.com/flyteorg/flytectl/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flytectl/commit/5aefa07cbab99770cf3f642f6cd18e63006bb89a|5aefa07c>

- refactor: remove redundant YAML packages (#387) flyteorg/flytectl

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytekit/commit/606e0c4e024c62151d760f8aa7fb06af7986c98e|606e0c4e>

- Use non-root user in default flytekit image (#1417) flyteorg/flytekit

Release - v0.6.30 New release published by flyte-bot Changelog • 5aefa07 refactor: remove redundant YAML packages (#387) flyteorg/flytectl

1 new commit pushed to

<https://github.com/flyteorg/homebrew-tap/tree/main|main>

by flyte-bot

<https://github.com/flyteorg/homebrew-tap/commit/c523cc38aa06be1ef300f0383be44a4041460682|c523cc38>

- Brew formula update for flytectl version v0.6.30 flyteorg/homebrew-tap

#2927 [Core feature] Support mapping `principal` to Kubernetes namespace Issue created by hankfanchiu Motivation: Why do you think this is important? At Stripe, Flyte users will be able to register and then launch workflows on an ad hoc basis, without second party review. Within these workflows, some tasks may access sensitive data. From a security perspective, we must be able to thread the identity of a Flyte user through Flyte Admin and Kubernetes to downstream, internal services or resources. To do so for any task execution, we must be able to run a Kubernetes workload in a Kubernetes namespace that is scoped to the user. Then, within this task that makes the outbound request to an internal service or resource, we would first extract the username/principal from the namespace, and then propagate this identity encoded in an access token. Goal: What should the final outcome look like, ideally? In Flyte Admin's `namespace_mapping` configuration, add support for

{{ principal }}

in the template. Then, an example scenario would be as such, given this Flyte Admin configuration:

namespace_mapping:
  template: "{{ principal }}"

When the user

hankfanchiu

triggers a task execution, the Kubernetes workload would run in the

hankfanchiu

namespace. Describe alternatives you've considered None Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2951 [Core feature] Force-enable Flyte Admin's auth endpoints by config, without actually enabling authentication Issue created by hankfanchiu Motivation: Why do you think this is important? At Stripe, we plan to implement custom, internally consistent authorization, without setting up Flyte's built-in authentication, by: 1. Introducing a Flyte Admin middleware that evaluates some

x-stripe-*

request header containing an access token; 2. Updating every Flyte client to pass this header. We would like to use Flyte Admin's `auth.httpAuthorizationHeader` and `auth.grpcAuthorizationHeader` configurations to customize the request header as some

x-stripe-*

. However, currently, these configurations are only evaluated when the `security.useAuth` configuration is enabled, as seen in the following: • `server.newHTTPServer()` → `auth.GetHTTPRequestCookieToMetadataHandler()` • `server.newGRPCServer()` → `auth.GetAuthenticationCustomMetadataInterceptor()` Goal: What should the final outcome look like, ideally? Introduce a new Flyte Admin configuration that allows the

auth.httpAuthorizationHeader

and

auth.grpcAuthorizationHeader

configurations to be used, without actually enabling the

security.useAuth

configuration. This new configuration could perhaps be named

security.forceUseAuthHeaders

. The default value would be

false

. We could optionally consider that enabling both

security.useAuth

and this new configuration is invalid, causing a panic. Describe alternatives you've considered None Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3337 [BUG] Request failed with status code 429 grpc: received message larger than max when opening Flyte Console Issue created by pradithya Describe the bug Opening workflow execution page is failing with error "Request failed with status code 429 grpc: received message larger than max when opening" Expected behavior Flyte console should shows workflows execution properly Additional context to reproduce The issue seems to happen due to

/api/v1/executions/

API response returning an execution object that has a huge

closure.error.message

payload. Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by andresgomezfrr

<https://github.com/flyteorg/flytekit-java/commit/305495e32dcd458b5bc0a8d15dfc82212dffa26b|305495e3>

- Refactor SdkBindindData factory methods (#193) flyteorg/flytekit-java

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by github-actions[bot]

<https://github.com/flyteorg/flytekit-java/commit/b9694501e3a4929c955a9de289ff7fb38cd06492|b9694501>

- [maven-release-plugin] prepare release 0.4.3 flyteorg/flytekit-java

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by github-actions[bot]

<https://github.com/flyteorg/flytekit-java/commit/b217cd66d56fc44e921f12574798e7243263a191|b217cd66>

- [maven-release-plugin] prepare for next development iteration flyteorg/flytekit-java

Release - 0.4.3 New release published by andresgomezfrr What's Changed • Refactor SdkBindindData factory methods by @narape in #193 Full Changelog: 0.4.2...0.4.3 flyteorg/flytekit-java

#3338 [BUG] - Spark Executors can't access the code from the Driver with Fast Registration Issue created by TheoLacour-Artefact Describe the bug Spark Executors can't access the code from the Spark Driver when new code is packaged with Fast Registration for Spark Tasks that use a

map

function. The Spark Executors can only access the code from the image it pulled. Expected behavior The Spark Executors should use the same code as the Spark Driver - which is the code that was pushed using Fast Registration. Additional context to reproduce 1. Following this exemple : use the code from https://docs.flyte.org/projects/cookbook/en/stable/auto/integrations/kubernetes/k8s_spark/pyspark_pi.html , building a Docker image like this one https://docs.flyte.org/projects/cookbook/en/stable/auto/integrations/kubernetes/k8s_spark/index.html#step-2-environment-setup 2.

pyflyte --pkgs my_code --image my_image --force

3.

flytectl register files --project my_project --domain development --archive flyte-package.tgz --outputLocationPrefix <s3://my_bucket> --k8sServiceAccount my_account --version 0.0.1

4. Go to the Flyte Console, run the workflow : it works as expected, the output of

my_spark

is close to 3.14 5. Then, change the code from the function

f

in step 1: now,

f

always returns 1. 6. I updated my code but I don't need to update my environnement, so I can use Fast Registration :

pyflyte --pkgs my_code --image my_image --force --fast

7.

flytectl register files --project my_project --domain development --archive flyte-package.tgz --outputLocationPrefix <s3://my_bucket> --k8sServiceAccount my_account --version 0.0.2

8. Go to the Flyte Console, run the workflow's new version : the output of

my_spark

is close to 3.14, but I was expecting it to be 4. Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3339 [BUG] Structured Dataset BQ decoder only reads partial data for big datasets (>100MB) Issue created by gitgraghu Describe the bug The Structured Dataset BigQuery decoder reads only partial data for big datasets (>100MB). Currently the read session is not configured with a max stream setting. Due to this, for larger datasets, multiple streams will be used to read the data. However the BQ decoder only reads the first stream as shown in code below:

read_session = client.create_read_session(parent=parent, read_session=requested_session)
    stream = read_session.streams[0]
    reader = client.read_rows(stream.name)
    frames = []
    for message in reader.rows().pages:
        frames.append(message.to_dataframe())
    return pd.concat(frames)

Expected behavior The BQ decoder should be able to read all data for bigger datasets as well. This decoder code needs to be changed to either set the max stream to 1 in the read session or read from all streams as below snippet:

frames = []
   for stream in read_session.streams:
       reader = client.read_rows(stream.name)
       for message in reader.rows().pages:
           frames.append(message.to_dataframe())
   if len(frames) > 0:
       df = pd.concat(frames)
   else:
       schema = pyarrow.ipc.read_schema(
           pyarrow.py_buffer(read_session.arrow_schema.serialized_schema)
       )
       df = schema.empty_table().to_pandas()
   return df

Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyte/commit/d60c9af85a59ebb4c2265f76cb082b992078a309|d60c9af8>

- Update Flyte components (#3336) flyteorg/flyte

#3340 [Docs] Add User Guide section: Debugging Flyte Issue created by cosmicBboy Description The purpose of this issue is to add a section or page to the docs user guide that explains how to inspect and debug a Flyte cluster. This can be scoped to using a

flytectl demo

cluster, though the commands should transfer over to most any other Flyte cluster. Here are example topics that would fit in this section: • Scoped down to

flytectl demo

cluster • Getting error logs from failing pods • Getting cluster configuration values (e.g. project, domain, task-level resources) • Building and pushing a custom docker image to a

flytectl demo

cluster • Move the contents from the Troubleshooting guide here Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flytesnacks/commit/fb33975ff3df43ba6cce954b25bebc15c9795a49|fb33975f>

- Deprecate FlyteSchema (#942) flyteorg/flytesnacks

#676 fix: execution view children fetch on demand refactor Pull request opened by olga-union Signed-off-by: Olga Nad olga@union.ai TL;DR Replaced recursive task execution fetch logic with on demand to avoid front-loading ExecutionNodeViews component with inner layer requests, which significantly slow down page render for big workflows. Type ☐ Bug Fix ☐ Feature ☐ Plugin ☑︎ Chore Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description • Removed childGroups fetch (recursive) from

ExecutionNodeViews

page • Moved childGroups fetch to

onToggle

in Table and Timeline components - requests trigger when user clicked on expander • Moved map tasks fetch to details page to back fill one-off nodeExecutions data, and to

ReactFlowGraphComponent

to batch them for all known nodeExecutions, so the graph is rendered properly (this will slow down the graph tab, but based on user's feedback, it's ok, if graph tab takes some time to load). Known issues ☐ some unit tests are broken ☑︎ add childGroups fetch in graph on node click ☑︎ fix rerender of graph on childGroups fetch ☑︎ fix

compiledWorkflowClosure

to check for

is_dynamic

field (available in node execution's metadata) instead of pulling multiple requests on repeat ☐ test filters in Table tab ☑︎ display parent node as

nested

in graph to indicate it has children ☐ child task type stays in Loading state in the Details Sidebar Follow-up issue NA Test workflows • normal

core.control_flow.subworkflows.parent_wf

• dynamic

<http://core.control_flow.dynamics.wf|core.control_flow.dynamics.wf>

• nested parent

core.control_flow.subworkflows.nested_parent_wf

• map task

<http://flyte.workflows.example_map_task.wf|flyte.workflows.example_map_task.wf>

• small fan-out dynamic

<http://workflows.branching_dynamic_example.wf|workflows.branching_dynamic_example.wf>

flyteorg/flyteconsole DCO: DCO ✅ 1 other check has passed 1/2 successful checks

Release - flytesnacks v0.3.168 New release published by flyte-bot Changelog • fb33975 Deprecate FlyteSchema (#942) flyteorg/flytesnacks

#3342 [BUG] Type transformer fails on local runs when list is composed within the workflow definition Issue created by bstadlbauer Describe the bug The following example serializes, registers and runs fine remotely. However, when trying to run it locally it fails:

from typing import List

from flytekit import task, workflow


@task
def my_task_1() -> int:
    return 1


@task
def my_task_2(integers: List[int]) -> int:
    return sum(integers)


@workflow
def subworkflow(integers: List[int]) -> int:
    return my_task_2(integers=integers)


@workflow
def my_workflow() -> int:
    result = my_task_1()
    return subworkflow(integers=[result])


if __name__ == "__main__":
    print(f"Running {__file__} main...")
    print(my_workflow())

with:

Traceback (most recent call last):
  File "/Users/bstadlbauer/workspace/pachama/protocol-baseline/flyte/baseline/demo.py", line 29, in <module>
    print(my_workflow())
  File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/workflow.py", line 258, in __call__
    return flyte_entity_call_handler(self, *args, **input_kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/promise.py", line 1050, in flyte_entity_call_handler
    result = cast(LocallyExecutable, entity).local_execute(child_ctx, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/workflow.py", line 273, in local_execute
    function_outputs = self.execute(**kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/workflow.py", line 708, in execute
    return exception_scopes.user_entry_point(self._workflow_function)(**kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 198, in user_entry_point
    return wrapped(*args, **kwargs)
  File "/Users/bstadlbauer/workspace/pachama/protocol-baseline/flyte/baseline/demo.py", line 24, in my_workflow
    return subworkflow(integers=[result])
  File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/workflow.py", line 258, in __call__
    return flyte_entity_call_handler(self, *args, **input_kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/promise.py", line 1042, in flyte_entity_call_handler
    return cast(LocallyExecutable, entity).local_execute(ctx, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/workflow.py", line 269, in local_execute
    kwargs[k] = Promise(var=k, val=TypeEngine.to_literal(ctx, v, t, self.interface.inputs[k].type))
  File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 767, in to_literal
    lv = transformer.to_literal(ctx, python_val, python_type, expected)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 924, in to_literal
    lit_list = [TypeEngine.to_literal(ctx, x, t, expected.collection_type) for x in python_val]  # type: ignore
  File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 924, in <listcomp>
    lit_list = [TypeEngine.to_literal(ctx, x, t, expected.collection_type) for x in python_val]  # type: ignore
  File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 751, in to_literal
    transformer.assert_type(python_type, python_val)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 90, in assert_type
    raise TypeTransformerFailedError(f"Type of Val '{v}' is not an instance of {t}")
flytekit.core.type_engine.TypeTransformerFailedError: Type of Val 'Resolved(o0=scalar {
  primitive {
    integer: 1
  }
}
)' is not an instance of <class 'int'>

Expected behavior The workflow should succeed locally. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3343 Add ip alb ingress target type for EKS values Pull request opened by fvde Tracking issue This change does not close an open Issue per se, but to I had to make this change in my config to get the ALB creation to work at all. See this Slack Thread for additional background: https://flyte-org.slack.com/archives/C01P3B761A6/p1671215817793359 Describe your changes Route traffic for EKS deployment directly from ALB to pods. For details: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/ingress/annotations/#target-type

<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: ip

Check all the applicable boxes ☐ I updated the documentation accordingly. Not sure if needed, seems to be referenced directly via eks-values.yaml mostly. ☐ All new and existing tests passed. ☑︎ All commits are signed-off. Screenshots Note to reviewers flyteorg/flyte ✅ All checks have passed 9/9 successful checks

Release - Flyte v1.4.0-b0 milestone release New release published by flyte-bot Flyte v1.4.0-b0 Changelog Pod Templates and changes to the sandbox experience (mainly around configuration reloading). A full changelog is going to come in the official release. flyteorg/flyte

Deployment to github-pages by flyte-bot flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyteadmin/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyteadmin/commit/32c8dbdc3d8936c3fed9d783129bbe1047a88cc4|32c8dbdc>

- Use ResourceManager to aid in validating task resources (#517) flyteorg/flyteadmin

#3065 [BUG] project/domain resource limits don't overwrite global task_resource_defaults Issue created by flixr Describe the bug Raising task_resources limits for a project/domain doesn't work if the limit is higher than the global default limits in task_resource_defaults.yaml configmap. Expected behavior If an admin configures higher defaults/limit for a project/domain as per https://docs.flyte.org/en/latest/deployment/cluster_config/general.html#task-resources those should overwrite the global default settings from the configmap. Additional context to reproduce 1. E.g. ith global task_resource_defaults.yaml configmap:

task_resources:
  defaults:
    cpu: 500m
    memory: 500Mi
    storage: 500Mi
  limits:
    cpu: 8
    gpu: 1
    memory: 10Gi
    storage: 1Gi

2. Set a higher limit for flytesnacks/development, tra.yml

defaults:
    cpu: "1"
    memory: 1Gi
limits:
    cpu: "12"
    memory: 25Gi
project: flytesnacks
domain: development

3. verify:

flytectl get task-resource-attribute -p flytesnacks -d development

{
  "project": "flytesnacks",
  "domain": "development",
  "defaults": {
    "cpu": "1",
    "memory": "2Gi"
  },
  "limits": {
    "cpu": "12",
    "memory": "25Gi"
  }
}

4. Try to run task with e.g. 20Gi memory requested:

Requested MEMORY default [20Gi] is greater than current limit set in the platform configuration [10Gi]. Please contact Flyte Admins to change these limits or consult the configuration

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

Release - v1.1.73 New release published by flyte-bot Changelog • 32c8dbd Use ResourceManager to aid in validating task resources (#517) flyteorg/flyteadmin

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flytesnacks/commit/44a2c483ce593566de357ae658bdb108299028b4|44a2c483>

- Remove deploying workflows page (#952) flyteorg/flytesnacks

1 new commit pushed to

<https://github.com/flyteorg/community/tree/main|main>

by davidmirror-ops

<https://github.com/flyteorg/community/commit/9ceea42e4b24fb31a7b5cf4a4a6f421fe53d0c8f|9ceea42e>

- Governance v1 flyteorg/community