#4348 Tune sandbox readiness checks to ensure that sandbox is fully accessi… Pull request opened by jeevb …ble when marked as ready Describe your changes The envoy proxy on

sandbox-bundled

was missing a healthcheck, and was incorrectly marked as ready before the listening port was even open. This causes CI to occasionally fail if using operations that depend on a ready sandbox state. Check all the applicable boxes ☐ I updated the documentation accordingly. ☑︎ All new and existing tests passed. ☑︎ All commits are signed-off. Screenshots Note to reviewers flyteorg/flyte GitHub Actions: build-and-push-sandbox-bundled-image ✅ 19 other checks have passed 19/20 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by jeevb

<https://github.com/flyteorg/flyte/commit/5b32a42746d666077df80b0c2266cf287e964e12|5b32a427>

- Tune sandbox readiness checks to ensure that sandbox is fully accessible when marked as ready (#4348) flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytectl/tree/master|master>

by jeevb

<https://github.com/flyteorg/flytectl/commit/422a8e73e7c5e09b12d5318d7f9c123c706f9953|422a8e73>

- Misc cleanups to aesthetics (#441) flyteorg/flytectl

#442 Increase sleep seconds when waiting for sandbox Pull request opened by jeevb TL;DR Increases time spent sleeping before the sandbox is considered ready Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytectl GitHub Actions: Dry Run Goreleaser ✅ 6 other checks have passed 6/7 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytectl/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flytectl/commit/d2979452fef41b419429a6c8f6e99f6999aa0c08|d2979452>

- Increase sleep seconds when waiting for sandbox (#442) flyteorg/flytectl

Release - v0.7.9 New release published by flyte-bot Changelog • d297945 Increase sleep seconds when waiting for sandbox (#442) • 422a8e7 Misc cleanups to aesthetics (#441) flyteorg/flytectl

1 new commit pushed to

<https://github.com/flyteorg/homebrew-tap/tree/main|main>

by flyte-bot

<https://github.com/flyteorg/homebrew-tap/commit/9a9a1c37dc567289141708ad8381e93060fd8b5d|9a9a1c37>

- Brew formula update for flytectl version v0.7.9 flyteorg/homebrew-tap

1 new commit pushed to

<https://github.com/flyteorg/flytectl/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flytectl/commit/1350bfa10018ab4b14990ceefe88b1caf49aba9e|1350bfa1>

- #minor Updated Sandbox config, with automated data configuration (#440) flyteorg/flytectl

Release - v0.8.0 New release published by flyte-bot Changelog • 1350bfa #minor Updated Sandbox config, with automated data configuration (#440) flyteorg/flytectl

1 new commit pushed to

<https://github.com/flyteorg/homebrew-tap/tree/main|main>

by flyte-bot

<https://github.com/flyteorg/homebrew-tap/commit/cdfe41b3891b5cb7f77f6e175e735620364c3129|cdfe41b3>

- Brew formula update for flytectl version v0.8.0 flyteorg/homebrew-tap

#4349 [BUG] Etcdserver: request is too large error causing never-ending tasks and log spamming Issue created by deadlycoconuts Describe the bug We’ve observed this issue whereby the FlytePropeller isn’t failing workflows that cannot be written to the etcd store. These workflows get stuck in an enqueue/resync/fail loop which repeats itself every minute or so indefinitely, even though they are shown as ‘failed’ on the FlyteConsole. This also floods our logging services with redundant logs. We tried toggling the

useOffloadedWorkflowClosure

flag but unfortunately it didn’t help resolve the etcd errors that we’ve been seeing from the FlytePropeller. We’ve restarted both the FlyteAdmin and the FlytePropeller deployments after updating the configs but that didn’t seem to stop the existing executions from throwing the etcd errors continuously. In the end we had to delete the affected executions one by one manually to stop them from being scheduled endlessly. There are some related issues and fixes that we found from 2-3 years ago which do not seem to work anymore, so we’re wondering what might’ve changed since: Related Issues • #363 • #511 Related Fixes • flyteorg/flytepropeller#240 In particular, it seems like this if condition is no longer working as expected: flyte/flytepropeller/pkg/controller/workflowstore/passthrough.go Line 99 in </flyteorg/flyte/commit/c6476cc178e576a4b7853d5d2adcb54081848974|c6476cc> Expected behavior Tasks should never end up in a state where they are retried endlessly and should be terminated with an error if there is an issue updating their state. Additional context to reproduce No response Screenshots Here are some images to illustrate the problem that I’ve described: 1. Affected workflow - this was first executed at 2.17 AM (SGT/HKT) on the 7th of October and failed at around 2.34 AM; it was however being restarted endlessly ever since 2. FlytePropeller logs when the workflow first failed 3. FlytePropeller logs 11 days after the initial failure Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1214 second iteration: docs restructure Pull request opened by samhita-alla This PR updates examples in the data types and io section. flyteorg/flytesnacks GitHub Actions: Mark github pre-release as Release GitHub Actions: Publish artifacts to github release GitHub Actions: Create Prerelease GitHub Actions: Bump Version GitHub Actions: Serialize & Register Flytesnacks workflow / Register example to sandbox ✅ 25 other checks have passed 25/30 successful checks

#1130 update environment setup guide Pull request opened by samhita-alla This PR corrects the workflow name in the environment setup guide and updates the instructions for triggering remote code execution in the case of ImageSpec, as the specified registry isn't accessible to everyone. flyteorg/flytesnacks GitHub Actions: Mark github pre-release as Release GitHub Actions: Publish artifacts to github release GitHub Actions: Create Prerelease GitHub Actions: Bump Version ✅ 26 other checks have passed 26/30 successful checks

#93 update tests Pull request opened by samhita-alla In accordance with the second iteration of docs restructure. PR: flyteorg/flytesnacks#1214 flyteorg/boilerplate ✅ All checks have passed 2/2 successful checks

#83 update tests Pull request opened by samhita-alla In accordance with the second iteration of docs restructure. PR: flyteorg/flytesnacks#1214 flyteorg/flytetools ✅ All checks have passed 2/2 successful checks

#4350 update tests Pull request opened by samhita-alla Tracking issue Describe your changes In accordance with the second iteration of docs restructure. PR: flyteorg/flytesnacks#1214 Check all the applicable boxes ☐ I updated the documentation accordingly. ☐ All new and existing tests passed. ☐ All commits are signed-off. Screenshots Note to reviewers flyteorg/flyte GitHub Actions: build-and-push-single-binary-image GitHub Actions: generate_helm GitHub Actions: docs GitHub Actions: compile ✅ 8 other checks have passed 8/12 successful checks

#4351 [Docs] FlyteRemote example throws an error Issue created by davidmirror-ops Description While runnning the FlyteRemote example locally, the result is the following error message:

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "missing project"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:127.0.0.1:30080 {created_time:"2023-11-02T05:53:05.142633-05:00", grpc_status:3, 
grpc_message:"missing project"}"

There's no reference to look further into FlyteRemote Config. A manual search leads to the FlyteRemote page but even adding explicitly the config like this:

remote = FlyteRemote(config= Config.for_endpoint(endpoint="localhost:30088", config_file="$HOME/.flyte/config.yaml"),default_project="flytesnacks",default_domain="development")

leads to an error:

FileNotFoundError: [Errno 2] No such file or directory: '$HOME/.flyte/config.yaml'

I'm sure there's something really simple I'm missing but it's nowhere explained in the docs. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#4352 Chore: Ensure Stalebot doesn't close issues we've not yet triaged. Pull request opened by brndnblck Describe your changes Small change to the stalebot config to exclude issues labeled "untriaged". Check all the applicable boxes ☑︎ I updated the documentation accordingly. ☑︎ All new and existing tests passed. ☑︎ All commits are signed-off. Screenshots N/A flyteorg/flyte ✅ All checks have passed 10/10 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyte/commit/5301da09de12c26008b321ad768680e33d014d96|5301da09>

- Chore: Ensure Stalebot doesn't close issues we've not yet triaged. (#4352) flyteorg/flyte

#4353 Do not automatically close stale issues Pull request opened by eapolinario Describe your changes This PR disables prevents stalebot from closing issues automatically. We're going to implement a step where the core maintainers engage with stale issues instead. As per the stalebot docs, any negative number indicates that stalebot does not close issues/PRs. Check all the applicable boxes ☐ I updated the documentation accordingly. ☐ All new and existing tests passed. ☐ All commits are signed-off. Screenshots Note to reviewers flyteorg/flyte ✅ All checks have passed 10/10 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by kumare3

<https://github.com/flyteorg/flyte/commit/a1d182b3690555b784764a59e1268f55b2a083b8|a1d182b3>

- Do not automatically close stale issues (#4353) flyteorg/flyte

#4354 [BUG] Keyring errors when intialising flyte remote client Issue created by Tom-Newton Describe the bug Get a keyring error when initialising a simple flyte remote client. It seems to only happen on some systems and the exact error varies slightly. I've seen

InitError: Failed to create the collection: Prompt dismissed..

KeyringLocked: Failed to unlock the collection!

Expected behavior I'm not trying to use any authentication so I would expect it to ignore any exceptions about the system keyring. Alternatively we could add

AuthType.NONE

so that all keyvault related code can be avoided. Additional context to reproduce On my system I can reproduce with:

from flytekit.configuration import Config
from flytekit.remote import FlyteRemote

config = Config.for_sandbox()
remote = FlyteRemote(config=config)
remote.client

This is SSHing from one linux system to another linux system. Both are used with a Desktop setup so they probably have the GNOME keyring installed. I tried using all the different Auth types but they all failed for one reason or another (this is largely expected because we do not have any authentication configured on our flyte deployment). Full stack trace: stack_trace.txt Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#4328 Support Azure Cloud Provider (via Stow) Pull request opened by cgrass Describe your changes 1. Update configmap and helper to support Azure cloud storage. 2. Update values with sample config and regen helm doc readme. Check all the applicable boxes • [X ] I updated the documentation accordingly. • [X ] All new and existing tests passed. • [X ] All commits are signed-off. Screenshots Note to reviewers This will allow users of the

flyte-binary

chart to configure Azure storage accounts. The stow version bump is needed in order to use Workload Identity. This PR does not include an implementation of RemoteURLInterface, which would be required before using signed URLs in Azure. A potential fix is being discussed in slack and should not be difficult, but I'm not confident that support for signed URLs is required. A storage key or workload identity needs to be configured for this to work end-to-end with task pods and azure blob storage via Stow. A workflow identity can be defined in the PodTemplate like this example, and the service can be deployed using a values.yaml similar to here. Should that configuration be documented in some other way? flyteorg/flyte GitHub Actions: build-and-push-sandbox-bundled-image GitHub Actions: sandbox-bundled-functional-tests ✅ 13 other checks have passed 13/15 successful checks

#3190 [Core feature] Add support Hex plugin Issue created by pingsutw Motivation: Why do you think this is important? Hex is a modern Data Workspace. It makes it easy to connect to data, analyze it in collaborative SQL and Python-powered notebooks, and share work as interactive data apps and stories. Hex creates a chart cell that allows editors to interactively explore and aggregate a dataframe, creating rich visualizations to share with app users. We can add a Hex plugin to run the notebook in the Hex workspace and display the interactive chart on Flyte Deck. chart-cell-demo-4190de769b111e6e9fecd7506d0fc18b.mp4 Goal: What should the final outcome look like, ideally? • Have an interactive chart on Flyte Deck • Submit a job to Hex to run the notebook in Hex workspace. Describe alternatives you've considered NA Propose: Link/Inline OR Additional context • https://learn.hex.tech/docs/logic-cell-types/display-cells/chart-cells • https://github.com/hex-inc/airflow-provider-hex Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3186 [FlyteCTL Feature] Make default output of `flytectl config init` consistent with the demo sandbox config Issue created by cosmicBboy Describe the feature/command for FlyteCTL Currently, the output of

flytectl config init

is:

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///localhost:30081
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level:

But the config produced by

flytectl demo start

is:

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: localhost:30080
  authType: Pkce
  insecure: true
console:
  endpoint: <http://localhost:30080>
logger:
  show-source: true
  level: 0

This may be confusing for new users who would expect the two configs to be consistent Provide a possible output or UX example The output of of

flytectl config init

should be the same as

config-sandbox.yaml

Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3184 [Core feature] Add a new state called CreatingCluster Issue created by pingsutw Motivation: Why do you think this is important? Some tasks like Ray and Spark on Databricks will create an ephemeral cluster before running the task. Adding a

ClusterCreating

state for tasks would be helpful for people to know what's going on underneath. It is also much easier to debug if an operator fails to create a new cluster. In addition, cluster creation time and current task running time can be measured.it Goal: What should the final outcome look like, ideally? Users will know Flyte is creating a cluster when the node is stared Describe alternatives you've considered NA Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#2999 [Core feature] Add support native python type in the condition task Issue created by pingsutw Motivation: Why do you think this is important? For now, The variable in the

_if

in the

condition

must be a promise, and can't use a native python type in the if statement.

@dynamic
def d1() -> bool:
    a = t1()
    return (
        conditional("train_estimator")
        .if_(a==True)
        .then(t2())
        .else_()
        .then(t2()))

@dynamic
def d1(a: bool) -> bool:
    return (
        conditional("train_estimator")
        .if_(a==True)  # <- failed because the a isn't a promise
        .then(t2())
        .else_()
        .then(t2()))

Goal: What should the final outcome look like, ideally? Both promise and native python types should be supported in the conditions Describe alternatives you've considered Users must create a task to convert it to a promise and use it in the conditions.

@task
def convert_to_promise(a: bool) -> bool:
    return a

Propose: Link/Inline OR Additional context https://flyte-org.slack.com/archives/CREL4QVAQ/p1666110935856779 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3063 [FlyteCTL Feature] Make flytectl config init respect config flag Issue created by hajapy Describe the feature/command for FlyteCTL The global

--config

cli flag is not respected by

flytectl config init

. Instead, this config always writes to

~/.flyte/config.yaml

. The presence of this global flag and yet it being ignored is confusing. (Relatedly, could flytectl suppress describing global flags that are meaningless to a specific subcommand in its help output?) Provide a possible output or UX example

flytectl config init --config my_config.yaml

creates a file

my_config.yaml

in current directory. This feature can help make the

flytectl config init

command more generally useful in scenarios where users may wish to store more than one config and reference appropriate configs in subsequent commands. It also happens that

flytectl config init

requires user input if the default path already exists and this could be problematic in CI workflows. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3166 [Housekeeping] Remove deprecated numpy aliases Issue created by eapolinario Describe the issue A few aliases defined in

numpy

were deprecated in v1.20.0 and finally removed in

v1.24.0

(check Deprecations in https://github.com/numpy/numpy/releases/tag/v1.20.0). We should remove the mentions to those deprecated aliases from the FlyteSchemaTransformer. What if we do not do this? Users who install

numpy>=1.24.0

will see this error in flytekit:

/tmp/tmp.pgqfAvNZOC/venv/lib/python3.10/site-packages/flytekit/types/schema/types.py:324: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.  (This may have returned Python scalars in past versions.
  _np.bool: SchemaType.SchemaColumn.SchemaColumnType.BOOLEAN,  # type: ignore
Traceback (most recent call last):
  File "/tmp/tmp.pgqfAvNZOC/venv/bin/pyflyte", line 5, in <module>
    from flytekit.clis.sdk_in_container.pyflyte import main
  File "/tmp/tmp.pgqfAvNZOC/venv/lib/python3.10/site-packages/flytekit/__init__.py", line 195, in <module>
    from flytekit.types import directory, file, numpy, schema
  File "/tmp/tmp.pgqfAvNZOC/venv/lib/python3.10/site-packages/flytekit/types/schema/__init__.py", line 1, in <module>
    from .types import (
  File "/tmp/tmp.pgqfAvNZOC/venv/lib/python3.10/site-packages/flytekit/types/schema/types.py", line 314, in <module>
    class FlyteSchemaTransformer(TypeTransformer[FlyteSchema]):
  File "/tmp/tmp.pgqfAvNZOC/venv/lib/python3.10/site-packages/flytekit/types/schema/types.py", line 324, in FlyteSchemaTransformer
    _np.bool: SchemaType.SchemaColumn.SchemaColumnType.BOOLEAN,  # type: ignore
  File "/tmp/tmp.pgqfAvNZOC/venv/lib/python3.10/site-packages/numpy/__init__.py", line 284, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'bool'. Did you mean: 'bool_'?

Related component(s) flytekit Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3131 [Plugin] Add dbt list task to dbt plugin Issue created by brandon-segal The dbt plugin should have the ability to run dbt ls statements to allow users to list out models that should be included in a given select statement. This kind of functionality is widely used in other orchestrators including Airflow to be able to orchestrate dbt tasks at the model level. This kind of behavior would allow the user to identify what models are going to be run and delegate the orchestration and retry logic to flyte rather than use something like dbt deferment which may be more complicated due to the need to preserve state through the manifest.json files. flyteorg/flyte