Deployment to github-pages by flyte-bot flyteorg/flyte

Release - v1.4.0 New release published by eapolinario What's Changed • Support checkpointing in local mode from cached tasks by @bethebunny in #1457 • Deprecate FlyteSchema by @pingsutw in #1418 • Use scarf images by @eapolinario in #1434 • add undocumented objects/functions to flytekit api ref by @cosmicBboy in #1502 • Use non-root user in default flytekit image by @pingsutw in #1417 • Fix PyTorch transformer by @samhita-alla in #1510 • Fix mypy errors by @pingsutw in #1313 • Compile the workflow only at compile time by @pingsutw in #1311 • Get the origin type when serializing dataclass by @pingsutw in #1508 • Remove requirements files by @eapolinario in #1511 • Delay initialization of SynchronousFlyteClient in FlyteRemote by @eapolinario in #1514 • Make flytekit comply with PEP-561 by @pingsutw in #1516 • Flytekit Auth system overhaul and pretty printing upgrade by @kumare3 in #1458 • Create non-root user after apt-get by @eapolinario in #1519 • Add root pyflyte reference to docs by @eapolinario in #1520 • Drop support for python 3.7 by @eapolinario in #1521 • DuckDB plugin by @samhita-alla in #1419 • add string as a valid input by @samhita-alla in #1527 • Add back attempt to use existing serialization settings when running by @wild-endeavor in #1529 • update configuration docs, fix some docstrings by @cosmicBboy in #1530 • Revert "Make flytekit comply with PEP-561 (#1516)" by @eapolinario in #1532 • Failed to initialize FlyteInvalidInputException by @pingsutw in #1534 Full Changelog: v1.3.2...v1.4.0 flyteorg/flytekit

Release - v1.3.3 New release published by eapolinario What's Changed • Pin fsspec by @wild-endeavor in #1540 Full Changelog: v1.3.2...v1.3.3 flyteorg/flytekit

Release - v1.4.1 New release published by eapolinario What's Changed • Pin fsspec to <=2023.1 by @wild-endeavor in #1537 Full Changelog: v1.4.0...v1.4.1 flyteorg/flytekit

Release - v1.2.10 New release published by eapolinario What's Changed • Backport 1.4 onto v1.2 release branch by @wild-endeavor in #1542 Full Changelog: v1.2.9...v1.2.10 flyteorg/flytekit

👋 Can someone help review https://github.com/flyteorg/flytekit/pull/1545? We use a lot of remote entities, so this is a blocking issue to us. Thank you. 🙏

Seeing a somewhat weird UI/caching bug. I have a workflow that launches a dynamic with cache enabled on the tasks that the dynamic runs. Cache works properly when launching fresh from console, but when I click the "Relaunch" button, cache is hit for the first few tasks (outside of the dynamic), but not for any of the tasks in the dynamic 🤔 On closer inspection, the inputs are identical for the dynamic in both executions (where cache worked, and cache did not work). I then looked at the inputs to the workflow in both executions, and it looks like when I click "Relaunch" and copy the inputs, the order of the keys are different, but their contents are identical to the original run. Would that somehow mess up the cache?

#3386 [BUG] Multiple definitions of the following tasks were found error for remote tasks Issue created by sonjaer Describe the bug When running

pyflyte package

, we get an error like:

Multiple definitions of the following tasks were found: ['ShiftDatetimeByDurationTaskV0', 'LookupFixedRangeBigQueryTaskV1', 'LookupBigQueryTaskV1']

where those tasks are remote tasks used within a workflow. The error is introduced In flytekit versions after 1.2.7 (probably 1.2.8) and is maybe due to that

_should_register_with_admin

function no longer filters out remote tasks entities as FlyteTask superclass changed from:

class FlyteTask(hash_mixin.HashOnReferenceMixin, RemoteEntity, _task_model.TaskTemplate):

to

class FlyteTask(hash_mixin.HashOnReferenceMixin, RemoteEntity, TaskSpec):

def _should_register_with_admin(entity) -> bool:
    """
    This is used in the code below. The translator.py module produces lots of objects (namely nodes and BranchNodes)
    that do not/should not be written to .pb file to send to admin. This function filters them out.
    """
    return isinstance(
        entity, (task_models.TaskSpec, _launch_plan_models.LaunchPlan, admin_workflow_models.WorkflowSpec)
    )

Expected behavior Should not get an error of duplicate tasks defined for remote tasks when running pyflyte package Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

This message was deleted.

Release - [v1.4] Patch - Prefer FLYTE_ prefixed env vars, don't register remote New release published by wild-endeavor What's Changed • [v1.4] Patch in recent changes by @wild-endeavor in #1551 #1523 #1545 #1538 flyteorg/flytekit

Hey, Do you have an idea of what has changed in the way we use Oauth2 with Flytekit? I saw that it was reworked recently and it seems like now it’s taking the

scopes

defined in FlyteAdmin correctly. I am asking because when I use Flytekit

v1.3.1

, I only get the scope

['all']

for the

PlatformConfig

PlatformConfig(endpoint='<http://flyteurl.com|flyteurl.com>', insecure=False, insecure_skip_verify=False, console_endpoint=None, command=None, client_id='github-client', client_credentials_secret='nice_secret', scopes=['all'], auth_mode='ClientSecret')

And when I use Flytekit

v1.4.1

then I get different scopes

/.venv/lib/python3.9/site-packages/flytekit/clients/auth/authenticator.py(193)get_token()
-> if scopes is not None:
(Pdb) scopes
['offline', 'all']

Flytekit

v1.4.1

correctly reflects the scopes we have but I was wondering how come

v1.3.1

has different scopes?

I'm trying to use datetimes in a dataclass as input to a workflow like this:

@dataclass_json
@dataclass
class TrainingArgs:
  latest_timestamps: List[datetime.datetime] = field(default_factory=lambda: [])

But now I'm getting a schema warning (I haven't tried to run it yet):

{
  "asctime": "2023-03-15 15:13:19,393",
  "name": "flytekit",
  "levelname": "WARNING",
  "message": "Failed to extract schema for object <class 'TrainingArgs'>, (will run schemaless) error: unsupported field type <fields._TimestampField(dump_default=datetime.datetime(2023, 3, 15, 15, 13, 19, 383752), attribute=None, validate=None, required=False, load_only=False, dump_only=False, load_default=<marshmallow.missing>, allow_none=False, error_messages={'required': 'Missing data for required field.', 'null': 'Field may not be null.', 'validator_failed': 'Invalid value.'})>If you have postponed annotations turned on (PEP 563) turn it off please. Postponedevaluation doesn't work with json dataclasses"
}

Does anyone have an idea how to get this working?

#3470 [Core feature] change with_overrides to update metadata as well Issue created by jsonporter Motivation: Why do you think this is important? While investigating a different issue born from a a user request to update the execution detail views (list, graph, timeline) to display values passed to

with_overrides

-- we realized this was not an ideal UX/UI because it would mean these views would then be displaying the

nodeId

rather than human-readable function names (eg,

n0-0-n2-0-n0-0-n1

vs

hello_person

). This is because while parsing the

compiledWorkflowClosure

we must choose to display based off either the

nodeId

or the

metadata.name

and a node will always have a

nodeId

so we can't do a conditional. Example: the user was requesting to display the

nodeId

'hello-bob' here instead of the function name 'hello_person' from

metadata.name

but we see in the second example that this is actually less useful in most cases.

"id": "hello-bob",
    "metadata": {
        "name": "hello_person",
        "retries": {}
      }

"id": "n0",
    "metadata": {
        "name": "people",
        "retries": {}
},

As a solution we want to update

with_overrides

to update both the

nodeId

and the

metadata.name

field -- this better matches the user intention of

with_overrides

and in those cases display the overridden name while also preserving the better UX for most/standard executions. Goal: What should the final outcome look like, ideally? When a user defines an entity in flytekit and uses

with_overrides

, it will update both the

nodeId

and

metadata.name

for that execution. Describe alternatives you've considered NA Propose: Link/Inline OR Additional context #3370 flyteorg/flytekit#1344 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

Deployment to github-pages by flyte-bot flyteorg/flyte

#3473 [BUG] Secret name should be validated on flytekit side and propeller should handle auto aborting/cancelling it Issue created by pmahindrakar-oss Describe the bug Currently workflows using non-conformed secrets cause error when propeller tries to create pods using the created CRD

flytepropeller-1 flytepropeller E0315 17:26:22.419072       1 workers.go:102] error syncing 'flyte-attendant-development/fbf61da4211b749988f0': failed at Node[n1]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [Invalid] failed to create resource, caused by: Pod "fbf61da4211b749988f0-n1-0" is invalid: [spec.containers[0].env[26].name: Invalid value: "_FSEC_ARN:AWS:SECRETSMANAGER:US-EAST-2:356633062068:SECRET:_OPENAI_API_KEY-QA0K0U": a valid environment variable name must consist of alphabetic characters, digits, '_', '-', or '.', and must not start with a digit (e.g. 'my.env-name',  or 'MY_ENV.NAME',  or 'MyEnvName1', regex used for validation is '[-._a-zA-Z][-._a-zA-Z0-9]*'), spec.containers[0].env[26].valueFrom.secretKeyRef.name: Invalid value: "arn:aws:secretsmanager:us-east-2:356633062068:secret:": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. '<http://example.com|example.com>', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')]

There is no progress that propeller can make with this workflow . This ticket tracks. • Adding validation in flytekit to disallow non-conforming secret names to be registered • Propeller should auto abort/cancel such tasks and send the validation error over so that admin can showcase the issue and the UI can render it • Existing workflows which bypassed this check should some how be flagged Expected behavior Disallow incorrect workflows and propeller should abort if any such incorrect workflows fall through the cracks and flyteadmin and UI should be able to show the right issue in the workflow Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

hi folks! looking for some documentation on ray integration - esp as it pertains to persistent clusters. can't find my last question on this since it was more than 90d ago. even the original RFC would suffice, the public docs don't go into detail on limitations/architecture

Deployment to github-pages by flyte-bot flyteorg/flyte

Deployment to github-pages by flyte-bot flyteorg/flyte

I am trying something a bit unconventional. Given that several task is performing modification to a

FlyteDirectory

and output the same

FlyteDirectory

. Is it possible to force all the

FlyteDirectory

to be backed by the same GCS path? Essentially sharing a directory content across tasks.

@task
def task(work_dir: FlyteDirectory) -> FlyteDirectory:
    # do something in work_dir
    # potentially add files
    return work_dir

@workflow
def workflow():
    work_dir = init_workdir()
    work_dir = task(work_dir)
    work_dir = task(work_dir)
    work_dir = task(work_dir)

Release - [Beta] FlyteFile and FlyteDirectory streaming support New release published by wild-endeavor What's Changed • prefer FLYTE_ prefixed AWS creds env vars by @flixr in #1523 • Filter out remote entity when generating pb by @honnix in #1545 • Pyflyte docs formatting by @bryanwweber in #1538 • Data subsystem by @wild-endeavor in #1526 • Less strict docker dependency versions by @jcugat in #1536 • Pluck retry from flytekit and into sagemaker by @eapolinario in #1411 • Update the pypi wait by @wild-endeavor in #1554 • Stream Directories and Files using Flyte by @kumare3 in #1512 New Contributors • @bryanwweber made their first contribution in #1538 • @jcugat made their first contribution in #1536 Full Changelog: v1.4.2...v1.4.3b0 flyteorg/flytekit

#3052 [Housekeeping] Remove unused retry dependency in flytekit Issue created by hajapy Describe the issue Remove the unused

retry

library dependency declared by flytekit: https://github.com/flyteorg/flytekit/blob/3b5d15acb5389d99b86ba342a77e1cab5ce8546d/setup.py#L68 This library is not used anywhere within flytekit that I can find searching for

import retry

or

from retry

, except in the sagemaker plugin: https://github.com/flyteorg/flytekit/blob/3b5d15acb5389d99b86ba342a77e1cab5ce8546d/plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker/distributed_training.py#L8 The dependency could be moved to the sagemaker plugin's requirements so only users of that plugin need it installed. Though, given the unmaintained state of the retry library, I'd advise against that and suggest to look for a maintained alternative (reretry? https://github.com/leshchenko1979/reretry) What if we do not do this? All users of flytekit end up installing an unmaintained library (see invl/retry#36, not updated since 2016), which also ends up bringing with it unnecessary dependency on

py

which itself has been the source of lots of security alerting pytest-dev/py#287 Related component(s) flytekit Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

Deployment to github-pages by flyte-bot flyteorg/flyte

Is it possible to retrieve workflow name from within a task? I am browsing trough

FlyteContext

but can’t see any suitable property available.

#3312 [Core feature] Support Python 3.11 Issue created by cosmicBboy Motivation: Why do you think this is important? Currently flytekit does not support python 3.11 for many reasons, the mains ones being that: 1. upstream dependencies in the flytekit dev environment do not support 3.11 currently, such as

tensorflow

and others don't support 3.11 2. certain parts of the the flytekit codebase (such as the parts that use dataclasses) need to be updated to support 3.11 Python 3.7 will be EOL'd on June 2023, which will ease the transition to 3.11 because we can unpin certain restrictions. Goal: What should the final outcome look like, ideally? A user should be able to install and use flytekit on python 3.11 Describe alternatives you've considered NA Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

Release - [Beta] Partials, deviceflow, pyflyte run multiple files, python 3.11 support New release published by wild-endeavor What's Changed • Make

FlyteFile

compatible with

Annotated[..., HashMethod]

by @AdrianoKF in #1544 • move FlyteSchema deprecation warning to initialization method by @cosmicBboy in #1558 • add pod_template and pod_template_name arguments for ContainerTask by @flixr in #1515 • Pass locally defined scopes to RemoteClientConfigStore by @franco-bocci in #1553 • TypeTransformer for TensorFlow model by @samhita-alla in #1562 • Remove flytekit-fsspec plugin from default dockerfile by @wild-endeavor in #1561 • Device auth flow / Headless auth by @kumare3 in #1552 • support python 3.11 by @cosmicBboy in #1557 • url encode secret in client credentials flow by @wild-endeavor in #1566 • Python run multiple files by @pingsutw in #1559 • General Partial support in flytekit and multi-list support in flytekit by @kumare3 in #1556 New Contributors • @franco-bocci made their first contribution in #1553 Full Changelog: v1.5.0b0...v1.5.0b1 flyteorg/flytekit

cc @Nicholas LoFaso ^^ you can try the beta - it has the partial map support 😄

for multi-list caching though you need backend update - cc @Dan Rammer (hamersaw)

How does flyte identify system error? Was trying ShellTask and the script I am providing is invalid, but instead of returning user error it’s classifying it as system error.

#3557 [BUG] Unnecessary warning in response to every pyflyte command Issue created by tsheiner Describe the bug 1. execute any valid pyflyte command 2. see in response the following warning

{"asctime": "2023-03-30 10:54:22,803", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}

Expected behavior That I only get this warning if I am doing something that involves FlyteSchema Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte