Running into a strange error when deploying flyte: • Using: opta from latest commit • Cloud provider: AWS • Region: eu-north-1 Had to bump to

flyteadmin:v0.6.110

and

flytescheduler:v0.6.110

to get around the db credentials problem Sören described above.

{"json":{"src":"factory.go:97"},"level":"warning","msg":"Failed to initialize new gizmo aws subscriber with config [{Config:{AccessKey: MFASerialNumber: Region:eu-north-1 RoleARN: SecretKey: SessionToken: EndpointURL:\u003cnil\u003e} QueueName:flyte-dev-service-flyte-notifcationsQueue QueueOwnerAccountID:[REDACTED] QueueURL: MaxMessages:\u003cnil\u003e TimeoutSeconds:\u003cnil\u003e SleepInterval:\u003cnil\u003e DeleteBufferSize:\u003cnil\u003e ConsumeBase64:0x398494f}] and err: InvalidAddress: The address <https://sqs.eu-north-1.amazonaws.com/> is not valid for this endpoint.\n\tstatus code: 404, request id: 470cb213-271f-572e-b819-e88989f14c89","ts":"2022-03-03T11:55:26Z"}
{"json":{"src":"shared.go:23"},"level":"warning","msg":"Failed [InvalidAddress: The address <https://sqs.eu-north-1.amazonaws.com/> is not valid for this endpoint.\n\tstatus code: 404, request id: 470cb213-271f-572e-b819-e88989f14c89] on attempt 0 of 0","ts":"2022-03-03T11:55:26Z"}

The queue exists:

aws sqs get-queue-url --queue-name flyte-dev-service-flyte-notifcationsQueue | jq '.QueueUrl' -r
<https://sqs.eu-north-1.amazonaws.com/[REDACTED]/flyte-dev-service-flyte-notifcationsQueue>

Hey is this not possible?

flytectl create execution --relaunch wfnu23ufbw3fab -p flyte-java-build-test -d production --targetDomain development

@Yee @Eduardo Apolinario (eapolinario): do y'all think this is still necessary? https://github.com/flyteorg/flytekit/blob/170f5af72b52997730bd2f8c83596418a632683a/plugins/flytekit-k8s-pod/flytekitplugins/pod/task.py#L122

@jeev what do you think?

We're seeing flytepropeller auth failures after installation/upgrade (via helm) with a message like this:

E1220 16:10:36.391892       1 workers.go:102] error syncing 'mandant1-development/f3359d6b5cd941830000': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]

It leaves the system in a weird state because there's no hard failure. flytepropeller ist running after all so our monitoring does not trigger an alarm, but no workflow execution is happening. After a flytepropeller restart, it suddenly starts working again. So after some digging I found out why this happens in our setup:

flyte-secret-auth

is populated with

.Values.secrets.adminOauthClientCredentials.clientSecret

during installation which is set to the placeholder

foobar

in

values.yaml

. We set the secret value dynamically though with a helm hook during installation because we need to fetch the real client-secret from Keycloak. That happens only after flytepropeller is deployed and it seems that flytepropeller does not reload the secret on changes. Since

flyte-secret-auth

is managed by helm, this happens again on every

helm upgrade

. I see mainly two (non exclusive) ways to improve this behavior: • Remove the default

clientSecret

and only create

flyte-secret-auth

via helm if the value is actually set. Only mount

flyte-secret-auth

if external auth is enabled. That would cause flytepropeller to fail to start until

flyte-secret-auth

is created by other means. • Trigger a flytepropeller reload when

flyte-secret-auth

changes. Any thoughts on this? Happy to contribute here but I'd like to discuss the best way forward with you first.

Hi! I have a question regarding flyte support for tensorflow data. I have generated a

tf.data.Dataset

in one of the tasks and return it as input for next task, but got data type error (when pickle serialize this data)`tensorflow.python.framework.errors_impl.InvalidArgumentError'>: Cannot convert a Tensor of dtype variant to a NumPy array` , I'm kind of new to both tensorflow and flyte, so wondering is there any guidance on how to resolve this issue?

Hi, testing Flyte in EKS, I can register the workflows but can’t run it. When execution workflow, my pod returned this error. The docs saying something about Flyte User Role but I don’t know where to assign it. I used flyte-binary helm chart to deploy flyte.

Failed to get data from \<ns3://meta-bucket/flytesnacks/development/HF4IUC4ZOJ77UBKLLP77H4SY24======/fastf>\na252ae25deeecc9b853db87173e84cb.tar.gz
          to /root/ (recursive=False).\n\nOriginal exception: Access Denied\n"

Hi all, quick question, I used to be able to run workflows with

StructuredDataset

inputs from the UI by just passing a link to the parquet file as input, but recently (as of 1.4.0 I think) I get an error message (screenshot attached). Is that something that is intentional i.e. you won't be providing support anymore?

Hi! Is there a way to make caching using the

Annotated[..., HashMethod(...)]

work with

FlyteFile

? It doesn't work out-of-the-box, failing an `issubclass` check in flytekit/types/file/file.py. However, I have put together a two-line patch (can submit an MR) for that file that would allow the use of the annotated type alias with a custom hash function. Or am I chasing a red herring here and there is an easier way to get content-aware caching for `FlyteFile`s? Thanks! 🙏 Here's the example workflow (expected behavior is that

print_file

is executed only when either its inputs change, or the signature/`cache_version` is bumped -- if I omit the custom hash method, the cache is hit regardless if the inputs have changed in content):

def calc_hash(f: FlyteFile) -> str:
    ...


CachedFlyteFile = Annotated[FlyteFile, HashMethod(calc_hash)]


@task
def write_file() -> CachedFlyteFile:
    print("write_file")
    local_path = "data.parquet"

    df = pd.DataFrame(data={"a": [1, 2, 3], "b": [3, 4, 5]})
    df.to_parquet(local_path)

    return FlyteFile(local_path, remote_path="s3://...")


@task(cache=True, cache_version="1")
def print_file(file: FlyteFile) -> None:
    file.download()
    print(pd.read_parquet(file))


@workflow
def wf() -> None:
    f = write_file()
    print_file(file=f)


if __name__ == "__main__":
    wf()
    wf()  # don't expect to see output from `print_file`

Hello! 👋 I have asked about problems with authentication (Azure oidc) earlier. This week I tried creating an entire new deployment using flyte-binary instead. However, this gives me the same problems during the redirect callback. In short:

Error generating encrypted accesstoken cookie [SECURE_COOKIE_ERROR] Error creating secure cookie, caused by: securecookie: the value is too long

Error setting encrypted JWT cookie [SECURE_COOKIE_ERROR] Error creating secure cookie, caused by: securecookie: the value is too long

Any advice on how I can continue is much appreciated.