GitHub
05/10/2023, 4:35 AMGitHub
05/10/2023, 6:44 AMGitHub
05/10/2023, 8:43 AMauth.externalAuthServer = true
in the values.yaml file, and if auth.internal.clientId
is undefined (defaults to flytepropeller
), the flyte-binary pod will query the external authorization server with the flytepropeller
parameter as the clientId.
This results in failed auth to the server when scheduling workflows/tasks/notebook tasks through pyflyte or the web console. Workflows and tasks will show up as UNKNOWN
in the web console, and the flyte-binary pod will continuously make requests to the auth server - too many retries(? Have counted potentially 60 within a span of a minute to auth0,).
Specifically, within the flyte-binary pod: if you exec into the flyte
container in the pod at path: /etc/flyte/config.d/000-core.yaml
or a similar path, the relevant parameters within the admin field aren't set correctly.
Or alternatively, add more documentation to clearly outline this behaviour so future users are aware of how these parameters are set - specifically within the context of integrating Auth0.
Tldr;
Granted, this could be classified as a documentation issue, but perhaps the internal
key could be renamed to something along the lines of propeller
or dataplane
(not sure if this is the correct terminology) in order to clarify what is being configured with internal.
Expected behavior
When auth.enableAuthServer: false
, it might be a good idea to make internal.clientId
be a required parameter. In addition to that, it might help to indicate to users that this will then correspond to 000-core.yaml
and will be used by the scheduler for workflow pods.
The current documentation implies that this only needs to be set for internal use and makes no mention of it being linked to external.
One alternative is if auth.enableAuthserver: false
, the Helm chart template could derive the values for 000-core.yaml
-- admin.clientId
from a different field in the values.yaml - thirdpartyconfig.flyteclient is one suggestion.
Relevant path in the helm chart is templates/configmap.yaml
Additional context to reproduce
## values.yaml
auth:
enabled: true
enableAuthServer: false
internal:
#### Note that clientId is not set - will default to flytepropeller which is unwanted behaviour
clientSecret: <client secret here> #
clientSecretHash: <client hash>
oidc:
baseUrl: <external auth server>
clientId: <clientId>
clientSecret: <client secret>
flyteClient:
clientId: <clientId>
redirectUri: <http://localhost:53593/callback>
scopes:
- read:client_grants
audience: <audience of your api>
authorizedUris:
- put the necessary uris here
## in flyte-binary pod in K8S - /etc/flyte/config.d/000-core.yaml
admin:
clientId: flytepropeller ### This would ideally be the clientId for auth server. Can be set with internal.clientId but unclear as to how to affect this specific parameter with current docs
endpoint: localhost:8089
insecure: true
catalog-cache:
endpoint: localhost:8081
insecure: true
type: datacatalog
cluster_resources:
standaloneDeployment: false
templatePath: /etc/flyte/cluster-resource-templates
logger:
show-source: true
level: 1
propeller:
create-flyteworkflow-crd: true
webhook:
certDir: /var/run/flyte/certs
localCert: true
secretName: flyte-backend-flyte-binary-webhook-secret
serviceName: flyte-backend-flyte-binary-webhook
servicePort: 443
## /etc/flyte/config.d/004-auth.yaml
data:
auth:
userAuth:
openId:
clientId: AUTH0_CLIENT_ID
baseUrl: AUTH0_BASE_URL
scopes:
- profile
- openid
- offline_access
appAuth:
authServerType: External
externalAuthServer:
baseUrl: AUTH0_BASE_URL
metadataUrl: .well-known/openid-configuration
allowedAudience: AUTH0_AUDIENCE
thirdPartyConfig:
flyteClient:
clientId: AUTH0_CLIENT_ID
redirectUri: <http://localhost:53593/callback>
audience: AUTH0_AUDIENCE
scopes:
- read:client_grants
authorizedUris:
- <http://flyteadmin:80>
- <http://flyteadmin.flyte.svc.cluster.local:80>
- <http://flyteadmin.mlops-services.svc.cluster.local:80>
- <http://HELM_NAME-flyte-binary.flyte.svc.cluster.local:80>
- <http://HELM_NAME-flyte-binary.flyte.svc:80>
- <http://HELM_NAME-flyte-binary.flyte:80>
- <http://HELM_NAME-flyte-binary:80>
- <http://localhost:8089>
- other URIs can go here as well
server:
security:
secure: false
useAuth: true
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
05/10/2023, 9:08 AMauth.appAuth.authServerType: External
and auth.appAuth.exernalAuthServer.baseUrl
& metadataUrl
& allowedAudience
- during a helm install
or helm upgrade --values values.yaml
command. These settings are necessary for a working auth0 integration.
We may potentially be misguided about the procedure, but we found that we had to retrieve the configmap.yaml with kubectl; parse some fields with yq
and `sed`; and then reapply with kubectl
and then do kubectl rollout restart
in order to get auth working on our EKS cluster with this flyte-binary chart.
It would be highly appreciated if these values were exposed within the helm/template/values flows for easier end-user configuration.
Expected behavior
Flyte-binary exposes these configuration values:
appAuth:
authServerType: External
externalAuthServer:
baseUrl: AUTH0_BASE_URL
metadataUrl: .well-known/openid-configuration
allowedAudience: AUTH0_AUDIENCE
thirdPartyConfig:
flyteClient:
clientId: AUTH0_CLIENT_ID
redirectUri: <http://localhost:53593/callback>
audience: AUTH0_AUDIENCE
scopes:
- read:client_grants
within values.yaml
as well as the templates/configmap.yaml # 004-auth.yaml
Addendum: would also be nice to expose server.security.secure
as well for easy deployments
Necessary to set these values for working auth0 deployment at the moment.
Additional context to reproduce
Apply this values.yaml block to helm deployment
auth:
enabled: true
enableAuthServer: false
internal:
clientId: <>
clientSecret: <>
clientSecretHash: <>
oidc:
baseUrl: <>
clientId: <>
clientSecret: <>
flyteClient:
clientId: <>
redirectUri: <http://localhost:53593/callback>
scopes:
- read:client_grants
audience: <>
authorizedUris:
- authorizedURIs go here
Results in failed deployment when using Auth0
The following block is the Rrequired 004-auth.yaml to pipe into flyte-binary in order to make auth0 work.
Filling in the provided Helm values without changing the template results in failed deployment with CrashLoopBackOff due to misconfiguration.
data: |
auth:
userAuth:
openId:
clientId: AUTH0_CLIENT_ID
baseUrl: AUTH0_BASE_URL
scopes:
- profile
- openid
- offline_access
appAuth:
authServerType: External
externalAuthServer:
baseUrl: AUTH0_BASE_URL
metadataUrl: .well-known/openid-configuration
allowedAudience: AUTH0_AUDIENCE
thirdPartyConfig:
flyteClient:
clientId: AUTH0_CLIENT_ID
redirectUri: <http://localhost:53593/callback>
audience: AUTH0_AUDIENCE
scopes:
- read:client_grants
authorizedUris:
- AUTH0_AUDIENCE
- LOADBALANCER_URL
- AUTH0_BASE_URL
- <http://LOADBALANCER_GRPC_URL.svc.aiq-index.com>
- <https://LOADBALANCER_GRPC_URL.svc.aiq-index.com>
- <http://flyteadmin:80>
- <http://flyteadmin.flyte.svc.cluster.local:80>
- <http://flyteadmin.mlops-services.svc.cluster.local:80>
- <http://HELM_NAME-flyte-binary.flyte.svc.cluster.local:80>
- <http://HELM_NAME-flyte-binary.flyte.svc:80>
- <http://HELM_NAME-flyte-binary.flyte:80>
- <http://HELM_NAME-flyte-binary:80>
- <http://localhost:8089>
server:
security:
secure: false
useAuth: true
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
05/10/2023, 9:36 AMaudience
parameter for the ClientSecret/ClientCredentials authorization flow w/ Auth0.
The ClientConfig
, PlatformConfig
, ClientCredentialsAuthenticator
classes, the get_authenticator
function, and the token_client.get_token
method are not currently configured to accept and pass around the required parameter, and the final get_token method never receives it as an input arg, nor does it pass it into the POST request when requesting a token.
This results in a failed auth request as Auth0 requires an audience parameter in the body of the request.
Expected behavior
Allow the end-user to set the "audience" parameter either in <flyteconsoleurl>/config/v1/flyte_client
or the config.yaml
file, and have the pyflyte/flytekit package use this in its auth request for Auth0 integration.
Will link fork with changes soon
Additional context to reproduce
Use an Auth0 external authorization server and attempt to run a pyflyte run
command with the following config values:
###004-auth.yaml configmap
data: |
auth:
userAuth:
openId:
clientId: AUTH0_CLIENT_ID
baseUrl: AUTH0_BASE_URL
scopes:
- profile
- openid
- offline_access
appAuth:
authServerType: External
externalAuthServer:
baseUrl: AUTH0_BASE_URL
metadataUrl: .well-known/openid-configuration
allowedAudience: AUTH0_AUDIENCE
thirdPartyConfig:
flyteClient:
clientId: AUTH0_CLIENT_ID
redirectUri: <http://localhost:53593/callback>
audience: AUTH0_AUDIENCE
scopes:
- read:client_grants
authorizedUris:
- uris
server:
security:
secure: false
useAuth: true
config.yaml
admin:
# For GRPC Endpoints you might want to use dns:///flyte.example.com
#endpoint: dns:///localhost:8089
endpoint: dns:///<endpointhere>
authType: ClientSecret
clientId: <>
clientSecretLocation: <>
scopes:
- read:client_grants
#- offline --------------------- set offline and all if desired, but not strictly necessary
#- all
audience: <> --------------------- deriving it from here is nice, but not strictly necessary as config/v1/flyte_client exists
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
05/10/2023, 11:54 AMcloud_events.yaml: |
cloudEvents:
enable: true
gcp:
region: us-east-2
eventsPublisher:
eventTypes:
- all # or node, task, workflow
topicName: my-topic
type: gcp
When including such a config in the helm values file, the following error is raised in flyteadmin:
{"json":{},"level":"fatal","msg":"caught panic: project id is required [goroutine 1 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\<http://ngithub.com/flyteorg/flyteadmin/pkg/rpc/adminservice.NewAdminServer.func1()|ngithub.com/flyteorg/flyteadmin/pkg/rpc/adminservice.NewAdminServer.func1()>\n\t/go/src/github.com/flyteorg/flyteadmin/pkg/rpc/adminservice/base.go:74 +0x88\npanic({0x2273960, 0xc0013f1dd0})\n\t/usr/local/go/src/runtime/panic.go:838 +0x207\<http://ngithub.com/flyteorg/flyteadmin/pkg/async/cloudevent.NewCloudEventsPublisher(|ngithub.com/flyteorg/flyteadmin/pkg/async/cloudevent.NewCloudEventsPublisher(>{0x2c04440, 0xc000128000}, {0x1, {0xc00051b4e8, 0x3}, {{0xc00051b470, 0xb}}, {{0x0, 0x0}}, {{0x0, ...}, ...}, ...}, ...)\n\t/go/src/github.com/flyteorg/flyteadmin/pkg/async/cloudevent/factory.go:61 +0x905\<http://ngithub.com/flyteorg/flyteadmin/pkg/rpc/adminservice.NewAdminServer({0x2c04440|ngithub.com/flyteorg/flyteadmin/pkg/rpc/adminservice.NewAdminServer({0x2c04440>?, 0xc000128000}, 0xc0006a3b80, {0x2c0cb60, 0xc00058c6c0}, {0x0, 0x0}, {0x0, 0x0}, 0xc000b563c0, ...)\n\t/go/src/github.com/flyteorg/flyteadmin/pkg/rpc/adminservice/base.go:103 +0x845\<http://ngithub.com/flyteorg/flyteadmin/pkg/|ngithub.com/flyteorg/flyteadmin/pkg/>
When setting the projectId
instead of the region as done here, this error disappears.
However, the resulting yaml file in the flyteadmin configmap still contains aws config:
❯ k -n flyte get configmaps flyte-admin-base-config -o yaml
apiVersion: v1
data:
cloud_events.yaml: "cloudEvents: \n aws:\n region: us-east-2\n enable: true\n
\ eventsPublisher:\n eventTypes:\n - all\n topicName: <my-topic>\n
\ gcp:\n projectId: <my-project>\n type: gcp\n"
Despite the unintended aws config, the published events do make it to GCP pub/sub and can be pulled in the cloud console.
However, even though the messages can successfully be pulled from the pub sub topic/subscription, flyteadmin always shows error logs:
{"json":{"exec_id":"b862f22a535442bda2e4","node":"n0"},"level":"error","msg":"Failed to publish a message with key [flyteidl.admin.TaskExecutionEventRequest] and message [Context Attributes,\n specversion: 1.0\n type: com.flyte.resource.flyteidl.admin.TaskExecutionEventRequest\n ... ] and error: context canceled","ts":"2023-05-10T11:46:42Z"}
{"json":{"exec_id":"b862f22a535442bda2e4","node":"n0"},"level":"error","msg":"Failed to send message [event:\u003ctask_id:\u003cresource_type:TASK ... ] with error: context canceled","ts":"2023-05-10T11:46:42Z"}
Expected behavior
• The documentation should not say to add the region for GCP but the projedId.
• When configuring GCP, the resulting configmap should not contain AWS config.
• Flyteadmin should not show errors even though the published events make it to pub/sub and can be pulled from there.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
05/10/2023, 2:02 PMtask_logs:
plugins:
logs:
templates:
- displayName: <name-to-show>
templateUris:
- ...
messageFormat: "json" # "unknown" | "csv" | "json"
However, if one sets a messageFormat
, flytepropeller
crashes with this error message:
Use "flyte-propeller [command] --help" for more information about a command.
time="2023-05-10T13:44:33Z" level=error msg="\n\n\n1 error(s) decoding:\n\n* cannot parse 'templates[0].messageFormat' as int: strconv.ParseInt: parsing \"json\": invalid syntax"
Without messageFormat
everything works.
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
05/10/2023, 2:26 PM<https://github.com/flyteorg/flyteidl/tree/master|master>
by EngHabu
<https://github.com/flyteorg/flyteidl/commit/5e89823156a3dd616f57d8c9ac7495bf15b8ede6|5e898231>
- Rename user_identity to execution_identity (#402)
flyteorg/flyteidlGitHub
05/10/2023, 2:55 PMScreenshot 2023-05-10 at 9 33 29 AM▾
Screenshot 2023-05-10 at 9 33 18 AM▾
GitHub
05/10/2023, 3:05 PMGitHub
05/10/2023, 4:40 PMGitHub
05/10/2023, 6:26 PMfrom flytekit import kwtypes, task
from flytekit.extras.tasks.shell import OutputLocation, ShellTask
from flytekit.types.file import FlyteFile
st = ShellTask(
name="shell_task",
debug=True,
script="""
touch {inputs.filename}
""",
inputs=kwtypes(filename=str),
output_locs=[OutputLocation(var="outfile", var_type=FlyteFile, location="{inputs.filename}")],
)
@task
def t1_task( ) -> FlyteFile:
outfile = st(filename='test.txt')
return outfile
if __name__ == "__main__":
print(f"let's run a shelltask and return a FlyteFile")
outfile = t1_task()
flyteorg/flyteGitHub
05/10/2023, 8:21 PMGitHub
05/10/2023, 8:26 PMGitHub
05/10/2023, 9:06 PMGitHub
05/10/2023, 9:07 PM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by ursucarina
<https://github.com/flyteorg/flyteconsole/commit/5caa0abc12acad8e12acd58f039c97ff0cd9e3e7|5caa0abc>
- fix: content panel captures focus (#754)
flyteorg/flyteconsoleGitHub
05/10/2023, 9:39 PM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by jsonporter
<https://github.com/flyteorg/flyteconsole/commit/29babaa4d9a814f130b055da8bdc2ebdd101a667|29babaa4>
- feat: runtime metrics UI (#747)
flyteorg/flyteconsoleGitHub
05/10/2023, 9:54 PMGitHub
05/10/2023, 10:03 PM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by ursucarina
<https://github.com/flyteorg/flyteconsole/commit/a7d7e1c2c351fc6d99106f9a2b2a1d0c64f639e2|a7d7e1c2>
- chore: remove disable_auth from docs (#753)
flyteorg/flyteconsoleGitHub
05/10/2023, 10:25 PMGitHub
05/10/2023, 10:29 PM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by ursucarina
<https://github.com/flyteorg/flyteconsole/commit/d985dfd158b513f057dc2ae7a9f20a387d716f7f|d985dfd1>
- fix: missing dependency (#755)
flyteorg/flyteconsoleGitHub
05/10/2023, 11:07 PMGitHub
05/11/2023, 3:50 AM<https://github.com/flyteorg/flytekit/tree/master|master>
by peridotml
<https://github.com/flyteorg/flytekit/commit/26d1f290ab9e800acfd9af08595a82c3b9f962b6|26d1f290>
- [Issue-3617] Enables FlyteFiles, FlyteDirectors, and StructuredDatasets inputs in papermill plugin (#1612)
flyteorg/flytekitGitHub
05/11/2023, 4:32 AMGitHub
05/11/2023, 4:54 AMGitHub
05/11/2023, 7:32 AMflytekit.task
are not hinted as PythonFunctionTask
as a result of its return typing, so there are spurious type errors when attempting to register such tasks on FlyteRemote
objects.
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
Here you can see an explanation of the problem and the fix:
import flytekit
import flytekit.configuration as flyte_config
import flytekit.remote
@flytekit.task
def my_task1() -> int:
return 0
@flytekit.task()
def my_task2() -> int:
return 0
@flytekit.task(task_config=5)
def my_task3() -> int:
return 0
my_task4 = flytekit.task(lambda: 0)
remote = flytekit.remote.FlyteRemote(...) # type: ignore
serialization_settings = flyte_config.SerializationSettings(...) # type: ignore
# before
reveal_type(my_task1) # Type of "my_task1" is "((...) -> Any) | PythonFunctionTask"
remote.register_task(my_task1, serialization_settings) # error: Argument of type "((...) -> Any) | PythonFunctionTask" cannot be assigned to parameter "entity" of type "PythonTask" in function "register_task"
reveal_type(my_task2) # Type of "my_task2" is "() -> int"
remote.register_task(my_task2, serialization_settings) # error: Argument of type "() -> int" cannot be assigned to parameter "entity" of type "PythonTask" in function "register_task"
reveal_type(my_task3) # Type of "my_task3" is "() -> int"
remote.register_task(my_task3, serialization_settings) # error: Argument of type "() -> int" cannot be assigned to parameter "entity" of type "PythonTask" in function "register_task"
reveal_type(my_task4) # Type of "my_task4" is "((...) -> Any) | PythonFunctionTask"
remote.register_task(my_task4, serialization_settings) # Argument of type "((...) -> Any) | PythonFunctionTask" cannot be assigned to parameter "entity" of type "PythonTask" in function "register_task"
# after
reveal_type(my_task1) # Type of "my_task1" is "PythonFunctionTask"
remote.register_task(my_task1, serialization_settings) # OK
reveal_type(my_task2) # Type of "my_task2" is "PythonFunctionTask[T@task]"
remote.register_task(my_task2, serialization_settings) # OK
reveal_type(my_task3) # Type of "my_task3" is "PythonFunctionTask[int]"
remote.register_task(my_task3, serialization_settings) # OK
reveal_type(my_task4) # Type of "my_task4" is "PythonFunctionTask"
remote.register_task(my_task4, serialization_settings) # OK
I accomplished this fix by using typing.overload
to hint a different return type based on whether _task_function
is passed to flytekit.task
or not.
If nothing is passed as in the case of my_task2
or my_task3
, we hint the return type as Callable[[Callable[..., Any]], PythonFunctionTask[T]]
... that is a callable that accepts a single argument that is any function and returns a PythonFunctionTask
.
If something is passed for _task_function
as in my_task1
or my_task4
, we hint the return type as PythonFunctionTask[T]
.
Tracking Issue
N/A
Follow-up issue
N/A
flyteorg/flytekit
Codecov: 71.20% (-0.02%) compared to e44b802
Codecov: 11.11% of diff hit (target 71.22%)
✅ 28 other checks have passed
28/30 successful checksGitHub
05/11/2023, 7:43 AMGitHub
05/11/2023, 8:11 AMAnnotated[FlyteFile, HashMethod(...)]
.
See the discussion on Slack for additional details.
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
Using an annotated FlyteFile type as an input to a task was previously impossible due to an exception being raised in FlyteFilePathTransformer.to_python_value
.
This MR applies the fix previously used in FlyteFilePathTransformer.to_literal
to permit using annotated FlyteFiles as either inputs or outputs of a task (see MR #1544), by unwrapping the origin type of the annotated type.
Tracking Issue
flyteorg/flyte#3424
Follow-up issue
NA
flyteorg/flytekit
✅ All checks have passed
2/2 successful checksGitHub
05/11/2023, 9:28 AMGitHub
05/11/2023, 1:57 PM