Hello everyone, Is it possible to use Git hooks with Flyte🤔? Essentially, I want to do is to trigger a workflow when there is any modifications to let say a yaml file. Thanks!

Is there a way to exclude certain arguments in a

@flytekit.dynamic

task from being part of the cache calculation? Is the alternative approach described in https://github.com/flyteorg/flyte/issues/3058 (using Annotated + HashMethod) the current best way to do it?

👋 we launched a workflow on the UI setting max parallelism and the Flyte UI in the execution shows the

Parallelism

attribute as

5

but the workflow executed with more than 5 tasks in parallel. Are we misunderstanding the parallelism attribute or is there a bug here?

I'm currently trying to enhance the functionality of my tasks by decorating tasks. I am trying to use FlyteFile within the decorators themselves. The key thing is that the decorators will not be returning the

FlyteFile

, but the output of the function which it's decorating. For example:

def my_decorator(fn):
    @wraps(fn)
    def _wrapper(*args, **kwargs):
        out = fn(*args, **kwargs)
        ... # process and save 'out' to some local file 
        file_handler = FlyteFile(path=some_path, remote_path=remote_path)
        return out
    return _wrapper

@task
@my_decorator
def some_func(value: int) -> int:
    return value * 2

However, it seems that when I use it in this way, the file does not get uploaded to the

remote_path

specified. I'm not very clear about how the internals of

FlyteFile

works, but I was wondering if the file will only be uploaded if the

FlyteFile

were created and returned directly within the decorated function itself? Is there any other way that I could still get this to work within my decorator function instead? Thanks!

Hello, I am running bigquery task locally. I am using the latest flytekit 1.7.0 and I have flytekitplugins-bigquery 1.7.0 and its dep *google-cloud-bigquery==*3.11.3, fsspec==2023.3.0 etc. installed locally. The bigquery_task succeeded in retrieving dataset from bigquery via select statement. However, it failed to convert it to pd.DataFrame in a subsequent task. error: Protocol not known: bq stack trace:

/venv/lib/python3.8/site-packages/fsspec/registry.py:209 in get_filesystem_class
   if protocol not in registry:
        if protocol not in known_implementations:
           raise ValueError("Protocol not known: %s" % protocol)

My code:

# from typing import Tuple
try:
    from typing import Annotated
except ImportError:
    from typing_extensions import Annotated


import pandas as pd
from flytekit import task, workflow, StructuredDataset, kwtypes
from flytekitplugins.bigquery import BigQueryConfig, BigQueryTask
import google.cloud.bigquery


bigquery_task = BigQueryTask(
    name="sql.bigquery.test",
    inputs=kwtypes(version=int),
    query_template="SELECT * FROM `bigquery-public-data.crypto_dogecoin.transactions` WHERE version = @version LIMIT 2;",
    task_config=BigQueryConfig(ProjectID=""),
    output_structured_dataset_type=pd.DataFrame

)

@task
def preproc(df: pd.DataFrame) -> None:
    print(df.head())


@workflow
def wf(version: int = 1) -> None:
    preproc(df = bigquery_task(version=version))

Do you have any idea?

Hello, I’m trying to install flyte cluster in existing EKS using instruction in https://docs.flyte.org/en/v1.0.0/deployment/aws/manual.html but I found one of the pod failed to start kubectl -n flyte get pod NAME READY STATUS RESTARTS AGE datacatalog-86f5c9c887-9l4lc 1/1 Running 0 23m datacatalog-86f5c9c887-ldcn6 1/1 Running 0 23m flyte-pod-webhook-5c67fbfb8c-v4sxc 1/1 Running 0 23m flyteadmin-66996975b9-5l9tm 1/1 Running 0 23m flyteadmin-66996975b9-sr45q 1/1 Running 0 23m flyteconsole-757c979578-97rbr 1/1 Running 0 23m flyteconsole-757c979578-s965v 1/1 Running 0 23m flytepropeller-7bcb7fc885-rpht9 0/1 CrashLoopBackOff 9 (2m1s ago) 23m flytescheduler-985cd58f5-2pqq5 1/1 Running 0 23m syncresources-86dd57b889-xr5w2 1/1 Running 0 23m in the log of pod: kubectl -n flyte logs flytepropeller-7bcb7fc885-rpht9 time=“2023-06-30T122505Z” level=info msg=------------------------------------------------------------------------ time=“2023-06-30T122505Z” level=info msg=“App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2023-06-30 122505.883856281 +0000 UTC m=+0.048754070]” time=“2023-06-30T122505Z” level=info msg=------------------------------------------------------------------------ time=“2023-06-30T122505Z” level=info msg=“Detected: 4 CPU’s\n” {“json”{},“level”“warning”,“msg”:“defaulting max ttl for workflows to 23 hours, since configured duration is larger than 23 [23]“,”ts”“2023 06 30T1225:06Z”} {“json”{},“level”“warning”,“msg”:“stow configuration section missing, defaulting to legacy s3/minio connection config”,“ts”“2023 06 30T1225:06Z”} I0630 122506.541469 1 leaderelection.go:248] attempting to acquire leader lease flyte/propeller-leader... I0630 122506.571316 1 leaderelection.go:258] successfully acquired lease flyte/propeller-leader {“json”{},“level”“error”,“msg”:“Cannot set default plugin [agent-service] for task types [[bigquery_query_job_task]] when it is not configured to be an enabled plugin. Please double check the flytepropeller config.“,”ts”“2023 06 30T1225:06Z”} {“json”{},“level”“error”,“msg”:“Failed to finalize enabled plugins. Error: cannot set default plugin [agent-service] for task types [[bigquery_query_job_task]] when it is not configured to be an enabled plugin”,“ts”“2023 06 30T1225:06Z”} {“json”{},“level”“panic”,“msg”:“cannot set default plugin [agent-service] for task types [[bigquery_query_job_task]] when it is not configured to be an enabled plugin”,“ts”“2023 06 30T1225:06Z”} panic: (*logrus.Entry) 0xc0008b4850

Hi, Im running into an issue when trying to set up a dev environment by following this guide: https://github.com/flyteorg/flyte/pull/3811 I have the demo environment with Flyte running (all Pods Running with

flytectl demo start --dev

), and was able to

make compile

and test all the different components. Now on the final step when running

flyte start --config flyte_local.yaml

it fails, due to wanting to access

/home/{user}/.flyte/k3s/k3s.yaml

which is a broken symlink (as described here as well: https://github.com/flyteorg/flyte/issues/3645)

❯ flyte start --config flyte_local.yaml
INFO[0000] Using config file: [flyte_local.yaml]        
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins] updated. No update handler registered.","ts":"2023-06-30T14:39:46+02:00"}
...
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [webhook] updated. No update handler registered.","ts":"2023-06-30T14:39:46+02:00"}
{"json":{"src":"start.go:63"},"level":"info","msg":"Running Database Migrations...","ts":"2023-06-30T14:39:46+02:00"}
{"json":{"src":"start.go:124"},"level":"error","msg":"Failed to create controller manager. error building Kubernetes Clientset: Error building kubeconfig: stat /home/user/.flyte/k3s/k3s.yaml: no such file or directory","ts":"2023-06-30T14:39:46+02:00"}
{"json":{"src":"start.go:185"},"level":"panic","msg":"Failed to start Propeller, err: error building Kubernetes Clientset: Error building kubeconfig: stat /home/user/.flyte/k3s/k3s.yaml: no such file or directory","ts":"2023-06-30T14:39:46+02:00"}
panic: (*logrus.Entry) 0xc000510380

goroutine 61 [running]:
<http://github.com/sirupsen/logrus.(*Entry).log(0xc000510310|github.com/sirupsen/logrus.(*Entry).log(0xc000510310>, 0x0, {0xc0014de0a0, 0x9f})
        /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:259 +0x487
<http://github.com/sirupsen/logrus.(*Entry).Log(0xc000510310|github.com/sirupsen/logrus.(*Entry).Log(0xc000510310>, 0x0, {0xc00105be68?, 0x1?, 0x1?})
        /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:293 +0x4f
<http://github.com/sirupsen/logrus.(*Entry).Logf(0xc000510310|github.com/sirupsen/logrus.(*Entry).Logf(0xc000510310>, 0x0, {0x3145be9?, 0x0?}, {0xc000a8e230?, 0x0?, 0x0?})
        /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:338 +0x85
<http://github.com/sirupsen/logrus.(*Entry).Panicf(0x3fdac40|github.com/sirupsen/logrus.(*Entry).Panicf(0x3fdac40>?, {0x3145be9?, 0x416947?}, {0xc000a8e230?, 0x2a74b80?, 0x1?})
        /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:376 +0x34
<http://github.com/flyteorg/flytestdlib/logger.Panicf({0x3fdac40|github.com/flyteorg/flytestdlib/logger.Panicf({0x3fdac40>?, 0xc000c55310?}, {0x3145be9, 0x22}, {0xc000a8e230, 0x1, 0x1})
        /root/go/pkg/mod/github.com/flyteorg/flytestdlib@v1.0.17/logger/logger.go:188 +0x64
<http://github.com/flyteorg/flyte/cmd/single.glob..func4.2()|github.com/flyteorg/flyte/cmd/single.glob..func4.2()>
        /home/user/git/flyte/flyte/cmd/single/start.go:185 +0xbe
<http://golang.org/x/sync/errgroup.(*Group).Go.func1()|golang.org/x/sync/errgroup.(*Group).Go.func1()>
        /root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x64
created by <http://golang.org/x/sync/errgroup.(*Group).Go|golang.org/x/sync/errgroup.(*Group).Go>
        /root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa5

Is it possible to limit access to particular emails using the built-in authorization server?

Simple, but from python how can I grab all executions in a

project

x

domain

that are in the state

Running

? I was looking over the

FlyteRemote

docs but didnt see any batch version of

fetch_execution

with any filters or anything.

Does anyone have experience with TLS/SSL between the Ingress and Flyte? 🔒 🔒 I’m currently trying to get a reference implementation to work for deploying Flyte on GCP with the GKE Ingress controller (instead of nginx) wich would allow us to use Google Identity Aware Proxy (IAP) for authentication. IAP requires the user to login “at the load balancer”, before a request reaches the backend, instead of being redirected to the login screen by the backend. Because this is more secure it’s a requirement by our company (and actually also my previous one) to use IAP. To make gRPC backends work with the GKE Ingress, HTTP2 needs to be enabled for the backend. Flyte already does so in the gcp helm values. Using HTTP2 between the load balancer and the backend requires TLS between lb and backend. So I’m trying to deploy Flyte with this config:

configmap:
  adminServer:
    server:
      httpPort: 8088
      grpcPort: 8089
      security:
        secure: false #true
        ssl:
          certificateFile: "/etc/tls/cert.pem"
          keyFile: "/etc/tls/key.pem"

Certificate is self-signed and mounted into the flyteadmin pod via a secret. I can’t talk to flyteadmin though and would appreciate help getting this to work 🙂 Details in 🧵

hello all, I have

remote = FlyteRemote(
    config=Config.for_endpoint(endpoint='localhost:30080'),
    default_project="flytesnacks",
    default_domain="development",
)

flyte_wf = remote.fetch_workflow(
    name="flyte.workflows.extract_flyte.extract_drs_data",
    project="flytesnacks",
    domain='development',
)

flyte demo running locally fine

but when I run this i get

details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30080: Ssl handshake failed: SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30080: Ssl handshake failed: SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER {created_time:"2023-06-30T16:35:10.548801-04:00", grpc_status:14}"

I ran flyte

Flyte is ready! Flyte UI is available at <http://localhost:30080/console> 🚀 🚀 🎉

curl localhost:30080/api/v1/projects

works too 🤔

@Yue Shang has left the channel

Hi! I have a question regarding flyte support for tensorflow data. I have generated a

tf.data.Dataset

in one of the tasks and return it as input for next task, but got data type error (when pickle serialize this data)`tensorflow.python.framework.errors_impl.InvalidArgumentError'>: Cannot convert a Tensor of dtype variant to a NumPy array` , I'm kind of new to both tensorflow and flyte, so wondering is there any guidance on how to resolve this issue?

Hi there! How to increase the allowed output size of the node?

RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [k8s-array]: output file @[] is too large [255540150] bytes, max allowed [10485760] bytes

I am evaluating Flyte for data engineering. When is Flyte not a good choice (compared to Dagster - another candidate)? The data science group uses Metaflow and they are actively advocating for us to adopt it too. Is Flyte comparable to data science tools like Metaflow apart from the common workflow orchestration bit?

Hi ! I am working on a task that is gonna be run in map task config, each instance will generate few files and will store them in /tmp/some_path, it returns a flyte directory from that path. however I keep getting this error:

Cannot convert from /tmp/some_path to typing.Optional[flytekit.types.directory.types.FlyteDirectory]\n\nSYSTEM ERROR! Contact platform administrators."

please note that sometimes it works . does it have to use context path instead ?

Hey all, I programmatically launch multiple executions and store the FlyteWorkflowExecution objects in a map. Later I iterate over them to check FlyteWorkflowExecution.is_done, but it’s always False, even when the executions have finished with success or failed. The .closure.phase is always 0, too (if it matters). Has anyone used is_done before to track workflows status, and can check if I’m doing it right? Code attached inside this thread, thanks

Looks like some documentation links are broken. I was on this page and when I opend

activates or deactivates

the schedule url, I got 404 error.

#ask-the-community We have been using Flyte in our production cluster for the past 6 months and now we are experiencing slowness in the system which we can see in the logs with the below messages SLOW SQL >= 200ms *[237.313ms] [rows:0] SELECT * FROM "node_executions" WHERE "node_executions"."execution_project" = 'samuel-predictive-handover' AND "node_executions"."execution_domain" = 'development' AND "node_executions"."execution_name" = 'a8ktq27nxlfzg2xd95w7' AND "node_executions"."node_id" = 'n3' LIMIT 1* Are there any housekeeping instructions to delete workflows off of the S3 as well as postgres DB for workflow executions older than 30 days so that we could manage the above issue???

In addition to the above, there have been many instances of flytectl running to create a namespace but it shows up in the k8s system only after 2-3 minutes. The delay is causing issues in the project deployment process. in which we use flytectl to deploy project in case the namespace doesnt exist it will create else it will register

we recently upgraded our flyte binary in EKS to

1.7.0

. we started the process of updating our docker images to use the latest

flytekit

as well with:

RUN pip install boto3 awscli flytekit==1.7.0

but we’re getting conflicting dependencies related to `fsspec`:

RROR: Cannot install flytekit and flytekit==1.7.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    flytekit 1.7.0 depends on fsspec>=2023.3.0
    s3fs 2023.1.0 depends on fsspec==2023.1.0

this works if we downgraded back to

flytekit==1.5.0

Hi, community. I am a software engineer intern from Taiwan who have 1 year of work experience at Synology, National Academia Sinica and Industrial Technology Research Institute. I am interested in being a contributor to flyte, can anyone assign a python issue to me ? Thank you very much. BTW, I am reading the flyte oss roadmap. https://github.com/orgs/flyteorg/projects/3/views/7

Hi quick question, Id like to populate a variable that is used as environment variable in a task from a configuration input; something like:

FOO = "some_default_value"

@task(environment={"FOO": FOO})
def some_task():
    return "bar"

def populate_vars_from_config(config):
    print(config.keys())
    FOO = config['foo']

@workflow
def some_workflow(config: dict):
    populate_vars_from_config()
    some_task()

And run it with

pyflyte run ./some_path some_workflow --config ./config.json

However that gives me an exception because its a Promise:

Failed with Unknown Exception <class 'AttributeError'> Reason: Error encountered while executing 'some_workflow':
  'Promise' object has no attribute 'keys'

Any suggestions how to approach this in a better way? 🙂 I guess I could just set them myself within the task, but would be nice to have them visible in the decorator.

Hello guys! Does gcs account need to have Delete permissions on buckets to be used in flyte files operations ? getting this error when using

FlyteDirectory(path='path/to/parent', remote_directory='<gs://datasets_bucket/path/to/parent>')

[0]: code:"SYSTEM:Unknown" message:"Traceback (most recent call last):\n\n      File \"/opt/venv/lib/python3.9/site-packages/flytekit/exceptions/scopes.py\", line 165, in system_entry_point\n        return wrapped(*args, **kwargs)\n      File \"/opt/venv/lib/python3.9/site-packages/flytekit/core/base_task.py\", line 588, in dispatch_execute\n        raise TypeError(msg) from e\n\nMessage:\n\n    Failed to convert outputs of task 'workflows.wfname.map_extract_f6482939eed2867de4faf3d9d475fe60' at position 0:\n  Failed to put data from /tmp/flyte-_2xcubb6/sandbox/local_flytekit/d8544a75c07b4b0a894056feafcf0bee/path/to/parent to <gs://datasets_bucket/path/to/parent> (recursive=True).\n\nOriginal exception: Forbidden: <https://storage.googleapis.com/upload/storage/v1/b/datasets_bucket/o>\naccount_name does not have storage.objects.delete access to the Google Cloud Storage object.\n\nSYSTEM ERROR! Contact platform administrators." kind:SYSTEM 
... and many more.

why delete when it is trying to upload? and if the directory path/to/parent does not exist or empty, does it delete contents of gcs corresponding path ?

Crossposting here. https://flyte-org.slack.com/archives/CREL4QVAQ/p1688481913445659

Hi there! I'm trying to register flyte workflow on the demo flyte sandbox I've deployed the sandbox on remove server and it available on domain

flyte.sandbox

This is my

.flyte/config.yaml

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: flyte.sandbox:30080
  authType: Pkce
  insecure: true
console:
  endpoint: <http://flyte.sandbox:30080>
logger:
  show-source: true
  level: 0
storage:
  connection:
    endpoint: flyte.sandbox:30002

But I got an error, with cmd

pyflyte register workflows

pyflyte register workflows                      
Running pyflyte register from /Users/dqunbp/projects/test-project/example-flyte with images ImageConfig(default_image=Image(name='default', fqn='<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>', tag='py3.10-1.6.1'), images=[Image(name='default', fqn='<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>', tag='py3.10-1.6.1')]) and image destination folder /root on 1 package(s) ('/Users/dqunbp/projects/sbxrmxd/example-flyte/workflows',)
Registering against flyte.sandbox:30080
Detected Root /Users/dqunbp/projects/test-project/example-flyte, using this to create deployable package...
No output path provided, using a temporary directory at /var/folders/nx/scmwbdl92j12dxc0wh1gkljw0000gn/T/tmp_87706wb instead
Failed with Unknown Exception <class 'requests.exceptions.ConnectionError'> Reason: HTTPConnectionPool(host='localhost', port=30002): Max retries exceeded with url: /my-s3-bucket/flytesnacks/development/RSXYOG4EB6Y3JGW235M236VHII%3D%3D%3D%3D%3D%3D/fastf5549d7958839100393a4ea553562a7c.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230705%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230705T053532Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=21dff243075588b3185a3863034a8cfb3d373459eb7081ea348bb71e78b079f4 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x110d04850>: Failed to establish a new connection: [Errno 61] Connection refused'))
HTTPConnectionPool(host='localhost', port=30002): Max retries exceeded with url: /my-s3-bucket/flytesnacks/development/RSXYOG4EB6Y3JGW235M236VHII%3D%3D%3D%3D%3D%3D/fastf5549d7958839100393a4ea553562a7c.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230705%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230705T053532Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=21dff243075588b3185a3863034a8cfb3d373459eb7081ea348bb71e78b079f4 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x110d04850>: Failed to establish a new connection: [Errno 61] Connection refused'))

Is it something wrong with my config.yaml ?