Hi all, building my first more complex flyte workflow and I have a couple of questions on best practices. I have a workflow defined in

project_1

that includes a dynamic workflow. The dynamic workflow starts a launchplan defined in

project_2

(see attached sketch). This is my code structure:

|- project_1 /
|  |- pyproject.toml
|  |- project_1 /
|  |  |- __init__.py
|  |  |- workflows.py
|  |  |- tasks.py
|  |  |- models.py
|  |  |- ...
|- project_2 /
|  |- pyproject.toml
|  |- project_2 /
|  |  |- __init__.py
|  |  |- workflows.py
|  |  |- tasks.py
|  |  |- models.py
|  |  |- ...
|- Dockerfile
|- docker_build.sh

What I'm doing right now: • add

project_2

as a dependency to project one • build a docker image with

project_2

installed as a dependency to project 1 • package + register • run This approach requires rebuilding the docker image with every execution because fast register will not realise that the launchplan defined in

project_2

should have a different version than the rest of the workflow and it fails while trying to fetch the workflow. Another option would be to do something like:

remote = FlyteRemote(config=Config.auto())
launchplan = remote.fetch_launch_plan(...)

but that would require my pod to be able to reach flyteadmin because it's inside a dynamic workflow, Is there anything I can do to avoid rebuilding the docker image for every run? Am I doing something wrong? Any advice is much appreciated. Another issue is that the way I run things now makes the UI fail to expand the dynamic workflow and doesn't provide any link to the newly spawned workflows so I can't reliably track the intermediate states or see why something failed (if it did). (see screenshot, the purple task is dynamic). It also leads to all sorts of mixed status reports (see other screenshot).

Hi team.. so flytekit needs pyarrow >= 4.0.0 except pyarrow (required by awswrangler) has this issue for versions > 2.0.0 that I’m currently running into when trying to chunk and read a large parquet file.. Has anyone else in the group encountered this and if yes how did you get around it?

Hello! I'd like to set up Flyte on Coreweave. Is there documentation available for setting up Flyte on non-AWS, non-GCP k8s clusters?

#CP2HDHKE1 Has anyone tried installing Flyte in a service mesh like Istio. What I am trying is to setup Istio with an oauth provider which then authenticates users to flyte....

Basically, input variable client is not a data json but a custom class object, therefore is not appropriate to be an input or output in a task is what I figured. How do I initialize a shared object in the workflow script to be used by all the tasks?

anyone else use flytectl with M1 chips? getting a segfault when running

Cross posting from https://flyte-org.slack.com/archives/C01P3B761A6/p1654537014728059 to see if anybody here might know 🙂 . I tried restarting the

flytepropeller

deployment and it’s hard to tell if the system is picking up the changes:

$ kubectl logs -f deployment/flytepropeller -nflyte | grep -i logs
Found 6 pods, using pod/flytepropeller-785bcc6f6d-sr8bl
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.logs] updated. No update handler registered.","ts":"2022-06-08T16:32:15Z"}
{"json":{"src":"viper.go:396"},"level":"debug","msg":"Config section [plugins.logs] hasn't changed.","ts":"2022-06-10T22:36:33Z"}

From looking at the plugins — it seems like adding that config should just automatically work

I archived my launch plans with

flytectl update launchplan -p flytesnacks -d development <launch plan name> --version <launch plan version> --archive

There's a success message "updated launchplan successfully on <launch plan name>" and the Flyte console doesn't show anything under Schedules anymore but my workflow is still being executed every hour. Am I missing something or is this a bug?

I seem to get this error often. I’m trying

hyper_parameters: Tuple[int, int, float, float, float]

raise RestrictedTypeError(f"Transformer for type {self.python_type} is restricted currently")
flytekit.core.type_engine.RestrictedTypeError: Transformer for type <class 'tuple'> is restricted currently

Hi all! Could you please help me - can i use PythonPickledFile between raw containers? Right now i am trying to use this data type with method described here https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/raw_container.html and getting

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

when trying to read them by pickle.load(file): inside

with open(f'{input_dir}/dump_file') as file:
        raw_data = pickle.load(file)

it is saved without errors by:

with open(f'{output_dir}/dump_file', 'wb') as handle:
        pickle.dump(raw_data, handle)

by previous stage raw container

stage1 = ContainerTask(
    name="stage1",
    output_data_dir="/var/outputs",
    outputs=kwtypes(generation=PythonPickledFile),
    image="<http://registry.project.com/flyte/stage1:0.0.1|registry.project.com/flyte/stage1:0.0.1>",
    command=[
        "python",
        "stage1.py",
        "/var/outputs",
    ],
)

stage2 = ContainerTask(
    name="stage2",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(dump_file=PythonPickledFile),
    outputs=kwtypes(stage2_out=str),
    image="<http://registry.project.com/flyte/stage2:0.0.1|registry.project.com/flyte/stage2:0.0.1>",
    command=[
        "python",
        "stage2.py",
        "/var/inputs",
        "/var/outputs",
    ],
)

@workflow
def my_wf() -> str:
    generation = stage1()
    return stage2(dump_file=generation)

Can i pass PythonPickledFile like that?