• b

    Babis Kiosidis

    3 months ago
    Hey I am trying to pass a list of string arguments in a ContainerTask and add all of them in the
    arguments
    field but struggling a bit with understanding if this is possible. ๐Ÿงต
  • I have a generic ContainerTask that could receive various arguments
    def dbeam_ct():
        return ContainerTask(
            "dbeam",
            image="...",
            input_data_dir="/var/inputs",
            output_data_dir="/var/outputs",
            inputs=kwtypes(
                when=datetime.datetime,
                docker_container_args=List[str],
            ),
            command=["some-command],
            arguments=["--partition={{.Inputs.when}}"] + "{{.Inputs.docker_container_args}}",
        )
    is there a way i could pass the docker_container_args as a list of strings, and not as a string itself?
  • the error i see is
    TypeError: can only concatenate list (not "str") to list
  • Inputs will look like this:
    lp = LaunchPlan.create(
        "...",
        wf,
        fixed_inputs={
            "dockercontainer_args": [
                "--partitionColumn=whatever",
                "--output=<gs://some-bucket>",
                "--connectionUrl=jdbc:postgresql://...",
                "--table=executions",
            ]
        },
    )
  • notice the inputs List[str]
  • since there can be any number of inputs to this docker container, i would prefer if i didnt have to configure default values for each, but the user could define them somehow in the Launch Plan inputs
  • Should i wrap them in a workflow instead?
  • Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    @Babis Kiosidis you should be able to. you will have to read it and parse it from json yourself
  • @Babis Kiosidis at the moment, a collection is simply handled as a collection of files - https://github.com/flyteorg/flytecopilot/blob/43938f5382ae580639bb5934199c7ea7566b027d/data/download.go#L251. One per item in the collection
  • I think this can be improved, by handling list of primitive values differently
  • b

    Babis Kiosidis

    3 months ago
    ah you mean that I will need it and parse it in the container?
  • Should we create an issue for flytecopilot? I believe it's a solid use case for us (500+ workflows), there are a lot of teams that use docker images for certain operations already. In the non-flyte system we just schedule a docker image execution with any arguments passed from the user One good example is https://github.com/spotify/dbeam
  • Ideally we would like to also configure ENV VARs based on inputs. Which i think might not be possible right now
  • should i just try to do that by using the K8sPod plugin?
  • Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    You can use k8s pod plugin, but won't be able to do that with pod too right?
  • Would you be open to discussing the usecase
  • Copilot needs little more love
  • Would love to understand how it can solve your usecase and I think we would be more than happy to adapt
  • b

    Babis Kiosidis

    3 months ago
    Yeah I'm up for discussion
  • Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    @Babis Kiosidis let me know if you are around
  • b

    Babis Kiosidis

    3 months ago
    do you have some time today?
  • Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    How about 8:45 PDT ish?
  • b

    Babis Kiosidis

    3 months ago
    sure it works. Earlier works even better as well ๐Ÿ‘
  • Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    Earlier I drop my daughter
  • b

    Babis Kiosidis

    3 months ago
    ๐Ÿ‘