https://flyte.org logo
#flytekit
Title
# flytekit
b

Babis Kiosidis

05/05/2022, 10:18 AM
Hey I am trying to pass a list of string arguments in a ContainerTask and add all of them in the
arguments
field but struggling a bit with understanding if this is possible. ๐Ÿงต
I have a generic ContainerTask that could receive various arguments
Copy code
def dbeam_ct():
    return ContainerTask(
        "dbeam",
        image="...",
        input_data_dir="/var/inputs",
        output_data_dir="/var/outputs",
        inputs=kwtypes(
            when=datetime.datetime,
            docker_container_args=List[str],
        ),
        command=["some-command],
        arguments=["--partition={{.Inputs.when}}"] + "{{.Inputs.docker_container_args}}",
    )
is there a way i could pass the docker_container_args as a list of strings, and not as a string itself?
the error i see is
Copy code
TypeError: can only concatenate list (not "str") to list
Inputs will look like this:
Copy code
lp = LaunchPlan.create(
    "...",
    wf,
    fixed_inputs={
        "dockercontainer_args": [
            "--partitionColumn=whatever",
            "--output=<gs://some-bucket>",
            "--connectionUrl=jdbc:postgresql://...",
            "--table=executions",
        ]
    },
)
notice the inputs List[str]
since there can be any number of inputs to this docker container, i would prefer if i didnt have to configure default values for each, but the user could define them somehow in the Launch Plan inputs
Should i wrap them in a workflow instead?
k

Ketan (kumare3)

05/05/2022, 4:26 PM
@Babis Kiosidis you should be able to. you will have to read it and parse it from json yourself
@Babis Kiosidis at the moment, a collection is simply handled as a collection of files - https://github.com/flyteorg/flytecopilot/blob/43938f5382ae580639bb5934199c7ea7566b027d/data/download.go#L251. One per item in the collection
I think this can be improved, by handling list of primitive values differently
b

Babis Kiosidis

05/06/2022, 7:17 AM
ah you mean that I will need it and parse it in the container?
Should we create an issue for flytecopilot? I believe it's a solid use case for us (500+ workflows), there are a lot of teams that use docker images for certain operations already. In the non-flyte system we just schedule a docker image execution with any arguments passed from the user One good example is https://github.com/spotify/dbeam
Ideally we would like to also configure ENV VARs based on inputs. Which i think might not be possible right now
should i just try to do that by using the K8sPod plugin?
k

Ketan (kumare3)

05/06/2022, 1:52 PM
You can use k8s pod plugin, but won't be able to do that with pod too right?
Would you be open to discussing the usecase
Copilot needs little more love
Would love to understand how it can solve your usecase and I think we would be more than happy to adapt
b

Babis Kiosidis

05/06/2022, 2:34 PM
Yeah I'm up for discussion
k

Ketan (kumare3)

05/06/2022, 3:37 PM
@Babis Kiosidis let me know if you are around
b

Babis Kiosidis

05/10/2022, 2:12 PM
do you have some time today?
k

Ketan (kumare3)

05/10/2022, 2:33 PM
How about 8:45 PDT ish?
b

Babis Kiosidis

05/10/2022, 2:52 PM
sure it works. Earlier works even better as well ๐Ÿ‘
k

Ketan (kumare3)

05/10/2022, 3:20 PM
Earlier I drop my daughter
b

Babis Kiosidis

05/10/2022, 3:20 PM
๐Ÿ‘
79 Views