https://flyte.org
Join Slack
is there documentation on how to use container tasks beyond <https://docs.flyte.org/projects/cookboo...
# announcements
r
is there documentation on how to use container tasks beyond https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/raw_container.html . I'm trying with the sandbox env but not having luck. I've heard that it depends on the SYS_PTRACE capability, which may depend on one's K8s setup
That might not even be the issue. I'm getting this screenshot when I try to run it
...with no logs
the task definition looks like this
Copy code
fastqc_raw = ContainerTask(
    name="fastqc_container",
    image="<http://docker.io/staphb/fastqc:latest|docker.io/staphb/fastqc:latest>",
    command=[
        "fastqc", "--version", ">", "/var/outputs/version"
    ],
    outputs=kwtypes(version=str),
    output_data_dir="/var/outputs",
)
h
@ripe-sunset-39556, where did you hear about the need for 
SYS_PTRACE
? Can you replace the command you pass to the container with:
Copy code
command=[
        "sh",
        "-c",
        "fastqc --version | tee /var/outputs/version",
    ],
r
from Ketan, but I'll try what you have
same deal. No luck
h
interesting. Is this the only 
ContainerTask
in your workflow?
r
yeah
h
weird, I just tested this workflow in a brand new sandbox:
Copy code
from flytekit.core.task import task
from flytekit.core.workflow import workflow
from flytekit import ContainerTask, kwtypes, workflow

fastqc_raw = ContainerTask(
    name="fastqc_container",
    image="<http://docker.io/staphb/fastqc:latest|docker.io/staphb/fastqc:latest>",
    command=[
        "sh",
        "-c",
        "fastqc --version | tee /var/outputs/version",
    ],
    outputs=kwtypes(version=str),
    output_data_dir="/var/outputs",
)

@workflow
def wf() -> str:
    return fastqc_raw()
r
hmm, that is weird. Let me try verbatim what you are doing. It would be nice if there was something else going on
👍 1
ohh...interesting
that did work
that definitely raises some questions
ok, I've made some good progress here. Is there a way to get the raw container input type to be a flytefile? I want the command I call in the container to operate on a file I generate in a previous task
This doesn't work (perhaps unexpectedly) because the fast_qc file I think is just the s3 file path that I pass in from a previous task
Copy code
fastqc_raw = ContainerTask(
    name="fastqc_container",
    image="<http://docker.io/staphb/fastqc:latest|docker.io/staphb/fastqc:latest>",
    command=[
        "sh",
        "-c",
        "fastqc -t 1 -q -o /var/outputs/ /var/inputs/fastqc_file"
    ],
    input_data_dir="/var/inputs",
    inputs=kwtypes(fastqc_file=FlyteFile),
    outputs=kwtypes(out=FlyteDirectory),
    output_data_dir="/var/outputs",
)

@workflow
def wf(path: FlyteFile) -> FlyteDirectory:
    return fastqc_raw(fastqc_file=path)
h
Unfortunately 
FlyteFile
is a flytekit concept, so you'd have to pipe that into your raw container task (and use python to download the file locally). Another option is to have a script to download the file from s3 into your container prior to running 
fastqc
.
👍 1
170 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by