Hello colleagues, in my work using flyte, I am mostly using http.client module to execute API calls, and everything working good: ------------------------------------------------------ from flytekit import task, workflow import http.client @task def convert(): url = 'app-cloudburstapi-cloudburst-dev.azurewebsites.net' # Create a connection to the server conn = http.client.HTTPSConnection(url) # Send the POST request conn.request('POST', endpoint, json_data, headers) # Get the response response = conn.getresponse() job_id = response.read().decode('utf-8') ----------------------------------------------- Now I am going to implement on another @task the downloading installation of private application and running the application inside of the task (the target of application the download some files in particular working directory, for example "temp") and then I need to install azcopy and using it to download the content of folder "temp" to the Azure file share. The bash script is below: ------------------------------------------------------------------------------ # Download and install private application curl -k --proto '=https' --tlsv1.2 -sSf https://mesh.epam.net/scripts/eratost_install.sh | bash # Download and install azcopy, then cleanning the directory wget https://aka.ms/downloadazcopy-v10-linux tar -xvf downloadazcopy-v10-linux sudo rm -f /usr/bin/azcopy sudo cp ./azcopy_linux_amd64_*/azcopy /usr/bin/ sudo chmod 755 /usr/bin/azcopy # copy the content of folder 'temp' to Azure cloud share azcopy copy './test/*' 'https://stcbartifactsdev.file.core.windows.net/test?sv=2022-11-02&ss=f&srt' -------------------------------------------------------------------------------------- Could you advice, how more nice and cleverly implement it using Flyte. In another words, it is necessary to translate bash script into python using flyte's modules. I am using Flyte cluster in Azure Kubernetes. I just want to pick up the right approach.

have you tried using

ShellTask

?

yes, for example I have tried to execute such piece of code (see the dark screenshot), and eventually I am getting such error:

i believe container task might be a better construct to use: https://docs.flyte.org/en/latest/user_guide/customizing_dependencies/raw_containers.html. you can build a docker image and you needn't worry about the dependency installation.

look like everything is good, next step to test it on prod. env

you should be able to send a

FlyteFile

to your container task. example:

ct = ContainerTask(
    name="sed_task",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(infile=FlyteFile),
    outputs=kwtypes(out=FlyteFile),
    image="alpine",
    command=[
        "/bin/sh",
        "-c",
        "sed 's/o/a/g' /var/inputs/infile > /var/outputs/out",
    ],
)

Hi Samhita, thanks for your update: I need to upload the file with name simkit.yaml into the running container and then proceed it. Could you please look if I have correctly created container task and the "pyflute command". As I understand everything from local directory where pyflyte command executing, will be uploaded to the "/var/inputs" directory.

the command should have

infile

and

out

. that's where the files will be available. here's an example you can refer to: https://gist.github.com/pryce-turner/0a67f86febdc812c9a2a9e739c22eeca

hello colleagues, could you advice where am I wrong. My target is to upload the simkit.yaml file into the some internal directory during the container executing and then I am going to work with this filr. As I understand it should be upload into the "/var/inputs", but could you please look, what is wrong with the command:

your simkit.yaml will be available as

infile

and that's how you should access it in the command. have you seen the example i shared here?

my target is to upload simkit.yaml inside of the container (during container task executing) as a file, just read the content is not enough

okay but why don't i see the file being used in your command?

I temporary downloading it from share, will look into it later: but now, when I am trying to execute next task (simulate) after container, it is getting OOMKIlled. Where can I adjust resources?

here's how you can update your task resources: https://discuss.flyte.org/t/16379310/i-m-running-into-the-same-oomkilled-issue-referenced-in-this#ef902dc5-fb68-4cec-bd53-f1309ef2619e