https://flyte.org logo
#ask-the-community
Title
# ask-the-community
d

Denis Skakalski

02/22/2024, 1:24 PM
Hello colleagues, in my work using flyte, I am mostly using http.client module to execute API calls, and everything working good: ------------------------------------------------------ from flytekit import task, workflow import http.client @task def convert(): url = 'app-cloudburstapi-cloudburst-dev.azurewebsites.net' # Create a connection to the server conn = http.client.HTTPSConnection(url) # Send the POST request conn.request('POST', endpoint, json_data, headers) # Get the response response = conn.getresponse() job_id = response.read().decode('utf-8') ----------------------------------------------- Now I am going to implement on another @task the downloading installation of private application and running the application inside of the task (the target of application the download some files in particular working directory, for example "temp") and then I need to install azcopy and using it to download the content of folder "temp" to the Azure file share. The bash script is below: ------------------------------------------------------------------------------ # Download and install private application curl -k --proto '=https' --tlsv1.2 -sSf https://mesh.epam.net/scripts/eratost_install.sh | bash # Download and install azcopy, then cleanning the directory wget https://aka.ms/downloadazcopy-v10-linux tar -xvf downloadazcopy-v10-linux sudo rm -f /usr/bin/azcopy sudo cp ./azcopy_linux_amd64_*/azcopy /usr/bin/ sudo chmod 755 /usr/bin/azcopy # copy the content of folder 'temp' to Azure cloud share azcopy copy './test/*' 'https://stcbartifactsdev.file.core.windows.net/test?sv=2022-11-02&ss=f&srt' -------------------------------------------------------------------------------------- Could you advice, how more nice and cleverly implement it using Flyte. In another words, it is necessary to translate bash script into python using flyte's modules. I am using Flyte cluster in Azure Kubernetes. I just want to pick up the right approach.
s

Samhita Alla

02/23/2024, 6:54 AM
have you tried using
ShellTask
?
d

Denis Skakalski

02/23/2024, 7:28 AM
yes, for example I have tried to execute such piece of code (see the dark screenshot), and eventually I am getting such error:
as I understand it is necessary to install the module wget, I have tried it to do by adding to the command "apt-get install wget", but user flytekit does not have permissions to do it:
Maybe it is possible to install all necessary components during the configuration of container?
s

Samhita Alla

02/23/2024, 2:05 PM
i believe container task might be a better construct to use: https://docs.flyte.org/en/latest/user_guide/customizing_dependencies/raw_containers.html. you can build a docker image and you needn't worry about the dependency installation.
d

Denis Skakalski

02/26/2024, 10:10 AM
look like everything is good, next step to test it on prod. env
hello colleagues, could you advice: I have a container task, where I am using the custom container image and executing some command (azcopy for example). But during the workflow execution, I need to upload some .yml file from my local computer to the container for proceeding inside of container with them. On the picture I need somehow put inside of container the file value.yaml during "pyflyte run"
s

Samhita Alla

02/27/2024, 3:53 PM
you should be able to send a
FlyteFile
to your container task. example:
Copy code
ct = ContainerTask(
    name="sed_task",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(infile=FlyteFile),
    outputs=kwtypes(out=FlyteFile),
    image="alpine",
    command=[
        "/bin/sh",
        "-c",
        "sed 's/o/a/g' /var/inputs/infile > /var/outputs/out",
    ],
)
d

Denis Skakalski

02/28/2024, 11:09 AM
Hi Samhita, thanks for your update: I need to upload the file with name simkit.yaml into the running container and then proceed it. Could you please look if I have correctly created container task and the "pyflute command". As I understand everything from local directory where pyflyte command executing, will be uploaded to the "/var/inputs" directory.
when I have tried to execute it, I am getting the error
s

Samhita Alla

02/28/2024, 11:54 AM
the command should have
infile
and
out
. that's where the files will be available. here's an example you can refer to: https://gist.github.com/pryce-turner/0a67f86febdc812c9a2a9e739c22eeca
d

Denis Skakalski

02/29/2024, 10:34 AM
hello colleagues, could you advice where am I wrong. My target is to upload the simkit.yaml file into the some internal directory during the container executing and then I am going to work with this filr. As I understand it should be upload into the "/var/inputs", but could you please look, what is wrong with the command:
s

Samhita Alla

02/29/2024, 12:13 PM
your simkit.yaml will be available as
infile
and that's how you should access it in the command. have you seen the example i shared here?
also, what's the error you're seeing?
d

Denis Skakalski

02/29/2024, 12:17 PM
my target is to upload simkit.yaml inside of the container (during container task executing) as a file, just read the content is not enough
s

Samhita Alla

02/29/2024, 12:37 PM
okay but why don't i see the file being used in your command?
d

Denis Skakalski

02/29/2024, 3:03 PM
I temporary downloading it from share, will look into it later: but now, when I am trying to execute next task (simulate) after container, it is getting OOMKIlled. Where can I adjust resources?
13 Views