https://flyte.org logo
Join Slack
Powered by
Hi Flyte support team, I have a question now I'm ...
# flyte-support
n
Hi Flyte support team, I have a question now I'm trying to invoke AWS Batch jobs from the Flyte workflow, but I'm facing issues - the page shows here only https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-aws-batch/flytekitplugins/awsbatch/task.py like what if I want to have give my own job definition and job queue - there's no support for that?
f
there is no flyte support team
flyte is a community and folks here support out of free will 😄 - ❤️ thank you for being part of the community
n
Thank you Ketan for the heads up!, can you connect me with someone here who can help me with invoking AWS Batch jobs from my workflows I'm going through the github repo and the documentation but somehow I'm not able to do it
e
you need to extend the plugin for that if its missing, its super easy to extend though
n
Thank you Vikas, can you please explain more
like here how do you specify the job queue and job definition - do you specify it in the helm chart but then its not dynamic everytime I have to do a deployment from flytekitplugins.awsbatch import AWSBatchConfig config = AWSBatch( parameters={"codec": "mp4"}, platformCapabilities="EC2", propagateTags=True, retryStrategy={"attempts": 10}, tags={"hello": "world"}, timeout={"attemptDurationSeconds": 60}, ) @task(task_config=config) def t1(a: int) -> str: return str(a) https://www.union.ai/docs/flyte/integrations/external-service-backend-plugins/aws_batch_plugin/
e
you need to check the fflyte operator code which uses flyteidl as protobuf and generate the python code there since you are missng it there it wont understand, hope it clarifies
n
still not clear, do you have a documentation page, further what's the production way to invoke AWS Batch jobs from Flyte tasks in the workflow
I just used boto3 inside the task to connect to aws batch - but is this the production way? @task(container_image="123440395442.dkr.ecr.us-west-1.amazonaws.com/flyte-repo:boto3-v1") def call_batch_job() -> str: import boto3 batch = boto3.client("batch") batch.submit_job( jobName="my-flyte-job", jobQueue="main-queue", jobDefinition="sample-batch-job-def:1" ) return "Job submitted!" @workflow def wf() -> str: return call_batch_job()
f
@numerous-pizza-94528 we do have a connector for batch I think
a
@numerous-pizza-94528 here the docs show how to specify the job queue https://www.union.ai/docs/flyte/deployment/flyte-plugins/batch/#update-flyteadmin-configuration
n
Hi @average-finland-92144 yes I tried that way but doing that would be hard coding only a specific job definition and job queue for the pod I tried to use boto3 directly as part of the ImageSpec inside the task to connect to AWS Batch and execute the job
a
@numerous-pizza-94528 I guess that the extent of the current integration. Would you be open to contributing to make it more dynamic?
n
Yes I would like to do that @average-finland-92144, my manager asked me to do a POC on the data orchestration tools like Dagster, Flyte, Airflow and Prefect, we majorly have our AWS Batch jobs which we need to invoke from our workflows. We are planning to migrate to these tools from Step Functions, so far I liked Flyte
a
@numerous-pizza-94528 you could start by adding the queue_name as an additional argument here: https://github.com/flyteorg/flytekit/blob/8abecfe0bd5ecd0423ff814651167b9fc4d76170/plugins/flytekit-aws-batch/flytekitplugins/awsbatch/task.py#L[…]5 and probably extending the 
get_command
function in that same file to accept use the queue_name if provided:
Copy code
if self._task_config.queue_name:
        container_args.extend(["--queue-name", self._task_config.queue_name])
Let me know if you need help
5 Views
Open in Slack
PreviousNext
Powered by