I have 2 endpoints : ep1, ep2 ep1 has two projects...
# ask-ai
r
I have 2 endpoints : ep1, ep2 ep1 has two projects e1p1, e1p2 ep2 has two projects e2p1, e2p2 i have to fetch a task(t4 of n1) from a launch plan in ep1 it is as follows start->n0(t0->t1) ==> n1(t0->t1->t2->t3->t4->t5)->end in the above launch plan n0, n1 are the subworkflows what i need to do is to fetch the inputs of a previous execution of the above launchplan for task n1t3 and fetch the task t4 itself from ep1 now use the task fetched above and inputs for a previous particular execution and execute it on endpoint ep2
Copy code
remote1 = FlyteRemote(
        Config.for_endpoint("<http://flyte.us.cloud.ep1.com|flyte.us.cloud.ep1.com>"),
        default_domain=domain,
    )

remote2 = FlyteRemote(
        Config.for_endpoint("<http://flyte.us.cloud.ep2.com|flyte.us.cloud.ep2.com>"),
        default_domain=domain,
    )

executionId = 'excID'
# to fetch inputs
inputs = remote1.get(f"<flyte://v1/xxxxxxxx/development/{executionId}/n1-0-n4/i>")
resolved_inputs = inputs.literals

# to fetch the task
task = remote1.fetch_task(project=e1p1,
                             domain=domain,
                             name="xxx",
                             version=version)

execution = remote2.execute(
        task,
        inputs=resolved_inputs,
        # type_hints=type_hints,
        wait=True
    )
this is big picture of what i want to do But in this way when i try to get the inputs using remote.get() the inputs are in the following format
Copy code
inputs = {
    'doc_records': [[
        {'chunking_strategy': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/19b51b99bbb056e66111c600f130cb12/658946114c44a5f359a6cff4403da9df>', 'section_name': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/ffc62cdcf77a0aba9034ac7e18bcaa1f/5f5eeac0b52bdcf77051f3b99c6d1a0c>', 'chunk_limit': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/6c4b70e927a76414e7a8b546148a4e57/06ebc0904e06ba10f259820a25e44223>', 'chunked_curated_text': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/d4a75921505aef366460e687b1183da2/cb5fce120a28297eccda5911c7794c08>', 'chunk_type': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/d7b1cf55a08fbab1822ba2339e3500d3/cac0b82c45dc01001ddd2c4e6c748df1>', 'document_id': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/f4dd76fdce0b3505cb45f008df44f759/918014083a0264dbee763ebfa76458db>', 'chunk_context': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/d61ee65273443d1cc45382ce3913b48e/f059ea4dd46a16928eeecb28a376fee4>', 'curated_text': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/4f4e274e0c660a2472660a7b338280f2/8c9281071b99e948f5feafe861a7d9eb>', 'document_type': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/cb7c8671b56ec5c61c925149f5aca84f/bfce25c3b4f76f53d149dd4cd63c7aeb>', 'document_name': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/c69640c03fc5a1f94ac9448e0b623c3f/f0fc5b171b942484f9b472924621b6f2>', 'document_version': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/6357a7015b3ea42f35c92f23282e74c6/5f27a1965500114a93463d88588d0a0d>', 'curated_entity_text': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/3e4f6eff6c1889adfab8c01e90da09bf/e90ba3ffc39e1edd0238292d72621967>', 'chunk_order': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/650370c86650e0f7763f845059a56da4/9e06af055e3ba7ef2991bb605e2d6980>'},
        {'chunking_strategy': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/8e7e7753433250ddefa1a9a01c606338/4a40433e2bc6fcd99bd3f4615498a336>', 'curated_entity_text': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/ea0003a7e9f1e936fd9d54c08a633983/599661f8776fe2b3d53681c551b4e8a9>', 'chunk_limit': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/5ead26d0bf3ec10979414ee447b13569/00db5e7f4bc2fd2db8a6cb253952b5ac>', 'chunked_curated_text': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/3fe89932a24b7c78ea77bd081fbbf2db/4067b628fdaf2bfd0e423a0542da6f2f>', 'chunk_type': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/90a6739d03995ad6504fdb60d04ea0bd/4742a19148418eb42e898d5221a4e97c>', 'document_id': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/761b4eb9aebef3a95ecab266b9fba136/0381ce00ee5eff071b37a484704d318f>', 'curated_text': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/ab6cfe9b938e31c15842438ed07fa632/18a2dbfe35c6236f874a1a7abe8247ab>', 'chunk_context': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/f8369008b80877de94f9b84a1eccca08/154590a9c0dc64762e20af7a50b369da>', 'document_type': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/b5a0ed52a8e2ce4b133607edc7083e22/5cc1b959923d4559b4075ae81f835162>', 'document_name': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/0abddd7926c86519a04109b822ee8885/7d6facfa0ee2f6cea1d4abb366100118>', 'document_version': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/68ff9dc4275f815def3daa8589c0ced6/16978ba3059b58c35714880c53140881>', 'section_name': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/b2bf4d66911edbe5a113c39e5ba806c9/828c65a501ecf700e8837d458730f111>', 'chunk_order': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/b6c1e2affcf75be658e1e724c0852274/f02859a80635d1e41b9e4be3f85dcf1b>'}, 
....
With this way i'm facing issue with send the inputs can anyone help me in finding a way where i can perform the above thing
j
To achieve your goal of fetching inputs from a previous execution and executing a task on a different endpoint, you can follow these steps: 1. Fetch Inputs: Use
FlyteRemote
to fetch the inputs of a previous execution. Ensure that the inputs are properly resolved from their S3 URIs to actual data that can be used in the task execution. You might need to download the data from S3 and deserialize it if necessary. 2. Fetch Task: Use
FlyteRemote.fetch_task
to retrieve the task from the first endpoint. Ensure you have the correct project, domain, and task name. 3. Execute Task on Second Endpoint: Use
FlyteRemote.execute
on the second endpoint with the resolved inputs. Ensure that the inputs are in the correct format expected by the task. If you need more detailed guidance on any of these steps, please let me know. Sources: - FlyteRemote API - FlyteRemote Example 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
r
@ancient-wolf-19325 how to resolve the inputs properly is the question, cant flyte do that for me
@ancient-wolf-19325
j
To resolve inputs properly, Flyte typically handles the serialization and deserialization of inputs and outputs automatically when using Flyte's native types. However, when dealing with custom or complex types, especially those stored as URIs (e.g., S3 paths), you may need to manually handle the deserialization. 1. Download and Deserialize: If your inputs are stored as URIs pointing to files (e.g., on S3), you need to download these files and deserialize them into the appropriate Python objects. This can be done using libraries like
boto3
for S3. 2. Use Flyte's Type System: Ensure that your tasks and workflows are using Flyte's supported types, such as
FlyteFile
or
FlyteDirectory
, which can automatically handle file-based inputs and outputs. 3. Custom Type Transformers: If you have custom types, consider implementing a custom type transformer in Flytekit to handle the serialization and deserialization process automatically. Would you like more details on implementing any of these steps? Sources: - https://github.com/flyteorg/flyte/issues/5426 - https://github.com/flyteorg/flyte/blob/master/docs/user_guide/development_lifecycle/inspecting_executions.md 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
r
@ancient-wolf-19325 instead of fetching inputs using
remote.get()
can i fetch the FlyteTaskExecution and get the inputs?
j
Yes, you can fetch the
FlyteTaskExecution
and retrieve the inputs from it. Use
FlyteRemote
to fetch the task execution and then access the inputs. Here's a general approach: 1. Fetch Task Execution: Use
FlyteRemote.fetch_task_execution
to get the task execution object. 2. Access Inputs: Once you have the task execution object, you can access its inputs using the
inputs
attribute. If you need more detailed guidance on how to implement this, please let me know. Sources: - https://github.com/flyteorg/flyte/issues/2482 - https://discuss.flyte.org/t/11162571/is-there-a-way-to-locally-execute-a-task-or-workflow-using-t/ 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
r
@ancient-wolf-19325 but
FlyteRemote.fetch_task_execution
is not present right now
@ancient-wolf-19325
j
FlyteRemote.fetch_task_execution
is not explicitly mentioned in the provided data sources. Instead, you can use
FlyteRemote.fetch_execution
to retrieve the execution and then access the node executions to get the inputs for a specific task. Here's a general approach: 1. Fetch Execution: Use
FlyteRemote.fetch_execution
to get the execution object. 2. Sync Execution: Use
FlyteRemote.sync
to ensure the execution state is up-to-date, including all node executions. 3. Access Node Execution Inputs: Access the specific node execution corresponding to your task and retrieve its inputs. Would you like more detailed guidance on implementing this approach? Sources: - Inspecting Executions 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
r
@ancient-wolf-19325 why is
FlyteRemote.fetch_task_execution
removed?
j
I'm not able to find any data sources to answer your question. You might want to provide more context on where you encountered this method or any documentation references. If this doesn't help, please reach out to the Flyte support team for further assistance. 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
r
@ancient-wolf-19325 is
FlyteRemote.fetch_task_execution
present in flyte?
j
I'm not able to find any data sources to answer your question. You might want to specify the version of Flyte or Flytekit you are using. If this doesn't help, please reach out to the Flyte support team for further assistance. 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
r
@ancient-wolf-19325 which version of flytekit or Flyte has
FlyteRemote.fetch_task_execution
j
I'm not able to find any data sources to answer your question. You might want to check the Flytekit or Flyte release notes or documentation for any updates regarding
FlyteRemote.fetch_task_execution
. If this doesn't help, please reach out to the Flyte support team for further assistance. 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.