rhythmic-hamburger-56597
11/18/2024, 4:21 PMremote1 = FlyteRemote(
Config.for_endpoint("<http://flyte.us.cloud.ep1.com|flyte.us.cloud.ep1.com>"),
default_domain=domain,
)
remote2 = FlyteRemote(
Config.for_endpoint("<http://flyte.us.cloud.ep2.com|flyte.us.cloud.ep2.com>"),
default_domain=domain,
)
executionId = 'excID'
# to fetch inputs
inputs = remote1.get(f"<flyte://v1/xxxxxxxx/development/{executionId}/n1-0-n4/i>")
resolved_inputs = inputs.literals
# to fetch the task
task = remote1.fetch_task(project=e1p1,
domain=domain,
name="xxx",
version=version)
execution = remote2.execute(
task,
inputs=resolved_inputs,
# type_hints=type_hints,
wait=True
)
this is big picture of what i want to do
But in this way when i try to get the inputs using remote.get()
the inputs are in the following format
inputs = {
'doc_records': [[
{'chunking_strategy': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/19b51b99bbb056e66111c600f130cb12/658946114c44a5f359a6cff4403da9df>', 'section_name': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/ffc62cdcf77a0aba9034ac7e18bcaa1f/5f5eeac0b52bdcf77051f3b99c6d1a0c>', 'chunk_limit': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/6c4b70e927a76414e7a8b546148a4e57/06ebc0904e06ba10f259820a25e44223>', 'chunked_curated_text': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/d4a75921505aef366460e687b1183da2/cb5fce120a28297eccda5911c7794c08>', 'chunk_type': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/d7b1cf55a08fbab1822ba2339e3500d3/cac0b82c45dc01001ddd2c4e6c748df1>', 'document_id': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/f4dd76fdce0b3505cb45f008df44f759/918014083a0264dbee763ebfa76458db>', 'chunk_context': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/d61ee65273443d1cc45382ce3913b48e/f059ea4dd46a16928eeecb28a376fee4>', 'curated_text': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/4f4e274e0c660a2472660a7b338280f2/8c9281071b99e948f5feafe861a7d9eb>', 'document_type': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/cb7c8671b56ec5c61c925149f5aca84f/bfce25c3b4f76f53d149dd4cd63c7aeb>', 'document_name': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/c69640c03fc5a1f94ac9448e0b623c3f/f0fc5b171b942484f9b472924621b6f2>', 'document_version': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/6357a7015b3ea42f35c92f23282e74c6/5f27a1965500114a93463d88588d0a0d>', 'curated_entity_text': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/3e4f6eff6c1889adfab8c01e90da09bf/e90ba3ffc39e1edd0238292d72621967>', 'chunk_order': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/650370c86650e0f7763f845059a56da4/9e06af055e3ba7ef2991bb605e2d6980>'},
{'chunking_strategy': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/8e7e7753433250ddefa1a9a01c606338/4a40433e2bc6fcd99bd3f4615498a336>', 'curated_entity_text': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/ea0003a7e9f1e936fd9d54c08a633983/599661f8776fe2b3d53681c551b4e8a9>', 'chunk_limit': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/5ead26d0bf3ec10979414ee447b13569/00db5e7f4bc2fd2db8a6cb253952b5ac>', 'chunked_curated_text': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/3fe89932a24b7c78ea77bd081fbbf2db/4067b628fdaf2bfd0e423a0542da6f2f>', 'chunk_type': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/90a6739d03995ad6504fdb60d04ea0bd/4742a19148418eb42e898d5221a4e97c>', 'document_id': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/761b4eb9aebef3a95ecab266b9fba136/0381ce00ee5eff071b37a484704d318f>', 'curated_text': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/ab6cfe9b938e31c15842438ed07fa632/18a2dbfe35c6236f874a1a7abe8247ab>', 'chunk_context': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/f8369008b80877de94f9b84a1eccca08/154590a9c0dc64762e20af7a50b369da>', 'document_type': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/b5a0ed52a8e2ce4b133607edc7083e22/5cc1b959923d4559b4075ae81f835162>', 'document_name': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/0abddd7926c86519a04109b822ee8885/7d6facfa0ee2f6cea1d4abb366100118>', 'document_version': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/68ff9dc4275f815def3daa8589c0ced6/16978ba3059b58c35714880c53140881>', 'section_name': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/b2bf4d66911edbe5a113c39e5ba806c9/828c65a501ecf700e8837d458730f111>', 'chunk_order': '<s3://beta-flyte-prod-us/o2/autoexecution7254-n1-0-n3-0-n0-0/b6c1e2affcf75be658e1e724c0852274/f02859a80635d1e41b9e4be3f85dcf1b>'},
....
With this way i'm facing issue with send the inputs
can anyone help me in finding a way where i can perform the above thingjolly-florist-66728
11/18/2024, 4:21 PMFlyteRemote
to fetch the inputs of a previous execution. Ensure that the inputs are properly resolved from their S3 URIs to actual data that can be used in the task execution. You might need to download the data from S3 and deserialize it if necessary.
2. Fetch Task: Use FlyteRemote.fetch_task
to retrieve the task from the first endpoint. Ensure you have the correct project, domain, and task name.
3. Execute Task on Second Endpoint: Use FlyteRemote.execute
on the second endpoint with the resolved inputs. Ensure that the inputs are in the correct format expected by the task.
If you need more detailed guidance on any of these steps, please let me know.
Sources:
- FlyteRemote API
- FlyteRemote Example 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.rhythmic-hamburger-56597
11/18/2024, 4:22 PMrhythmic-hamburger-56597
11/18/2024, 4:27 PMjolly-florist-66728
11/18/2024, 4:27 PMboto3
for S3.
2. Use Flyte's Type System: Ensure that your tasks and workflows are using Flyte's supported types, such as FlyteFile
or FlyteDirectory
, which can automatically handle file-based inputs and outputs.
3. Custom Type Transformers: If you have custom types, consider implementing a custom type transformer in Flytekit to handle the serialization and deserialization process automatically.
Would you like more details on implementing any of these steps?
Sources:
- https://github.com/flyteorg/flyte/issues/5426
- https://github.com/flyteorg/flyte/blob/master/docs/user_guide/development_lifecycle/inspecting_executions.md 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.rhythmic-hamburger-56597
11/18/2024, 5:22 PMremote.get()
can i fetch the FlyteTaskExecution and get the inputs?jolly-florist-66728
11/18/2024, 5:22 PMFlyteTaskExecution
and retrieve the inputs from it. Use FlyteRemote
to fetch the task execution and then access the inputs. Here's a general approach:
1. Fetch Task Execution: Use FlyteRemote.fetch_task_execution
to get the task execution object.
2. Access Inputs: Once you have the task execution object, you can access its inputs using the inputs
attribute.
If you need more detailed guidance on how to implement this, please let me know.
Sources:
- https://github.com/flyteorg/flyte/issues/2482
- https://discuss.flyte.org/t/11162571/is-there-a-way-to-locally-execute-a-task-or-workflow-using-t/ 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.rhythmic-hamburger-56597
11/18/2024, 5:27 PMFlyteRemote.fetch_task_execution
is not present right nowrhythmic-hamburger-56597
11/18/2024, 5:27 PMjolly-florist-66728
11/18/2024, 5:27 PMFlyteRemote.fetch_task_execution
is not explicitly mentioned in the provided data sources. Instead, you can use FlyteRemote.fetch_execution
to retrieve the execution and then access the node executions to get the inputs for a specific task. Here's a general approach:
1. Fetch Execution: Use FlyteRemote.fetch_execution
to get the execution object.
2. Sync Execution: Use FlyteRemote.sync
to ensure the execution state is up-to-date, including all node executions.
3. Access Node Execution Inputs: Access the specific node execution corresponding to your task and retrieve its inputs.
Would you like more detailed guidance on implementing this approach?
Sources:
- Inspecting Executions 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.rhythmic-hamburger-56597
11/19/2024, 7:43 AMFlyteRemote.fetch_task_execution
removed?jolly-florist-66728
11/19/2024, 7:43 AMrhythmic-hamburger-56597
11/19/2024, 7:44 AMFlyteRemote.fetch_task_execution
present in flyte?jolly-florist-66728
11/19/2024, 7:44 AMrhythmic-hamburger-56597
11/19/2024, 7:45 AMFlyteRemote.fetch_task_execution
jolly-florist-66728
11/19/2024, 7:45 AMFlyteRemote.fetch_task_execution
. If this doesn't help, please reach out to the Flyte support team for further assistance. 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.