#3639 [BUG] Flyte Remote Execute fails with Dynamic Tasks Issue created by zeryx Describe the bug When attempting to use flytekit to execute a a remote workflow that contains a dynamic task; the following error is reported.

Traceback (most recent call last):
  File "/home/runner/.local/lib/python3.10/site-packages/grpc/_interceptor.py", line 274, in continuation
    response, call = self._thunk(new_method).with_call(
  File "/home/runner/.local/lib/python3.10/site-packages/grpc/_interceptor.py", line 301, in with_call
    return self._with_call(request,
  File "/home/runner/.local/lib/python3.10/site-packages/grpc/_interceptor.py", line 290, in _with_call
    return call.result(), call
  File "/home/runner/.local/lib/python3.10/site-packages/grpc/_channel.py", line 379, in result
    raise self
  File "/home/runner/.local/lib/python3.10/site-packages/grpc/_interceptor.py", line 274, in continuation
    response, call = self._thunk(new_method).with_call(
  File "/home/runner/.local/lib/python3.10/site-packages/grpc/_channel.py", line 1043, in with_call
    return _end_unary_response_blocking(state, call, True, None)
  File "/home/runner/.local/lib/python3.10/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.NOT_FOUND
	details = "missing entity of type TASK with identifier project:"flytetester" domain:"development" name:"workflows.bayesian_optimization_example.concurrent_trials" version:"cd16b9c4-27e6-44c1-9e8a-5542f5fa0ca4" "
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"missing entity of type TASK with identifier project:\"flytetester\" domain:\"development\" name:\"workflows.bayesian_optimization_example.concurrent_trials\" version:\"cd16b9c4-27e6-44c1-9e8a-5542f5fa0ca4\" ", grpc_status:5, created_time:"2023-05-03T22:35:43.500233657+00:00"}"
>

The code that generated this error is in the following gist, registration does seem to work correctly; execution fails when a workflow containing a dynamic task is located. When viewing the executed workflow with the UI; the tasks actually do complete. However when interacting with the admin api endpoint: https://hosted_environment.example/v1/tasks/project_name/domain_name/workflows.bayesian_optimization_example.concurrent_trials/cd16b9c4-27e6-44c1-9e8a-5542f5fa0ca4 the following error message can be seen:

{
  "error": "missing entity of type TASK with identifier project:\"flytetester\" domain:\"development\" name:\"workflows.bayesian_optimization_example.suggest_points\" version:\"cd16b9c4-27e6-44c1-9e8a-5542f5fa0ca4\" ",
  "code": 5,
  "message": "missing entity of type TASK with identifier project:\"flytetester\" domain:\"development\" name:\"workflows.bayesian_optimization_example.suggest_points\" version:\"cd16b9c4-27e6-44c1-9e8a-5542f5fa0ca4\" "
}

This remote workflow was registered to the flyte cluster using flytekit remote. Expected behavior all tasks are discoverable by the flyte cluster, and remote.wait() waits successfully until the execution completes, as would be the case with non-dynamic task based workflows. Additional context to reproduce if need to find a flytekit remote example, there's one in https://github.com/flyteorg/flytekit-python-template. You will need a hosted flyte cluster, and an App (Operator) that has access to said cluster.

git clone <https://github.com/flyteorg/flytekit-python-template> && cd flytekit-python-template
git checkout zeryx/bug/dynamic_remote_execute_failure
pip install -r requirements.txt
python integration.py 
python integration.py --host YOURCLUSTER.DOMAIN --client_id YOUR_APP_OPERATOR_ID --client_secret YOUR_APP_OPERATOR_SECRET --image-hostname <http://ghcr.io/zeryx/flytekit-python-template|ghcr.io/zeryx/flytekit-python-template> --image_suffix pr-1

This should attempt to register the bayesian-optimization to your cluster, and execute it. If successful; you will recieve the output response from the model, otherwise an error. Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte