https://flyte.org logo
Join Slack
Powered by
Hello, I am trying to use Sensors with Flyte agent...
# flyte-support
b
Hello, I am trying to use Sensors with Flyte agents but the sensor tasks fail with
Copy code
failed to get grpc connection with error: failed to exit idle mode: passthrough: received empty target in Build()
and I see the same error message in propeller
Copy code
Failed to create resource....
using 
v1.11.1-b1
version. These are my helm values(flyte-core)(relavant parts)
Copy code
flyteagent:
  enabled: true

configmap:  
  enabled_plugins:
    tasks:
      task-plugins:
        enabled-plugins:
          - agent-service
        default-for-task-types:
          sensor: agent-service
    plugins:
      agent-service:
        supportedTaskTypes:
          - sensor
agent logs
Copy code
Starting up the server to expose the prometheus metrics...                                                      
Starting the agent service...
Am I missing something out?
👀 1
t
cc @glamorous-carpet-83516 / @damp-lion-88352
d
Hi, can you help me try this example?
Copy code
from flytekit.sensor.file_sensor import FileSensor
from flytekit import task, workflow

sensor = FileSensor(name="test_sensor")

@task()
def t1():
    return

@workflow()
def wf():
    sensor(path="/") >> t1()

if __name__ == "__main__":
    wf()
try this.
Copy code
pyflyte run --remote sensor_example.py sensor --path "<s3://my-s3-bucket>"
b
I get the same error
d
Can you show me your agent image?
b
Copy code
image: <http://ghcr.io/flyteorg/flyteagent:1.11.0|ghcr.io/flyteorg/flyteagent:1.11.0>
d
Thanks
Do you have this in your config map?
Copy code
agent-service:
	defaultAgent:
	  defaultTimeout: 10s
	  endpoint: dns:///flyteagent.flyte.svc.cluster.local:8000
	  insecure: true
	  timeouts:
	    GetTask: 10s
	supportedTaskTypes:
	- default_task
It works for me
I think the main reason is about your grpc connection
b
I only had this part
Copy code
agent-service:
        supportedTaskTypes:
          - sensor
after updating with yours, it started working now!
❤️ 1
but this configuration is not aligned with the configuration documented https://docs.flyte.org/en/latest/deployment/agents/sensor.html#deployment-agent-setup-sensor
d
Thank you
I will discuss with Kevin how to update the doc
thanks
🙏 2
b
thank you as well!
d
Hi, @blue-ice-67112 I've created a PR here, thank you again. https://github.com/flyteorg/flyte/pull/5208
❤️ 1
looks more clear now : )
b
Awesome @damp-lion-88352 🙌
❤️ 1
b
apologies for leeching on this thread, i was hitting the same error and workaround after adding the same config. Still have issue with custom sync agent. From what i understood in the code, only async agent needs to implement the create method but not sync agent, did i configure anything wrongly? err msg
Copy code
USER::rpc error: code = Internal desc = failed to create caramlnotification task with error: 'CustomAgent' object has no attribute 'create'
helm
Copy code
enabled_plugins:
    tasks:
        enabled-plugins:
          - container
          - sidecar
          - k8s-array
          - agent-service
        default-for-task-types:
          container: container
          sidecar: sidecar
          container_array: k8s-array
          customagent: agent-service
    plugins:
      agent-service:
        supportedTaskTypes:
        - customagent
        defaultAgent:
          defaultTimeout: 10s
          endpoint: dns:///flyteagent.flyte.svc.cluster.local:8000
          insecure: true
          timeouts:
            GetTask: 10s
Task
Copy code
class CustomAgentTask(SyncAgentExecutorMixin, PythonTask):
    _TASK_TYPE = "customagent"
super().__init__(
            task_type=self._TASK_TYPE,
            ...
        )
Agent
Copy code
class CustomAgent(SyncAgentBase):
    _TASK_TYPE = "customagent"
    def __init__(self):
        super().__init__(task_type_name=TASK_TYPE)
I tried printing the agent registry locally while running 
pyflyte serve agent --port 8000
Copy code
agent is name: "Sensor"
supported_task_types: "sensor"
supported_task_categories {
  name: "sensor"
}
Starting Sensor that supports task categories ['sensor']

agent is name: "Custom Agent"
supported_task_types: "customagent"
is_sync: true
supported_task_categories {
  name: "customagent"
}
Starting Custom Agent that supports task categories ['customagent']

 agent is name: "ChatGPT Agent"
supported_task_types: "chatgpt"
is_sync: true
supported_task_categories {
  name: "chatgpt"
}
Starting ChatGPT Agent that supports task categories ['chatgpt']
b
@broad-train-34581 not sure if this will help, but I'll just dump my working configuration. I'm using FileSensor only at the moment
Copy code
plugins:
      agent-service:
        supportedTaskTypes:
          - sensor
        defaultAgent:
          defaultTimeout: 10s
          endpoint: dns:///flyteagent.flyte.svc.cluster.local:8000
          insecure: true
b
oh nvm ignore me, it work after i restarted the propeller, although i dont know why. Thanks @blue-ice-67112, i was hitting the same grpc error you had initially too
d
it work after i restarted the propeller
It's because propeller needs to read the 
supported task type
from the configmap again. It is not dynamically changing now.
👍 1
31 Views
Open in Slack
PreviousNext
Powered by