https://flyte.org logo
Join Slack
Powered by
Hi, I am trying to develop a new agent but I am st...
# flyte-connectors
c
Hi, I am trying to develop a new agent but I am struggling to get it to run in my sandbox I followed the instructions here. Once I schedule the workflow. The task is sent to the cluster but it fails immediately with error:
Copy code
RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [BadTaskSpecification] invalid TaskSpecification, unable to determine Pod configuration
I believe the it doesn't find the right agent? For deploying the agent in a regular cluster I will have to update the task and plugin settings as described here. How would I do that for sandbox cluster?
✅ 1
f
It seems config is wrong - cc @damp-lion-88352
d
Yes
Did you implement your agent metadata service?
Did you use single binary or your sandbox?
If you use single binary, did you start your agent server ?
If you are using your own cluster, can you update your propeller image to the latest?
And if you use it in your cluster, please make sure you update your agent image with your custom agent
c
Thanks for the reply. I did not implement an agent metadata service. This is where the documentation gets very sparse and I don't understand how I would do that. I tried running my task in my sandbox. I did update the agent image using my own docker image including the custom agent
❤️ 1
d
c
Yes, I used that example as a reference and implemented all those classes
I can run the task locally but not in the sandbox
d
Do you mind create a draft pr? I can take a look
Did you use the latest propeller image?
The latest propeller doesn’t need config
It can automatically fetech the latest available agent
And also, can you check the log in the agent pod?
c
Yes, will do over the weekend. I am on the latest version
d
It will show your available agent
Can you give me your log like this?
f9f7199a60b7f2fa5db99bd897f4d888.png
c
it only list sensor, not my custom agent
d
If it didn’t show at your agent pod, then your flytekit implementation might be wrong
Yes so this is not a config problem
You need to provide your metadata
I think you can offer your code and dockerfile
It will be faster
c
ok, I will have to strip out some of our proprietary code and will post something over the weekend. Thanks
d
No problem
If agent is useful, feel free to write a blog or come to the community sync to share your company’s use case, thank you so much
c
Here an example to reproduce my issue https://github.com/thomas-maschler/flyte_agent/pull/1
❤️ 1
d
No problem
Let me take a look
hi, @colossal-nightfall-74781
AgentRegistry.register(BigQueryAgent())
you should register your agent to your agent registry
c
I believe I do that in workflow.py
d
no
you should not do there
when you run 
pyflyte serve agent
you will not run 
workflow.py
hop on a call?
I can sycn with you for 10 minutes
this is not a hard problem, let's communicate efficiently
Copy code
from setuptools import setup

PLUGIN_NAME = "bigquery"

microlib_name = f"flytekitplugins-{PLUGIN_NAME}"

plugin_requires = ["flytekit>1.10.7", "google-cloud-bigquery", "flyteidl>1.10.7"]

__version__ = "0.0.0+develop"

setup(
    name=microlib_name,
    version=__version__,
    author="flyteorg",
    author_email="<mailto:admin@flyte.org|admin@flyte.org>",
    description="This package holds the Bigquery plugins for flytekit",
    namespace_packages=["flytekitplugins"],
    packages=[f"flytekitplugins.{PLUGIN_NAME}"],
    install_requires=plugin_requires,
    license="apache2",
    python_requires=">=3.8",
    classifiers=[
        "Intended Audience :: Science/Research",
        "Intended Audience :: Developers",
        "License :: OSI Approved :: Apache Software License",
        "Programming Language :: Python :: 3.8",
        "Programming Language :: Python :: 3.9",
        "Programming Language :: Python :: 3.10",
        "Topic :: Scientific/Engineering",
        "Topic :: Scientific/Engineering :: Artificial Intelligence",
        "Topic :: Software Development",
        "Topic :: Software Development :: Libraries",
        "Topic :: Software Development :: Libraries :: Python Modules",
    ],
    entry_points={"flytekit.plugins": [f"{PLUGIN_NAME}=flytekitplugins.{PLUGIN_NAME}"]},
)
2 small PR about flyte agent. 1. helm chart https://github.com/flyteorg/flyte/pull/5730 2. documentation https://github.com/flyteorg/flyte/pull/5731
🙌🏽 1
thank you for the feedback
c
Thanks a lot. I got this to run now. My main issue was that I didn't install the agent as a plugin. I updated my pyproject.toml and the path structure and got it to work. Other issues I encountered were type-related. I made sure that all agent metadata are JSON serializable and now it works.
❤️ 2
d
Glad it works! Feel free to ask more questions so I can figure out how to improve the documentation.
7 Views
Open in Slack
PreviousNext
Powered by