https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
flyte-bazel
  • r

    Rahul Mehta

    01/29/2023, 6:04 PM
    set the channel description: Channel for discussing Flyte & Bazel integration
  • r

    Rahul Mehta

    01/29/2023, 6:04 PM
    Just set this up to discuss Bazel <> Flyte integration & any particulars that come up during development, as well as any work to get
    rules_flyte
    back up to date/upstream some of our own work
  • s

    Sam Eckert

    01/29/2023, 6:16 PM
    I'd be happy to contribute to
    rules_flyte
    . The implementation I put together this week is fairly encapsulated from our stack simply because its so new -- though that is a double edged sword
  • r

    Rahul Mehta

    01/29/2023, 6:19 PM
    The only specifics we use are our internal py_binary/py_library targets. Historically we added linters as sidecar targets to that, though we’ve recently moved those to aspects. So we might be able to ditch the custom targets soon
  • r

    Rahul Mehta

    01/29/2023, 6:19 PM
    Other than that we’re close to the same state
  • v

    varsha Parthasarathy

    02/01/2023, 7:24 PM
    Joining late to the party, but we (Woven by Toyota/ex-level5) have also open sourced some parts of our monorepo.. Happy to conttibute/upstream our work as well.
  • b

    Brian Tang

    02/22/2023, 1:07 AM
    Would love to contribute as well. We based much of our integration at stripe on the bazel rules but will need some time to remove the stripe specific components
  • k

    Ketan (kumare3)

    02/22/2023, 4:11 AM
    Cc @Sam Eckert / @Rahul Mehta
  • r

    Rahul Mehta

    02/23/2023, 2:59 PM
    One thing that'd be awesome to share is how each of our implementations is handling serialization, registration & execution. Do you all have separate targets for each, or are any of the steps handled together? In our case, we handle both serialization & registration in a single
    .register
    target, and handle execution w/ a
    py_binary
    target that takes in the launch plan name & user-specified CLI args that are passed to
    flyte_remote.execute
    inside that binary. One interesting concern is that it requires the user to remember to register prior to attempting to launch an execution, and this is an area we'd really like to clean up and make more seamless. Curious if anyone else has different solutions for this?
    m
    • 2
    • 1
  • k

    Ketan (kumare3)

    03/17/2023, 2:33 PM
    Folks remember Flyte slack is free. Good thing would be to put things in a doc
  • m

    Mohd Shahid Khan Afridi

    03/20/2023, 3:55 PM
    Hi Guys, I am using bazel to build the flyte target but when I use bazel output binaries to serialize and register the workflows it lead to ModuleNotFoundError: No module named 'pyflyte' at the time of task execution. The error occurred because the value of
    resolver
    in task container argument changed from usual
    flytekit.core.python_auto_container.default_task_resolver
    to
    pyflyte.pypi_flytekit.site-packages.flytekit.core.python_auto_container.default_task_resolver
    . Did anyone face this issue ? is there a way to control this?
    y
    r
    • 3
    • 4
  • r

    Rahul Mehta

    03/24/2023, 2:41 PM
    @Sam Eckert also we figured out how to avoid the container_image step; since the entrypoint in our container delegates to the appropriate
    pyflyte-*
    command, just using the py3_image as-is worked (w/ the way that Flyte sets
    args
    for the pods)
  • r

    Rahul Mehta

    03/24/2023, 2:42 PM
    This shaved ~45-60s off build time, which adds up over time
Powered by Linen
Title
r

Rahul Mehta

03/24/2023, 2:42 PM
This shaved ~45-60s off build time, which adds up over time
View count: 1