Hi, we recently opened a pull request to address the following issue (inter-cluster communication between Flyte and custom Ray cluster). Can someone please review it? It adds to a product Spotify is building that is integral to our machine learning platform.

Hey @Kevin Su @Ketan (kumare3) we are trying to make Flyte work for our internal Flyte cluster setup. @Abdullah Mobeen opened this PR to enable the inter-cluster communication feature for Ray plugin. Could you guys help take a look? Thank you so much!

Thanks, reviewing

👍

Technically, the changes are similar to what Spotify did for the Flyte-Flink plugin. Our data infra team also added context to the issue I linked. Thanks!

cc @Dan Rammer (hamersaw) to review as well

timely 😄

Looks great, merged, thanks @Abdullah Mobeen! Now we would just need to update the flyteplugin dependency version in flytepropeller. Is this something you're looking for an immediate propeller release on or are you building your own image anyways?

Thanks a lot @Dan Rammer (hamersaw)! Yess -- Since we prefer to always stay on a stable Flyte release, it is better if we make a new release to cover this plugin

Stable Flyte release will be 1.4

@Ketan (kumare3), what’s the rough timeline for Flyte 1.4 release?

So 1.4 is the current stable, 1.5 will be end of month (maybe first week of April) since we switched to a monthly release cycle. I opened this PR to get the plugin updates merged into propeller and will make sure this is merged for the 1.5 release.

Thanks @Dan Rammer (hamersaw)!