Attention Airflow users, @tall-lock-23197 recently created a FlyteOperator in Airflow, and would love if some of you can upvote this operator - https://github.com/apache/airflow/pull/22646
π 1
π 1
freezing-airport-6809
04/06/2022, 4:46 AM
cc @lemon-fish-7011 / @gorgeous-iron-33877
π 1
t
tall-lock-23197
04/06/2022, 5:07 AM
Iβve started a discussion in the devlist: https://lists.apache.org/thread/b55g3gydgmqmhow6f7xzzbm5t0gmhs2x. Itβd be nice if some of you could share your thoughts. Only on getting some upvotes will the operator be merged into the Airflow repository.
Thanks in advance!
b
broad-monitor-993
04/06/2022, 2:49 PM
@tall-lock-23197 to clarify, do folks need to add β+1sβ or specific comments to the discussion thread you started?
So I would say all future providers (if they are using some external
service) will have to have system tests implemented. That means few
things: first of all, there should be systems tests, secondly, there
should be some way (credits, free accounts with enough capacity to run
regular testing in an automated way - donated to Airflow in order to
be able to support a provider). We have not yet discussed the last
point (credits and system tests being condition) but this seems like
we are heading towards - with Amazon and Google providers leading the
effort of implementing system tests (we already got credits and
commitments from both).
which I think is a good point before an operator is added to airflow. Is there currently a hosted flyte option that could be used to enable this?
f
freezing-airport-6809
04/06/2022, 3:55 PM
Flyte is an open source, linux foundation project, good thing is the test can be run on any machine with docker
π 1
g
gorgeous-iron-33877
04/06/2022, 4:02 PM
So they could be run with the credits provided by AWS/GCP. Sounds good
π 1
t
tall-lock-23197
04/06/2022, 4:48 PM
@tall-lock-23197 to clarify, do folks need to add β+1sβ or specific comments to the discussion thread you started?