Attention Airflow users, <@U01J90KBSU9> recently ...
# announcements
k
Attention Airflow users, @Samhita Alla recently created a FlyteOperator in Airflow, and would love if some of you can upvote this operator - https://github.com/apache/airflow/pull/22646
πŸŽ‰ 1
πŸ‘ 1
cc @Anand Swaminathan / @Max Payton
πŸ‘ 1
s
I’ve started a discussion in the devlist: https://lists.apache.org/thread/b55g3gydgmqmhow6f7xzzbm5t0gmhs2x. It’d be nice if some of you could share your thoughts. Only on getting some upvotes will the operator be merged into the Airflow repository. Thanks in advance!
n
@Samhita Alla to clarify, do folks need to add β€œ+1s” or specific comments to the discussion thread you started?
k
To the issue itself
m
I was reading through the linked discussion thread in the issue: https://lists.apache.org/thread/t1k3d0518v4kxz1pqsprdc78h0wxobg0 and I noticed they talked about testing the provider
Copy code
So I would say all future providers (if they are using some external
service) will have to have system tests implemented. That means few
things: first of all, there should be systems tests, secondly, there
should be some way (credits, free accounts with enough capacity to run
regular testing in an automated way - donated to Airflow in order to
be able to support a provider). We have not yet discussed the last
point (credits and system tests being condition) but this seems like
we are heading towards - with Amazon and Google providers leading the
effort of implementing system tests (we already got credits and
commitments from both).
which I think is a good point before an operator is added to airflow. Is there currently a hosted flyte option that could be used to enable this?
k
Flyte is an open source, linux foundation project, good thing is the test can be run on any machine with docker
πŸ‘ 1
m
So they could be run with the credits provided by AWS/GCP. Sounds good
πŸ‘ 1
s
@Samhita Alla to clarify, do folks need to add β€œ+1s” or specific comments to the discussion thread you started?
@Niels Bantilan, either one works.
115 Views