GitHub
07/06/2023, 3:21 PMGitHub
07/06/2023, 4:05 PM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by cosmicBboy
<https://github.com/flyteorg/flyte-conference-talks/commit/0194eb6dd6438410f64fee8520e0cf12a45c2fbc|0194eb6d>
- cleanup docker build, debug
flyteorg/flyte-conference-talksGitHub
07/06/2023, 4:36 PM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by cosmicBboy
<https://github.com/flyteorg/flyte-conference-talks/commit/b23ef278d74d5438cf168d14fde4705e39798dfd|b23ef278>
- debugging
flyteorg/flyte-conference-talksGitHub
07/06/2023, 5:25 PM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by cosmicBboy
<https://github.com/flyteorg/flyte-conference-talks/commit/ea0d7b04f8c125110fee1a8572c376911e1515b1|ea0d7b04>
- debugging
flyteorg/flyte-conference-talksGitHub
07/06/2023, 6:07 PM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by cosmicBboy
<https://github.com/flyteorg/flyte-conference-talks/commit/ccfab5b3da86323f07a643ab576d0ad3ed37e3ea|ccfab5b3>
- debugging
flyteorg/flyte-conference-talksGitHub
07/06/2023, 6:27 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by cosmicBboy
<https://github.com/flyteorg/flytekit/commit/ca1608e5dcfb4d84dd9b4fc9c206ad6e4f4e825c|ca1608e5>
- use getattr to access task_def.disable_deck in entrypoint (#1724)
flyteorg/flytekitGitHub
07/06/2023, 8:42 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flytekit/commit/f52526d47aa3175c021da84cbf08a97fc7da7dd8|f52526d4>
- Add a couple tests (#1722)
flyteorg/flytekitGitHub
07/06/2023, 8:53 PM<https://github.com/flyteorg/flyte/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flyte/commit/b46fbd28dfa4b71f0131063a757560c81b30460b|b46fbd28>
- Fix single binary config (#3808)
flyteorg/flyteGitHub
07/06/2023, 8:54 PM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by cosmicBboy
<https://github.com/flyteorg/flyte-conference-talks/commit/fdd7a6a643575ce52e66bd5eecd8f727364e5d21|fdd7a6a6>
- end to end tests pass
flyteorg/flyte-conference-talksGitHub
07/06/2023, 9:12 PM<https://github.com/flyteorg/flytectl/tree/master|master>
by katrogan
<https://github.com/flyteorg/flytectl/commit/47d6c6255e44b1b618c07575b2cbe78f3787aab3|47d6c625>
- Fix the error message for the update cmd (#411)
flyteorg/flytectlGitHub
07/06/2023, 9:42 PMGitHub
07/06/2023, 9:42 PM<https://github.com/flyteorg/homebrew-tap/tree/main|main>
by flyte-bot
<https://github.com/flyteorg/homebrew-tap/commit/5011a7e3d356977e95b6efefd2071d925c66b64b|5011a7e3>
- Brew formula update for flytectl version v0.6.41
flyteorg/homebrew-tapGitHub
07/06/2023, 9:53 PMImagePullBackoff
on a Pod
it immediately declares the task a failure. In most scenarios this may not be the most efficient behavior because FlytePropeller will then retry with a new Pod
(given retries are enabled) which will just attempt to pull the same image(s). By introducing a configurable grace period before FlytePropeller declares a task failure we allow k8s to continue attempting to pull the image. A grace period mechanism has precedent in other issues as well.
Goal: What should the final outcome look like, ideally?
FlytePropeller should wait a configurable period before declaring an ImagePullBackoff
a failure rather than immediately on first detection.
Describe alternatives you've considered
Leaving this as is causes unnecessary failures / retries.
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
07/06/2023, 10:59 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/26291f192b6760e23d2e6aff418a14dd8c1880a2|26291f19>
- Csvtransform (#1671)
flyteorg/flytekitGitHub
07/06/2023, 10:59 PMGitHub
07/07/2023, 12:04 AM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/30060324122760a57345722cea8adcb154875f49|30060324>
- add private registry support (#1705)
flyteorg/flytekitGitHub
07/07/2023, 12:06 AM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by cosmicBboy
<https://github.com/flyteorg/flyte-conference-talks/commit/e3383b3852c4933a1b3fb6a2cc2fbc017e0bc11f|e3383b38>
- updates
flyteorg/flyte-conference-talksGitHub
07/07/2023, 12:48 AM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by cosmicBboy
<https://github.com/flyteorg/flyte-conference-talks/commit/d6714b376e2c56083a1eb1d9fbbe4cd15f24ac43|d6714b37>
- fix notebook
flyteorg/flyte-conference-talksGitHub
07/07/2023, 5:59 AMflytectl demo start --dev
# Under flyte repo
make compile
POD_NAMESPACE=flyte ./flyte start --config flyte-single-binary-local.yaml
The following errors may occurs.
{"json":{"src":"start.go:63"},"level":"info","msg":"Running Database Migrations...","ts":"2023-07-07T05:13:37Z"}
{"json":{"src":"initialize.go:41"},"level":"warning","msg":"Database [flyte] does not exist, trying to create it now","ts":"2023-07-07T05:13:37Z"}
{"json":{"src":"database.go:123"},"level":"warning","msg":"Database [flyte] does not exist","ts":"2023-07-07T05:13:37Z"}
{"json":{"src":"database.go:136"},"level":"info","msg":"Creating database flyte","ts":"2023-07-07T05:13:37Z"}
{"json":{"src":"handle.go:59"},"level":"info","msg":"Creating Database flyte since it does not exist","ts":"2023-07-07T05:13:37Z"}
{"json":{"src":"initialize.go:54"},"level":"error","msg":"Failed to create DB flyte err ERROR: duplicate key value violates unique constraint \"pg_database_datname_index\" (SQLSTATE 23505)","ts":"2023-07-07T05:13:37Z"}
panic: ERROR: duplicate key value violates unique constraint "pg_database_datname_index" (SQLSTATE 23505)
It seems that during initialization, the process attempts to create the flyte database twice.
this isn't a big issue as it can be resolved by simply rerunning the command POD_NAMESPACE=flyte ./flyte start --config flyte-single-binary-local.yaml
.
It will not encounter the same issue on subsequent runs as the flyte database will already exist.
Expected behavior
No errors when first running POD_NAMESPACE=flyte ./flyte start --config flyte-single-binary-local.yaml
.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
07/07/2023, 6:33 AMImagePullBackoffGracePeriod
configuration option so that Flyte does not automatically declare Pods
with ImagePullBackoff
as failures, but rather waits a configurable duration for k8s to recover.
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☑︎ Code documentation added
☐ Any pending items have an associated Issue
Complete description
^^^
Tracking Issue
fixes flyteorg/flyte#3843
Follow-up issue
NA
flyteorg/flyteplugins
✅ All checks have passed
2/2 successful checksGitHub
07/07/2023, 7:03 AMGitHub
07/07/2023, 7:08 AMGitHub
07/07/2023, 7:13 AMGitHub
07/07/2023, 12:24 PMfrom datetime import timedelta
from airflow.utils import trigger_rule
from flytekit import task, workflow, ImageSpec
from airflow.sensors import time_sensor
from airflow.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperator, DataprocDeleteClusterOperator, DataprocSubmitSparkJobOperator
airflow_plugin = "git+<https://github.com/flyteorg/flytekit.git@487438ab59147879eded897674593a1eaee1c78b#subdirectory=plugins/flytekit-airflow>"
image_spec = ImageSpec(base_image="pingsutw/flytekit:v1", packages=["apache-airflow", airflow_plugin], apt_packages=["git"], registry="pingsutw")
@workflow
def wf():
create_cluster = DataprocCreateClusterOperator(
task_id="create_dataproc_cluster",
image_version="2.0.27-debian10",
storage_bucket="opta-gcp-dogfood-gcp",
master_machine_type="n1-highmem-32",
master_disk_size=1024,
num_workers=2,
worker_machine_type="n1-highmem-64",
worker_disk_size=1024,
region="us-west1",
cluster_name="flyte-dataproc",
project_id="dogfood-gcp-dataplane",
)
run_spark = DataprocSubmitSparkJobOperator(
job_name="spark_pi",
task_id="run_spark",
dataproc_jars=["file:///usr/lib/spark/examples/jars/spark-examples.jar"],
main_class="org.apache.spark.examples.JavaWordCount",
arguments=["<gs://opta-gcp-dogfood-gcp/spark/file.txt>"],
cluster_name="flyte-dataproc",
region="us-west1",
project_id="dogfood-gcp-dataplane",
)
delete_cluster = DataprocDeleteClusterOperator(
task_id="create_dataproc_cluster",
project_id="dogfood-gcp-dataplane",
cluster_name="flyte-dataproc",
region="us-west1",
retries=3,
retry_delay=timedelta(minutes=5),
email_on_failure=True,
trigger_rule=trigger_rule.TriggerRule.ALL_DONE
)
create_cluster >> run_spark >> delete_cluster
if __name__ == '__main__':
wf()
bashSensor
from airflow.sensors.bash import BashSensor
from flytekit import task, workflow, ImageSpec
airflow_plugin = "git+<https://github.com/flyteorg/flytekit.git@487438ab59147879eded897674593a1eaee1c78b#subdirectory=plugins/flytekit-airflow>"
image_spec = ImageSpec(base_image="pingsutw/flytekit:v1", packages=["apache-airflow", airflow_plugin], apt_packages=["git"], registry="pingsutw")
@task(container_image=image_spec)
def t1():
print("flyte")
@workflow
def wf():
sensor = BashSensor(task_id="Sensor_succeeds", bash_command="echo hello")
sensor >> t1()
if __name__ == '__main__':
wf()
HiveSensor
from airflow.providers.apache.hive.sensors.hive_partition import HivePartitionSensor
from flytekit import task, workflow, ImageSpec
airflow_plugin = "git+<https://github.com/flyteorg/flytekit.git@487438ab59147879eded897674593a1eaee1c78b#subdirectory=plugins/flytekit-airflow>"
image_spec = ImageSpec(base_image="pingsutw/flytekit:v1", packages=["apache-airflow", airflow_plugin], apt_packages=["git"], registry="pingsutw")
@task(container_image=image_spec)
def t1():
print("flyte")
@workflow
def wf():
sensor = HivePartitionSensor(table="flyte", schema="person")
sensor >> t1()
if __name__ == '__main__':
wf()
FileSensor
from airflow.sensors.filesystem import FileSensor
from flytekit import task, workflow, ImageSpec
airflow_plugin = "git+<https://github.com/flyteorg/flytekit.git@487438ab59147879eded897674593a1eaee1c78b#subdirectory=plugins/flytekit-airflow>"
image_spec = ImageSpec(base_image="pingsutw/flytekit:v1", packages=["apache-airflow", airflow_plugin], apt_packages=["git"], registry="pingsutw")
@task(container_image=image_spec)
def t1():
print("flyte")
@workflow
def wf():
sensor = FileSensor(task_id="id", filepath="/tmp/1234")
sensor >> t1()
if __name__ == '__main__':
wf()
TimeSensor
from datetime import datetime, timedelta
from pytz import UTC
from airflow.sensors.time_sensor import TimeSensor
from flytekit import task, workflow, ImageSpec
airflow_plugin = "git+<https://github.com/flyteorg/flytekit.git@487438ab59147879eded897674593a1eaee1c78b#subdirectory=plugins/flytekit-airflow>"
image_spec = ImageSpec(base_image="pingsutw/flytekit:v1", packages=["apache-airflow", airflow_plugin], apt_packages=["git"], registry="pingsutw")
@task(container_image=image_spec)
def t1():
print("flyte")
@workflow
def wf():
sensor = TimeSensor(task_id="fire_immediately", target_time=(datetime.now(tz=UTC)+timedelta(seconds=5)).time())
sensor >> t1()
if __name__ == '__main__':
wf()
PythonSensor
from airflow.sensors.python import PythonSensor
from flytekit import task, workflow, ImageSpec
airflow_plugin = "git+<https://github.com/flyteorg/flytekit.git@487438ab59147879eded897674593a1eaee1c78b#subdirectory=plugins/flytekit-airflow>"
image_spec = ImageSpec(base_image="pingsutw/flytekit:v1", packages=["apache-airflow", airflow_plugin], apt_packages=["git"], registry="pingsutw")
@task(container_image=image_spec)
def t1():
print("flyte")
def py_func():
print("airflow python sensor")
return True
@workflow
def wf():
sensor = PythonSensor(task_id="fire_immediately", python_callable=py_func)
sensor >> t1()
if __name__ == '__main__':
wf()
Tracking Issue
NA
Follow-up issue
NA
flyteorg/flytekit
GitHub Actions: build-plugins (3.11, flytekit-sqlalchemy)
GitHub Actions: build-plugins (3.11, flytekit-snowflake)
GitHub Actions: build-plugins (3.11, flytekit-polars)
GitHub Actions: build-plugins (3.11, flytekit-papermill)
GitHub Actions: build-plugins (3.11, flytekit-pandera)
GitHub Actions: build-plugins (3.11, flytekit-kf-tensorflow)
GitHub Actions: build-plugins (3.11, flytekit-kf-pytorch)
GitHub Actions: build-plugins (3.11, flytekit-kf-mpi)
GitHub Actions: build-plugins (3.11, flytekit-k8s-pod)
GitHub Actions: build-plugins (3.11, flytekit-huggingface)
GitHub Actions: build-plugins (3.11, flytekit-hive)
GitHub Actions: build-plugins (3.11, flytekit-envd)
GitHub Actions: build-plugins (3.11, flytekit-duckdb)
GitHub Actions: build-plugins (3.11, flytekit-dolt)
GitHub Actions: build-plugins (3.11, flytekit-deck-standard)
GitHub Actions: build-plugins (3.11, flytekit-dbt)
GitHub Actions: build-plugins (3.11, flytekit-data-fsspec)
GitHub Actions: build-plugins (3.11, flytekit-dask)
GitHub Actions: build-plugins (3.11, flytekit-bigquery)
GitHub Actions: build-plugins (3.11, flytekit-aws-sagemaker)
GitHub Actions: build-plugins (3.11, flytekit-aws-batch)
GitHub Actions: build-plugins (3.11, flytekit-aws-athena)
GitHub Actions: build-plugins (3.10, flytekit-whylogs)
GitHub Actions: build-plugins (3.10, flytekit-vaex)
GitHub Actions: build-plugins (3.10, flytekit-sqlalchemy)
GitHub Actions: build-plugins (3.10, flytekit-spark)
GitHub Actions: build-plugins (3.10, flytekit-ray)
GitHub Actions: build-plugins (3.10, flytekit-polars)
GitHub Actions: build-plugins (3.10, flytekit-papermill)
✅ 1 other check has passed
1/30 successful checksGitHub
07/07/2023, 1:18 PM<https://github.com/flyteorg/flyteadmin/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flyteadmin/commit/05981db6cef7f96122e5728963d5a80457ebc023|05981db6>
- Fix archive.replacements in goreleaser (#586)
flyteorg/flyteadminGitHub
07/07/2023, 1:45 PMBildschirmfoto_2023-07-07_um_15_03_07▾
GitHub
07/07/2023, 2:28 PMGitHub
07/07/2023, 2:35 PM<https://github.com/flyteorg/flytestdlib/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytestdlib/commit/53f62a741e0f697f94129e5541d027920f89f115|53f62a74>
- Update .goreleaser.yml (#158)
flyteorg/flytestdlibGitHub
07/07/2023, 2:36 PM<https://github.com/flyteorg/flytecopilot/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytecopilot/commit/10f74209f0f0141e4f28dbd40a2d378e77c41c28|10f74209>
- Update .goreleaser.yml (#61)
flyteorg/flytecopilotGitHub
07/07/2023, 2:36 PM<https://github.com/flyteorg/datacatalog/tree/master|master>
by eapolinario
<https://github.com/flyteorg/datacatalog/commit/5e2512ab65baf283fafd3dafcde04bf1a1c5a56c|5e2512ab>
- Fix archive.replacements in goreleaser (#109)
flyteorg/datacatalog