https://flyte.org
Join Slack
Does the flyte-binary helm chart come with support for scheduling / ~cron out of the box? Or are the...
# flyte-deployment
e
Does the flyte-binary helm chart come with support for scheduling / ~cron out of the box? Or are there additional components that need to be deployed?
a
I don't think so @echoing-carpenter-92090 I mean, the scheduler is a native component of Flyte, regardless of the deployment method. Adding a schedule to your Launch plan and activating it, should be enough: https://docs.flyte.org/projects/cookbook/en/latest/getting_started/run_schedule.html#scheduling-a-launch-plan
e
Thanks @average-finland-92144.. Hm. It appears as active in the console UI, but fetching the lp through flytectl is not encouraging..
Copy code
-latest
 -------------------------- ----------------------------------- ------ ------- ---------- -------- ---------
| VERSION                  | NAME                              | TYPE | STATE | SCHEDULE | INPUTS | OUTPUTS |
 -------------------------- ----------------------------------- ------ ------- ---------- -------- ---------
| 9xlF6CkNO1rB46vBvlGNuA== | schedule_test.print_time_workflow |      |       |          |        | o0      |
 -------------------------- ----------------------------------- ------ ------- ---------- -------- ---------
1 rows
a
can you share the launch plan definition?
e
Copy code
cron_lp_every_min_of_hour = LaunchPlan.get_or_create(
        name="my_cron_scheduled_lp",
        workflow=print_time_workflow,
        schedule=CronSchedule(schedule="*/5 * * * *")
    )
gratitude thank you 1
a
and did you activated it?
e
yes, through flytectl. That's when it appeared in the UI
a
and what's the symptom. it hasn't been executed? anything interesting on the logs?
e
Yes, there's no executions kicking off. I didnt' see anything in the flyte-binary pod
Anywhere else I can check?
f
It does come with scheduling
You just have to activate the schedule
I am working on a PR that will make this nicer
gratitude thank you 1
e
Ketan (kumare3) [8:50 PM]
You just have to activate the schedule
Is there anything in addition to registering the launchplan and then 
flytectl update launchplan ... --activate
?
f
No
e
Is this helpful at all?
Copy code
2023/09/13 14:14:31 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/callbacks.go:134
[0.946ms] [rows:0] SELECT * FROM "schedulable_entities"
{"json":{"src":"composite_workqueue.go:88"},"level":"debug","msg":"Subqueue handler batch round","ts":"2023-09-13T14:14:32Z"}
{"json":{"src":"composite_workqueue.go:98"},"level":"debug","msg":"Dynamically configured batch size [-1]","ts":"2023-09-13T14:14:32Z"}
{"json":{"src":"composite_workqueue.go:129"},"level":"debug","msg":"Exiting SubQueue handler batch round","ts":"2023-09-13T14:14:32Z"}
{"json":{"src":"composite_workqueue.go:88"},"level":"debug","msg":"Subqueue handler batch round","ts":"2023-09-13T14:14:33Z"}
{"json":{"src":"composite_workqueue.go:98"},"level":"debug","msg":"Dynamically configured batch size [-1]","ts":"2023-09-13T14:14:33Z"}
Inside the db, there is a table 
launch_plans
where I can see the state=1 for scheduled launch plan only, while the other LPs are state 0. The 
schedulable_entities
table is empty.
Is there some way to inspect the running processes inside the 
flyte-binary
deployment? From inside the container I see the following proc 
root         7  0.6  8.1 2079380 160480 ?      Sl   15:41   2:45 /usr/local/bin/flyte start --config /etc/flyte/config.d/*.yaml
Within the config files I can see the 
flyte.admin.disableScheduler: false
param..
f
there should be only one process
everything runs as a single process (go program) this is why flyte binary is small and light
i can try and run it locally
e
The workflow or the binary? That would be fantastic
f
the binary
flytectl demo start runs the binary
e
Everything seems so straightforward but my scheduler must not be configured properly? The schedule appears in the console, and the workflow executes manually just fine. The only hangup is there is no execution time, and when fetching the launchplan after updating with 
--activate
, there is no change in status, scheduled_time, etc.
f
i will be in a flight, then will try
e
thanks Ketan, I will try to local binary in the meantime
f
@echoing-carpenter-92090 I was able to try it out locally in the sandbox This is the xample code
Copy code
from datetime import datetime
from flytekit import task, workflow, LaunchPlan, CronSchedule


@task
def say_hello() -> str:
    return "Hello"


@workflow
def talker() -> str:
    return say_hello()


example_launch_plan = LaunchPlan.get_or_create(
    name="talk-every-minute",
    workflow=talker,
    schedule=CronSchedule(
        schedule="* * * * *",  # Following schedule runs every min
    ),
)
I used the latest pyflyte run
Copy code
pyflyte register backfill2.py --activate-launchplans
And it started running automatically
gratitude thank you 1
so scheduler is on by default
e
Your example worked for me local and remote. Thanks Ketan
a
Cool @echoing-carpenter-92090, so what's the main difference between your schedule definition and Ketan's example? is it maybe the cron syntax? Just trying to learn
e
That's a great question. I believe the syntax was effectively the same. I don't think it's the cron, these crons are dead simple. The only difference was that I bumped pyflyte to the 
main
commit from git, rather than the pip detected release. I've also only been using the 
pyflyte register .. --activate-launchplans
syntax. Previously I was using the 
pyflyte register..
, and then 
flytectl update launchplan ... --activate
style.
f
Really that does not make sense
Cc @tall-lock-23197 can me add this to the docs as a way to register and activate and also deactivate schedules
👍 1
20 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by