Stephen
04/01/2022, 9:48 AMlatest
for scheduled workflows? It’d be useful for us so that we wouldn’t have to archive/ activate launchplans for each change.Nicholas LoFaso
04/01/2022, 3:10 PMNicholas LoFaso
04/01/2022, 3:13 PMMiggy
04/01/2022, 4:32 PM[1/1] currentAttempt done. Last Error: USER::[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[flyte-copilotdownloader] terminated with exit code (1). Reason [Error]. Message:
--storage.cache.max_size_mbs int Maximum size of the cache where the Blob store data is cached in-memory. If not specified or set to 0, cache is not used
--storage.cache.target_gc_percent int Sets the garbage collection target percentage.
I took a look at the config in the docs and looks like we need to add these. Is there anything else I’m missing and will these default values work for our deployment?
co-pilot:
cpu: 500m
default-input-path: /var/flyte/inputs
default-output-path: /var/flyte/outputs
image: <http://cr.flyte.org/flyteorg/flytecopilot:v0.0.15|cr.flyte.org/flyteorg/flytecopilot:v0.0.15>
input-vol-name: flyte-inputs
memory: 128Mi
name: flyte-copilot-
output-vol-name: flyte-outputs
start-timeout: 1m40s
storage: ""
illarion Disabled
04/02/2022, 11:42 AM@task(container_image="<http://registry.name.com/project/image:tag|registry.name.com/project/image:tag>")
I am trying to build this example by myself on my local Flyte sandbox, but getting error messages:
ModuleNotFoundError: No module named 'flyte'
My workflow from example.py
@task(container_image="<http://registry.name.com/project/image:tag|registry.name.com/project/image:tag>")
def some_data_generation() -> PythonPickledFile:
with open(BASE_FILE_PATH) as file:
some_descriptors = json.load(file)
some_set = generate_some_set(some_descriptors[0])
with open(PICKLE_PATH, 'wb') as handle:
pickle.dump(some_set, handle)
return PICKLE_PATH
@task(container_image="<http://registry.name.com/project/image2:tag|registry.name.com/project/image2:tag>")
def load_pickle_dump(dump_file_path: PythonPickledFile) -> set:
with open(dump_file_path, 'rb') as handle:
return pickle.load(handle)
@workflow
def my_wf() -> set:
dump_file_path = some_data_generation()
return load_pickle_dump(dump_file_path=dump_file_path)
if __name__ == "__main__":
a = my_wf()
print(f"Running my_wf() {a}")
What was localised:
1. Workflow works fine without containers (without @task(container_image= )
2. Images works fine if workflow contain only one task with Docker image where included Flyte Workflow folder with this file (example.py)
3. Problem appears if i build Docker Image for first task without Flyte workflow files inside (but with initial data for first files to skip downloading).
4. I am sure - i can skip second task at all from this test case, problem should appear. Root cause - i have Docker image without Flute workflow (example.py) but it seems that this code is required inside the container to be executed. I do not understand how can i split example.py between two tasks if it should be executed actually outside the tasks (because this is a workflow, if should contain tasks inside it according to example)
Error:
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[j3nl8bafcr-n0-0] terminated with exit code (1). Reason [Error]. Message:
thon3.8/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 460, in execute_task_cmd
_execute_task(
File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
return wrapped(*args, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 327, in _execute_task
_task_def = resolver_obj.load_task(loader_args=resolver_args)
File "/opt/venv/lib/python3.8/site-packages/flytekit/core/python_auto_container.py", line 189, in load_task
task_module = importlib.import_module(task_module)
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'flyte'
As i see - it caused by Docker Container without a Flyte workflow. But if i will place workflow in first task - how it should be correctly splitted between two tasks? Should not include code for first task inside the second task image/container?
Am i on correct way at all?
Can you please advise a simple short workflow example on Python with two tasks on separate containers?Fabio Grätz
04/04/2022, 4:21 PMSandra Youssef
04/04/2022, 4:59 PMAlex Bain
04/04/2022, 7:12 PMv0.19.3
from an older version of Flyte. When we browse to Flyte Console, we get a new CORS error that we did not get before (see below). What configuration change can we make to fix it?
Access to XMLHttpRequest at '<https://avflyteadminhttp.pdx.l5.woven-planet.tech/api/v1/projects>' from origin '<https://avflyteconsole.pdx.l5.woven-planet.tech>' has been blocked by CORS policy: The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'. The credentials mode of requests initiated by the XMLHttpRequest is controlled by the withCredentials attribute.
(This ^^^^ is the error in my Chrome developer tools console output)Alex Bain
04/04/2022, 10:02 PMAlex Pozimenko
04/05/2022, 2:08 AMflyte-admin-executor-flytepropeller-execution.*
Have these been removed or renamed?illarion Disabled
04/05/2022, 12:25 PMUnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
when trying to read them by pickle.load(file):
inside
with open(f'{input_dir}/dump_file') as file:
raw_data = pickle.load(file)
it is saved without errors by:
with open(f'{output_dir}/dump_file', 'wb') as handle:
pickle.dump(raw_data, handle)
by previous stage raw container
stage1 = ContainerTask(
name="stage1",
output_data_dir="/var/outputs",
outputs=kwtypes(generation=PythonPickledFile),
image="<http://registry.project.com/flyte/stage1:0.0.1|registry.project.com/flyte/stage1:0.0.1>",
command=[
"python",
"stage1.py",
"/var/outputs",
],
)
stage2 = ContainerTask(
name="stage2",
input_data_dir="/var/inputs",
output_data_dir="/var/outputs",
inputs=kwtypes(dump_file=PythonPickledFile),
outputs=kwtypes(stage2_out=str),
image="<http://registry.project.com/flyte/stage2:0.0.1|registry.project.com/flyte/stage2:0.0.1>",
command=[
"python",
"stage2.py",
"/var/inputs",
"/var/outputs",
],
)
@workflow
def my_wf() -> str:
generation = stage1()
return stage2(dump_file=generation)
Can i pass PythonPickledFile like that?Hampus Rosvall
04/05/2022, 2:04 PM{
"asctime": "2022-04-05 13:58:10,552",
"name": "flytekit",
"levelname": "WARNING",
"message": "No config file provided or invalid flyte config_file_path flytekit.config specified."
}
Hampus Rosvall
04/05/2022, 5:00 PMFlyteAdmin
pod. Could it be related to oidc_client_secret
in the flyte-admin-secrets
, or do anyone have an idea on what could cause this?
flyteadmin-7498449bc8-pl6rw flyteadmin {"json":{"src":"token.go:84"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T16:53:17Z"}
flyteadmin-7498449bc8-pl6rw flyteadmin {"json":{"src":"handlers.go:237"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T16:53:17Z"}
flyteadmin-7498449bc8-pl6rw flyteadmin {"json":{"src":"token.go:104"},"level":"debug","msg":"Could not retrieve id token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2022-04-05T16:53:17Z"}
Hampus Rosvall
04/05/2022, 5:55 PMhttps://<host>/callback
and get stuck in some redirect loop where the FlyteAdmin
Pod keeps logging like below. Does anyone know what could cause this? Thanks 🙂
flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_idt] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_idt] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_at] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:71"},"level":"info","msg":"Could not detect existing cookie [flyte_rt]. Error: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie_manager.go:71"},"level":"info","msg":"Refresh token doesn't exist or failed to read it. Ignoring this error. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_rt], caused by: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_user_info] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"handlers.go:227"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"token.go:84"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"handlers.go:237"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"token.go:64"},"level":"debug","msg":"JWT parsing with claims failed failed to verify signature: failed to verify id token signature","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_at] cookie found","ts":"2022-04-05T17:46:04Z"}
flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:71"},"level":"info","msg":"Could not detect existing cookie [flyte_rt]. Error: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}
Ketan (kumare3)
04/06/2022, 4:45 AMAlex Bain
04/06/2022, 5:01 PMillarion Disabled
04/06/2022, 11:29 PMSamhita Alla
04/07/2022, 8:44 AMSandra Youssef
04/07/2022, 3:26 PMOrchestrating MLOps With Flyte▾
satyanarayana m
04/08/2022, 4:16 AMMike Ossareh
04/08/2022, 4:28 PMAlex Bain
04/08/2022, 6:08 PMmake lint
CI for flytecopilot
. Maybe also merge https://github.com/flyteorg/flytecopilot/pull/28 which looks fine. Then I can rebase my PR on top of these. If you could enable my GitHub user to trigger CI workflows that would also be great, my GH user is https://github.com/convexquad. @Haytham Abuelfutuh @Ketan (kumare3)Hampus Rosvall
04/11/2022, 8:34 AMflyte.config
?
• Is it best practice to assign a SA/IAM role for each task, or reuse the same identity on a workflow basis?
• What should be configured in flyte.config
vs ~/.flyte/config.yaml
?illarion Disabled
04/11/2022, 10:10 AMHampus Rosvall
04/12/2022, 11:20 AMworkflow.py
needs to access both data and train, do you still recommend this structure and reference everything in workflow.py
relative root? Probably not explaining the idea that well, but interested in what you think is the best practice with respect to workflows and relative imports/access to other folders containing data/code important to your project.
├── Dockerfile
├── data
│ └── some_folder_with_data
├── train
│ ├── train.py
├── docker_build_and_tag.sh
├── flyte
│ ├── __init__.py
│ └── workflows
│ ├── __init__.py
│ └── workflow.py
├── flyte.config
└── requirements.txt
Ketan (kumare3)
04/12/2022, 5:27 PMRobinson N. Costa
04/13/2022, 1:01 PMHaytham Abuelfutuh
04/14/2022, 8:14 AMflyte-admin-task.execution.manager-task.executions.terminated.counter
Erik H
04/15/2022, 7:18 AMAlessandro Liparoti
04/15/2022, 12:05 PMdocker pull <http://cr.flyte.org/flyteorg/flyteconsole-release:v0.19.3|cr.flyte.org/flyteorg/flyteconsole-release:v0.19.3>
it doesn’t find it. Any idea why?Alessandro Liparoti
04/15/2022, 12:05 PMdocker pull <http://cr.flyte.org/flyteorg/flyteconsole-release:v0.19.3|cr.flyte.org/flyteorg/flyteconsole-release:v0.19.3>
it doesn’t find it. Any idea why?Ketan (kumare3)
04/15/2022, 1:39 PM