Alex Pozimenko
06/16/2022, 9:26 PMHaytham Abuelfutuh
Alex Pozimenko
06/16/2022, 9:30 PMflyteadmin_version = "v1.1.21"
flyteconsole_version = "v1.1.0"
flytecopilot_version = "v0.0.26"
flytepropeller_version = "v1.1.12"
Nastya Rusina
06/16/2022, 9:34 PMAlex Pozimenko
06/16/2022, 9:39 PMcheckerboard_dynamic_tasks
is there one that supposed to have subtasksJason Porter
06/16/2022, 9:40 PMAlex Pozimenko
06/16/2022, 9:40 PMJason Porter
06/16/2022, 9:41 PMJason Porter
06/16/2022, 9:44 PMAlex Pozimenko
06/16/2022, 10:41 PMNastya Rusina
06/17/2022, 6:33 PMAlex Pozimenko
06/17/2022, 6:39 PMNastya Rusina
06/17/2022, 6:46 PMeugene jahn
06/21/2022, 4:43 PMAlex Pozimenko
06/21/2022, 6:33 PMEugene Jahn
06/21/2022, 6:42 PMeugene jahn
06/21/2022, 6:43 PMJason Porter
06/21/2022, 6:43 PMAlex Pozimenko
06/21/2022, 6:59 PMeugene jahn
06/21/2022, 7:04 PMAlex Pozimenko
06/21/2022, 8:56 PM{
"node_executions": [
{
"id": {
"node_id": "start-node",
"execution_id": {
"project": "avfleetscenes",
"domain": "dev",
"name": "f67f46d3206de43699b7"
}
},
"closure": {
"output_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-f67f46d3206de43699b7/start-node/data/0/outputs.pb>",
"phase": "SUCCEEDED",
"created_at": "2022-06-21T20:16:34.290172734Z",
"updated_at": "2022-06-21T20:16:34.290172734Z"
},
"metadata": {
"spec_node_id": "start-node"
}
},
{
"id": {
"node_id": "n0",
"execution_id": {
"project": "avfleetscenes",
"domain": "dev",
"name": "f67f46d3206de43699b7"
}
},
"input_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-f67f46d3206de43699b7/n0/data/inputs.pb>",
"closure": {
"error": {
"code": "RetriesExhausted|USER:Unknown",
"message": "[2/2] currentAttempt done. Last Error: USER::Traceback (most recent call last):\n\n File \"/root/.local/share/virtualenvs/root-BuDEOXnJ/lib/python3.8/site-packages/flytekit/exceptions/scopes.py\", line 203, in user_entry_point\n return wrapped(*args, **kwargs)\n File \"/root/flyte/flyte/tasks/fs1_training_data.py\", line 88, in get_annotation_info_task\n call_scene_reconstruction_binary(\n File \"/root/flyte/flyte/commands/scene_command.py\", line 51, in call_scene_reconstruction_binary\n subprocess_handler.run(binary=command, args=params, log_stdout=True)\n File \"/root/cli/cli/subprocess_handler.py\", line 53, in run\n raise subprocess.CalledProcessError(returncode=exit_code, cmd=command, output=stdout, stderr=stderr)\n\nMessage:\n\n Command '['annotation', 'query', '--verbose', '--data-source', 'FS1', '--label-source', 'SCALE', '--output-file', '/tmp/flyte/20220621_202627/sandbox/local_flytekit/39e65c520948189a3f2663c3f94842e3/annotations.csv', '--scale-project-name', 'panda_lfd_lidar', '--task-id', '60c72d8333efc20018b6ea0d']' returned non-zero exit status 1.\n\nUser error.",
"kind": "USER"
},
"phase": "FAILED",
"started_at": "2022-06-21T20:16:34.460611038Z",
"duration": "804.056366105s",
"created_at": "2022-06-21T20:16:34.354000459Z",
"updated_at": "2022-06-21T20:29:58.516977105Z"
},
"metadata": {
"spec_node_id": "n0",
"is_dynamic": true
}
}
]
}
{
"node_executions": [
{
"id": {
"node_id": "start-node",
"execution_id": {
"project": "avfleetscenes",
"domain": "dev",
"name": "xcipb7ytwp"
}
},
"input_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-xcipb7ytwp/start-node/data/inputs.pb>",
"closure": {
"phase": "SUCCEEDED",
"created_at": "2021-09-23T13:27:27.874059940Z",
"updated_at": "2021-09-23T13:27:27.874059940Z"
},
"metadata": {
"spec_node_id": "start-node"
}
},
{
"id": {
"node_id": "n0",
"execution_id": {
"project": "avfleetscenes",
"domain": "dev",
"name": "xcipb7ytwp"
}
},
"input_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-xcipb7ytwp/n0/data/inputs.pb>",
"closure": {
"output_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-xcipb7ytwp/n0/data/0/outputs.pb>",
"phase": "SUCCEEDED",
"started_at": "2021-09-23T13:27:28.357826621Z",
"duration": "11798.029477360s",
"created_at": "2021-09-23T13:27:28.089924503Z",
"updated_at": "2021-09-23T16:44:06.387304360Z"
},
"metadata": {
"is_parent_node": true,
"spec_node_id": "n0"
}
},
{
"id": {
"node_id": "n1",
"execution_id": {
"project": "avfleetscenes",
"domain": "dev",
"name": "xcipb7ytwp"
}
},
"input_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-xcipb7ytwp/n1/data/inputs.pb>",
"closure": {
"output_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-xcipb7ytwp/n1/data/0/outputs.pb>",
"phase": "SUCCEEDED",
"started_at": "2021-09-23T16:44:52.527074281Z",
"duration": "548.360438964s",
"created_at": "2021-09-23T16:44:52.317653506Z",
"updated_at": "2021-09-23T16:54:00.887512964Z"
},
"metadata": {
"spec_node_id": "n1"
}
},
{
"id": {
"node_id": "end-node",
"execution_id": {
"project": "avfleetscenes",
"domain": "dev",
"name": "xcipb7ytwp"
}
},
"input_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-xcipb7ytwp/end-node/data/inputs.pb>",
"closure": {
"phase": "SUCCEEDED",
"created_at": "2021-09-23T16:54:01.094210244Z",
"updated_at": "2021-09-23T16:54:01.446059558Z"
},
"metadata": {
"spec_node_id": "end-node"
}
}
]
}
{
"node_executions": [
{
"id": {
"node_id": "start-node",
"execution_id": {
"project": "avfleetscenes",
"domain": "dev",
"name": "f860a446514bb4e07be0"
}
},
"input_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-f860a446514bb4e07be0/start-node/data/inputs.pb>",
"closure": {
"phase": "SUCCEEDED",
"created_at": "2021-09-22T15:58:52.269739635Z",
"updated_at": "2021-09-22T15:58:52.269739635Z"
},
"metadata": {
"spec_node_id": "start-node"
}
},
{
"id": {
"node_id": "n0",
"execution_id": {
"project": "avfleetscenes",
"domain": "dev",
"name": "f860a446514bb4e07be0"
}
},
"input_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-f860a446514bb4e07be0/n0/data/inputs.pb>",
"closure": {
"error": {
"code": "RetriesExhausted|USER:Unknown",
"message": "[2/2] currentAttempt done. Last Error: USER::Traceback (most recent call last):\n\n File \"/root/.local/share/virtualenvs/root-BuDEOXnJ/lib/python3.8/site-packages/flytekit/common/exceptions/scopes.py\", line 203, in user_entry_point\n return wrapped(*args, **kwargs)\n File \"/root/flyte/flyte/tasks/fs1_training_data.py\", line 324, in generate_l5mldatastore_dataset_task\n call_scene_reconstruction_binary(\n File \"/root/flyte/flyte/commands/scene_command.py\", line 51, in call_scene_reconstruction_binary\n subprocess_handler.run(binary=command, args=params, log_stdout=True)\n File \"/root/cli/cli/subprocess_handler.py\", line 53, in run\n raise subprocess.CalledProcessError(returncode=exit_code, cmd=command, output=stdout, stderr=stderr)\n\nMessage:\n\n Command '['annotation', 'write-training-data-chunk-l5mldatastore', '/tmp/flytezs70k80t/local_flytekit/92f659b1ba7cba47388b85d2c3ca177f/', '--verbose', '--chunk-id', '217', '--dataset-name', 'fs1_stereo_dataset', '--dataset-version', '0.0.3071-main.32268f0_0.0.1', '--end-ts', '1621979365', '--filtered-tracks-pb', '/tmp/flytezs70k80t/local_flytekit/8f281133547f3cb71452f2ef735477d1/filtered_tracks.pb', '--mission-id', '1863992517161235894_9314680418666361737', '--obstacles-frame', 'camera', '--output-file', '/tmp/flytezs70k80t/20210922_171104/local_flytekit/740367bf4e8c77e6ae90c2d4b3f0aa40/l5mldatastore_chunk_metadata_217.json', '--partition-name', 'train', '--start-ts', '1621979337']' returned non-zero exit status 1.\n\nUser error.",
"kind": "USER"
},
"phase": "FAILED",
"started_at": "2021-09-22T15:58:52.479254644Z",
"duration": "4790.533657594s",
"created_at": "2021-09-22T15:58:52.358488635Z",
"updated_at": "2021-09-22T17:18:43.012912594Z"
},
"metadata": {
"is_parent_node": true,
"spec_node_id": "n0"
}
}
]
}
Haytham Abuelfutuh
katrina
eugene jahn
06/21/2022, 10:51 PMkatrina
is_parent_node
is indeed setAlex Pozimenko
06/21/2022, 11:22 PMif you don’t mind double confirming this, both the older and more recent executions ran the same version of the worklfow?@Haytham Abuelfutuh the versions are different (they're 2 months apart)
Haytham Abuelfutuh
Alex Pozimenko
06/21/2022, 11:42 PM{
"node_executions": [
{
"id": {
"node_id": "start-node",
"execution_id": {
"project": "avfleetscenes",
"domain": "dev",
"name": "a8djqk9pzmdgfdjf75lx"
}
},
"closure": {
"output_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-a8djqk9pzmdgfdjf75lx/start-node/data/0/outputs.pb>",
"phase": "SUCCEEDED",
"created_at": "2022-06-21T23:38:52.525455365Z",
"updated_at": "2022-06-21T23:38:52.525455365Z"
},
"metadata": {
"spec_node_id": "start-node"
}
},
{
"id": {
"node_id": "n0",
"execution_id": {
"project": "avfleetscenes",
"domain": "dev",
"name": "a8djqk9pzmdgfdjf75lx"
}
},
"input_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-a8djqk9pzmdgfdjf75lx/n0/data/inputs.pb>",
"closure": {
"phase": "RUNNING",
"started_at": "2022-06-21T23:38:52.669382566Z",
"created_at": "2022-06-21T23:38:52.616230119Z",
"updated_at": "2022-06-21T23:38:52.669382566Z"
},
"metadata": {
"is_parent_node": true,
"spec_node_id": "n0"
}
}
]
}
{
"node_executions": [
{
"id": {
"node_id": "n0-0-start-node",
"execution_id": {
"project": "avfleetscenes",
"domain": "dev",
"name": "a8djqk9pzmdgfdjf75lx"
}
},
"closure": {
"output_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-a8djqk9pzmdgfdjf75lx/n0/data/0/start-node/0/outputs.pb>",
"phase": "SUCCEEDED",
"created_at": "2022-06-21T23:41:52.895604301Z",
"updated_at": "2022-06-21T23:41:52.895604301Z"
},
"metadata": {
"retry_group": "0",
"spec_node_id": "start-node"
}
},
{
"id": {
"node_id": "n0-0-dn0",
"execution_id": {
"project": "avfleetscenes",
"domain": "dev",
"name": "a8djqk9pzmdgfdjf75lx"
}
},
"input_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-a8djqk9pzmdgfdjf75lx/n0/data/0/dn0/inputs.pb>",
"closure": {
"phase": "RUNNING",
"started_at": "2022-06-21T23:41:55.385688619Z",
"created_at": "2022-06-21T23:41:53.138938047Z",
"updated_at": "2022-06-21T23:41:55.385688619Z",
"workflow_node_metadata": {
"executionId": {
"project": "avfleetscenes",
"domain": "dev",
"name": "foyc3j4i"
}
}
},
"metadata": {
"retry_group": "0",
"spec_node_id": "dn0"
}
},
{
"id": {
"node_id": "n0-0-dn1",
"execution_id": {
"project": "avfleetscenes",
"domain": "dev",
"name": "a8djqk9pzmdgfdjf75lx"
}
},
"input_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avfleetscenes-dev-a8djqk9pzmdgfdjf75lx/n0/data/0/dn1/inputs.pb>",
"closure": {
"phase": "RUNNING",
"started_at": "2022-06-21T23:41:55.677555860Z",
"created_at": "2022-06-21T23:41:53.193659416Z",
"updated_at": "2022-06-21T23:41:55.677555860Z",
"workflow_node_metadata": {
"executionId": {
"project": "avfleetscenes",
"domain": "dev",
"name": "f4gdb2lq"
}
}
},
"metadata": {
"retry_group": "0",
"spec_node_id": "dn1"
}
}
]
}
katrina
Alex Pozimenko
06/22/2022, 4:55 PMkatrina
Jason Porter
06/22/2022, 6:22 PMis_parent_node
katrina
Alex Pozimenko
06/22/2022, 7:38 PMfrom flytekit import task, dynamic, workflow, Resources
from flytekit.core.node_creation import create_node
from typing import List, Tuple, NamedTuple
SceneLevelProcessingResults = NamedTuple("OP2",
num_scenes_published=int)
@task(requests=Resources(mem='4G'), retries=6)
def simulation_metadata_collect(run_id: str) -> pd.DataFrame:
"""Collect all task metadata for downstream workers"""
# experiment_task_metadata_df = ...
# return experiment_task_metadata_df
@dynamic
def checkerboard_dynamic_tasks(run_id: str,
number_shards: int) -> Tuple[List[int], List[int]]:
num_issues_from_shards = []
num_scenes_from_shards = []
for i in range(number_shards):
scene_level_processing_results = checkerboard_scene_level_processing(
run_id=run_id,
shard=i,
number_shards=number_shards)
num_issues_from_shards.append(scene_level_processing_results.num_scenes_published)
num_scenes_from_shards.append(scene_level_processing_results.num_scenes_published)
return num_issues_from_shards, num_scenes_from_shards
@task(requests=Resources(mem='10G'), retries=5)
def checkerboard_scene_level_processing(run_id: str,
shard: int = 0,
number_shards: int = 1,
) -> SceneLevelProcessingResults:
scene_pipeline = InitPipeline(run_id=run_id,
number_shards=number_shards,
shard=shard)
scenes: List[CheckerboardScene] = scene_pipeline.initialize_scenes()
# ...
return SceneLevelProcessingResults(num_scenes_published=len(scenes))
@workflow
def CheckerboardParallelBackendLaunch(run_id: str, number_shards: int = 1):
metadata_collect_task = create_node(simulation_metadata_collect, run_id=run_id)
# WF to calculate issues, dynamically scaled by amount of metrics to process
dynamic_tasks = create_node(checkerboard_dynamic_tasks,
run_id=run_id,
number_shards=number_shards)
metadata_collect_task >> dynamic_tasks
katrina
Alex Pozimenko
06/22/2022, 8:11 PMkatrina
"id": {
"node_id": "n1",
"execution_id": {
"project": "flytesnacks",
"domain": "development",
"name": "f629e8fae1aa143e1a14"
}
},
"input_uri": "<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f629e8fae1aa143e1a14/n1/data/inputs.pb>",
"closure": {
"output_uri": "<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f629e8fae1aa143e1a14/n1/data/0/outputs.pb>",
"phase": "SUCCEEDED",
"started_at": "2022-06-22T20:21:44.605794900Z",
"duration": "120.055171600s",
"created_at": "2022-06-22T20:21:44.517225200Z",
"updated_at": "2022-06-22T20:23:44.660965600Z"
},
"metadata": {
"is_parent_node": true,
"spec_node_id": "n1",
"is_dynamic": true
}
},
as expectedAlex Pozimenko
06/22/2022, 8:26 PMkatrina
Alex Pozimenko
06/22/2022, 8:28 PMkatrina
Alex Pozimenko
06/22/2022, 9:21 PMimport flytekit
@flytekit.dynamic(
requests=flytekit.Resources(mem='256Mi', cpu='1'),
)
def simple_batch_task(iterations: int, input_string: str) -> None:
for i in range(iterations):
identity_sub_task(input_string=input_string)
@flytekit.task(
requests=flytekit.Resources(mem='256Mi', cpu='1')
)
def identity_sub_task(input_string: str) -> str:
return input_string
@flytekit.workflow
def HelloWorldDynamicTaskWorkflow(input_string: str = 'Hello World',
iterations: int = 5):
simple_batch_task(iterations=iterations, input_string=input_string)
katrina
Alex Pozimenko
06/23/2022, 10:45 PMJason Porter
06/23/2022, 11:10 PMkatrina
Alex Pozimenko
06/23/2022, 11:43 PMkatrina
Alex Pozimenko
06/23/2022, 11:52 PM{
"node_executions": [
{
"id": {
"node_id": "start-node",
"execution_id": {
"project": "avexampleworkflows",
"domain": "dev",
"name": "alkhkhqw6qlxgrqq2lv9"
}
},
"closure": {
"output_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avexampleworkflows-dev-alkhkhqw6qlxgrqq2lv9/start-node/data/0/outputs.pb>",
"phase": "SUCCEEDED",
"created_at": "2022-06-23T20:37:45.792945109Z",
"updated_at": "2022-06-23T20:37:45.792945109Z"
},
"metadata": {
"spec_node_id": "start-node"
}
},
{
"id": {
"node_id": "n0",
"execution_id": {
"project": "avexampleworkflows",
"domain": "dev",
"name": "alkhkhqw6qlxgrqq2lv9"
}
},
"input_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avexampleworkflows-dev-alkhkhqw6qlxgrqq2lv9/n0/data/inputs.pb>",
"closure": {
"phase": "SUCCEEDED",
"started_at": "2022-06-23T20:37:45.909076702Z",
"duration": "310.470520303s",
"created_at": "2022-06-23T20:37:45.849547550Z",
"updated_at": "2022-06-23T20:42:56.379596303Z"
},
"metadata": {
"spec_node_id": "n0",
"is_dynamic": true
}
},
{
"id": {
"node_id": "end-node",
"execution_id": {
"project": "avexampleworkflows",
"domain": "dev",
"name": "alkhkhqw6qlxgrqq2lv9"
}
},
"input_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avexampleworkflows-dev-alkhkhqw6qlxgrqq2lv9/end-node/data/inputs.pb>",
"closure": {
"phase": "SUCCEEDED",
"created_at": "2022-06-23T20:42:56.461800234Z",
"updated_at": "2022-06-23T20:42:56.522386713Z"
},
"metadata": {
"spec_node_id": "end-node"
}
}
]
}
katrina
Alex Pozimenko
06/23/2022, 11:54 PM{
"node_executions": [
{
"id": {
"node_id": "start-node",
"execution_id": {
"project": "avexampleworkflows",
"domain": "dev",
"name": "a2n7dnlw76stjlpzqhq6"
}
},
"closure": {
"output_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avexampleworkflows-dev-a2n7dnlw76stjlpzqhq6/start-node/data/0/outputs.pb>",
"phase": "SUCCEEDED",
"created_at": "2022-06-23T23:56:16.891813217Z",
"updated_at": "2022-06-23T23:56:16.891813217Z"
},
"metadata": {
"spec_node_id": "start-node"
}
},
{
"id": {
"node_id": "n0",
"execution_id": {
"project": "avexampleworkflows",
"domain": "dev",
"name": "a2n7dnlw76stjlpzqhq6"
}
},
"input_uri": "<s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avexampleworkflows-dev-a2n7dnlw76stjlpzqhq6/n0/data/inputs.pb>",
"closure": {
"phase": "RUNNING",
"started_at": "2022-06-23T23:56:17.014704942Z",
"created_at": "2022-06-23T23:56:16.951710847Z",
"updated_at": "2022-06-23T23:56:17.014704942Z"
},
"metadata": {
"is_parent_node": true,
"spec_node_id": "n0"
}
}
]
}
katrina
Alex Pozimenko
06/24/2022, 5:55 PMHaytham Abuelfutuh
katrina
Haytham Abuelfutuh
<http://ghcr.io/flyteorg/flytepropeller:v1.1.15-patch1|ghcr.io/flyteorg/flytepropeller:v1.1.15-patch1>
for flytepropellerAlex Pozimenko
06/27/2022, 6:49 PMJason Porter
06/28/2022, 10:36 PMAlex Pozimenko
06/29/2022, 5:52 PMHaytham Abuelfutuh
Alex Pozimenko
06/29/2022, 8:32 PM{"json":{"exec_id":"a2xflpcg2x5zfpkfnlrk","ns":"dev","routine":"worker-8"},"level":"error","msg":"Failed to update workflow. Error [Operation cannot be fulfilled on <http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com> \"a2xflpcg2x5zfpkfnlrk\": the object has been modified; please apply your changes to the latest version and try again]","ts":"2022-06-29T20:29:00Z"}
E0629 20:29:00.729217 1 workers.go:102] error syncing 'dev/a2xflpcg2x5zfpkfnlrk': Operation cannot be fulfilled on <http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com> "a2xflpcg2x5zfpkfnlrk": the object has been modified; please apply your changes to the latest version and try again
Haytham Abuelfutuh
Alex Pozimenko
06/29/2022, 8:45 PMHaytham Abuelfutuh
Alex Pozimenko
06/29/2022, 8:48 PMHaytham Abuelfutuh
logger:
level: 6
show-source: true
You might see you already have “logger” there…Alex Pozimenko
06/29/2022, 8:50 PMHaytham Abuelfutuh
Alex Pozimenko
06/29/2022, 8:52 PMapiVersion: v1
kind: ConfigMap
metadata:
name: flyte-propeller-config
namespace: "prod"
data:
propeller: |-
propeller:
logger:
level: 6
show-source: true
kube-client-config:
qps: 100
burst: 50
timeout: 30s
rawoutput-prefix: "s3://${flyte_bucket_name}"
....
Haytham Abuelfutuh
Alex Pozimenko
06/29/2022, 9:49 PMHaytham Abuelfutuh
propeller:
apiVersion: v1
kind: ConfigMap
metadata:
name: flyte-propeller-config
namespace: "prod"
data:
propeller: |-
logger:
level: 6
show-source: true
propeller:
kube-client-config:
qps: 100
burst: 50
timeout: 30s
rawoutput-prefix: "s3://${flyte_bucket_name}"
....
Alex Pozimenko
06/29/2022, 10:01 PMHaytham Abuelfutuh
flyteadmin:
eventVersion: 2
katrina
Alex Pozimenko
07/01/2022, 8:06 PMkatrina
Alex Pozimenko
07/01/2022, 8:26 PMkatrina
Alex Pozimenko
07/01/2022, 8:32 PM