billions-midnight-10687
10/16/2023, 1:18 PMfreezing-airport-6809
billions-midnight-10687
10/16/2023, 2:16 PMfreezing-airport-6809
freezing-airport-6809
billions-midnight-10687
10/16/2023, 2:19 PMfreezing-airport-6809
freezing-airport-6809
damp-lion-88352
10/16/2023, 3:44 PMbillions-midnight-10687
10/16/2023, 6:13 PMbillions-midnight-10687
10/16/2023, 6:13 PMbillions-midnight-10687
10/16/2023, 6:13 PMbillions-midnight-10687
10/16/2023, 6:14 PMdamp-lion-88352
10/18/2023, 2:43 AMfreezing-airport-6809
freezing-airport-6809
damp-lion-88352
10/18/2023, 3:23 AMbillions-midnight-10687
10/31/2023, 5:03 PMdamp-lion-88352
11/01/2023, 2:24 AMfreezing-airport-6809
damp-lion-88352
11/01/2023, 4:45 AMdamp-lion-88352
11/01/2023, 4:50 AMbillions-midnight-10687
11/01/2023, 6:29 PMdamp-lion-88352
11/02/2023, 3:28 AMdamp-lion-88352
11/02/2023, 3:29 AMfreezing-airport-6809
damp-lion-88352
11/02/2023, 3:31 AMdamp-lion-88352
11/02/2023, 3:33 AMbillions-midnight-10687
11/03/2023, 1:09 AMdamp-lion-88352
11/03/2023, 1:11 AMbillions-midnight-10687
11/03/2023, 1:53 AMbillions-midnight-10687
11/03/2023, 1:56 AMbillions-midnight-10687
11/03/2023, 2:00 AMfreezing-airport-6809
damp-lion-88352
11/03/2023, 2:33 AMfreezing-airport-6809
freezing-airport-6809
damp-lion-88352
11/03/2023, 2:56 PMbillions-midnight-10687
11/03/2023, 11:37 PMdamp-lion-88352
11/04/2023, 8:42 AMdamp-lion-88352
11/04/2023, 8:42 AMglamorous-carpet-83516
11/05/2023, 4:44 AMdamp-lion-88352
11/05/2023, 6:25 AMdamp-lion-88352
11/05/2023, 3:53 PMbillions-midnight-10687
11/06/2023, 9:31 PMbillions-midnight-10687
11/06/2023, 9:31 PMfreezing-airport-6809
billions-midnight-10687
11/07/2023, 10:16 PMbillions-midnight-10687
11/07/2023, 10:17 PMglamorous-carpet-83516
11/07/2023, 10:28 PMglamorous-carpet-83516
11/07/2023, 10:33 PMglamorous-carpet-83516
11/13/2023, 7:23 PMpyflyte run spark.py wf # Run spark in local python process
pyflyte run --raw-output-data-prefix <s3://saprk/output> spark.py wf # We serialize the input and upload to s3, and trigger the databricks job.
The reason to do that because it’s easy to develop, and test agent locally. In addiction, you are able to run databricks task without flyte cluster.
my question is should we use different flag? like --agent, --hybrid. Any feedback is appreciatedglamorous-carpet-83516
11/13/2023, 7:23 PMsalmon-refrigerator-32115
11/13/2023, 7:37 PMtask_config=Spark(
spark_conf={
....
# The following is needed only when running spark task in dev's local PC. Also need to do this locally: export SPARK_LOCAL_IP="127.0.0.1"
"spark.hadoop.fs.s3a.access.key": "aaa",
"spark.hadoop.fs.s3a.secret.key": "bbb",
"spark.hadoop.fs.s3a.session.token": "ccc",
},
),
glamorous-carpet-83516
11/13/2023, 7:39 PMglamorous-carpet-83516
11/13/2023, 7:40 PMThat would be very nice to have especially when we can do this locally for spark tasks that runs on k8s spark operator by passing the AWS S3 creds locally like this.so you want to run a spark workflow locally, but run the spark job on k8s?
glamorous-carpet-83516
11/13/2023, 7:40 PMsalmon-refrigerator-32115
11/13/2023, 7:41 PMsalmon-refrigerator-32115
11/13/2023, 7:41 PMsalmon-refrigerator-32115
11/13/2023, 7:42 PMglamorous-carpet-83516
11/13/2023, 7:43 PMsalmon-refrigerator-32115
11/13/2023, 7:44 PMglamorous-carpet-83516
11/13/2023, 7:46 PMpyflyte run spark.py wf # Run spark in local python process
pyflyte run --raw-output-data-prefix <s3://saprk/output> spark.py wf # We serialize the input and upload to s3, and submit the databricks job.
is this command confused to you? or should we add new flag, like pyflyte run --hybrid, --agent, or something elseglamorous-carpet-83516
11/13/2023, 7:47 PMfreezing-airport-6809
billions-midnight-10687
11/15/2023, 2:34 PMbillions-midnight-10687
11/15/2023, 2:36 PM