seunggs
01/01/2023, 4:13 AMFelix Ruess
01/02/2023, 2:40 PMGiacomo Dabisias IT
01/02/2023, 4:53 PMOphir Yoktan
01/02/2023, 7:02 PMPanos Strouth
01/03/2023, 10:45 AM{"json":{},"level":"error","msg":"Error creating auth context [AUTH_CONTEXT_SETUP_FAILED] Error creating oidc provider ....
flytectl or browser does not redirect to Cognito UI for authentication.Chandramoulee K V
01/03/2023, 1:00 PMspark
related doubt here.
Scenario:
Currently while executing the spark workflow The driver and the executors are being scheduled in different pods.
Eg: we have 1 driver (4 cores CPU 8GB Memory
) and 4 executors(4 cores CPU 8GB Memory each)
-> 1 node for 1 driver pod and 1 node to accommodate all 4 executor pods.
Here the node to accommodate the executors is very large as the request sent for the node is the summation of CPU's and memory of all 4 executors combined so the request is greater than 16 cores and 32 GB memory which will be inefficient going forward with more number of executors.
So is there a workaround/fix to make this scale horizontally i.e... spawn up 4 executor pods in separate nodes (or a combination of n nodes to hold n executor pods each node) so we will have nodes running in parallel instead of pods running in parallel inside a single very large node
?Seth Baer
01/03/2023, 4:07 PMDan Rammer (hamersaw)
01/03/2023, 5:37 PMDavid Cupp
01/03/2023, 6:33 PMLaunchPlanCreateRequest
?
Are both of these supported workflows, or is # 2 considered a bit of a hack?Rahul Mehta
01/04/2023, 7:17 AMMücahit
01/04/2023, 9:20 AMflyte:admin:database:postgres;*
to flyte:admin:admin:database:*
etc and it doesn’t seem to be that a quick search/replace can fix it. Do you have any up-to-date dashboard for Flyteadmin/control plane?
• I’ve looked at the monitoring docs, and it looks pretty neat, but I was wondering if you have any alert rules for checking system health?Robin Eklund
01/05/2023, 9:52 AM@task
def print_the_date(execution_date: str) -> str:
print(f"in print_the_date: execution_date={execution_date}")
return execution_date
@workflow
def my_wf(execution_date: str) -> dict[str, str]:
print(f"inside my_wf: execution_date={execution_date}")
print_the_date(execution_date=execution_date)
return {"execution_date": execution_date}
This works perfectly fine running locally like this:
pyflyte run workflow.py my_wf --execution_date 2023-01-01
But not sure how to do this on a scheduled basis, i have tried with this:
LaunchPlan.get_or_create(
name="my_lp",
workflow=my_wf,
schedule=CronSchedule(schedule="0 7 * * *"),
security_context=security_context,
default_inputs={
"execution_date": datetime.today().strftime('%Y-%m-%d %H:%M:%S')
}
)
But i understand this doesn't work because it created the default inputs in "compile-time" of the workflow(?).
Also i would like to be able to do like this when running locally:
pyflyte run workflow.py my_wf
But it seems Flyte are not supporting default input like this:
@workflow
def my_wf(execution_date: Union[str, None] = None) -> dict[str, str]:
...
Anyone who have done this before that can point me in the right direction, would appreciate the help!Felix Ruess
01/05/2023, 4:54 PMFelix Ruess
01/05/2023, 5:13 PMFrank Shen
01/05/2023, 7:50 PM@task(
container_image="<http://xyz.dkr.ecr.us-east-1.amazonaws.com/flyte-pyspark:latest|xyz.dkr.ecr.us-east-1.amazonaws.com/flyte-pyspark:latest>",
task_config=Spark(
spark_conf={...
}
),
)
def read_spark_df() -> pandas.DataFrame:
sess = flytekit.current_context().spark_session
spark_df = sess.read.parquet("<s3a://bucket/key.parquet>").toPandas()
df = pandas.DataFrame(spark_df)
return df
Frank Shen
01/05/2023, 7:52 PMFelix Ruess
01/05/2023, 8:39 PMFrank Shen
01/06/2023, 6:46 PM@task(
container_image="<http://xyz.dkr.ecr.us-east-1.amazonaws.com/flyte-pyspark:latest|xyz.dkr.ecr.us-east-1.amazonaws.com/flyte-pyspark:latest>",
task_config=Spark(
spark_conf={...
}
),
)
def read_spark_df() -> pandas.DataFrame:
sess = flytekit.current_context().spark_session
spark_df = sess.read.parquet("<s3a://bucket/key.parquet>").toPandas()
....
Frank Shen
01/06/2023, 6:49 PMjava.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:53)
Frank Shen
01/06/2023, 6:51 PMFrank Shen
01/06/2023, 10:38 PMFrank Shen
01/06/2023, 10:39 PMFrank Shen
01/06/2023, 10:40 PMRahul Mehta
01/06/2023, 11:37 PMFrank Shen
01/07/2023, 12:43 AMpy4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.io.IOException: Failed to connect to <http://nyxmmedina01741.wmad.warnermedia.com/10.217.173.85:55808|nyxmmedina01741.wmad.warnermedia.com/10.217.173.85:55808>
Stephen
01/09/2023, 10:45 AM<http://ad.datadoghq.com/|ad.datadoghq.com/><CONTAINER_IDENTIFIER>.logs: '[<LOG_CONFIG>]'
-> <CONTAINER_IDENTIFIER>
here needs to be identical to the container name from which to scrape the logs.
When running Flyte tasks the container name is generated from the execution ID which cannot be known in advance. It is possible to configure default annotations for all Flyte workflow pods centrally. It is even possible to inject things like project name and domain dynamically into these default annotations but it is not possible to dynamically inject the container name(s).Niels Bantilan
01/09/2023, 2:37 PMFabio Grätz
01/09/2023, 4:00 PMargs
of both pods specify the same values for:
- --output-prefix
- gs://.../metadata/propeller/sandbox-development-f6695ca08aa47490c859/n0/data/0
- --raw-output-data-prefix
- gs://.../xq/f6695ca08aa47490c859-n0-0
- --output-prefix
- gs://...metadata/propeller/sandbox-development-f6695ca08aa47490c859/n0/data/0
- --raw-output-data-prefix
- gs://.../xq/f6695ca08aa47490c859-n0-0
In case both return the same value (as ist assumably often the case), it shouldn’t matter if both write. But in case I want to return a metric which might be slightly different for each worker, is it random which one I get?Samhita Alla
Felix Ruess
01/09/2023, 5:52 PM