• Haytham Abuelfutuh

    Haytham Abuelfutuh

    2 months ago
    set the channel description: Integration w/ Databricks
  • c

    Christopher Chalcraft

    2 months ago
    Hello all. Chris from databricks here. Nothing official yet but just scoping.
    c
    1 replies
    Copy to Clipboard
  • Ketan (kumare3)

    Ketan (kumare3)

    2 weeks ago
    cc @Evan Sadler
  • Ketan (kumare3)

    Ketan (kumare3)

    3 days ago
    @Mehtab Mehdi can you paste your question here
  • m

    Mehtab Mehdi

    2 days ago
    i wrote the following code to integrate with flyte and databricks, could you please help me to resolve it import datetime import random,json from operator import add import flytekit from flytekit import Resources, task, workflow from pyspark import SparkContext from operator import add from flytekitplugins.spark import Spark from airflow.providers.databricks.operators.databricks import DatabricksRunNowOperator @task( task_config=DatabricksRunNowOperator(task_id='notebook_run', tasks=[{ "spark.driver.memory": "1000M", "spark.executor.memory": "1000M", "spark.executor.cores": "1", "spark.executor.instances": "2", "spark.driver.cores": "1", "new_cluster": {"spark_version": "2.1.0-db3-scala2.11", "num_workers": 2}, "notebook_task": {"notebook_path": "/Users/airflow@example.com/PrepareData"}, }])) def hello_spark(partitions: int) -> float: print("Starting Spark with Partitions: {}".format(partitions))

    n = 100000 * partitions

    sess = flytekit.current_context().spark_session
    m
    Ketan (kumare3)
    4 replies
    Copy to Clipboard