Apache Spark cluster installation on Kubernetes | Apache Spark Helm Chart

Описание

#spark #kubernetes #helm

Apache Spark is one of the most widely used open-source clusters for big data processing. Installing Spark on Kubernetes using a helm chart is a great way to quickly deploy and manage a Spark cluster in a production environment. In this video, we will discuss the process of setting up a Spark cluster on Kubernetes using a helm chart and the various options available to customize your cluster.

Please subscribe by clicking the below link to support my channel:
https://bit.ly/3gM6elC

Commands to execute in kubernetes master node.
./helm repo add bitnami-repo https://charts.bitnami.com/bitnami
./helm install spark-release bitnami-repo/spark

If you want to scale up worker nodes please use this command
------------------------------------------------------------------------------------------------------
./helm upgrade spark-release bitnami-repo/spark --set worker.replicaCount=5

To expose the spark master pod use this command
------------------------------------------------------------------------------------
kubectl port-forward --address 0.0.0.0 pod/spark-release-master-0 30010:8080

To access spark master
---------------------------------------
http://10.0.0.10:30010/


spark submit example
-------------------------------------
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://spark-release-master-0.spark-release-headless.default.svc.cluster.local:7077 --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 examples/jars/spark-examples_2.12-3.3.1.jar 10

spark kubernetes
spark kubernetes tutorial
spark kubernetes configuration
spark kubernetes helm
spark kubernetes client mode
install spark on kubernetes
spark submit kubernetes
spark bitnami
spark deployment on kubernetes
apache spark kubernetes
spark helm
spark configuration
helm charts kubernetes
helm commands
kubernetes helm
kubernetes helm tutorial
spark submit
spark submit command
spark submit jar file example
spark submit in cluster mode
spark on kubernetes
kubernetes
spark
helm chart tutorial
helm kubernetes
helm tutorial
pyspark tutorial