Accessing Additional Spark LogsΒΆ
Apart from accessing the logs from the QDS UI, you can also access the following logs, which reside on the cluster, to identify the errors and exceptions in Spark jobs failures:
| Logs | Location | Desciption |
|---|---|---|
| Spark History | hdfs://master:9000/spark-history |
This location contains one file per one Spark application and it contains the Spark event logs. |
| Spark History Server Logs | The spark-yarn-org.apache.spark.deploy.history.HistoryServer-1-localhost.localdomain.log file is stored at /media/ephemeral0/logs/spark. |
The Spark history server logs are stored only on the master node of the cluster. |
| Spark Event Logs | The Spark eventlog files are stored at <scheme><defloc>/logs/hadoop/<cluster_id>/<cluster_inst_id>/spark-eventlogs |
|
Spark History: The Spark history is stored at
hdfs://master:9000/spark-history. This location contains one file per one Spark application and it contains the Spark event logs.Spark History Server Logs: The
spark-yarn-org.apache.spark.deploy.history.HistoryServer-1-localhost.localdomain.logfiles are stored at/media/ephemeral0/logs/spark. The Spark history server logs are stored only on the master node of the cluster.Spark Event Logs: The
Spark eventlogfiles are stored at<scheme><defloc>/logs/hadoop/<cluster_id>/<cluster_inst_id>/spark-eventlogswhere:- scheme is the Cloud-specific URI scheme:
s3://for AWS;wasb://oradl://for Azure;oci://for Oracle OCI. - defloc is the default storage location for the QDS account.
- cluster_id is the cluster ID as shown on the Clusters page of the QDS UI.
- cluster_inst_id is the cluster instance ID. You should contact Qubole Support to obtain the cluster instance ID.
- scheme is the Cloud-specific URI scheme: