StreamingLens Insights always showing "Streaming Query State: NONEWBATCHES" in Logs.

104
1
github

StreamingLens Insights always showing "Streaming Query State: NONEWBATCHES" in Logs.

Hi All,

I am using StreamingLens in my spark structure streaming application but it's always showing same logs .BatchId is getting updated but Streaming Query State: NONEWBATCHES remains same. can someone suggest why the State and recommendations are not updating in logs.

**|||||||||||||||||| StreamingLens Insights |||||||||||||||||||||||||

BatchId: 344

Analysis Time: 00s 000ms

Expected Micro Batch SLA: 120s 000ms

Batch Running Time: 00s 000ms

Critical Time: 00s 000ms

Streaming Query State: NONEWBATCHES

||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

21/10/01 15:50:04 WARN QueryInsightsManager: Streaming Lens failed key not found: BatchDescription(e68c3c2c-6d5f-469e-864a-)**

Spark Submit Command:

**spark-submit \

--verbose \

--name SparkStreamingLens \

--num-executors 1 \

--jars /home/abc/jars/spark-streaminglens_2.11-0.5.3.jar,\

/home/abc/jars/kafka-clients-0.10.2.1.jar, \

--master yarn \

--deploy-mode cluster \

--driver-cores 1 --driver-memory 2G --executor-cores 1 --executor-memory 2G \

--supervise --class com.data.datalake.SparkStreamingLens \

/home/abc/jar/SparkStreamingLens-spark-utility_2.11-1.0.jar **

Kindly Guide if is there anything needs to change here.

https://github.com/qubole/streaminglens/blob/master/src/main/scala/com/qubole/spark/streaminglens/common/results/AggregateStateResults.scala

https://github.com/qubole/streaminglens/blob/master/src/main/scala/com/qubole/spark/streaminglens/common/results/StreamingCriticalPathResults.scala

As in Project (com.qubole.spark.streaminglens.QueryInsightsManager) below code is available to fetch the insights.

**| |||||||||||||||||| StreamingLens Inisights |||||||||||||||||||||||||

| BatchId: ${results.batchId}

| Analysis Time: ${pd(results.analysisTime)}

| Expected Micro Batch SLA: ${pd(results.streamingCriticalPathResults.expectedMicroBatchSLA)}

| Batch Running Time: ${pd(results.streamingCriticalPathResults.batchRunningTime)}

| Critical Time: ${pd(results.streamingCriticalPathResults.criticalTime)}

| Streaming Query State: ${results.streamingCriticalPathResults.streamingQueryState.toString}

| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

""".stripMargin)**

Here taking all the details from streamingCriticalPathResults and here only code available for NONEWBATCH State

**case class StreamingCriticalPathResults(expectedMicroBatchSLA: Long = 0,

batchRunningTime: Long = 0,

criticalTime: Long = 0,

streamingQueryState: StreamingState.Value = StreamingState.NONEWBATCHES)**

Kindly Suggest.

@abhishekd0907 @itsvikramagr @shubhamtagra @jsensarma @mjose007

Hi, strange, it may be worth using another software. Perhaps you will find solutions to the problem here livestream studio, on this site everything is written about streams, platforms, and software that are recommended to use. I think you will solve the problem as soon as possible, good luck.