企业官网建设流程全解析

spark序列化报错

org.apache.spark.SparkException:Kryo serialization failed:Buffer overflow.Available:0,required:61186304.To avoidthis,increase spark.kryoserializer.buffer.max value.at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:300)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:313)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)

问题解决

the problem is that kryo does not have enough room in its buffer.

• You should be adjustingspark.kryoserializer.buffer.maxin your properties file,
• or use--conf "spark.kryoserializer.buffer.max=128m"in your spark-submit command.