How do I limit write operations to 1k records/sec?
0
Currently, I am able to write to database in the batchsize of 500. But due to the memory shortage error and delay synchronization between child aggregator and leaf node of database, sometimes I am running into Leaf Node Memory Error. The only solution for this is if I limit my write operations to 1k records per second, I can get rid of the error. dataStream .map(line => readJsonFromString(line)) .grouped(memsqlBatchSize) .foreach { recordSet => val dbRecords = recordSet.map(m => (m, Events.transform(m))) dbRecords.map { record => try { Events.setValues(eventInsert, record._2) eventInsert.addBatch } catch { case e: Exception => logger.error(s"error adding batch: ${e.getMessag