Cfrgtkky

I have a large DynamoDB (not huge around 20GB) that holds items with expiry time stamps.
We are having 32 parallel scanner reading the database while other entries are added to database from other parts of the system.
This is the code that we have at the moment

    val scanRequest = ScanRequest

        .builder()

        .tableName(tableName)

        .filterExpression("expiryDate < :date")

        .expressionAttributeValues(mapOf(":date" to expiredThreshold).asAttributeValueMap())

        .totalSegments(totalSegments)

        .segment(segment)

        .limit(1000)

        .exclusiveStartKey(lastEvaluatedKey)

        .build()



    val scanResult = client.scan(scanRequest).asDeferred().await()



    lastEvaluatedKey = scanResult.lastEvaluatedKey()

I have several concerns regarding this pattern.

1. Would we ever reach the end of each segment by having this in a while or not? If so, how can we know if the end has been reached or not?


2. Would the items that are being added from other part go before the lastEvaluatedKey or everything would go after?


3. What would be a good indication for starting reading again from the beginning of the segment?