Datastore : Handling heavy insertions
Note: This is a case study to share what I faced, this might not be new to you and may have a better solution then I find. (You can share you solution here 😊) .
At first sight, datastore looks very easy to work with. And it is if insertions have to be done at a low rate. But when insertions are done at a high rate you come to find out certain issues which are not because your program is faulty but because you don't read the ( long detailed ) documentation of datastore, that itself has certain restrictions you might never hear before.
Timeout Problems
There are two kinds of timeouts
1. Call error 11: Deadline exceeded
2. API error 5 (datastore_v3: TIMEOUT)
Call error 11: Deadline exceeded
This kind of time out happens when your datastore operation takes time more than 60 sec to finish the operation.
For example, you are inserting too much data with a putMulti function. That takes time more then 60s insert.
API error 5 (datastore_v3: TIMEOUT)
This mostly occurs due to write contention.
When…
At first sight, datastore looks very easy to work with. And it is if insertions have to be done at a low rate. But when insertions are done at a high rate you come to find out certain issues which are not because your program is faulty but because you don't read the ( long detailed ) documentation of datastore, that itself has certain restrictions you might never hear before.
Timeout Problems
There are two kinds of timeouts
1. Call error 11: Deadline exceeded
2. API error 5 (datastore_v3: TIMEOUT)
Call error 11: Deadline exceeded
This kind of time out happens when your datastore operation takes time more than 60 sec to finish the operation.
For example, you are inserting too much data with a putMulti function. That takes time more then 60s insert.
API error 5 (datastore_v3: TIMEOUT)
This mostly occurs due to write contention.
When…
