Showing posts from January, 2019

Datastore : Handling heavy insertions

Note: This is a case study to share what I faced, this might not be new to you and may have a better solution then I find. (You can share you solution here 😊) . 

At first sight, datastore looks very easy to work with. And it is if insertions have to be done at a low rate. But when insertions are done at a high rate you come to find out certain issues which are not because your program is faulty but because you don't read the ( long detailed ) documentation of datastore, that itself has certain restrictions you might never hear before.

Timeout Problems
There are two kinds of timeouts
1. Call error 11: Deadline exceeded
2. API error 5 (datastore_v3: TIMEOUT)

Call error 11: Deadline exceeded

This kind of time out happens when your datastore operation takes time more than 60 sec to finish the operation. 
For example, you are inserting too much data with a putMulti function. That takes time more then 60s insert.

API error 5 (datastore_v3: TIMEOUT)

This mostly occurs due to write contention.


Cursor based navigation for datatable

I was working on a module where data was comming from datastore and paginated using cursor. If you are not familier with datastore, then you can refer to this link.

In case of datastore we just have cursor to get next set of data. Though there is offset availabe in datastore its costly. So to be cost effective we have only option available is cursor.

Datastore decodes cursor and in response returns data associated with that cursor.

Now, If you are using bootstrap data-table you might find it cofusing that how to pass the cursor to your queryparameter.

Even I still don't get answer of how I can change data-queryparamter dynamically to update cursor on the url.

But instead I got another simple solution.

HTML Snippet
<table id="table" data-toggle="table" > <thead> <tr> // your data table fields </tr> </thead> <tbody></tbody> </table> <nav aria-label="Page navigation example" style…