Posts

Go-boiler : smaller and simplest starter kit

-
Image
About project

In my every project, I was using the same dependency every time like 
Cloudykit/Jet for HTML templatinggorilla-mux for routing gorilla-csrf  for csrf protection.


Even using some fix helping functions like view, json and htmlString for rendering , and staticRoutes and dynamicRoutes for route registrations.
So, I decided to make a starter-kit including minimal things I was using every time for the basic setup of the project.
That's go-boiler.
I know that every project has different requirements, so instead of giving a full-fledge boiler-kit go-boiler gives only those libraries you have to use anyway ( or similar one). There is nothing that you need to remove or even replace.
For example, This starter kit does not provide any database library, because it will be bad to assume that we will be using MySQL every time, we may have to use mongo even in some case.
In fututre, even this kit will only include libraries that are a basic requirement and no one need to remove it.
Source Co…
Post a Comment
Read more

Customize Scrollbar look

-
You can customize the look of scrollbar using CSS.
Here is a snippet of sample custom scrollbar UI,

( I have introduced :hover  because I want to show scrollbar on an element only when user hover on that element )




1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19*::-webkit-scrollbar{width:5px;display:none;background-color:#666666;}*:hover::-webkit-scrollbar{width:5px;display:block;background-color:#666666;}*::-webkit-scrollbar-track{background:#ddd;}*::-webkit-scrollbar-thumb{background:#666;border-radius:20px;}
Post a Comment
Read more

YTDownload ( Server )

-
Image
About project 
YTDownload ( Server ) is a server-based application. Its aim is to download video from youtube in all format and qualities provided by youtube. 
It uses a web browser for its UI. The core logic of the server is written in goLang. 

Initially, It was created as a web service. We were hosting this site through VPS but due to lack of resources, I have to shut it down. 

YTDownload is now open sourced, you can get it from source-forge.






Built withYTDownload ( Server ) Project is built using below frameworks, libraries and core GoLang.
VueJS ( JavaScript FrameWork ) Spectre ( Css FrameWork ) YTDL ( GoLang Youtube download Library ) Gorilla Mux ( GoLang HTTP Routing Library )
Pros
It's a server-based application. So you just have to start the server and you will be able to use it from any device having a browser and connected to the same network using local address of the server.No need to install 

Cons 
Downloads with a single thread, so downloading speed will be less than those other d…
Post a Comment
Read more

Datastore : Handling heavy insertions

-
Note: This is a case study to share what I faced, this might not be new to you and may have a better solution then I find. (You can share you solution here 😊) . 

At first sight, datastore looks very easy to work with. And it is if insertions have to be done at a low rate. But when insertions are done at a high rate you come to find out certain issues which are not because your program is faulty but because you don't read the ( long detailed ) documentation of datastore, that itself has certain restrictions you might never hear before.



Timeout Problems
There are two kinds of timeouts
1. Call error 11: Deadline exceeded
2. API error 5 (datastore_v3: TIMEOUT)


Call error 11: Deadline exceeded

This kind of time out happens when your datastore operation takes time more than 60 sec to finish the operation. 
For example, you are inserting too much data with a putMulti function. That takes time more then 60s insert.


API error 5 (datastore_v3: TIMEOUT)

This mostly occurs due to write contention.

When…
Post a Comment
Read more

Cursor based navigation for datatable

-
I was working on a module where data was comming from datastore and paginated using cursor. If you are not familier with datastore, then you can refer to this link.

In case of datastore we just have cursor to get next set of data. Though there is offset availabe in datastore its costly. So to be cost effective we have only option available is cursor.

Datastore decodes cursor and in response returns data associated with that cursor.

Now, If you are using bootstrap data-table you might find it cofusing that how to pass the cursor to your queryparameter.

Even I still don't get answer of how I can change data-queryparamter dynamically to update cursor on the url.

But instead I got another simple solution.

HTML Snippet
<table id="table" data-toggle="table" > <thead> <tr> // your data table fields </tr> </thead> <tbody></tbody> </table> <nav aria-label="Page navigation example" style…
Post a Comment
Read more

datastore to bigquery data export

-
Google Cloud Provides a ready made script to run scheduled export from datastore.
I am using that export data as input to BigQuery.

To get it done, I had created two crons,

First for calling data export operationSecond for loading that exported data to BigQuery

Problem I was facing was that export was taking some time more time than I have assumed, So Bigquery loading cron was getting initiated before data export opertion get completed, and result bigquery loading was failing.

Solution was to make those two cron job as a single one which works as chain. On successful completion of task one second gets called.

So, I just modified the schedule export script provided by google, which exports files, and then checks if operation is successfully done or not fixed time intervals in loop, and if operation is found to be done successfully, it calls BigQuery load task.

Here is gist
Post a Comment
Read more

partial search with firestore

-
Image
You can provide partial search [ type ahead facility ] with firestore. It's not that much streight forward and also not feasible in case of large amount of data that has to support the partial search.

Here is just a sample of how you provide partial search with firestore.

Logic behind the search is a custom kind of indexing that enables partial search for us. You need to create a collection of documents called indexes which contains indexes generated for the fields you want to search over.



First reqirement is to create every possible partial search strings.
Below function returns array of the possible indexes generate after chopping out input string character by character.


/* storing searchabel fields */function _chopped_object(iString) { var searchables = {} for (i = 0; i < iString.length; i++) { var string_part = iString.substr(0, iString.length - i) searchables[string_part] = true } return searchables }
Output of _chopped_object(manish) will be,
{
manish :true
man…
Post a Comment
Read more