mchampaneri.in

Posts

Go-boiler : smaller and simplest starter kit

- April 29, 2019

Project Name
Go-boiler

Project Descrtiption
Simple, small & easy to get started boiler-plate(starter-kit) written in golang for creating website/webapps.

External libraries
Cloudykit/Jet ( HTML templating engine)gorilla/mux ( HTTP Routing )gorilla/csrf ( CSRF )

Source code
https://github.com/mchampaneri/go-boiler

Documentation
https://mchampaneri.github.io/go-boiler

Customize Scrollbar look

- March 25, 2019

You can customize the look of scrollbar using CSS.
Here is a snippet of sample custom scrollbar UI,

( I have introduced :hover because I want to show scrollbar on an element only when user hover on that element )

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19*::-webkit-scrollbar{width:5px;display:none;background-color:#666666;}*:hover::-webkit-scrollbar{width:5px;display:block;background-color:#666666;}*::-webkit-scrollbar-track{background:#ddd;}*::-webkit-scrollbar-thumb{background:#666;border-radius:20px;}

YTDownload ( Server )

- March 02, 2019

Project Name
YTDownload

Project Descrtiption
YTDownload ( Server ) is a server-based application. Its aim is to download video from youtube in all format and qualities provided by youtube.
It uses a web browser for its UI. The core logic of the server is written in goLang.

External libraries & dependcies
VueJS ( JavaScript FrameWork )Spectre ( Css FrameWork )YTDL ( GoLang Youtube download Library )Gorilla Mux ( GoLang HTTP Routing Library )Pros
It's a server-based application. So you just have to start the server and you will be able to use it from any device having a browser and connected to the same network using local address of the server.No need to install

Cons
Downloads with a single thread, so downloading speed will be less than those other downloaders provides.Can not use without the local server running.No mobile platform for the server is supported
Source code
https://sourceforge.net/projects/ytdownloader-xyz/

Download link
https://ytdownloader-xyz.sourceforge.io/

Datastore : Handling heavy insertions

- January 24, 2019

Note: This is a case study to share what I faced, this might not be new to you and may have a better solution then I find. (You can share you solution here 😊) .

At first sight, datastore looks very easy to work with. And it is if insertions have to be done at a low rate. But when insertions are done at a high rate you come to find out certain issues which are not because your program is faulty but because you don't read the ( long detailed ) documentation of datastore, that itself has certain restrictions you might never hear before.

Timeout Problems
There are two kinds of timeouts
1. Call error 11: Deadline exceeded
2. API error 5 (datastore_v3: TIMEOUT)

Call error 11: Deadline exceeded

This kind of time out happens when your datastore operation takes time more than 60 sec to finish the operation.
For example, you are inserting too much data with a putMulti function. That takes time more then 60s insert.

API error 5 (datastore_v3: TIMEOUT)

This mostly occurs due to write contention.

When…

Cursor based navigation for datatable

- January 02, 2019

I was working on a module where data was comming from datastore and paginated using cursor. If you are not familier with datastore, then you can refer to this link.

In case of datastore we just have cursor to get next set of data. Though there is offset availabe in datastore its costly. So to be cost effective we have only option available is cursor.

Datastore decodes cursor and in response returns data associated with that cursor.

Now, If you are using bootstrap data-table you might find it cofusing that how to pass the cursor to your queryparameter.

Even I still don't get answer of how I can change data-queryparamter dynamically to update cursor on the url.

But instead I got another simple solution.

HTML Snippet
<table id="table" data-toggle="table" > <thead> <tr> // your data table fields </tr> </thead> <tbody></tbody> </table> <nav aria-label="Page navigation example" style…

datastore to bigquery data export

- December 07, 2018

Google Cloud Provides a ready made script to run scheduled export from datastore.
I am using that export data as input to BigQuery.

To get it done, I had created two crons,

First for calling data export operationSecond for loading that exported data to BigQuery

Problem I was facing was that export was taking some time more time than I have assumed, So Bigquery loading cron was getting initiated before data export opertion get completed, and result bigquery loading was failing.

Solution was to make those two cron job as a single one which works as chain. On successful completion of task one second gets called.

So, I just modified the schedule export script provided by google, which exports files, and then checks if operation is successfully done or not fixed time intervals in loop, and if operation is found to be done successfully, it calls BigQuery load task.

Here is gist

partial search with firestore

- November 28, 2018

You can provide partial search [ type ahead facility ] with firestore. It's not that much streight forward and also not feasible in case of large amount of data that has to support the partial search.

Here is just a sample of how you provide partial search with firestore.

Logic behind the search is a custom kind of indexing that enables partial search for us. You need to create a collection of documents called indexes which contains indexes generated for the fields you want to search over.

First reqirement is to create every possible partial search strings.
Below function returns array of the possible indexes generate after chopping out input string character by character.

/* storing searchabel fields */function _chopped_object(iString) { var searchables = {} for (i = 0; i < iString.length; i++) { var string_part = iString.substr(0, iString.length - i) searchables[string_part] = true } return searchables }
Output of _chopped_object(manish) will be,
{
manish :true
man…

Search This Blog