mchampaneri.in

About project 
YTDownload ( Server ) is a server-based application. Its aim is to download video from youtube in all format and qualities provided by youtube. 
It uses a web browser for its UI. The core logic of the server is written in goLang. 

Initially, It was created as a web service. We were hosting this site through VPS but due to lack of resources, I have to shut it down. 

YTDownload is now open sourced, you can get it from source-forge.






Built withYTDownload ( Server ) Project is built using below frameworks, libraries and core GoLang.
VueJS ( JavaScript FrameWork ) Spectre ( Css FrameWork ) YTDL ( GoLang Youtube download Library ) Gorilla Mux ( GoLang HTTP Routing Library )
Pros
It's a server-based application. So you just have to start the server and you will be able to use it from any device having a browser and connected to the same network using local address of the server.No need to install 

Cons 
Downloads with a single thread, so downloading speed will be less than those other d…

Note: This is a case study to share what I faced, this might not be new to you and may have a better solution then I find. (You can share you solution here 😊) . 

At first sight, datastore looks very easy to work with. And it is if insertions have to be done at a low rate. But when insertions are done at a high rate you come to find out certain issues which are not because your program is faulty but because you don't read the ( long detailed ) documentation of datastore, that itself has certain restrictions you might never hear before.



Timeout Problems
There are two kinds of timeouts
1. Call error 11: Deadline exceeded
2. API error 5 (datastore_v3: TIMEOUT)


Call error 11: Deadline exceeded

This kind of time out happens when your datastore operation takes time more than 60 sec to finish the operation. 
For example, you are inserting too much data with a putMulti function. That takes time more then 60s insert.


API error 5 (datastore_v3: TIMEOUT)

This mostly occurs due to write contention.

When…

I was working on a module where data was comming from datastore and paginated using cursor. If you are not familier with datastore, then you can refer to this link.

In case of datastore we just have cursor to get next set of data. Though there is offset availabe in datastore its costly. So to be cost effective we have only option available is cursor.

Datastore decodes cursor and in response returns data associated with that cursor.

Now, If you are using bootstrap data-table you might find it cofusing that how to pass the cursor to your queryparameter.

Even I still don't get answer of how I can change data-queryparamter dynamically to update cursor on the url.

But instead I got another simple solution.

HTML Snippet
<table id="table" data-toggle="table" > <thead> <tr> // your data table fields </tr> </thead> <tbody></tbody> </table> <nav aria-label="Page navigation example" style…

Google Cloud Provides a ready made script to run scheduled export from datastore.
I am using that export data as input to BigQuery.

To get it done, I had created two crons,

First for calling data export operationSecond for loading that exported data to BigQuery

Problem I was facing was that export was taking some time more time than I have assumed, So Bigquery loading cron was getting initiated before data export opertion get completed, and result bigquery loading was failing.

Solution was to make those two cron job as a single one which works as chain. On successful completion of task one second gets called.

So, I just modified the schedule export script provided by google, which exports files, and then checks if operation is successfully done or not fixed time intervals in loop, and if operation is found to be done successfully, it calls BigQuery load task.

Here is gist

You can provide partial search [ type ahead facility ] with firestore. It's not that much streight forward and also not feasible in case of large amount of data that has to support the partial search.

Here is just a sample of how you provide partial search with firestore.

Logic behind the search is a custom kind of indexing that enables partial search for us. You need to create a collection of documents called indexes which contains indexes generated for the fields you want to search over.



First reqirement is to create every possible partial search strings.
Below function returns array of the possible indexes generate after chopping out input string character by character.


/* storing searchabel fields */function _chopped_object(iString) { var searchables = {} for (i = 0; i < iString.length; i++) { var string_part = iString.substr(0, iString.length - i) searchables[string_part] = true } return searchables }
Output of _chopped_object(manish) will be,
{
manish :true
man…

Firestore returning no records yet, I see them in consoleCollection not retuning any element.... ... 
You might facing issues written above. Then it might be this case.

You might getting empty response in this kind of query though there are elements in firestore.


db.collection(collection1).doc(doc1) .collection(collection2).doc(doc2) .collection(collection3).get() .then( function (querySnapshot) { querySnapshot.forEach(function (doc) { console.log(doc.id, " = ", doc.data()); } )
I face the same problem. Though data was stored sucessfully ( I was guessing, which was wrong actully ).  To verify that your data is not stored as you expceted you can do check this.

If key of your data is in italic font then your data is not stored or is marked as deleted [ that was happening in my case ]. See in image.


Now you might thinking, that I have created element and  not deleted it then why it's showing as deleted.

Truth is that if you want to create neste…