GSoC 2016 - Add MongoDB to Tajo Storage

Introduction

The purpose of this blog post is to integrate and describe my contribution to Apache Tajo under Google Summer of Code Program 2016. Further this post describe issues I faced while implementing the MongoDB storage module for Apache Tajo and what are the possible/required improvements.

Apache Tajo.

Apache Tajo™ is a big data warehouse for Apache Hadoop. The key idea is that it supports SQL (actually Relational Database Management Systems) on top of Hadoop file system in a distributed manner. If you are more interested start it from here. 


Introduction to the Project - Add MongoDB Support for Apache Tajo
Is it only works as a data warehouse for Hadoop? of course not. Since it has a generic structure of reading and writing data, it is possible to connect other data storage systems to Apache Tajo. In other words Tajo will work as a big data warehouse for other storage systems too. All you need is a storage module for particular storage system. In that case, Tajo already contains default storage modules for HDFS, RDBMS(example:- MySQL, PostgreSQL), Amazon S3. My project was to implement a storage module for MongoDB. Then users can connect their mongodb databases to Apache Tajo and perform queries on them.

Commitment

• I implemented the module for mongodb storage(tajo-storage-mongodb)  inside the tajo-storage module.  Source code of that module can be found here. https://github.com/janakact/tajo/tree/TAJO-2079/tajo-storage/tajo-storage-mongodb 
  

• Commits I added to the Apache Tajo code base can be found here. Since the code is not yet merged to the original repository, I have added the link in my forked repository. https://github.com/janakact/tajo/commits/TAJO-2079?author=janakact&page=1

• TAJO-2079 is the branch name for the issue. Build status of that branch can be found at Travis-CI. https://travis-ci.org/janakact/tajo

• Further I added a user guide document to Apache Tajo Docs. It contains basic instructions on how to configure a MongoDB Table Space and how to set-up a MongoDB cluster using Apache Tajo. https://github.com/janakact/tajo/blob/TAJO-2079/tajo-docs/src/main/sphinx/storage_plugins/mongodb.rst

For more details on the module I implemented, you can refer to my blog posts.

• http://janakact.blogspot.com/2016/06/implemented-components-at-mid-term.html
• http://janakact.blogspot.com/2016/05/the-first-week-gsoc.html 

 

Issues to be Solved and Future Development

Table Scheme Problem
Apache Tajo tajo can handel two kinds of table spaces. 

• MetaData provided table spaces
• Here, the table-space itself provide the meta data of tables. It provides data such as,
• List of tables in the database
• Table Schemes
• Table Statistics (Ex:- Number of rows)  
• For examples MySQL Table Space, PostgreSQL Table Space can be taken. They are well structured db systems. They contain all the metadata required so table space itself can provide meta data
• File Spaces (Table Spaces which do not provide metadata) 
• Apache Tajo's primitive functionality is to provide SQL on top of Hadoop (or any other file system). File systems can't provide metadata themselves because they do not keep schemes, statistics inside them. 
• Therefore Tajo maintains a catalog which contains meta-data of the table space, schema details, statistics, etc. 

MongoDB is on other hand do not have a scheme, but it can provide some kind of metadata such as,

• The list of tables(Collections actually, Tajo tables are mapped into collections in MongoDB)
• Table statistics

Therefore it was encouraged to implement MongoDB table space as a meta data provided table-space, but MongoDB does not maintain a schema. So schema details can't be provided by the table space. 

At the end of the day this leads not to maintain the schema details by anyone.  Catalog do not maintain it because meta-data is provide by table-space, table-space can't maintain it because MongoDB is schema less. 

We need to solve this problem. For reading purpose, this doesn't matter too much but for data insertion schema is really important. Because of this issue, even though appender is implemented mongodb-tablespace still do not support insert queries. 

Support Nested Columns
Apache Tajo supports nested columns in its tables. MongoDB also supports nested columns as Embedded Documents.  The current implementation of tajo-storage-mongodb do not support nested columns. It should be implemented in the future. 

Query Testing for Insertion
Because of the schema problem mongodb table space do not support insert queries. All the internal work for data insertion to mongodb table space is done, therefore ASAP the schema problem is resolved the Query Testing for Insertion should be enabled(test-code is already added). 

Improving Performance 
Performance of the module can be improved in various aspects. For a example higher level projections and filtering can be push-downed to the MongoDB for better performance.

So that's it. :-) 

Last but not least I want to thank Jaehwa Jung(my mentor), Jihoon Son and other members of Apache Tajo community for helping and immensely guiding me throughout the project.

I got a great exposure to many technologies and concepts. It was one of the best experiences in my life (As a programmer it's the best experience up to now ;-) and I am planning to contribute more in the future.  

Automate the build and run in local machine - GSoC 2016

Wrote a small bash script today. Just few lines. It do these things specifically,

• Build tajo-storage-mongodb module and copy the .jar file to the snapshot. The snap shot is already configured to use mongo storage.
• Remove the logs. 
• Start tajo. 
• Wait for a little and open the log file with gedit. 
• Stop tajo 

Lol. It is really a small script, but it simplified my work a lot.

Anyway I was able to run tajo with this configurations. 

Of course table space don't do anything yet but seeing something like this makes me really happy. :D

The First Week ( GSoC )

The beginning of the coding period was not actually rushing as expected. This week was allocated to discuss the architecture of the module, with my mentors. Actually it was done a long before. Of course still there are questions regarding the architecture but they can't be solved before hand. They will be solved during the implementation. It's agile guys!

Project at the moment

Created a new module for mongodb storage plugin which is going to be implemented throughout the summer by me ;) 

Created the following main classes by implementing those interfaces and abstract classes.

• MongoDbTableSpace
• MongoDbFragment
• MongoDbScanner 
• MongoDbAppender

Also implemented a class called ConnectionInfo to keep MongoDB connection. When I implement it copplied a lot from the JDBC connection info class. Thank you blrunner. Hope you will not be mad at me about that. ;)

Problems and Solutions

Let's discuss about some questions came across in the first week. The first question was regarding the newly created module. When I buld using mvn command it says the module was build successfully but the relevant jar was not in the snapshot. I couldn't find why was it. Actually I build it several time (around 10 times) by changing pom.xml file several times. Problem was not with the mvn configurations. The module was build in the module directory, but it should be copied into the snapshot directory. It is done by a command in pom.xml of tajo-storage module. Anyway I added the lien and it started to work fine.

The next question is replication. It is something complex. ;) The thing is that in configurations for hdfs you can provide multiple hosts. MongoDB also can have multiple hosts as replica. Should the storage plugin I write include that functionality? If so, how the URI passed a question. For a table space details will of the table is given as a URI. By default java URI don't allowed multiple hosts. Then how hdfs do that? It is something to be studied.

Optional

I setup the Travis for my GitHub account. It is cool. I mean great. It can be name as one of the coolest things provided in the internet. Traivis automatically build the project in my GirHub repositories. We can configure it with travis.yml. And the best thing is it is completely free for opensource projects. :D :D 

The Simple Contribution - GSoC 2016

Got a reply from the mongo community. Seems like I have to learn a lot about mapping document based databases to column based databases.

Yesterday something marvelous happened. My mentor asked me to do a commit. Actually he told me how to do it. First I couldn't even understand the issue, but somehow he explained it really well. I made the changes in my repo yesterday, today I  make the pull request. Lol, I should have done it yesterday, but I had doubts. Anyway Travis the bot is doing tests automatically. I don't have to worry about that. 😂 I think what I edited do not effect unit tests or integration tests but Jaehwa said that after testing with MariaDB server, he'll prepare to commit my patch.

Further I got an email from a student(Subashini Hariharan) who is doing her Masters. She wants to add Cassandra plugin for Tajo as her Master's project. I don't know whether it is enough for that but I think it's a great idea. I think it is possible and will be easier to do compared to the MongoDB. So I introduced her to my mentors.

Still I need to understand the storage module architecture. It can't be much complicated. I want understand how to map Mongo Collections to Tajo tables. Thing is that we don't have much time. So many assignments and submissions. I am going to go through Storage Module again today. That's it for today. 

Moving On - GSoC 2016

So far, I filled Google Tax form. It get rejected once and submitted again. Hope this time it will not be rejected. In the development process still no progress. The thing is that I am busy with a lot of academic works these days. Poor me!

Also I have interviews for internships in the next semester. What a tragedy. Anyway, I have to get prepared for it too.

But I found some solutions for above scenarios. Part of my semester project is to develop a web service to collect data from a mobile application. Yeah I have to develop mobile app too ;)
I chose to develop the web service using java and MongoDB. ;-)
Now I have to use mongo driver for that. I am using it as a pre-project to get familiar with mongo java driver. Today I tried it with several applications, works fine. I am positive. :D :D

Now I am gonna connect a MySQL database in my local machine with the Tajo in the virtual machine. ;) Thanks to VMware, it has a virtual network card too. I can just ping from my physical machine to virtual machine. Funny. ;)  I am following Jaehwa's docs.

Finally, GSoC Payoneer account thing is really complicated. I always have doubts about that. What to do. This looks like a diary, isn't it? Never mind. It's public anyway.

The First Commit - GSoC 2016

This is something I should have started while before. Last month I applied for Google Summer of Code. My project was to add MongoDB support for Apache Tajo. The thing is that I got selected. I got the email today.

:D :D Yaai.. Congratulations to me! ;) ;)

https://summerofcode.withgoogle.com/projects/#4541666853650432

From today onward, I am gonna log my contribution in this blog daily or weekly. Before moving to that here we go with the background and my behaviors up to now.

This is not the first time I applied for GSoC. I did it last year but I didn't get in.

"Keep trying, don't settle.!"

This year, I started to go through the organization lists last month when they were announced. Yeah, Apache is a giant one. I am not sure why I chose this project. It's like the only project I understand from the given list by Apache Community. ;)  https://issues.apache.org/jira/browse/TAJO-2079?filter=12334770

I added a comment there. It's great. I got the reply quickly. Then I followed the guide lines. Registered to the mailing list, and waited. I didn't tried to start a new thread because I wanted to know what were the discussions going on at he moment.

Waited few days but didn't get any email from the subscription. :/ why is that?

Seems like it's better to start a thread. Yeah. I started a thread with the "Add MongoDB to Tajo Storage". Then I got the reply. Ok, no need to go into details. It was Jihoon Son, Jaehwa Jung were the people who helped me through mailing list.

Then they gave me permission to edit the confluence wiki of Apache and I started creating my proposal there. https://cwiki.apache.org/confluence/display/TAJO/Add+MongoDB+to+Tajo+Storage+-+Proposal

I had a great help from Jaehwa and Jihoon during the application period.

Meanwhile I setup the development environment. First I setup it in an Ubuntu virtual machine, but it had no enough resources. Then I had to install it as a parallel boot. Now it is time to start development. :D