With Bitlab Studio’s doors open, I finally bid farewell to Akshara with a heavy heart…

The History

The last two times I quit my job, despite both the companies helping me grow and explore tremendously on a personal and career level, I had enough frustration built up over various reasons which served as a valid excuse for my decision. However this time, it is slightly different.

Ask me about my first job at HasGeek anytime, and I will tell you how I grew with them from a college graduate who was afraid to speak English, who was afraid to travel, to someone who could confidently strike up a conversation with any stranger; to a person who pursued his career interests; to a person who made his dream of flying in a plane come true.

Ask me about my second job at Eventifier anytime, and I will tell you how gracious they were to accept me into their company as their first employee despite me having almost 0 skills in industry level programming. I’ve written about it in detail already. The opportunity that they gave me changed my life, and I will be forever thankful to them.

However, what didn’t quite make it onto the blog during my days at Eventifier was the fact that I had my first ever international trip! And that too accompanied by giving my first ever talk at a conference! Which was at Singapore. We’ll get to that in a bit.

Life during the KLP years + one of the biggest failures in my life

I joined Akshara Foundation‘s Karnataka Learning Partnership (KLP) back in November 2014 with a 15-hr per week schedule. This was one month before my marriage when I need some financial fortification. While during the succeeding months the folks at Eventifier slowly began to realize that it was probably a bad idea to let one of their full time employee take up a part time gig on the side, at that time, they felt it best that I made that decision. Goes to show how understanding and empathetic the three of them were.

The engagement with KLP remained as it is until April 2016, when I quit Eventifier and moved over to KLP on a 4-day per week contract. It has been so until now. Oh, and during that April is when my ThinkPad saved me from lightning by taking the hit itself and led me to evolution:

The nature of work, while relaxed, was very result oriented and gratifying. Our entire work is open source as well. We had a completely remote team, which allowed me to move out of Bangalore. I never truly paid attention to setting up a proper office setup at home and spent most of my time typing away on the bed. Believe it or not, this was my posture every day for better part of the year:

laptop-bed

See how happy I am!

My DjangoCon US trip and presentation happened in between this. I was doing a non-trivial amount of work with Django REST Framework which helped me come up with a presentation.

It was only after that conference that it hit me the lengths people would go to, to setup a proper working space at home. Even with that inspiration, this was as far as I got, and that too only for 2 – 3 hours a day:

The failure:

That happiness soon turned into one of my biggest failures in life. By October-ish, I started feeling slightly stagnant in terms of technology. Well, maybe not really. I guess I was just getting complacent. I kept an eye out for opportunities and ended up landing a part time gig with one of my programming heroes, Anand Chitipothu, to work along with him at Rorodata.

Mistake number 1: Taking up a 3-day per week part time gig while I already had a 4-day per week engagement with Akshara (even if the nature of work was just trivial maintenance at that point).

We got on fine at the beginning. I hadn’t even anticipated the trouble with time management. This ended up in me staying up till 2am or 3am in the mornings. My sleep cycle screwed up considerably.

Mistake number 2: Failing to realize quickly that the work pace and style at an NGO and at bootstrapping startup are completely different.

I was relaxed. I would sleep if I felt sleepy, I would goof off if I couldn’t concentrate, which all ended up in the work piling up from both the companies day after day. This started creating frustration for my colleagues as it was almost as if they couldn’t count on me to finish my job in time.

I remember arrogantly thinking the compliment that my first boss, Kiran, gave me while I was at HasGeek: “You’re good at getting shit done”.

Mistake number 3: I got extremely complacent. With that, even if I missed a day’s work, I would think “They aren’t going to fire me right? I am that good. Everyone is dying to hire me”.

With this attitude, lacking any proper working setup at home, a 7-day per week work schedule, on February 20th, Rorodata decided to terminate my contract. I wasn’t surprised as I remember telling my family that it wasn’t working out and probably I was going to get fired (although my ego never truly believed that).

However, during that week’s PyCon Pune, I had a long conversation with Nigel, who helped me figure out what all I needed to do to setup a proper work space at home.

I had a new monitor, cleaned up and made an office room, bought an office chair and everything setup. That’s when I had to leave Rorodata.

Life goes on…

That was a blow, a very severe one, to my ego. I never quite came to terms with it really. It hurt bad. Real bad. I realized I sucked big time. Almost twice a week for the next 3-4 months after that, I would sulk away into a corner brooding about the opportunity that I had completely destroyed. It was a dream job, and I just… sigh…

Even though that nagged me all the time, I still had to keep up my work at KLP. The notion of being fired and unemployment scared me. I got my shit together and concentrated on work like never before. I learned my lesson.

The last few months

We had our yearly meeting during March 15th. The fact that everyone involved, at the core, worked to create meaning for Children’s education, and was not working for profit or make a huge exit, gave the work culture an interesting atmosphere.

The slack conversations and personal chats almost revolved around personal and practical problems in the society rather than the next coolest thing happening in the tech world.

The main difference I felt was not having that feeling of “your boss is good and friendly, but at the end of the day wants you to work really hard so she/he can make huge lump of money sometime down the line”. That wasn’t there. At all.

Work hard so that we can bring about sustainable change which will impact children’s education. If you couldn’t connect with that, you won’t be able to work long at this place.

It was relaxing, it was fun, and most importantly it was extremely gratifying. As I used to say, “Now I can show my Mom and Dad what I am really doing with my skills!”.

The last few months had been tremendous. We were evolving. From Karnataka Learning Partnership, we were growing into India Learning Partnership. A huge database unification process along with introducing a data scientist into the team has stirred the organization in quite an exciting path.

Oh, and also I got a chance to attend and speak at DjangoCon Europe.

I was researching on managing multi-state data with a single Django app and allowing organizational access one fine day when….

Enter Bitlab Studio!

Remember me mentioning my first ever international trip and conference talk? Well, the organizer of PyCon Singapore that year was Mr. Martin Brochhaus. I follow him on Twitter. Games, bitcoins and Singapore+Python is what I usually see from him. On June 8th, I simply, just like that, decided to DM him about what the scene in Singapore is and if there would be any opportunity there for a Django guy like me.

We had a brief chat where he talked about the state of tech and what people usually made around those parts. We ended the chat by him telling that he would keep an eye out for me and asking me to send him a resume, if I had one.

22 days passed by before I remembered that I hadn’t sent him my Resume. On June 30th, I sent it to him, he said he’d talk to his team and get back to me. I was like “wut”.

We did a video call two days after, everyone got to know each other and they asked me to jump in! By the coincidence of coincidences, it was exactly during those few months that they were pondering to hire a new employee.

The now and the future

Like I mentioned at the very beginning of this post, that last two times I quit, I had enough frustration built up. However in July, with a heavy heart, I told the KLP team about my opportunity at Bitlab studio.

As much they were worried about letting me go, they understood that this was an offer I couldn’t refuse and a good opportunity for my career growth. They allowed me to work for 3 hrs a day for the last two months with Bitlab, which I did.

Marvelous Martin (founder), Tremendous Tobi (Co-founder) and Delightful Dan (first employee) are extremely cool cats to work with. (* cough * * cough *… edited to avert waterboarding threats). The fact that Martin mentored both of them to a large extent reflects in their conversations with me. They know that I need time to get accustomed and to grow. Patiently explaining frontend technology to a 3-year experienced backend dev like me takes extreme restraint – and they have that.

So yeah, there it is! They are happy with my work over the last two months and now I am a Bitlab Studio employee! It is a completely remote gig with a yearly retreat. I am learning, working and getting trained on ReactJS already while I continue to help them develop and maintain a client’s infrastructure.

Super excited and super happy about this opportunity. I realize my drawbacks, things that I need to work on improving and to keep myself motivated for a remote working engagement now. While I hope, pray and work towards making this the best that it can be, the cracks are already beginning to show. 😛

I wish my teammates at KLP the best. They definitely have an exciting path ahead, both impact and technology wise. I am sure they will scale out and realize the dream of an India Learning Partnership in the near future.

Twitter comments:

Advertisements

7 facts about the VAC & US Consulate Visa Center, Chennai.

Very recently, I had the opportunity to apply for a US Visa, and thankfully, get it approved as well. However, it was not without its troublesome and frustrating moments, a few of which I’d like to list down so that you can brace yourselves.

Keep in mind that the interview happens across two days:

The first day is at the VAC, behind the Good Shepherd square on the Kodambakam high road. They will scan your fingerprint and then take your mug shot.

The second day is at The Consulate itself. You’ll have your Visa interview here.

1. The VAC respects your time slot.

When you reach the VAC, you will be greeted with a humongous queue outside the walls. Worry not. The watchman at the gate respects the time slots that people applied for. If you just walk up to the front of the queue and show your appointment confirmation, he will let you through provided your time slot is within the next half an hour.

2. The VAC allows mobile phones inside.

You can take your cell phones inside. At one of the gates, they will ask you to take it out, and switch it off in front of them. You will be scanned thoroughly and asked to display any metallic object on you, including keys, your wallet and even your belt, if you have one.

3. The sign!

You’ll see a very… peculiar sign board outside the gates saying “DS form correction done very quickly and very cheap. Contact auto stand”. A couple of us had a nice laugh reading that loud again and again. Apparently, the first step at the VAC is checking for discrepancy in your DS forms. I did not see anyone being sent back, but make sure to have exactly the same details both on your passport and your DS form.

4. The Consulate does not care about your time slot.

I had booked the time slot for 8:00AM. When I reached near the embassy at around 7:35AM, there was this massive queue outside the walls, outside the barricade on the side walk. This was apparently the queue to just get inside the barricade that takes you to the door that will lead you inside the walls of the Consulate.

I went and queued up. A minute later, I simply asked the person in front of me whether he was there for the 8:00AM slot. He graciously replied that he was there for the 10:30AM appointment! I felt a shudder down my spine. Frustrated, I walked over onto the front through the highway (yes, we are queuing on the sidewalk of a highway) just beside the queue. Before I could ask the person standing there in a purple tucked in shirt and black pants regarding time slots, a couple of police officers came and shoved the few of us there on the highway back to the rear of the queue. All of them only spoke Tamil, so whatever I tried to communicate in English fell on deaf ears and they waved us all back to the rear.

Within these 5 minutes, the queue had grown to a +15 people. I went and stood behind them, patiently. By the time the queue was half done (it was 7:52AM then), a certain gentleman came up from the rear of the queue and asked me my time slot. Upon hearing my reply, he said his was at 8:30AM and asked me whether he could stand behind me in the queue. After a minute, I asked him to hold my place. I again went over the front and boldly stepped up to the man in purple shirt.

He was a very gentle and calm person who had just the right words for any sort of query you put to him.

He’d say: “Queue”.

I tried phrasing my concern of the 8:00AM appointment in three different ways to which I got the “Queue” answer all three times. I went back and joined the queue. All this while, all the other time slot people ranging from 8:30AM to 11:00AM went ahead in front of me.

5. The Consulate strictly forbids you from taking in any mobile phones, bags, pen drives, etc.

Even though this was clearly written on the appointment confirmation, I thought it was ridiculous that they would really expect people to not show up with cell phones. I mean, I had just taken an Uber to get to the place!

Suffice to say, after passing the initial purple shirt guy and getting inside the barricade, the security guard outside the front door thoroughly searches everyone and just says “No cell phones, pen drives and bags allowed inside”. No debate there.

6. The “Personal belongings deposit counter!”

Ha, this is great. While you would notice a small metal cart being manned by two shabbily dressed old gentlemen with these words above written on them standing in your way while you make your way into the barricade, you’ll most probably brush away the feeling that it has anything to do with you. Well, like it or not, if you have any belongings with you that is not allowed inside the embassy, then your only option is to drop it off at this small metal cart!

Unless of course, you have another person (friend / family) waiting for you on the highway being hauled by the police every other minute or so.

Suffice to say, I had to give my Moto G3 and Nokia 2690 phones at this counter in order to get into the embassy. They charge 10 bucks for safe keeping. They will print two receipts with your passport number on them, one of which you will have to sign which they will keep, and the other to keep with you.

NOTE: You might find people desperately hunting for 10 rupees to pay these safe keeping gentlemen. If you happen to see anyone like that, have a mind to just give the 10 rupees and help them out.

7. Special language queues.

I think that we enter our primary language or “Language in which I would like to have my Visa interview in” at some point while filling up our application form. The thing is, if your application has this “language” field as anything other than “English”, then you will be taken out of the primary queue, and given another language queue, for each languages (Malayalam, Kannada, Telugu, Tamil). These queues have seats. So you can sit and relax while waiting for your chance while all the “English” language people have to wait in this huge queue waiting for at least 40 minutes to get your chance.

Well, that’s about it. Just thought of sharing these points so that you know these are for real.

Apart from these, I booked my accommodation at St. Xavier’s guest house for 650 bucks a night. I must say these people put the word “budget” in “budget hotel”! It was good enough for a one night stay, but if you’re the type who wants everything to be crystal clean and well serviced, then this might not be the place for you.

Also, while you’re waiting for the interview in the queue, you can actually see others’ interviews being conducted right in front of you at all the counters. You will see interesting questions, emotions, rejections and acceptances as well. Be strong and be confident. If your case is solid, then you don’t need to worry.

I got my B1/B2 Visa and am pretty excited about attending & speaking at Djangocon in two weeks!

Many thanks to Elizabeth from Chinnocio for helping me out with the Visa application process.

Help me pick a story.

At times when you least expect it, is when you suddenly get a whiff of a lovely idea for a story. I’ve had more than a few such instances and even though a few of those ideas have materialized into stories on this blog, I’ve lost even more due to putting it off for another time.

This is the way I found in order to let those ideas remain, so that I have one to pick from whenever I feel like writing. Below, you can find the ideas that I have in mind right now. As much as this is just a reminder/reference for me, I’m putting this up as a blog post so that you can tell me which story you would like me to write first. Leave a comment if you feel like it.

Currently started drafts:
——————————-
1. A long life, and a few last words.

Last words of an old Tree.

2. The Lost Ones.

The land of characters made up by authors, who never make it to books.

Lingering ideas:
———————

1. The Seed of Eternity.

His journey seeking the seed of eternity. Believed to contain the entire history of Earth, passed on from generation to generation. A certain plant/tree or a community of them are responsible for guarding and nourishing the seed of eternity within each generation before passing it onto the next. His quest leads him to places unknown, where he learns the language of the Green and listens to the stories they have to say.

2. The Village of Kadur.

A remote village located at the foot of a mountain range. The arrival of the supply truck that reaches Kadur once every year is considered to be the Holy day. Celebrations and Festivals happen around the day the truck arrives. Even the elders of the village remember the truck reaching Kadur on the same day every year since their childhood. The story speaks about the mysterious driver, and about that day when the truck does not show up…

3. His Journey to Salvation. (Half baked story idea)

Software Engineers, heroes of the digital world. The days of yore and the stories of brave knights and brave adventurers setting out on dangerous journeys to slay the dragon or fetch the Excalibur are over. With planes, automobiles and arms, they are heroes no more. The world today is at the fingertip of people who command the greatest knowledge over the digital world. Their nemeses fight hidden and are never known. Join our hero as he sets off on his quest to brave the odds and slay the nemesis once and for all, saving our World and restoring peace to the era.

A review on Neil Gaiman’s Neverwhere.

Prologue
————-

It was during the September of 2014 that I enjoyed watching Will Smith movies so much that I stumbled upon the Fresh Prince of Bel-Air. From then on since last week, the “purpose”, the “thing”  that I made sure I did everyday was to watch at least one episode of The Fresh Prince. I did not enjoy it as much as I did Mind Your Language, but you do know how these TV series grow on you, don’t you? Free time meant finishing off as much as I can of the series.

What a sad life.

I remember back in my 11th grade how our TV got burnt from a lightning. I got so bored back then that I took a Calvin & Hobbes comic lying around in the junk (yes, I had left it in the junk box) and read it. I got hooked onto “reading” from that moment on. However, over the past one year, reading had come down to technical documentation, random blog posts from Twitter, etc. It was when I finished The Fresh Prince and I saw I had free time in a day, that I simply picked up Neil’s Neverwhere and flipped through it.

It was amazing.

I must say the first book that I read by Neil Gaiman was Smoke & Mirrors. I was not too impressed, not to say confused, suffice to say I kept it back nicely in my stash after the first few pages. I bought and read Ocean At the End of the Lane afterwards, which I must say, I did not quite enjoy again, although I finished it. I read Sandman and I absolutely loved it, but it was a bit too expensive to keep buying and it was a comic book.

Having had that experience with Neil’s books, I came to the conclusion that his writing is not as engaging and interesting as his talks and speeches. Even then, he did inspire me. You can understand how much if you read through A Fortunate Evening. It was only last week that I found out he had endorsed the exact story that I had written, already through this cartoon of his: https://www.youtube.com/watch?v=pEKheZs2dkg. Thanks to my brother Noufal for sharing it with me.

Why I am so motivated to write this review is because I spent 7 hours yesterday reading. 3 and 4 hours continuously. The last time I did that was with Lord of The Rings, almost 4 years back.

Review
———–

In that cartoon of his, he says that one of the few sentences that any writer loves the most to hear from their readers is, “What happens next?”. The moment that question arises in a readers mind, the writer gets that grin on his face, thinking, “Now I’ve got you in the palm of hands. BUHAHAHAH!”.

First few pages into Neverwhere, that is exactly the question that came into my mind. Even with the last two pages remaining, I could not contain my curiosity. I postponed a meeting with an institution by half an hour just to get through to the end of the book.

One of the most interesting things that struck me about the book was the way his imagination was working. If you are ‘just’ a passionate reader, you’ll zip right through the book, traversing a terrible world, experiencing things that you might have never imagined before. However, if you’ve ever flexed your creative muscles, trying to talk about the non-existent, trying to convince people of it, then you will see what I mean by saying I found it of interest to see how his imagination worked.

During several instances while going through the book, not because the writing was not engaging, but because I had tried to walk down the same road of writing fiction, I was intentionally able to disconnect, take a few steps back and look at what was happening. You could immediately feel the way Neil was having fun, taking advantage of his creative liberties. Being a reader, you will never feel the occurrences to be vague, but the moment you try to see it is a fiction story, you could seem him using the elements around him to build upon what he has. The feeling of connect was truly exhilarating.

The intertwined story telling really built up the excitement. It was almost like that Guitar solo when Joker was upto something in those Nolan Batman movies. From the reading time of switching between two story lines, you could judge how far the climax of that specific part was. This itself gives you the urge to keep going forward, if the story itself is not enough.

The story ending was not one of the best that I have read, but that doesn’t bother me one bit. A good read allows you to create a universe and characters inside it, and relate to them as if you had met them just two days back. I would read any book to get that feeling, which Neverwhere definitely imparted. You will find yourself a lovely sister in Door, two terrible menaces in Croup and Vandemar and an unassuming random-everyday Joe in Richard Mayhew. Although the book did not have any ‘extreme’ moments, so to speak, as there were in Lord of The Rings, it still gave a pleasant reading experience.

If you are a lover of fiction/fantasy, then Neverwhere is definitely recommended.

Epilogue
———–

Not particularly in relation with this specific book, but when I finished reading the story, I realized that words was one of the greatest gifts that God has given humanity. The more you learn how to understand words, and how to wield it, the stronger you become as a person.

Uninvited savior.

Getting down from the bus, he slowly walked over to his office, which was a ten minute walk away. It was early in the morning, and he could see the health conscious people running their rounds in the parks, the shop owners lighting up the small lamp in front of their deities, hoping and praying for a good day of business, the birds leaving their nests in search of their day’s fill, the servants of the home walking their masters’ dogs and like so, the city slowly waking up and coming to life.

What hurt him most was seeing those little kids, on such cold mornings, with their heavy bags and sleepy eyes, waiting for their ride on the express way to hell. Many of them called it school. He knew of parents who just wanted their kids out of their way so that they could work, enjoy and have their own lives. He could not help but wonder then why they brought these poor little souls into this world in the first place. He hated school.

His stride was slow. There was no hurry. Unlike the other days, today he had no song on his lips. He was lost in thought thinking of the suffering that many kids had to face in the name of education. To make things worse, child molesters lurked in every nook and corner. It was only recently that a drug gang was busted near an Upper Primary School in his village. Evil had its roots so close to our loved ones, closer than us, pampering and calling to them in their most vulnerable and private moments.

All of a sudden, a van turned around the corner and screeched past him. The ghastly face of the children inside told him all that he needed to know.  In an instant, he picked up a big branch that was lying close to him and threw it at the van. The bewildered driver applied the brakes and tried to turn into the next lane. However, within a few seconds, he had jumped on the fence, ran across and caught up with the van.

He banged on the driver’s door, making the van come to a complete stand still. The screams of the children within got his adrenaline pumped up, and he pulled open the door. He caught the driver by his collar and threw him out of the van, yelling at him.

“You nasty little piece of shite. You ain’t kidnapping anyone today!”

The commotion had got few of the neighbours out. They ran and caught him while he was landing punch after punch on the driver’s skinny face, whose slightly protruded teeth was bleeding by then.

“What are you doing!? That’s the driver taking those kids to school!”, shouted one of them.

“Let go of me! I just saved those kids from this sick kidnapper!”, he retorted.

“Are you crazy?”

“Ask them!”, he shouted and ran over to the back of the van. He opened the door and asked, “Don’t worry, don’t worry. Everything’s fine now. Uncle has taken care of the bad man. Tell me, how did he get you?”

“We are going to school uncle. We got in the van from our homes”, one of the kids replied, to which all the other kids nodded approvingly in unison.

“But.. but.. I saved you… You don’t have to. You can get out.. I saved you…”

The realization slowly dawned on him. He lowered his gaze, and fell on his knees, and sat there, a defeated man.

Weekend PythonExpress workshop at BMSIT, Bangalore.

Arun taking it away!

That’s Arun. Jovial, cool, slightly crazy and a nice guy who is as curious as a 5 year old all the time about the things happening around him.

Santosh, the celebrity of BSMIT.

That’s Santosh. Don’t get him started on being sarcastic. He’ll just have too much fun. A pure geek since his college days, there are very few people around whom he is not acquainted with.

Yours truly.

Last but not the least, yours truly.

Arun, Santosh and I planned to do a beginner level Python workshop at the BMSIT Engineering college as a part of the Python Express initiative. Karthik, who was the organizer, was only too happy to welcome us. The three of us decided to use Anand’s Python Practice Book as a guide. We were planning to cover until Object Oriented Programming. However, there was no point in rushing through.

The idea was to get the kids comfortable with Python. Both Arun and Santosh realized this very well. During the introductory session, which was handled by Arun, he quickly gauged that most of the students were not even comfortable using the interpreter. He took his time, teaching them about variables, strings and conditional statements, by giving them enough exercises to work on as well as using his incredible humour and charm to keep the crowd engaged.

Arun - The man.

Only Santosh and I knew it was his first ever session. No one could have guessed. He was in the zone and a couple of kids came and personally thanked him for making the introduction so welcoming that they were motivated to sit through the entire session. One of them even skipped lunch to stay back!

Way to go Arun!

He finished his session by 12:20PM when we broke for lunch. The lunch was exceptionally good and we had a full stomach by the end of it. We got back and it was about time for Santosh to start his session.

The poor guy had a really sore throat. We tried to arrange speakers, but they were even less audible than one’s voice. In a room with ~100 young, energetic, curious youth, you had to shout at the top of your voice to have your voice heard. That is exactly what Santosh did.

Santosh in charge.

He lost himself among the crowd and did not care about his voice. He had a keen sense of understanding the audience well and dynamically changing his presentation style to suit them.

We had a box of chocolates around. Every time someone finishes a problem first, they are awarded a delicious chocolate! We really needed to buy a bigger box.

Gauging the exhaustion on the face of many, we decided to wrap up our workshop around 4 by finishing off a quick peep into file handling.

I must say it felt good to have been back at a college. When I was roaming around the lab, I noticed most of them earnestly taking down notes in a notebook out of the fear that they would be missing some point. To one of the guys, I asked,

“Hey, will you really be referring back to these notes again?”

He: “Absolutely! Not all of it, but many points in it”.

“Alright”, I said and went around.

While I was passing this young man a second time, he called me and said,

“To be honest, no. I don’t think I will be referring to this at all”

“Thanks for not lying to me”, I said with a smile.

It was good to meet Karthik, who was the one coordinating the entire thing. It was interesting to know he was a Linux Kernel lover who was cracking his head on getting deep into it. Dharshan, his friend, one with a very soothing personality, was a great help in getting us around the venue as well as with the setup.

I did not get to know many of their names, but I remember Aranya, a college student who is an “investment consultant”. Yeah, no kidding. I have his card right here. Also Utkash, who was really keen on getting to understand the intricacies of the Internet. Then there was Kunal, who was from the EEE stream, but interested in programming. A very enthusiastic lad.

All in all, it was a day well spent. Malaysia was extremely fun (long story) and after hanging around the front gate for a while, our cab came at around 5 and we were on our way.

I should do this more often.

 

An introduction to Redis – PyCon Singapore 2014.

The following is the transcript of the talk “Redis – What, why and where” that I gave at PyCon Singapore 2014. You can find the slides down below. Try as I might, I was not able to embed the slides from slides.com. So I have shared the links.

My talk was on Friday, 20th June, 2014 at 1:00PM.

—-

Ladies and gentlemen,

Do you know what my prayer was the moment I knew I got my talk selected? That I would not be allocated a slot right after lunch. Yet here we are.

You must be wondering why a dude from India has come all the way over to Singapore and is giving a talk on Redis at a Python conference. Well, I believe you’ll have the answers to those questions by the time I am done with my talk. This is intended for a beginner level audience and as such, if you have already implemented redis in your stack, then you might be a little disappointed.

There are times when, in your Django web application, you need a certain specific data to be saved. Let me give you an example. Let us say you are gathering all the tweets for the Football World Cup. You hit the Twitter API and tweets are pouring in by the second. How do you keep a counter? Of course, put a Python variable in the loop and keep incrementing.

tweets = fetch_tweets(hashtag = "#WorldCup2014") #Use the Twython Library
count = 0
for tweet in tweets:
    entities = process_tweet(tweet)
    count = count + 1

The only problem is that if another process/view wants to display it, it won’t be able to access it.

Which means you should have persistence. If you’re using Postgres or any other SQL database for that matter, you could have a field that would allow you to keep the count or maybe do a count(*) on your Tweets model each time you want to get the total number of tweets.

#Assume you have defined a model Tweet
count = Tweet.objects.all().count()

The count(*) option is going to get your SQL query to execute quite slow once you have about 20000 rows or so.

#Assume you have defined a model Stat to store the count which has a field tweet_count
Stat.objects.get(hashtag = "#WorldCup2014").update(tweet_count = F('tweet_count') + 1)

The next option being to increment the count within the Postgres field. This has an immense potential to lead you into race conditions and thereby screwing up your count.

So a fast, reliable and persistent solution is to have redis. Believe it or not, you can use this as an actual Database because of its persistence. All you need to do is to get the redis server up and running on your machine, use the redis-py Python library to increment a “key” by one each time a new tweet comes in. You don’t even need to “initialize” the key. The increment command creates a key if it is not already present and increments it. Really neat. Hence, redis is a persistent key-value based NoSQL Data storage.

import redis #We are using the redis-py library
r = redis.StrictRedis()

tweets = fetch_tweets(hashtag = "#WorldCup2014")
for tweet in tweets:
    entities = process_tweet(tweet)
    r.incr("tweets_count", amount = 1)

count = r.get("tweets_count")

Now, persistence is not the only thing that makes Redis useful. Suppose you just don’t stop with counting tweets. You count the pictures, videos and other links form within them. Also, you are doing the same with Facebook as well. Now you have two sources and their corresponding fields. Intuitively, a dictionary comes to mind. Name of one dictionary would be “Twitter” and the other one “Facebook”. Each of them will have fields “statuses”, “photos”, “links”, etc.

Guess what? Redis has a dictionary data type and let’s you do exactly this. The various types of in-built data types that it provides is fantastic. People tend to call it the data structure server due to this reason.

import redis
r = redis.StrictRedis()

tweets = fetch_tweets(hashtag = "#WorldCup2014")
for tweet in tweets:
    entities = process_tweet(tweet)
    r.hincrby("Twitter", "tweets_count", amount = 1)
    if "photo" in entities:
        r.hincrby("Twitter", "photo_count", amount = 1)
    if "video" in entities:
        r.hincrby("Twitter", "video_count", amount = 1)
    if "link" in entities:
        r.hincrby("Twitter", "link_count", amount = 1)

twitter_photos_count = r.hget("Twitter", "photo_count")
...

posts = fetch_fb_posts(hashtag = "#WorldCup2014")
for post in posts:
    entities = process_post(post)
    r.hincrby("Facebook", "posts_count", amount = 1)
    if "photo" in entities:
        r.hincrby("Facebook", "photo_count", amount = 1)
    if "video" in entities:
        r.hincrby("Facebook", "video_count", amount = 1)
    if "link" in entities:
        r.hincrby("Facebook", "link_count", amount = 1)

fb_photos_count = r.hget("Facebook", "photo_count")
...

It supports 5 data types comprising of strings, sets, dictionaries, sorted sets and lists.

So, one, the persistence and two, the data types. These two are what makes Redis special.

Narcissism
———-

Oh and incidentally, I am Haris Ibrahim K. V. and I am from the southern most state of India called Kerala. I work as a Computer Science Engineer at a small company called Eventifier. I’ve been a Python developer only since the past 7 months and hence, have relatively lesser experience when it comes to programming. Although I have organized conferences and workshops by myself, as a part of my earlier job, this is my first ever talk at one. So there might be a few rusty edges. Do bare with me. Also, as a hobby and passion, I love writing.

Alright, enough with the narcissism. Let’s get back to business.

Redis stores its data in a Big In-Memory dictionary where they keys can only be strings, but the values can be any of the 5 data types that we mentioned earlier. Each of these data structures have their own implementation which will come to later. Let us go back to a few more use cases where you can use redis.

LEADER BOARD (using sorted sets)

Let’s talk about leader board. What I am trying to do here is to give you examples that cover all the 5 data structures that Redis provides so that you will know what to use where and why. Leader board. I am sure you are familiar with the concept of leaderboard, but for those among you who are not, it is place where the top 10 of something is shown. Top 10 or 20, it does not matter. But a list of entities sorted based on their rank.

An example should clarify this right away. Let’s go back to the football world cup example. The tweets are pouring in. Boy, reminds me of monsoon back at home. Anyway, You want to show the most retweeted tweets in descending order of their retweet count. This will give you an idea of what is trending for that particular hashtag. Now, what do you do? This is where the “sorted set” data type comes into picture. As the name suggests, it is a set, but sorted.

What is this sorted based on? Ah yes. So when you hear a sorted set, the picture that should come into your mind is a key with a value as a list of tuples. I use “tuples” in a loose sense. Once you have that picture in mind, this is how the structure would look like:

key: (score member) (score member)

All you need to do is to define a key called “trending_tweets” and then use the “zadd” redis command to specify the score as the number of retweets and the member as the “tweet text + username” or something.

import redis
r = redis.StrictRedis()

tweets = fetch_tweets(hashtag = "#WorldCup2014") #Use the Twython Library
count = 0
for tweet in tweets:
    entities = process_tweet(tweet)
    r.zadd("trending_tweets", tweet.retweet_count, tweet.text)

trending_tweets = r.zrange("trending_tweets", 0, -1)

You could also store the tweet ids as the members and just do a query on your SQL database to fetch tweets with those particular ids. This would work much better since sorted set is a set and it will be expensive to maintain uniqueness on members if they are huge chunks of text.

import redis
r = redis.StrictRedis()

tweets = fetch_tweets(hashtag = "#WorldCup2014") #Use the Twython Library
count = 0
for tweet in tweets:
    entities = process_tweet(tweet)
    t = Tweet.objects.create(tweet = tweet)
    r.zadd("trending_tweets", tweet.retweet_count, t.id)

trending_tweets = r.zrange("trending_tweets", 0, -1)
popular_tweet_list = []
for tweet_id in trending_tweets:
    popular_tweet_list.append(Tweet.objects.get(id = tweet_id))

To retrieve the top 10, use the “zrange” command and specify the indices. That should get you going.

CACHING (using list)

This introduces a new data type as well as a useful feature.

Redis allows you to set “expire” on certain keys. You can specify the key name and the number of seconds in which the key should expire. You might have already guessed it. Yes, you can implement a caching mechanism with this. The timeout remains valid as long as you only “alter” the keys using operations such as increment, add, etc. However, if you set the key once more or delete it, the deal is off. No timeout for you.

The way to implement this would be to first know what value want to be cached. Save that value into redis with a key. Call expire(key, seconds) and you’re done. What goes hand in hand with this is the TTL command. Known as Time To Live. As you could guess, this gives you the time left before a certain key expires. It returns -2 if the key has expired or -1 if an expire has not been set on the key to begin with. Pretty handy.

Let’s go back to the Football world cup tweets example once again. Suppose you want to showcase the photos that got retweeted the most every 5 minute or so. You might have to do something like fetching the popular tweets, get the corresponding photo url, push them into a list and set an expiry on that list’s name.

import redis
r = redis.StrictRedis()

tweets = fetch_tweets(hashtag = "#WorldCup2014") #Use the Twython Library
count = 0
for tweet in tweets:
    entities = process_tweet(tweet)
    t = Tweet.objects.create(tweet = tweet)
    r.zadd("trending_tweets", tweet.retweet_count, t.id)

trending_tweets = r.zrange("trending_tweets", 0, -1)
popular_tweet_list = []
for tweet_id in trending_tweets:
    popular_tweet_list.append(Tweet.objects.get(id = tweet_id))

if r.ttl("trending_photos") in [-1, -2]:
    for tweet in popular_tweet_list:
        r.rpush("trending_photos", tweet.media_url)
        trending_photos = r.lrange("trending_photos", 0, -1)
        r.expire("trending_photos", 120) #Expire in 2 minutes
else:
    trending_photos = r.lrange("trending_photos", 0, -1)

The list is a double ended list actually. You can insert at the left or the right. Accordingly you can pop from either side as well.

CREDITS

The first person whom I would like to thank is someone who deserves much more than me to be up on this stage and give this talk. However, he usually prefers to be behind the scenes, getting things done and motivate people to do things. He is my colleague and the CTO of the company I work for, Mr Nazim Zeeshan and there he is.

The second would be Sripathi. There is a company called HasGeek back in India who organizes technology conferences and workshops. They had organized a Redis miniconf recently where Sripathi gave a talk on Redis Memory optimization. What I am going to present next is from his inspiration.

Last but not the least, the PyCon Singapore team who organized and made this a reality. Kudos to them!

INTERNAL DATA TYPES

This is something that I picked up from what Sripathi explained. I confess I’m not an expert on this but thought it would spark a few minds if presented. Redis stores all that we talked about right now internally using 6 different data types.

Refer to the slides and video for this part.

—-

Slides:

http://slides.com/harisibrahimkv/redis-what-why-and-where

Video:

https://archive.org/details/IntroductionToRedis