With Bitlab Studio’s doors open, I finally bid farewell to Akshara with a heavy heart…

The History

The last two times I quit my job, despite both the companies helping me grow and explore tremendously on a personal and career level, I had enough frustration built up over various reasons which served as a valid excuse for my decision. However this time, it is slightly different.

Ask me about my first job at HasGeek anytime, and I will tell you how I grew with them from a college graduate who was afraid to speak English, who was afraid to travel, to someone who could confidently strike up a conversation with any stranger; to a person who pursued his career interests; to a person who made his dream of flying in a plane come true.

Ask me about my second job at Eventifier anytime, and I will tell you how gracious they were to accept me into their company as their first employee despite me having almost 0 skills in industry level programming. I’ve written about it in detail already. The opportunity that they gave me changed my life, and I will be forever thankful to them.

However, what didn’t quite make it onto the blog during my days at Eventifier was the fact that I had my first ever international trip! And that too accompanied by giving my first ever talk at a conference! Which was at Singapore. We’ll get to that in a bit.

Life during the KLP years + one of the biggest failures in my life

I joined Akshara Foundation‘s Karnataka Learning Partnership (KLP) back in November 2014 with a 15-hr per week schedule. This was one month before my marriage when I need some financial fortification. While during the succeeding months the folks at Eventifier slowly began to realize that it was probably a bad idea to let one of their full time employee take up a part time gig on the side, at that time, they felt it best that I made that decision. Goes to show how understanding and empathetic the three of them were.

The engagement with KLP remained as it is until April 2016, when I quit Eventifier and moved over to KLP on a 4-day per week contract. It has been so until now. Oh, and during that April is when my ThinkPad saved me from lightning by taking the hit itself and led me to evolution:

The nature of work, while relaxed, was very result oriented and gratifying. Our entire work is open source as well. We had a completely remote team, which allowed me to move out of Bangalore. I never truly paid attention to setting up a proper office setup at home and spent most of my time typing away on the bed. Believe it or not, this was my posture every day for better part of the year:

laptop-bed

See how happy I am!

My DjangoCon US trip and presentation happened in between this. I was doing a non-trivial amount of work with Django REST Framework which helped me come up with a presentation.

It was only after that conference that it hit me the lengths people would go to, to setup a proper working space at home. Even with that inspiration, this was as far as I got, and that too only for 2 – 3 hours a day:

The failure:

That happiness soon turned into one of my biggest failures in life. By October-ish, I started feeling slightly stagnant in terms of technology. Well, maybe not really. I guess I was just getting complacent. I kept an eye out for opportunities and ended up landing a part time gig with one of my programming heroes, Anand Chitipothu, to work along with him at Rorodata.

Mistake number 1: Taking up a 3-day per week part time gig while I already had a 4-day per week engagement with Akshara (even if the nature of work was just trivial maintenance at that point).

We got on fine at the beginning. I hadn’t even anticipated the trouble with time management. This ended up in me staying up till 2am or 3am in the mornings. My sleep cycle screwed up considerably.

Mistake number 2: Failing to realize quickly that the work pace and style at an NGO and at bootstrapping startup are completely different.

I was relaxed. I would sleep if I felt sleepy, I would goof off if I couldn’t concentrate, which all ended up in the work piling up from both the companies day after day. This started creating frustration for my colleagues as it was almost as if they couldn’t count on me to finish my job in time.

I remember arrogantly thinking the compliment that my first boss, Kiran, gave me while I was at HasGeek: “You’re good at getting shit done”.

Mistake number 3: I got extremely complacent. With that, even if I missed a day’s work, I would think “They aren’t going to fire me right? I am that good. Everyone is dying to hire me”.

With this attitude, lacking any proper working setup at home, a 7-day per week work schedule, on February 20th, Rorodata decided to terminate my contract. I wasn’t surprised as I remember telling my family that it wasn’t working out and probably I was going to get fired (although my ego never truly believed that).

However, during that week’s PyCon Pune, I had a long conversation with Nigel, who helped me figure out what all I needed to do to setup a proper work space at home.

I had a new monitor, cleaned up and made an office room, bought an office chair and everything setup. That’s when I had to leave Rorodata.

Life goes on…

That was a blow, a very severe one, to my ego. I never quite came to terms with it really. It hurt bad. Real bad. I realized I sucked big time. Almost twice a week for the next 3-4 months after that, I would sulk away into a corner brooding about the opportunity that I had completely destroyed. It was a dream job, and I just… sigh…

Even though that nagged me all the time, I still had to keep up my work at KLP. The notion of being fired and unemployment scared me. I got my shit together and concentrated on work like never before. I learned my lesson.

The last few months

We had our yearly meeting during March 15th. The fact that everyone involved, at the core, worked to create meaning for Children’s education, and was not working for profit or make a huge exit, gave the work culture an interesting atmosphere.

The slack conversations and personal chats almost revolved around personal and practical problems in the society rather than the next coolest thing happening in the tech world.

The main difference I felt was not having that feeling of “your boss is good and friendly, but at the end of the day wants you to work really hard so she/he can make huge lump of money sometime down the line”. That wasn’t there. At all.

Work hard so that we can bring about sustainable change which will impact children’s education. If you couldn’t connect with that, you won’t be able to work long at this place.

It was relaxing, it was fun, and most importantly it was extremely gratifying. As I used to say, “Now I can show my Mom and Dad what I am really doing with my skills!”.

The last few months had been tremendous. We were evolving. From Karnataka Learning Partnership, we were growing into India Learning Partnership. A huge database unification process along with introducing a data scientist into the team has stirred the organization in quite an exciting path.

Oh, and also I got a chance to attend and speak at DjangoCon Europe.

I was researching on managing multi-state data with a single Django app and allowing organizational access one fine day when….

Enter Bitlab Studio!

Remember me mentioning my first ever international trip and conference talk? Well, the organizer of PyCon Singapore that year was Mr. Martin Brochhaus. I follow him on Twitter. Games, bitcoins and Singapore+Python is what I usually see from him. On June 8th, I simply, just like that, decided to DM him about what the scene in Singapore is and if there would be any opportunity there for a Django guy like me.

We had a brief chat where he talked about the state of tech and what people usually made around those parts. We ended the chat by him telling that he would keep an eye out for me and asking me to send him a resume, if I had one.

22 days passed by before I remembered that I hadn’t sent him my Resume. On June 30th, I sent it to him, he said he’d talk to his team and get back to me. I was like “wut”.

We did a video call two days after, everyone got to know each other and they asked me to jump in! By the coincidence of coincidences, it was exactly during those few months that they were pondering to hire a new employee.

The now and the future

Like I mentioned at the very beginning of this post, that last two times I quit, I had enough frustration built up. However in July, with a heavy heart, I told the KLP team about my opportunity at Bitlab studio.

As much they were worried about letting me go, they understood that this was an offer I couldn’t refuse and a good opportunity for my career growth. They allowed me to work for 3 hrs a day for the last two months with Bitlab, which I did.

Marvelous Martin (founder), Tremendous Tobi (Co-founder) and Delightful Dan (first employee) are extremely cool cats to work with. (* cough * * cough *… edited to avert waterboarding threats). The fact that Martin mentored both of them to a large extent reflects in their conversations with me. They know that I need time to get accustomed and to grow. Patiently explaining frontend technology to a 3-year experienced backend dev like me takes extreme restraint – and they have that.

So yeah, there it is! They are happy with my work over the last two months and now I am a Bitlab Studio employee! It is a completely remote gig with a yearly retreat. I am learning, working and getting trained on ReactJS already while I continue to help them develop and maintain a client’s infrastructure.

Super excited and super happy about this opportunity. I realize my drawbacks, things that I need to work on improving and to keep myself motivated for a remote working engagement now. While I hope, pray and work towards making this the best that it can be, the cracks are already beginning to show. ๐Ÿ˜›

I wish my teammates at KLP the best. They definitely have an exciting path ahead, both impact and technology wise. I am sure they will scale out and realize the dream of an India Learning Partnership in the near future.

Twitter comments:

Advertisements

7 facts about the VAC & US Consulate Visa Center, Chennai.

Very recently, I had the opportunity to apply for a US Visa, and thankfully, get it approved as well. However, it was not without its troublesome and frustrating moments, a few of which I’d like to list down so that you can brace yourselves.

Keep in mind that the interview happens across two days:

The first day is at the VAC, behind the Good Shepherd square on the Kodambakam high road. They will scan your fingerprint and then take your mug shot.

The second day is at The Consulate itself. You’ll have your Visa interview here.

1. The VAC respects your time slot.

When you reach the VAC, you will be greeted with a humongous queue outside the walls. Worry not. The watchman at the gate respects the time slots that people applied for. If you just walk up to the front of the queue and show your appointment confirmation, he will let you through provided your time slot is within the next half an hour.

2. The VAC allows mobile phones inside.

You can take your cell phones inside. At one of the gates, they will ask you to take it out, and switch it off in front of them. You will be scanned thoroughly and asked to display any metallic object on you, including keys, your wallet and even your belt, if you have one.

3. The sign!

You’ll see a very… peculiar sign board outside the gates saying “DS form correction done very quickly and very cheap. Contact auto stand”. A couple of us had a nice laugh reading that loud again and again. Apparently, the first step at the VAC is checking for discrepancy in your DS forms. I did not see anyone being sent back, but make sure to have exactly the same details both on your passport and your DS form.

4. The Consulate does not care about your time slot.

I had booked the time slot for 8:00AM. When I reached near the embassy at around 7:35AM, there was this massive queue outside the walls, outside the barricade on the side walk. This was apparently the queue to just get inside the barricade that takes you to the door that will lead you inside the walls of the Consulate.

I went and queued up. A minute later, I simply asked the person in front of me whether he was there for the 8:00AM slot. He graciously replied that he was there for the 10:30AM appointment! I felt a shudder down my spine. Frustrated, I walked over onto the front through the highway (yes, we are queuing on the sidewalk of a highway) just beside the queue. Before I could ask the person standing there in a purple tucked in shirt and black pants regarding time slots, a couple of police officers came and shoved the few of us there on the highway back to the rear of the queue. All of them only spoke Tamil, so whatever I tried to communicate in English fell on deaf ears and they waved us all back to the rear.

Within these 5 minutes, the queue had grown to a +15 people. I went and stood behind them, patiently. By the time the queue was half done (it was 7:52AM then), a certain gentleman came up from the rear of the queue and asked me my time slot. Upon hearing my reply, he said his was at 8:30AM and asked me whether he could stand behind me in the queue. After a minute, I asked him to hold my place. I again went over the front and boldly stepped up to the man in purple shirt.

He was a very gentle and calm person who had just the right words for any sort of query you put to him.

He’d say: “Queue”.

I tried phrasing my concern of the 8:00AM appointment in three different ways to which I got the “Queue” answer all three times. I went back and joined the queue. All this while, all the other time slot people ranging from 8:30AM to 11:00AM went ahead in front of me.

5. The Consulate strictly forbids you from taking in any mobile phones, bags, pen drives, etc.

Even though this was clearly written on the appointment confirmation, I thought it was ridiculous that they would really expect people to not show up with cell phones. I mean, I had just taken an Uber to get to the place!

Suffice to say, after passing the initial purple shirt guy and getting inside the barricade, the security guard outside the front door thoroughly searches everyone and just says “No cell phones, pen drives and bags allowed inside”. No debate there.

6. The “Personal belongings deposit counter!”

Ha, this is great. While you would notice a small metal cart being manned by two shabbily dressed old gentlemen with these words above written on them standing in your way while you make your way into the barricade, you’ll most probably brush away the feeling that it has anything to do with you. Well, like it or not, if you have any belongings with you that is not allowed inside the embassy, then your only option is to drop it off at this small metal cart!

Unless of course, you have another person (friend / family) waiting for you on the highway being hauled by the police every other minute or so.

Suffice to say, I had to give my Moto G3 and Nokia 2690 phones at this counter in order to get into the embassy. They charge 10 bucks for safe keeping. They will print two receipts with your passport number on them, one of which you will have to sign which they will keep, and the other to keep with you.

NOTE: You might find people desperately hunting for 10 rupees to pay these safe keeping gentlemen. If you happen to see anyone like that, have a mind to just give the 10 rupees and help them out.

7. Special language queues.

I think that we enter our primary language or “Language in which I would like to have my Visa interview in” at some point while filling up our application form. The thing is, if your application has this “language” field as anything other than “English”, then you will be taken out of the primary queue, and given another language queue, for each languages (Malayalam, Kannada, Telugu, Tamil). These queues have seats. So you can sit and relax while waiting for your chance while all the “English” language people have to wait in this huge queue waiting for at least 40 minutes to get your chance.

Well, that’s about it. Just thought of sharing these points so that you know these are for real.

Apart from these, I booked my accommodation at St. Xavier’s guest house for 650 bucks a night. I must say these people put the word “budget” in “budget hotel”! It was good enough for a one night stay, but if you’re the type who wants everything to be crystal clean and well serviced, then this might not be the place for you.

Also, while you’re waiting for the interview in the queue, you can actually see others’ interviews being conducted right in front of you at all the counters. You will see interesting questions, emotions, rejections and acceptances as well. Be strong and be confident. If your case is solid, then you don’t need to worry.

I got my B1/B2 Visa and am pretty excited about attending & speaking at Djangocon in two weeks!

Many thanks to Elizabeth from Chinnocio for helping me out with the Visa application process.

Help me pick a story.

At times when you least expect it, is when you suddenly get a whiff of a lovely idea for a story. I’ve had more than a few such instances and even though a few of those ideas have materialized into stories on this blog, I’ve lost even more due to putting it off for another time.

This is the way I found in order to let those ideas remain, so that I have one to pick from whenever I feel like writing. Below, you can find the ideas that I have in mind right now. As much as this is just a reminder/reference for me, I’m putting this up as a blog post so that you can tell me which story you would like me to write first. Leave a comment if you feel like it.

Currently started drafts:
——————————-
1. A long life, and a few last words.

Last words of an old Tree.

2. The Lost Ones.

The land of characters made up by authors, who never make it to books.

Lingering ideas:
———————

1. The Seed of Eternity.

His journey seeking the seed of eternity. Believed to contain the entire history of Earth, passed on from generation to generation. A certain plant/tree or a community of them are responsible for guarding and nourishing the seed of eternity within each generation before passing it onto the next. His quest leads him to places unknown, where he learns the language of the Green and listens to the stories they have to say.

2. The Village of Kadur.

A remote village located at the foot of a mountain range. The arrival of the supply truck that reaches Kadur once every year is considered to be the Holy day. Celebrations and Festivals happen around the day the truck arrives. Even the elders of the village remember the truck reaching Kadur on the same day every year since their childhood. The story speaks about the mysterious driver, and about that day when the truck does not show up…

3. His Journey to Salvation. (Half baked story idea)

Software Engineers, heroes of the digital world. The days of yore and the stories of brave knights and brave adventurers setting out on dangerous journeys to slay the dragon or fetch the Excalibur are over. With planes, automobiles and arms, they are heroes no more. The world today is at the fingertip of people who command the greatest knowledge over the digital world. Their nemeses fight hidden and are never known. Join our hero as he sets off on his quest to brave the odds and slay the nemesis once and for all, saving our World and restoring peace to the era.

Weekend PythonExpress workshop at BMSIT, Bangalore.

Arun taking it away!

That’s Arun. Jovial, cool, slightly crazy and a nice guy who is as curious as a 5 year old all the time about the things happening around him.

Santosh, the celebrity of BSMIT.

That’s Santosh. Don’t get him started on being sarcastic. He’ll just have too much fun. A pure geek since his college days, there are very few people around whom he is not acquainted with.

Yours truly.

Last but not the least, yours truly.

Arun, Santosh and I planned to do a beginner level Python workshop at the BMSIT Engineering college as a part of the Python Express initiative. Karthik, who was the organizer, was only too happy to welcome us. The three of us decided to use Anand’s Python Practice Book as a guide. We were planning to cover until Object Oriented Programming. However, there was no point in rushing through.

The idea was to get the kids comfortable with Python. Both Arun and Santosh realized this very well. During the introductory session, which was handled by Arun, he quickly gauged that most of the students were not even comfortable using the interpreter. He took his time, teaching them about variables, strings and conditional statements, by giving them enough exercises to work on as well as using his incredible humour and charm to keep the crowd engaged.

Arun - The man.

Only Santosh and I knew it was his first ever session. No one could have guessed. He was in the zone and a couple of kids came and personally thanked him for making the introduction so welcoming that they were motivated to sit through the entire session. One of them even skipped lunch to stay back!

Way to go Arun!

He finished his session by 12:20PM when we broke for lunch. The lunch was exceptionally good and we had a full stomach by the end of it. We got back and it was about time for Santosh to start his session.

The poor guy had a really sore throat. We tried to arrange speakers, but they were even less audible than one’s voice. In a room with ~100 young, energetic, curious youth, you had to shout at the top of your voice to have your voice heard. That is exactly what Santosh did.

Santosh in charge.

He lost himself among the crowd and did not care about his voice. He had a keen sense of understanding the audience well and dynamically changing his presentation style to suit them.

We had a box of chocolates around. Every time someone finishes a problem first, they are awarded a delicious chocolate! We really needed to buy a bigger box.

Gauging the exhaustion on the face of many, we decided to wrap up our workshop around 4 by finishing off a quick peep into file handling.

I must say it felt good to have been back at a college. When I was roaming around the lab, I noticed most of them earnestly taking down notes in a notebook out of the fear that they would be missing some point. To one of the guys, I asked,

“Hey, will you really be referring back to these notes again?”

He: “Absolutely! Not all of it, but many points in it”.

“Alright”, I said and went around.

While I was passing this young man a second time, he called me and said,

“To be honest, no. I don’t think I will be referring to this at all”

“Thanks for not lying to me”, I said with a smile.

It was good to meet Karthik, who was the one coordinating the entire thing. It was interesting to know he was a Linux Kernel lover who was cracking his head on getting deep into it. Dharshan, his friend, one with a very soothing personality, was a great help in getting us around the venue as well as with the setup.

I did not get to know many of their names, but I remember Aranya, a college student who is an “investment consultant”. Yeah, no kidding. I have his card right here. Also Utkash, who was really keen on getting to understand the intricacies of the Internet. Then there was Kunal, who was from the EEE stream, but interested in programming. A very enthusiastic lad.

All in all, it was a day well spent. Malaysia was extremely fun (long story) and after hanging around the front gate for a while, our cab came at around 5 and we were on our way.

I should do this more often.

ย 

An introduction to Redis – PyCon Singapore 2014.

The following is the transcript of the talk “Redis – What, why and where” that I gave at PyCon Singapore 2014. You can find the slides down below. Try as I might, I was not able to embed the slides from slides.com. So I have shared the links.

My talk was on Friday, 20th June, 2014 at 1:00PM.

—-

Ladies and gentlemen,

Do you know what my prayer was the moment I knew I got my talk selected? That I would not be allocated a slot right after lunch. Yet here we are.

You must be wondering why a dude from India has come all the way over to Singapore and is giving a talk on Redis at a Python conference. Well, I believe you’ll have the answers to those questions by the time I am done with my talk. This is intended for a beginner level audience and as such, if you have already implemented redis in your stack, then you might be a little disappointed.

There are times when, in your Django web application, you need a certain specific data to be saved. Let me give you an example. Let us say you are gathering all the tweets for the Football World Cup. You hit the Twitter API and tweets are pouring in by the second. How do you keep a counter? Of course, put a Python variable in the loop and keep incrementing.

tweets = fetch_tweets(hashtag = "#WorldCup2014") #Use the Twython Library
count = 0
for tweet in tweets:
    entities = process_tweet(tweet)
    count = count + 1

The only problem is that if another process/view wants to display it, it won’t be able to access it.

Which means you should have persistence. If you’re using Postgres or any other SQL database for that matter, you could have a field that would allow you to keep the count or maybe do a count(*) on your Tweets model each time you want to get the total number of tweets.

#Assume you have defined a model Tweet
count = Tweet.objects.all().count()

The count(*) option is going to get your SQL query to execute quite slow once you have about 20000 rows or so.

#Assume you have defined a model Stat to store the count which has a field tweet_count
Stat.objects.get(hashtag = "#WorldCup2014").update(tweet_count = F('tweet_count') + 1)

The next option being to increment the count within the Postgres field. This has an immense potential to lead you into race conditions and thereby screwing up your count.

So a fast, reliable and persistent solution is to have redis. Believe it or not, you can use this as an actual Database because of its persistence. All you need to do is to get the redis server up and running on your machine, use the redis-py Python library to increment a “key” by one each time a new tweet comes in. You don’t even need to “initialize” the key. The increment command creates a key if it is not already present and increments it. Really neat. Hence, redis is a persistent key-value based NoSQL Data storage.

import redis #We are using the redis-py library
r = redis.StrictRedis()

tweets = fetch_tweets(hashtag = "#WorldCup2014")
for tweet in tweets:
    entities = process_tweet(tweet)
    r.incr("tweets_count", amount = 1)

count = r.get("tweets_count")

Now, persistence is not the only thing that makes Redis useful. Suppose you just don’t stop with counting tweets. You count the pictures, videos and other links form within them. Also, you are doing the same with Facebook as well. Now you have two sources and their corresponding fields. Intuitively, a dictionary comes to mind. Name of one dictionary would be “Twitter” and the other one “Facebook”. Each of them will have fields “statuses”, “photos”, “links”, etc.

Guess what? Redis has a dictionary data type and let’s you do exactly this. The various types of in-built data types that it provides is fantastic. People tend to call it the data structure server due to this reason.

import redis
r = redis.StrictRedis()

tweets = fetch_tweets(hashtag = "#WorldCup2014")
for tweet in tweets:
    entities = process_tweet(tweet)
    r.hincrby("Twitter", "tweets_count", amount = 1)
    if "photo" in entities:
        r.hincrby("Twitter", "photo_count", amount = 1)
    if "video" in entities:
        r.hincrby("Twitter", "video_count", amount = 1)
    if "link" in entities:
        r.hincrby("Twitter", "link_count", amount = 1)

twitter_photos_count = r.hget("Twitter", "photo_count")
...

posts = fetch_fb_posts(hashtag = "#WorldCup2014")
for post in posts:
    entities = process_post(post)
    r.hincrby("Facebook", "posts_count", amount = 1)
    if "photo" in entities:
        r.hincrby("Facebook", "photo_count", amount = 1)
    if "video" in entities:
        r.hincrby("Facebook", "video_count", amount = 1)
    if "link" in entities:
        r.hincrby("Facebook", "link_count", amount = 1)

fb_photos_count = r.hget("Facebook", "photo_count")
...

It supports 5 data types comprising of strings, sets, dictionaries, sorted sets and lists.

So, one, the persistence and two, the data types. These two are what makes Redis special.

Narcissism
———-

Oh and incidentally, I am Haris Ibrahim K. V. and I am from the southern most state of India called Kerala. I work as a Computer Science Engineer at a small company called Eventifier. I’ve been a Python developer only since the past 7 months and hence, have relatively lesser experience when it comes to programming. Although I have organized conferences and workshops by myself, as a part of my earlier job, this is my first ever talk at one. So there might be a few rusty edges. Do bare with me. Also, as a hobby and passion, I love writing.

Alright, enough with the narcissism. Let’s get back to business.

Redis stores its data in a Big In-Memory dictionary where they keys can only be strings, but the values can be any of the 5 data types that we mentioned earlier. Each of these data structures have their own implementation which will come to later. Let us go back to a few more use cases where you can use redis.

LEADER BOARD (using sorted sets)

Let’s talk about leader board. What I am trying to do here is to give you examples that cover all the 5 data structures that Redis provides so that you will know what to use where and why. Leader board. I am sure you are familiar with the concept of leaderboard, but for those among you who are not, it is place where the top 10 of something is shown. Top 10 or 20, it does not matter. But a list of entities sorted based on their rank.

An example should clarify this right away. Let’s go back to the football world cup example. The tweets are pouring in. Boy, reminds me of monsoon back at home. Anyway, You want to show the most retweeted tweets in descending order of their retweet count. This will give you an idea of what is trending for that particular hashtag. Now, what do you do? This is where the “sorted set” data type comes into picture. As the name suggests, it is a set, but sorted.

What is this sorted based on? Ah yes. So when you hear a sorted set, the picture that should come into your mind is a key with a value as a list of tuples. I use “tuples” in a loose sense. Once you have that picture in mind, this is how the structure would look like:

key: (score member) (score member)

All you need to do is to define a key called “trending_tweets” and then use the “zadd” redis command to specify the score as the number of retweets and the member as the “tweet text + username” or something.

import redis
r = redis.StrictRedis()

tweets = fetch_tweets(hashtag = "#WorldCup2014") #Use the Twython Library
count = 0
for tweet in tweets:
    entities = process_tweet(tweet)
    r.zadd("trending_tweets", tweet.retweet_count, tweet.text)

trending_tweets = r.zrange("trending_tweets", 0, -1)

You could also store the tweet ids as the members and just do a query on your SQL database to fetch tweets with those particular ids. This would work much better since sorted set is a set and it will be expensive to maintain uniqueness on members if they are huge chunks of text.

import redis
r = redis.StrictRedis()

tweets = fetch_tweets(hashtag = "#WorldCup2014") #Use the Twython Library
count = 0
for tweet in tweets:
    entities = process_tweet(tweet)
    t = Tweet.objects.create(tweet = tweet)
    r.zadd("trending_tweets", tweet.retweet_count, t.id)

trending_tweets = r.zrange("trending_tweets", 0, -1)
popular_tweet_list = []
for tweet_id in trending_tweets:
    popular_tweet_list.append(Tweet.objects.get(id = tweet_id))

To retrieve the top 10, use the “zrange” command and specify the indices. That should get you going.

CACHING (using list)

This introduces a new data type as well as a useful feature.

Redis allows you to set “expire” on certain keys. You can specify the key name and the number of seconds in which the key should expire. You might have already guessed it. Yes, you can implement a caching mechanism with this. The timeout remains valid as long as you only “alter” the keys using operations such as increment, add, etc. However, if you set the key once more or delete it, the deal is off. No timeout for you.

The way to implement this would be to first know what value want to be cached. Save that value into redis with a key. Call expire(key, seconds) and you’re done. What goes hand in hand with this is the TTL command. Known as Time To Live. As you could guess, this gives you the time left before a certain key expires. It returns -2 if the key has expired or -1 if an expire has not been set on the key to begin with. Pretty handy.

Let’s go back to the Football world cup tweets example once again. Suppose you want to showcase the photos that got retweeted the most every 5 minute or so. You might have to do something like fetching the popular tweets, get the corresponding photo url, push them into a list and set an expiry on that list’s name.

import redis
r = redis.StrictRedis()

tweets = fetch_tweets(hashtag = "#WorldCup2014") #Use the Twython Library
count = 0
for tweet in tweets:
    entities = process_tweet(tweet)
    t = Tweet.objects.create(tweet = tweet)
    r.zadd("trending_tweets", tweet.retweet_count, t.id)

trending_tweets = r.zrange("trending_tweets", 0, -1)
popular_tweet_list = []
for tweet_id in trending_tweets:
    popular_tweet_list.append(Tweet.objects.get(id = tweet_id))

if r.ttl("trending_photos") in [-1, -2]:
    for tweet in popular_tweet_list:
        r.rpush("trending_photos", tweet.media_url)
        trending_photos = r.lrange("trending_photos", 0, -1)
        r.expire("trending_photos", 120) #Expire in 2 minutes
else:
    trending_photos = r.lrange("trending_photos", 0, -1)

The list is a double ended list actually. You can insert at the left or the right. Accordingly you can pop from either side as well.

CREDITS

The first person whom I would like to thank is someone who deserves much more than me to be up on this stage and give this talk. However, he usually prefers to be behind the scenes, getting things done and motivate people to do things. He is my colleague and the CTO of the company I work for, Mr Nazim Zeeshan and there he is.

The second would be Sripathi. There is a company called HasGeek back in India who organizes technology conferences and workshops. They had organized a Redis miniconf recently where Sripathi gave a talk on Redis Memory optimization. What I am going to present next is from his inspiration.

Last but not the least, the PyCon Singapore team who organized and made this a reality. Kudos to them!

INTERNAL DATA TYPES

This is something that I picked up from what Sripathi explained. I confess I’m not an expert on this but thought it would spark a few minds if presented. Redis stores all that we talked about right now internally using 6 different data types.

Refer to the slides and video for this part.

—-

Slides:

http://slides.com/harisibrahimkv/redis-what-why-and-where

Video:

https://archive.org/details/IntroductionToRedis

A sunny Saturday at BeaglesLoft.

Siva sent me, Krace, Kartik and Sayan an email asking whether we would be available on the 7th of June to volunteer for the first offline Django meetup. I was only too happy to receive the invitation and replied saying “I believe I can make it”.

The next mail in my inbox is where I found TechBuilders. The email was from the BangPypers mailing list posted by someone called Niranjan. This is the link that was in the mail:

http://techbuildersbayesianreasoning.splashthat.com/

Even during my time at HasGeek last year, I used to keep wondering why isn’t there any learning related to Math happening among all these Computer geeks who were working on Python, JS, Ruby, etc. I even had a decent conversation regarding this with the one person whom I found to be interested in the Math aspect of computers. His name is Abhijith and we became friends at the Fifth Elephant conference last year when he signed up to volunteer for it.

Suffice to say, visiting that link, when I saw that these people were trying to bring Math and Computer Science together, I knew it was something that I could not miss at any cost. I sent Siva and the rest of them an email then and there itself saying I had stumbled upon this TechBuilders meeting and might not be able to make it for the Django workshop.

I love teaching and hence was extremely upset about missing the Django workshop. However, on the other hand, I felt like the TechBuilders people had read my mind. It was, as Paulo Coelho would say, a calling. I could not resist going. Also, I had to give up on my Saturday writing as well.

It was being hosted at Haggle’s office. The people working at Haggle were the ones behind BeaglesLoft (a playground for creators and innovators) and also behind TechBuilders, their initiative to teach the Bangalore tech community something that it is lacking. The office was just a 5 minute walk away from my home.

The mail which we received from Asya, the quick witted community manager at BeaglesLoft, on the day before had asked all of us to be there at the venue exactly at 10:30AM and not to follow the “Indian Standard Time”. Little did they realize the inevitable force they were reckoning with. The meeting started at 11:00AM.

The event was supposed to start off with Sandipan from JustDial giving a talk on how they were using Bayesian theorem at their company. Unfortunately, he had some emergency and could not make it. So Niranjan, who is the founder of Haggle, took the stage and started off by introducing us to what the whole deal was about.

The thing that I liked about Niranjan was that he was not pretentious. He really observed Math was not a part of the IT culture, along with the liberal arts being treated as a completely separate entity as well. He wanted to create an atmosphere where these things would co-exist and would value each other’s importance. There, he was doing it.

Not just that. I have heard many people twisting their words to indirectly mean “spread the word”. Niranjan directly told us to do it. His conviction to doing this impressed me. Apart from taking the initiative to build the community, I must say he is a really good teacher too. He taught me Math and that, is amazing.

If you were to meet me before my 4th year of college, I would have told you, without question, that I was going to become a Math teacher. So when he talked about Mass Probability function and the Bayesian theorem in a way that I could understand after more than 2 years of staying away from it, it felt really great.

You must read his series of blog posts on Bayesian Reasoning here: http://beaglesblog.tumblr.com/tagged/techbuilders

We were asked to read them before attending the meetup. Having been the college kid, I put it to the last moment as usual. An hour before the meetup! I finished off all the posts within 45 minutes and it was time to leave in order to reach the venue on time. That dreadful feeling of not having revised what you had learned that dawns upon you on the morning of the exam day was on me. I know, it is funny. But to know that it was not something to worry about, made me feel even more excited to attend the gathering.

Towards the end of his session, he proposed a few use cases where Bayesian reasoning could be applied so that we could break up into teams and work on modelling them.

One was about a Rikshaw driver. Suppose you were one and someone came and asked you to take him to Jayanagar, how would you apply Bayesian reasoning to know whether it would be profitable for you to take him there.

Second one was about the problem given on the blog itself, identifying a person whom you meet in the US as being from Bangalore or not.

The third one was the famous Monty Hall problem. Even though I say it is famous, it was the first time I was hearing of it. It is an interesting problem which makes you realize why Math ain’t your gut feeling. It is a bit crazy, but yeah, read it.

We decided to then split up into three teams of 5 each. The decision was followed by an interesting 5 minutes of trying to figure out an algorithm to split us up. Whether the count should start from 1 and go until 5 before the 15 us were through or whether it should start form 1 until 3 until all of us were through. The confusion was funny enough to have while we were learning Math!

I was in team 2 consisting of:

Sandeep, an IIITB graduate who was going to join Haggle in a few months. He was sharp. The moment we gathered around a table to “brain-buzz”, he came up with this idea of building a recommendation system which would analyse the social media streams of users and figure out what sort of restaurants he preferred to eat out of.

Ashray, who was working with Haggle already. A strong and silent person, I would say. He was as keen as the rest of us on learning together.

Ashutosh, who is Sandeep’s junior at IIITB. He is awesome. When I was struggling to get the basics really strong, he took my pen and paper from me and taught me the reasoning from step 1 patiently, with examples and proper explanation. I hope to see more of him over the coming days.

Last but not the least, Fasil. I would define him as exuberant, but not the BSing kind. He was very outspoken but knew exactly well what he was speaking about. He was working on his own startup.

By the time we had discussed and modelled our recommendation system, it was time for presentations.

Oh, and I forgot to mention the drinks and biscuits that were there all along! No, no, seriously. What kind of a chump would I be if I did not mention this after eating 6 of those delicious cream biscuits right under the nose of my team mates while they were busy building the recommendation system!

Asya, Reya and Tania made sure we had the best atmosphere for thinking and solving the problems at hand. These are the times when I really see the importance of good community managers. They make other people’s lives easier. I never saw myself like that when I was at HasGeek I guess. I just hope others did at least.

It was time for the presentations and team 1 was the first one to go in front. They had build a model around detecting the person who was sarcastic. After analysing manually a few 100s of a person’s tweets and identifying the sarcastic ones in them, each person was assigned a probability of being sarcastic based on how many times he was sarcastic among his past 100 tweets.

This was done for more than a few users. After having built the prior data, when a new tweet came in, you could use the Bayesian reasoning to find out what was the probability of that tweet being sarcastic given it was from a particular user. They had a few numbers as well for demoing this.

Second one was us. Well, I have already explained what we did. The interesting point that Niranjan made was to use more than just words for our probability calculation. Because if we were to just look at words like “Pizza”, “Burger”, etc, then we would miss out on differences between sentences like “I hate pizza” and “I love pizza”.

Once ours was concluded, team 3 came in. They had a funny use case. I have learnt to take things in a lighter note and I hope people don’t jump around reading the use case. It was about the probability of a girl going out with you given the fact that she smiled at you. As funny as this was, for a few of them to think of something like this, would mean that the social media that we have today would have already gone miles ahead in terms of taking advantage ofย  us on similar terms. It was scary.

Niranjan came up to conclude the presentations. This is where he asked us to spread the word and help build the community. He left the rest of the afternoon as an open invitation to do anything sitting together or to move out.

They were taking memberships for the community and I “sold my soul”, as Asya put it. We hung out with each other for an hour or more, getting to know each other better.

I met Samarth, a smart lad who was a Hardware hacker by passion working at Infosys. His face was familiar and there was only one question that I could ask him about it. “Were you there at any HasGeek events?”. Yep, he was there for Droidcon 2013.

Then there was Vamsee, who was a kindred soul when it came to people calling him “Vamshee” adding that all-too-horrible “h” right in the middle! We shared our grief with each other over how inconsiderate people were towards our feelings.

Then there was Ashutosh, Jha (because I really can’t remember his other part of the name), Fasil, Prateek, who asked me, “Hey, aren’t you that guy who wrote that Eventifier blog post? That was amazing”. I was so happy! Jon from Minsh was there. It was good to meet him after such a long time. He was the first few geeks whom I interacted with as soon as I had joined HasGeek. Definitely a part of what made me grow.

We shook hands and were about to leave when I met this unassuming young fellow at the stairs.

“Hey, don’t I know you?”

“I am Rishab. Umm.. Do you know me?”

I unleashed my secret weapon once again.

“Were you at any HasGeek event before?”

“Oh… Were you at MetaRefresh 2013?”

“Yeah, I was a part of the organizing committee”

“Okay. Maybe you heard about that guy who gave a talk on CSSDeck?”

“Oooh! It was you! Now I remember… Cool man”

So that was him. He had generated a whole lot of buzz with his flash talk at that conference. He said he was working on his own startup now. I bid him goodbye and was on my way.

Now I have an excuse to learn Math. I hope these folks keep at it. It was amazing.

A short review on “On Writing” by Stephen King

When Krace first called me asking, “Hey, what is your take on Stephen King novels?”, I was pretentious.

“Although I have not read any of his works, I must say I have heard not-so-good reviews about them”.

I’m sure God won’t forgive me for saying that. I thought he wanted to buy a book for himself and hold true to his resolution of reading as much as possible. On my last day at HasGeek, I was surprised and happy when he gifted me with Stephen’s “On Writing”.

This was 7 months ago. That is how long my reading has suffered. Religiously, I would take this book with me every time I went home in the hope that I would read it from there. Alas, that never happened. During one of my visits, I was talking to my Dad about his childhood days in the hope of recreating the history of my village through a story. I got what I wanted from him and finished writing the first two paragraphs, setting the scene. My sister saw this unfinished piece of work lying on my bed. She read it. Being a voracious reader, she had already finished reading Stephen’s book. She came to me and said,

“I advice you to read ‘On Writing’ before you start off with this piece”.

I did not pay heed to that advice and hence the book remained unread and I did not make much progress with my story either.

The last time I remember when I could not put a book down and *had* to continue reading it was when I was going through Lord Of The Rings. That was during my second year of college and hence almost 3 years ago. Today morning is when I finally found that same spirit again.

The rooftop garden of my office is a lovely place. There is no wifi reception there to begin with. No distractions. It is high enough for one to consider the street sounds to be meager background noise. I reached there about 8 in the morning, made myself comfortable on one of the straw chairs, adjusted the cushion, pulled another chair to put my leg on, took out the book and started reading. I had finished almost 130 pages out of the 285 already.

The next three and half hours were magical. Stephen describes writing as telepathy. For the ignorant, that would sound like a cool word filled with philosophical mumbo jumbo. However, once you start seeing his study with the writing table, the unfinished manuscripts stacked neatly in the drawers, him sitting on the chair with door of the room closed, writing away, you realize you went back 16 years in time. Or maybe he came 16 years into the future.

The image of his life and his struggles become so vividly clear in your mind that there were moments when I really felt like I was with him during those days. Of course, if it was a writing describing his life story, that should be called a biography. This is not so. As he puts it himself, this book describes how a writer is formed. It maps to any budding author’s life. After all, no author had a red carpet laid in front him to be a part of the ministry of authors. They all struggled to get there.

Inspiration is overrated in the modern world. Many find that as an excuse. They wait as if it is someone else’s responsibility to get them inspired. Although websites like zenpencils and the kind are doing a pretty good job at it, the point you are missing is that inspiration is not the solution to your problem of creativity. It is only a part of the puzzle. Grit, determination and perseverance are those which will get you where you want to reach.

Coming back from the slight detour, even though there are more than a few moments in the book where you can imbibe tremendous inspiration from, Stephen gives absolutely practical advices so as to what makes a great writing and what constructs are the worst that you could use.

I quote, “The road to hell is paved with adverbs”.

Those words are going to hit you like a truck the next time you write ‘he said sarcastically’.

For the past few years, many people have told me to get rid of using the passive tone in my writing. Upon asking why that is, I never got a concrete explanation. Stephen just bursts forth with anger and criticisms against using the passive tone in your writing unnecessarily.

I quote, “It’s weak, it’s circuitous and it’s frequently torturous, as well. How about this: ‘My first kiss will always be recalled by me as how my romance with Shanya was begun’. Oh, man – who farted, right? A simpler way to express this idea – sweeter and more forceful, as well – might be this: ‘My romance with Shanya began with our first kiss. I’ll never forget it’. I’m not in love with this because it uses ‘with’ twice in four words, but at least we’re out of that awful passive voice.”

He goes on dissecting the construct even further and giving us more reasons to hate and crucify the passive tone. It is for the timid writers, he says. For those are afraid they are not able to convey what they want to. I was always afraid and I still am. However, I am happy that I wrote the last sentence instead of ‘I have always been afraid’.

I would not recommend this book to those who are interested in factual writing. As much as it offers practical advises like the ones above, it revolves around imagination and creative writing. Taking in things from around you and converting them into what you want them to be. Digging for the fossils, as he puts it. Be patient, be careful. Dig slowly and you will get the entire piece unscathed.

One thing becomes absolutely clear from this book. If you have a passion for writing, the only thing that is stopping you from doing it is your lame excuses. You will want to believe they are real excuses so that you can convince people. But they are lame. You are just lazy to write. You would rather be entertained and wait for your all important inspiration to shower upon you. Get rid of the fear. Don’t be afraid. Just write.

Thank you Krace for gifting me this book. You knew what I wanted more than I knew it myself.