Vector databases have exploded in the past two years. There are both open-source and managed options, including Pinecone, Milvus, Qdrant, and Weaviate, which often claim greater scale, flexibility, and speed than traditional search platforms.

When we set out to design our AI retrieval systems, we began with first principles, not vendor promises. Our research showed that Elasticsearch struck the right balance of performance, cost, operability, and long-term maintainability – at least for our needs, and given our experience operating it at scale.

In this post we share our rationale, and two years of operational results running Fin, the market’s most advanced AI agent.

Initial System: In-Memory Retrieval

In early versions, each customer’s content and embeddings lived on S3. For each inference request, we:

1. Retrieved the customer’s S3 object

2. Loaded all vectors into memory

3. Ran a brute-force KNN search

This design was a good starting point:

• The “database” could handle any number of requests or spikes

• Iteration was simple

• Our scientists could experiment easily, leveraging familiar tools like notebooks and pandas

It also was battle-tested to some degree, having served our previous ‘Resolution Bot’ product for years (which used much smaller datasets).

But as we added support for new content types, and began generating content from conversations, the number of embeddings for each customer soared. Soon, S3 download and deserialization times began to dominate request latency— we started to see spikes of 15 seconds just to load vectors for our largest customers, before even starting the search or LLM inference.

Requirements & Constraints

Prior to committing to a particular technology, we defined concrete constraints:

• Scale: Target of 100M+ embeddings of 768 dimensions, accounting for accelerating growth

• Cost: Consider infrastructure and ongoing operational burden

• Filtering support: Must be able to filter retrieval (by language, content type, audience, etc. to support our platform and permissions)

Other nice-to-haves included:

• Run Less Software: We want to avoid undifferentiated heavy lifts

• Full text search: We might want to move to hybrid search in the future

• Predictability: Known scaling properties, failure modes, and monitoring

Looking back, even our optimistic scale estimate of 100 million embeddings was much too low!

Surveying the Landscape

We evaluated Pinecone, Milvus, Qdrant, Weaviate, and Elasticsearch. All had the basic features we required, so cost initially seemed like the main differentiator. At the time, we had about 20 million embeddings; we projected costs for 5x that scale—100 million embeddings.

Several promising open-source solutions offered theoretical top-end performance, but with potentially increased operability risks. And the managed services tended to be more expensive — Pinecone and Milvus, for example, were estimated to be 2x the cost of their open-source alternatives.

In the end, the primary differentiator for us was operational familiarity and proven reliability. Elasticsearch, already core to several business-critical Intercom systems, presented clear advantages here.

Benchmarking: Vector Search in Elasticsearch

We began to look at standard benchmarks:

• Elasticsearch nightly vector search benchmarks (2M embeddings, 768D): search latency between 100ms (nightly-so_vector-script-score-query-match-all-latency) and 200ms (nightly-so_vector-script-score-query-acceptedAnswerId-filter71%-latency).

• ANN-Benchmarks: Elasticsearch sat in the center of the pack, neither the fastest nor the slowest.

And verified these against our in-house benchmarks on production-like datasets, which gave similar results:

Overall, this performance was good enough for us. For an AI Agent, the bottleneck will generally be the LLM’s latency, which is measured in seconds. Even if search latency dropped to zero, we only stand to save ~200ms. This would be imperceptible to the user in most cases.

Decision Criteria: Why We Chose Elasticsearch

Beyond raw benchmark numbers, the following factored heavily in the final decision:

Low Infrastructure Cost

Just accounting for the infrastructure, we estimated Elasticsearch would cost us the least to run and the gap would widen as we scaled up, especially compared to the Managed offerings.

Initially, it would be even cheaper to get started as we planned to run on a shared large scale Elasticsearch cluster that Intercom used for multiple use cases. And we could migrate to a separate infrastructure when necessary.

Low Onboarding Cost

Since we would not be running a new distributed system, we didn’t need new runbooks and could avoid common scaling/maintenance issues and failure modes.

We could also reuse most of the tooling around running a database: disaster recovery, snapshots, replication.

Subject-Matter Expertise

Elasticsearch is one of the “core technologies” at Intercom. We have experience running it and have subject matter experts within the company (1, 2) who have familiarity with how ES scales. They can proactively manage cluster health, be able to tweak capacity or performance by playing with multiple levers like the number of shards in an index, size of the data nodes etc.

Filterable Hybrid Search

The ability to compose vector and structured filters with existing ES query DSL is a requirement most vector-first DBs do not easily fulfill. With Elasticsearch, we can combine structured filters and vector search with full-text search to improve relevance of the results in the future.

Possibility to move to Approximate KNN in the Future

Since Exact KNN performed adequately for us, we could start with it and avoid having to worry about recall. This simplified the migration. In the future, we could move to ANN for faster results.

Production Results

Two years later, our scale exceeded even our most ambitious forecasts by 3x—a welcome problem.

Note: The actual number of embeddings we have is ~600 Million as we are continually experimenting with different embedding models or chunking strategies. The cost here is for this total architecture: 600 Million embeddings, replicated once, with raw data and metadata. Since we don’t currently create an index for ANN, which usually requires as much memory/disk space as the size of embeddings, we use 300 Million as the number for a fair cost comparison with other vendors.

Compared to other vendors, this setup costs us at least 3 times less (comparing just the unblended cost for fairness). For example: Qdrant’s estimate is ~$30k/month (not accounting for capacity needed for experimentation). Pinecone would be even costlier.

Specifics of our Architecture

• Architecture

  • Data nodes: 9 x i4g.4xlarge

  • Client/Master nodes: 3 each x c6g.xlarge

• Index

  • Embeddings are partitioned into 15 indexes, each with 30 primary shards and a replica.

  • Refresh Interval: 2 seconds

  • The embeddings field is a dense_vector with index=False.

• Ingestion

  • Ingest on every create/update.

  • Average request processes 3 contents (articles, file etc.), which are chunked and bulk-ingested.

  • In the future, we can buffer content updates for a few seconds before processing them to improve ingestion efficiency if needed.

• Querying

  • Exact vector-search using script_score and functions for vector fields.

  • Queries use filters for locale, content that should be visible/invisible to a user etc.

Business Impact

The migration from S3/in-memory to Elasticsearch drove customer-visible response time improvements:

The system has also absorbed 10x increases in customer data volume and query traffic without architectural change, cluster downtime, or significant operational incident. Most importantly, all operational practices—from tuning, upgrades, snapshotting, disaster recovery, to scaling—were well understood from prior ES experience.

Conclusion: Practical Takeaways

When picking tools or infrastructure—whether it’s for vector search, databases, or anything else—these principles worked well for us:

• Leverage what your team already knows. Familiar tools let you move faster, reduce onboarding, incidents, and painful outages. If your team already has expertise in a given technology, that’s a huge advantage.

• Expect trade-offs. Cutting-edge technology (like vector databases) might promise slightly lower latency or higher throughput, but come with new operational risks, documentation gaps, and complex migrations. Make sure those trade-offs are worth it for your business.

• Don’t underestimate the costs outside infrastructure. The time spent learning, debugging, and supporting a new system often outweighs the dollar savings or benchmark wins.

Our experience is not a call to avoid new technology that’s better suited for your situation, but to reassure you that a “boring” choice might not necessarily be the wrong one.

Learn when ORMs actually evaluate queries

Look at the Rails code below and choose the right option:

engineers = Employee.where(role: 'engineer').all

if engineers.count > 0
    send_welcome_email(employee_ids: engineers.map(&:id))
end

You can skip to the next section if you answered correctly1

When you construct a query to retrieve objects, ORMs do not immediately run a query on the database. So, the below line of Rails code has no effect:

engineers = Employee.where(role: 'engineer').all

This code returns a “Constructed Query” object. This is an `ActiveRecord::Relation` in Rails, a `QuerySet` in Django etc. (readers with PHP experience, help me with the Laravel equivalent). ORMs allow you to chain, filter, slice these “Constructed Query” objects further without running a database query until you force the query to be evaluated. This evaluation is forced when you actually try to access the rows. For example, here are some ways to force an evaluation:

# convert the object to a list
engineers.to_a

# print the rows
puts "#{engineers}"

# iterate over the rows
engineers.each do { |engineer| do_something(engineer) }

The takeaway:

Most methods that fetch multiple records don’t evaluate until forced. Most methods that fetch a single value/record evaluate immediately. Please read the documentation for your framework to learn more: Rails, Django.

Back to the send_welcome_email example: the correct answer is 2 queries.

engineers = Employee.where(role: 'engineer').all

# 1st query: COUNT(*)
if engineers.count > 0
    # 2nd query: SELECT *
    send_welcome_email(employee_ids: engineers.map(&:id))
end

This knowledge about ORMs being lazy is not crucial because you can reduce the number of queries that hit your database, but because you can inadvertently create subtle bugs if you don’t know this. Here are two examples that I have seen in production:

1. No welcome emails

In our send_welcome_email example, if a concurrent request updated the role for all engineers AFTER you check the count, but BEFORE you fetch IDs, you will send an empty list to `send_welcome_email`. This could have unintended effects: maybe the method doesn’t handle empty lists.

engineers = Employee.where(role: 'engineer').all

if engineers.count > 0
    # --> concurrent requests updated all role=engineer to role=agent
    # --> EXCEPTION! send_welcome_email doesn't handle empty list
    send_welcome_email(employee_ids: engineers.map(&:id))
end

I like to remember this class of bugs by the acronym: TOCTTOU. More technically, this is the Phantom effect.

2. No farewell emails

Look at the below code, and see if you can find a problem with it:

# Select all undeleted employees who exited before today
past_employees = Employee.where(deleted: false, exit_ts: ..Time.now)

past_employees.update_all(deleted: true)
send_farewell_email(employee_ids: past_employees.map(&:id))

We can see now that no employee will be receiving a farewell email. That’s because past_employees is not a fixed list of records. Let me explain:

past_employees = Employee.where(deleted: false, exit_ts: ..Time.now)

This doesn't immediately fetch records from the database—it returns an ActiveRecord::Relation, which is lazy and only evaluated when needed, like during .each, .to_a, .map etc.

Then, when we do this:

past_employees.update_all(deleted: true)

This performs the update directly in the database, setting deleted: true for all matching employees at that moment.

send_farewell_email(employee_ids: past_employees.map(&:id))

This is the point when the past_employees relation is actually evaluated—but it now no longer matches any records, because they’ve all just been marked deleted: true. It is easy to make such mistakes if you unaware of ORM’s laziness.

One solution here would be to materialise the relation before we update the records, like:

past_employees = Employee.where(deleted: false, exit_ts: ..Time.now).to_a

Learn what your ORM does on .save()

Consider this scenario and choose the right option:

* Fetch an employee record
* Change any column's value. Eg: employee.salary = 50000
* employee.save()

There is no one correct answer, which makes it more dangerous! The answer will depend on your framework/ORM:

• Rails tracks the fields you change and only updates them by default.

• Django’s default ORM updates all the fields by default (you can specify which fields to update if you want)

  • SQLAlchemy, although, has the same behaviour as Rails

• Laravel has the same behaviour as Rails.

My take is that Rails’ default is “less surprising”, and makes for fewer headaches in a large codebase. Imagine your team built an admin UI that allows employees to change their name. This UI is backed by some code like this:

employee = Employee.find(id: x)
employee.name = new_name
employee.save

Unknown to you, the finance team has built a UI for Human Resource personnel to update an employee’s salary. That code looks like:

employee = Employee.find(id: x)
employee.salary = new_salary
employee.save

Now pretend that we serve a request for each of these UIs concurrently. Both the requests will receive the same data for the Employee, change the fields and run the .save() request.

With Rails, there is no conflict by default: each request updates the field they changed.
With Django’s default, depending on the update that hit the database last, the employee will either have the new name (but old salary) or the new salary (but old name). This defect is classed as Lost Update (or TOCTTOU).

The takeaway

Django’s default isn’t wrong, it just means that one needs to be more careful when updating whole objects. Wherever possible, use update_fields to only send the changed fields to your database. Also, race conditions will need to be top-of-mind for you every time. You can protect your important writes with locks, but you might end up with a lot of them in a large codebase.

Learn when you can avoid locks

Locks are convenient but shouldn’t be used willy-nilly. Locking doesn’t come for free. There are other ways to make updates safely without race conditions. Let’s look at an example.

You run a gaming website and maintain a high score for each game. Your website is busy and you get many concurrent requests for updating the high score. After a few lost updates, you started using locks to prevent it from happening again:

def update_high_score(game, new_high_score)
    game.with_lock("FOR UPDATE") do
        return if game.high_score < new_high_score

        game.update!(high_score: new_high_score)
    end
end

This is called a “Pessimistic Lock”: you prevent any other process to update the record while you are updating it. Pessimistic locks are simple to understand but can cause deadlocks or slow down your application (because requests are waiting for the lock to be released).
You can flip this to an Optimistic Lock by checking for the state as you commit your change. Optimistic locks are even more useful in cases where we don’t need to retry a failed update: if there is already a higher score than the one you were updating, you don’t need to retry.

Our update_high_score method can be changed to use an Optimistic Locking strategy called a “Conditional Update”. That way, we can remove the lock and move the condition “update only if the new high score is greater than the current one” to the database directly:

def update_high_score(game, new_high_score)
    game.class.where(id: game.id).where(
        "high_score < #{new_high_score}"
    ).update_all(
        "high_score = #{new_high_score}"
    )
end

The takeaway

If you are not used to them, optimistic locks can seem backward. Pessimistic locks are simpler, you lock the row and do what you want. But once you get the hang of it, you will start seeing it like solving a puzzle and enjoy changing your locks where it makes sense. Most databases support conditional updates. For example: DynamoDB.

Atomic updates

This is not an example for Optimistic locking, but I have sometimes seen locks being used to increment values when no locking is required. It seems obvious to use some atomic operations in cache like databases (like Redis or Memcached), but might not be the first thing that comes to mind for non-cache databases even though most of them support atomic operations.

In our gaming website example, if you want to increment the number of times a game has been played, you can handle race conditions using a Pessimistic lock:

def increment_played_count(game)
    game.with_lock("FOR UPDATE") do
        game.played_count += 1
        game.save
    end
end

But you can also just increment the value in the database directly by running an atomic update:

def increment_played_count(game)
    game.class.where(id: game.id).update_all(
        "played_count = played_count + 1"
    )
end

Fin.

P.S. - If you enjoy reading about race conditions and how to cause subtle bugs, I wrote a post about it that you will like:

I spent my last two years in Intercom working with the Core Technologies team, with Elasticsearch being the core technology I focussed on. As part of my work, I got to build and consult other teams building search experiences in the Intercom product. If you use Intercom, and you have searched for conversations, you have come across my work. To get an idea of what I helped build, this article about Intercom’s Inbox Search is useful: Inbox search and filter.

I have now moved on to the AI Group in Intercom, but I wanted to note down learnings from my time building these search experiences. In addition to other people finding these useful, I find myself coming back to these articles when I need to in the future.

Even though we use Elasticsearch at Intercom, the advice here will hold for Opensearch too.

Start with a contract

This is the most important learning. Write down the cases that you are and aren’t covering.

In most areas of building software, you start with functional and non-functional requirements. If you are building a webpage, you have a design you have to implement. If you are building the backend for a website, you first agree on the API that you will use to interact with the frontend.

With search interfaces, I have found that the requirements tend to be more hand-wavy. On the surface, the goal is easy: The results should be relevant. But how is “relevance” defined? As we start digging, we start defining “relevance” for our use-case:

• Should we support spelling errors?

• How should we handle special characters?

• For a multi-term input, should results where the terms exist in the same order as the input be ranked higher?

• And how much weight should the “age” of a result play in the ordering?

I enjoy this process of figuring out what’s important for the user. It is crucial to note down the conclusions you arrive at after disambiguating the requirements. You might have to make tradeoffs as you build the experience. The contract helps you document the tradeoffs you have made and why.

Once you can see all the requirements together, you also have a better chance at designing your mapping and queries to support the requirements in the best manner.

Add tests for the contract

I have found that it is easy to cause regressions by seemingly small changes to the query/mapping. Simple tests that ingest a few dummy documents and confirm that the results are as expected can do wonders for reducing these regressions and allowing you to iterate with speed.

Understand the Text Analysis pipeline

Elasticsearch performs text analysis when indexing or searching text fields. Based on your requirements, you might have to tune the analysis pipeline for your specific needs: Should you remove stop words? Should you tokenize on whitespace? Should you convert your tokens to n-grams?

Elasticsearch has a helpful section in their documentation about Text Analysis: Text analysis. My advice is to get familiar with this part of the documentation, specially Anatomy of an analyzer. I created a simple diagram to remind me of the concept. This is how text analysis works:

Given an input, this is how the output will be generated.

Knowing how the text analysis works will help you decide the schema of your fields and writing a search query that works for you. It also helps you debug your queries and search results when someone complains in the future (“why is this document not being returned”).

Do you really need to support Typos?

When we think of supporting misspelled words, we think it will help people find apple when they typed aple by mistake. But it also means that you could return bat when someone actually needed bar. Typo tolerance can also return irrelevant results if the user input can be a number: So if someone searches for 98765, they probably don’t want 88765.

Supporting typos is useful in many cases but it comes with a cost: queries handling misspellings are usually slower to run and can lead to lower quality of results.

My suggestion is to avoid typo tolerance unless your use-case would really benefit from it. If you do have to support typos, consider setting the fuzziness such that only mistakes in longer words are supported, as people tend to make fewer mistakes in shorter words.

Don’t be afraid to use multiple fields

As you go over the different cases you uncovered while creating the contract, it will become clear that ingesting the data into a single field might not be enough.

For example: If a user searches for running, you want to return results containing run. But when the user searches for "running", you want to return results that contain the word running exactly. This could be difficult to do using a single field.

Usually something I have seen working is having one field that indexes your text as is (without even removing stop words), and another field analyzed for similarity matching (whatever similarity means for you: removing stop words, stemming, etc.).

Not all text is the same

Your user might be expecting different behaviour based on the text they are searching over. Smaller text fields (like blog titles, movie titles) require different analysis than longer text fields (like blog content, movie synopsis).

For example: if the user knows they have a title like The legend of Tarzan, and they input the, they would expect Tarzan to be in the results. But if you removed stop words during the text analysis, the legend of Tarzan will remain a myth. This can be frustrating for the user.

For searching over smaller text fields, a simple WYSIWYG approach might be good enough and it might make sense to skip most of the text analysis: skip stemming, removing stop words, removing punctuations etc. Also consider using an edge-n-gram Index analyser and standard Search analyser for such a use-case. (Elasticsearch usually advises against using different analysers for indexing and search, but mentions this specific case as a good candidate for going against the suggestion: Here).

For longer text fields, running a more involved text analysis can improve results quality. And if a user’s search for the doesn’t return all the blogs that contain the word the in them, the behaviour is still intuitive for them.

Increasing relevance by using multiple clauses

For improving user experience, you will want to rank more relevant results higher. For example: for a multi-term input, you may want to show those results first where all the terms are present in the same order as in the input.

You can use should clauses in your query to add weight to more relevant results. For this particular example, you can add a should clause with a phrase match to boost the relevance of documents that contain the words in the same order.

When you can’t depend on score

If you can sort results by score, things become easier. Your search query could still match some results with low-relevance, but that might be okay because they would be buried deep behind more relevant results.

But some use-cases require results to be sorted by other fields (frequently, created_at). For example: while searching for blog titles, it will be easier for the user to go through all the results if they are sorted by their published date. But if your query returns both highly and somewhat-relevant blogs, and the results were sorted by published date, you might end up showing irrelevant (but recent) results at the top!

In such cases, you want to tighten your query to only return results that are perfect matches for the user input. This allows for the results to be sorted by any field and still be relevant for the user.

Minimum Should Match

One of the ways of only matching relevant results is by using minimum_should_match and using a value like 2<90% which would only return those documents where most of the words from the input are present. Documentation: minimum_should_match parameter.

Hope it helps 👋

I have derived a bulk of these learnings and examples from DDIA itself, and all the credit for that work goes to Martin Kleppmann.

First, some basics

Before we start, I want to quickly go over some basics so that a wider audience can follow along more easily.

• Transaction

  • A “transaction” groups several reads and writes together into one logical unit. Conceptually, all the reads and writes in a transaction are executed as one operation: either the entire transaction succeeds (commits successfully) or it fails (aborts and rolls back).

  • For a longer explanation, this might be helpful: What is a database transaction?

• Commit and Rollback

  • From this doc:

    The process of placing into effect in the database the updates made by a transaction is called commit. The process of invalidating the updates made by a transaction is called rollback

• Race Conditions or Concurrency Defects

  • Roughly, if two or more transactions simultaneously access (read or write) the same data, these transactions will be considered concurrent. Based on what operations these transactions perform on the data, we might run into anomalous behaviour like losing writes, or reporting incorrect data, or permanently making the data inconsistent.

  • To get a better idea, take a look at this example: Race condition example

I have strictly kept the scope of this article to be an introduction to defects that can occur because of concurrency. Resolving or avoiding these defects is a larger topic that I might discuss in a later post. For now, I just wanted to make the reader aware of these problems. Knowing that a problem exists is often the first step towards dealing with it.

Let’s begin

Dirty Read

Tl;dr: If a transaction can read data written by another transaction that is not yet committed (or aborted), this read is called a “Dirty Read”.

Changes made by an ongoing transaction only become permanent once the transaction commits. If a Database allows other transactions to read updates made by a still uncommitted/incomplete transaction, it can lead to unintended outcomes:

1. If an in-progress transaction makes several changes to the database, and dirty reads are not prevented, other concurrent transactions might be able to see some of these changes while the rest are still being made. This is confusing to users and could cause other transactions to take incorrect decisions.

2. A transaction can abort, requiring its writes to be rolled back. If the database allows dirty reads, another transaction might see data that was later rolled back. This can cause bugs and issues that are hard to reason about.

Let's look at an example: Nancy has $1000 in her bank, split equally between two accounts:

• Account A: Her general account that she uses for receiving and spending money

• Account B: Her savings account that she uses for saving money every month (she was saving for a trip to Rome, but we all know nobody is going anywhere in 2020).

Nancy decides to transfer 100$ from Account A to Account B. The Bank makes this transfer by carrying out a transaction with two operations:

1. Subtract 100$ from Account A

2. Add 100$ to Account B

Let’s assume that Nancy has her Bank’s app open and she is looking at her Account balances while the transfer is happening. Thankfully, her Bank’s database prevents dirty reads. This is what she sees:

While looking at the app, Nancy always only sees that data that is committed to the database, and never a partial state.

But if the database did not prevent dirty reads, this is what she could see:

For a moment there, she gets back an incorrect balance which is lower than what she should have. This happened because the database made the partial changes from her transfer transaction visible to Nancy’s transaction, even before it was committed.

In this particular case, there wasn’t any lasting harm done. She would see her correct balance if she were to refresh again, but not all cases would be this harmless.

Thankfully, dirty reads are prevented by default by all major databases.

Related reads:

• Read Committed in Postgresql

• Read Committed in MySQL

• MongoDB doesn’t prevent dirty reads by default. Check out Read Uncommitted

Dirty Write

Tl;dr: If a transaction can overwrite data written by another transaction that is not yet committed (or aborted), this read is called a “Dirty Write”.

Similar to dirty reads, dirty writes happen when a transaction can access and overwrite data written by a still uncommitted/incomplete transaction. Dirty writes should be prevented because they can lead to messed up partial states caused by the intermingling of writes made by multiple transactions. These states would be difficult to reason about and could cause terrible things to happen.

Let's look at an example: Ginika and Ranyinudo are software engineers. Their managers (different people) want to set up a meeting with them. The meeting system makes two database writes while booking a room:

1. It sets the room’s name to reflect the meeting

2. It adds the participant to the room

Now, Ginika’s manager wanted to schedule a regular 1-1 with her, whereas Ranyinudo’s manager wanted to tell her that she is getting promoted (she is going to be a Senior)! They both proceed to book the same room for the same time. As these examples go, their requests for booking the room were concurrent, and their database does not prevent dirty writes. This is what happens:

Because the database allows transactions to overwrite uncommitted data written by other transactions, we have reached this incorrect partial state. Ginika’s Manager overwrote the Room’s name, and Ranyinudo’s Manager overwrote the participant. In this case, disaster was averted because humans are involved, Ranyinudo’s manager told her of the database mishap and all was fine. But in the wild, this could lead to a lot problems.

Again, thankfully, dirty writes are prevented by default by all major databases.

Is preventing dirty writes enough?

At this point, astute readers might notice that even if dirty write was prevented, one of the transaction would still overwrite the other. And that is correct. But at least the final state would not be a partial state.

In our example, if dirty writes are prevented, the room will be set for an “Appraisal” with Ranyinudo as a participant because Ranyinudo’s transaction committed last. If Ginika’s transaction would have committed after Ranyinudo’s, the room would say “1-1” with Ginika as the participant. Notice how the final state isn’t an inter-mingled mess of these 4 writes.

Even though this works out fine for us in this case, this overwriting is still a problem. We will explore this more in Lost Update.

Related reads:

• Read Committed in Postgresql

• Read Committed in MySQL

• Read Uncommitted in MongoDB

Read Skew

Tl;dr: A transaction (typically, long running) may read different parts of the database at different points in time causing a Read Skew.

If each query in a transaction can always see the latest committed data, some queries would read an older version of the data, whereas some would read from a newer version. This can result in inconsistencies best explained by an example.

Let’s go back to Nancy’s example where she was transferring 100$ from Account A to B. Now let’s assume that while this transfer is happening, the bank is running a scheduled daily backup of the database. This backup transaction is carried out by querying for all the account balances in the bank. Since a bank usually has a lot of accounts, the backup takes hours to complete.

In a database susceptible to Read Skew, this scenario might occur:

According to this horrible horrible backup (we are the bank), Nancy has a total of $1100 instead of $1000. This happened because when the backup transaction read the balance in Account B, it read the updated version of the balance. Since Account A’s balance was read before the transfer, we now have an inconsistency. If this backup is needed to be restored, this inconsistency will become permanent.

Similar to this backup example, Read Skew can cause nonsensical data to be returned by analytical queries that often need to scan over a large number of rows.

Unfortunately, not all databases prevent this defect by default and you should consider verifying this for your database.

Related reads:

• Postgresql doesn’t prevent this by default. Check out Repeatable Read.

• MySQL’s InnoDB engine prevents it by default. Check out Repeatable Read.

Lost Update

Tl;dr: Two or more concurrent transactions might read the same value from the database, modify it, and write it back, not including the modification made by the other concurrent transactions. This could lead to updates getting lost.

Lost updates is one of the most common defect that happens due to race conditions. If you have been developing software for a few years, there is a high chance you have caused or experienced bugs because of this problem.

It is an easy defect to cause. Luckily, it is also easy to resolve in many cases. Any situation where we read a value from the database, modify it, and then set it back, could fall prey to this defect. This set of operations can be called a “read-modify-write” cycle. I like to remember it by the fun abbreviation: TOCTTOU.

A few scenarios where updates can get lost are:

1. Incrementing a counter

2. Updating an Account Balance

3. Updating a complex value like a JSON or a list.

Without troubling Nancy further, let's look at a common example. Assume that we store a list of IDs in redis (or memcached, or any other database). There are two concurrent processes trying to add another ID to this list. Sounds like a recipe for Lost Updates:

Both the processes here read the same list, appended their IDs and then tried writing it back to the Database. Since Process A updated the value last, the update made by Process B will get lost. This tactic of “Last Write Wins” is often used by databases for resolving such conflicts.

Most of the times it is up to the application developer to identify and prevent these kind of lost updates. Being a common problem, there are multiple ways to avoid it:

1. using atomic operations (like INCR in Redis),

2. doing a “compare-and-set” operation (set id_list = new-value if id_list = old-value)

3. or taking an explicit lock on the value you are trying to modify.

Write Skew

Tl;dr: Write Skew is a generalisation of the Lost Update problem. It is also caused by a “read-modify-write” cycle, but the objects getting modified could be different.

A generalisation of the lost update problem is when two or more concurrent transactions read the same objects, and then update some of those objects (the transactions might be updating different objects). If the transactions update the same object, a lost update can happen. If they are updating different objects, a write skew can happen. Write skews are also a very common defect that you would have come across in the wild.

It is easier to explain with an example. Omphile wants to go to the supermarket and she requests for a cab using the popular ride-hailing app, Sober. The app broadcasts her request to all available drivers, and any driver can accept the request. When a driver accepts a request, the app first checks if an ongoing ride for the customer already exists. If not, the app adds a ride for the given combination of customer and driver.

Conveniently, two drivers accepted Omphile’s request at the same time. If the application developers aren’t careful, this is how a write skew can occur:

Here, when the drivers checked if an ongoing ride exists or not, both got the right response: No ride exists. Both of them then proceeded to add a ride. Write skew.

Other common scenarios where this problem can surface is when booking a meeting room in a similar fashion to our ride-hailing app, or assigning a username for a user etc. Some of these problems can be resolved by adding a database constraint or taking a lock on the search results. But, like in our example, it is not always possible to take a lock because our query checks for the absence of a row, and therefore there is nothing to take a lock on.

Phantoms

A related concept here is the Phantom effect: It occurs when a write in one transaction changes the result of a search query in another transaction. In our example, If Driver A’s write would have occurred before Driver B’s search, the result would be different.

👋

We discussed a lot of defects that can be caused if concurrent requests are executed. In reality, the situation is not as grim as it might look. There are a lot of reliable ways to avoid these defects. My aim here was to give you the vocabulary that helps you identify potential scenarios where these defects can occur in your systems. I hope this has helped.

If this topic fascinates you, I highly recommend you give DDIA a read.

Martin starts the book with an example that he goes on to refactor in the first chapter. Sensing an insane opportunity, I stopped right away and copied this original code in Ruby. I then went ahead, wrote down my thoughts and refactored the code like I would if it was an actual thing I was working on. Once I was done, I went back to the book and compared Martin’s notes against mine, and learnt from his thought process and code. I think this is the best way to read this chapter.

If you are someone who enjoys this, I highly recommend you give this a try too. And if not the book, use this blog to power your review.

The Original Code, and provided Context

(Psssst… You can find the full original code here: Github)

We have a Theatre Company that perform plays in events. They charge customers based on the type of the play, and also provide a “Volume Credit” for future discounts. The Company stores data about their plays (plays.json) and bills (invoices.json) in JSON files.

The simple code below is used to print a bill from an invoice. This is the code we will attempt to refactor. Assume that this code is part of a much larger system and we are refactoring while adding a feature to print bills in HTML. Moreover, the Company has plans to perform more types of plays in the future.

# statement.rb

require 'json'

def statement(invoice, plays)
  total_amount = 0
  volume_credits = 0
  result = "Statement for #{invoice['customer']}\n"

  invoice['performances'].each do |perf|
    play = plays[perf['playID']]

    this_amount = 0

    case play['type']
    when 'tragedy'
      this_amount = 40000
      if perf['audience'] > 30
        this_amount += 1000 * (perf['audience'] - 30)
      end
    when 'comedy'
      this_amount = 30000
      if perf['audience'] > 20
        this_amount += 10000 + 500 * (perf['audience'] - 20)
      end
      this_amount += 300 * perf['audience']
    else
      raise Exception("Unknown type: #{play['type']}")
    end

    volume_credits += [perf['audience'] - 30, 0].max
    # Add extra credit for every 10 comedy attendees
    if 'comedy' == play['type']
      volume_credits += (perf['audience'] / 5).floor
    end

    result += "  #{play['name']}: $#{this_amount / 100} (#{perf['audience']} seats)\n"
    total_amount += this_amount
  end

  result += "Amount owed is $#{total_amount / 100}\n"
  result += "You earned #{volume_credits} credits\n"
end

plays_json = JSON.parse(File.read("./plays.json"))
invoices_json = JSON.parse(File.read("./invoices.json"))
result = statement(invoices_json.first, plays_json)
puts result

output

Statement for
  Hamlet: $650 (55 seats)
  As You Like It: $580 (35 seats)
  Othello: $500 (40 seats)
Amount owed is $1730
You earned 47 credits

Note: If you are planning to do this yourself first, stop here. Come back once you have refactored the original code.

My attempt at Refactoring

(Psssst… You can find the refactored code here: Github)

Reading through the code, this is what stood out for me:

1. The statement method returns a formatted text right now. I would like it to return a structured statement that I can format later as I wish, for example: for sending Slack notifications, emails. Additionally, it simplifies writing tests for this and the formatter methods.

2. The code that adds extra volume_credits for comedy plays, the comment states that we add extra credit for every “10” attendees, but the code adds a credit for every “5” attendees. Either this comment has gone stale, or there is a bug in the code. Either way, we can make the code more readable and remove the comment. (Aside: I like antirez’s blog about writing comments a lot.)

3. Overall, it is difficult to understand what is going on in the code. I think we can make it more readable easily.

4. The method is doing too much, calculating volume_credit and amount for each type of play, and creating the text. It would be better if I can hand over this calculation to a different class that would do the needful, and my method just creates the text.

5. Because of this “doing too much”, the code has become messy. It will become still messier if the calculation was different for each Play too (and not same for Plays of the same type).

6. Also, looks like that the calculation is in cents and is being converted to dollars in the text. The code should make this explicit. Maybe I could rename the variable to be amount_in_cents?

7. The code right now assumes audience to be present and to be a number. Maybe we should put in validations that asserts for this truth? Although it seems out of scope for our current exercise.

Extract out the Calculation logic

I moved the code for calculating amount and volume credits to a different class and file. I also separated the calculation for the two types of plays.

In the future, adding new play types will just need a new class to be added.

# statement_calculators.rb

class PlayTypeBaseStatementCalculator
  def initialize(play_id:, audience:)
    @play_id = play_id
    @audience = audience
  end

  def amount
    raise NotImplementedError
  end

  def volume_credits
    raise NotImplementedError
  end
end

class TragedyStatementCalculator < PlayTypeBaseStatementCalculator
  ...
end

class ComedyStatementCalculator < PlayTypeBaseStatementCalculator
  ...
end

I also implemented the calculation in a more verbose manner. I think this improves the readability of the code.

# statement_calculators.rb

...

class TragedyStatementCalculator < PlayTypeBaseStatementCalculator
  def amount
    fixed_charge = 40_000
    total_additional_charge = 0

    included_attendee_count = 30
    charge_per_extra_person = 1000

    if @audience > included_attendee_count
      extra_attendee = @audience - included_attendee_count
      total_additional_charge += charge_per_extra_person * extra_attendee
    end

    fixed_charge + total_additional_charge
  end

  def volume_credits
    min_attendee_count = 30

    @audience > min_attendee_count ? @audience - 30 : 0
  end
end

class ComedyStatementCalculator < PlayTypeBaseStatementCalculator
  def amount
    fixed_charge = 30_000
    additional_charge_per_person = 300
    total_additional_charge = additional_charge_per_person * @audience

    included_attendee_count = 20
    charge_per_extra_person = 500
    additional_fixed_charge_if_extra_attendee = 10_000

    if @audience > included_attendee_count
      extra_attendee = @audience - included_attendee_count
      total_additional_charge += (charge_per_extra_person * extra_attendee) + additional_fixed_charge_if_extra_attendee
    end

    fixed_charge + total_additional_charge
  end

  def volume_credits
    min_attendee_count = 30
    extra_credit_for_every_n_attendee = 5

    credits = @audience > min_attendee_count ? @audience - 30 : 0
    credits += (@audience / extra_credit_for_every_n_attendee).floor

    credits
  end
end

Return the right calculator for a Play

I wanted to separate out the logic behind fetching the right calculator for a play, so I created a getter for it.

# statement_calculators.rb

...

def get_calculator(play, performance)
  audience = performance['audience']
  play_id = performance['play_id']

  case play['type']
  when 'tragedy'
    TragedyStatementCalculator.new(play_id: play_id, audience: audience)
  when 'comedy'
    ComedyStatementCalculator.new(play_id: play_id, audience: audience)
  else
    raise Exception
  end
end

Calculate the Statement

Keeping calculation out of the formatting logic, I created a method that loops over the invoices, and generates a structure containing information about the bill.

This structured statement can now be used by any formatter.

# statement_calculators.rb

...

def calculate_statement(invoice, plays)
  statement_hash = { performances: [] }

  invoice['performances'].each do |performance|
    play_id = performance['playID']
    play = plays[play_id]

    calculator = get_calculator(play, performance)

    statement_hash[:performances].push(
      {
        play_id: play_id,
        play_name: play['name'],
        audience: performance['audience'],
        amount: calculator.amount,
        volume_credits: calculator.volume_credits,
      }
    )
  end

  statement_hash[:customer] = invoice['customer']
  statement_hash[:total_amount] = statement_hash[:performances].reduce(0) { |sum, perf| sum + perf[:amount] }
  statement_hash[:total_volume_credits] = statement_hash[:performances].reduce(0) { |sum, perf| sum + perf[:volume_credits] }

  statement_hash
end

Finally, generate the Bill

The original statement method is now text_statement and all it does is format the bill as needed. A new method, html_statement can now be defined similarly to format the bill in HTML.

# statement.rb

require 'json'

require './statement_calculators'

def text_statement(invoice, plays)
  statement_data = calculate_statement(invoice, plays)

  lines = ["Statement for #{statement_data['customer']}"]

  statement_data[:performances].each do |perf|
    lines.push("  #{perf[:play_name]}: $#{perf[:amount] / 100} (#{perf[:audience]} seats)")
  end

  lines.push("Amount owed is $#{statement_data[:total_amount] / 100}")
  lines.push("You earned #{statement_data[:total_volume_credits]} credits")

  lines.join("\n")
end

def html_statement(invoice, plays)
  nil
end

I will call the refactoring done at this point.

Although, I can’t seem to shake off this feeling that there is something more that can be done. I don’t think I am completely happy with the changes 😕 Let’s see what Sir Martin has to say.

Code Review, courtesy Martin Fowler

The process that Martin followed for refactoring the original code, in his words, was:

1. Decomposing the original function into a set of nested functions

2. Separate calculating and printing code, the two phases.

3. Introducing a polymorphic calculator for the calculation logic.

From what I have learnt in this chapter, in the hypothetical scenario in which Martin Fowler does a code review of my changes, this is what he would have to say:

1. “I like that you are using an intermediate structure to pass the calculation from one ‘phase’ to another. I also like that both of us are calling the new class a Calculator” - Thank you Martin, thank you. I try.

2. “Your methods might just be too verbose. Try implementing them without all that verbosity? Same with the class names.” - 🗒

3. “You could also maybe implement the volume_credits method in the base class as there seems to be some code that can be reused” - 🗒

Code after addressing Martin’s comments

(Psssst… You can find the final refactored code here: Github)

Based on Martin’s comments and how he went about his refactoring, these are the changes I made to my code

Rename the Calculator classes

I renamed the calculator classes. My names were too verbose and I didn’t like them a lot. I liked the names Martin gave his classes better.

# statement_calculators.rb

class StatementCalculator
  ...
end

class TragedyCalculator < StatementCalculator
  ...
end

class ComedyCalculator < StatementCalculator
  ...
end

...

Pass complete objects to the Calculator class

I am now passing complete performance and play objects to the calculator class. This is easier for the callers now as they can simply pass what they have, and the calculation logic can make use of what it needs.

1. I had previously opted to explicitly pass only those arguments that the Calculator class needed, and there is some benefit to it. For example: It becomes clearer what information Calculator needs. But then it can be argued that the caller doesn’t need to know how the Calculator class works?

2. If we are fearful of introducing breaking changes by omitting certain values from the passed object, values that the class might need, I don’t think making the caller do extra work is the solution. Either the Calculator class should validate the input it is getting, or this validation should be done while creating these objects, or both.

# statement_calculators.rb

class StatementCalculator
  def initialize(play, performance)
    @play_id = play['play_id']
    @audience = performance['audience']
  end
  ...
end

...

def get_calculator(play, performance)
  case play['type']
  when 'tragedy'
    TragedyCalculator.new(play, performance)
  when 'comedy'
    ComedyCalculator.new(play, performance)
  else
    raise Exception
  end
end

Simplify amount and volume_credits implementations

I re-implemented amount and volume_credits methods in the Calculator classes. I think I went overboard with the verbosity earlier and didn’t strike a good balance. I opted for longer names, that “felt” more readable/explanatory but, they had the opposite effect when I zoomed out.

The actual calculations were simple enough and just the expressions would do a good job explaining what is happening. I tried making it more readable by introducing variables to make the code self-documenting, but I now think that it was counterproductive.

I also realise that this is closer to the original definition of these methods, and I wasted my time for nothing. But hey, you wouldn’t know until you tried.

# statement_calculators.rb

...

class TragedyCalculator < StatementCalculator
  def amount
    result = 40_000
    result += 1000 * (@audience - 30) if @audience > 30

    result
  end

  def volume_credits
    [@audience - 30, 0].max
  end
end

class ComedyCalculator < StatementCalculator
  def amount
    result = 30_000
    result += 10_000 + 500 * (@audience - 20) if @audience > 20
    result += 300 * @audience

    result
  end

  def volume_credits
    base_credits = [@audience - 30, 0].max
    base_credits + (@audience / 5).floor
  end
end

...

Implement volume_credits in the parent Calculator class

I changed the base volume_credits method to implement the part that is common to both types of play.

1. I had previously opted to not do this because I thought it might be over-abstraction.

2. But after seeing Martin’s implementation, I find this current version to be much better. It makes the code easier to understand. The reader can just say, “Okay, so we are doing what we do for all the types PLUS this extra thing.”

# statement_calculators.rb

class StatementCalculator
  ...
  def volume_credits
    [@audience - 30, 0].max
  end
end

class TragedyCalculator < StatementCalculator
  ...
  # remove the definition of `volume_credits` here
end

class ComedyCalculator < StatementCalculator
  ...
  def volume_credits
    super + (@audience / 5).floor
  end
end

...

Calculate totals within the loop in calculate_statement

While in flow of making these above mentioned changes, I also made some minor modifications to the calculate_statement method.

1. Instead of looping over all the statements at the end to calculate the total_sum and total_volume_credits, I simply maintained a variable that I updated in each iteration of the loop and added that to my dictionary at the end.

2. This I think makes the intention clearer, and the reader won’t have to go through the reduce loop over again. total_amount += perf_statement[:amount] is beautifully simple.

Yes, I should have done this the first time itself 🤷‍♂️

# statement_calculators.rb

...

def calculate_statement(invoice, plays)
  statement_hash = { customer: invoice['customer'], performances: [] }

  total_amount = 0
  total_volume_credits = 0

  invoice['performances'].each do |perf|
    play_id = perf['playID']
    play = plays[play_id]

    calculator = get_calculator(play, perf)
    perf_statement = {
      play_id: play_id,
      play_name: play['name'],
      audience: perf['audience'],
      amount: calculator.amount,
      volume_credits: calculator.volume_credits,
    }

    total_amount += perf_statement[:amount]
    total_volume_credits += perf_statement[:volume_credits]

    statement_hash[:performances].push(perf_statement)
  end

  statement_hash.merge(
    total_amount: total_amount, total_volume_credits: total_volume_credits
  )
end

😌 I can now shake off that bad feeling from earlier. I think we did good today 👍

In Closing

Call me delusional, but it sure feels great to go through this process and learn from one of the foremost authority on the subject.

Another takeaway for me is around creating impact. Martin Fowler could teach people all that he has learnt one by one. It would be good for those few who were fortunate enough to be working with him directly and people like me would never get this chance. But he spent the time, wrote a book, and now his knowledge can educate thousands of people. That’s the kind of impact I hope to make in my life someday.

I would love it if someone gets inspired by this and decides to go through this exercise. Reach out if you do!

Ron introduces a simple notion of “The Natural Way” of Software Development: Focus on delivering value early and often.

Building software is like walking in a room where the floor is lava, but there is a path of cool, green, grass that you can walk on and be safe. This path exists and is the “Natural Way”. We can’t stay on this Natural Way all the time, but it is good to know that this path exists so you can try and come back to it when you wander too far off.

The Pyramid of Value

Value: What we want

Value starts when we ship the software. What if we shipped some valuable part sooner than the rest? Most users don’t use every feature of the product. Since most users don’t use all the features, a smaller set of feature can provide real value, and provide it sooner.

After that first release, we may need to follow up with the rest of the product. But there can be times when you released the first bit and then stopped. Releasing early gives Information which is value as well. Sometimes the most important information we can get is that we are doing the wrong thing. Fail fast.

While choosing what to work on, choose work that is low effort but high value. When we work like that, we can start delivering value early, and in most of the cases, end up providing more net value because we might choose to work on other high ROI tasks for some other product, or realise we don’t need some low ROI features etc.

Best value comes from small, value-focused features, delivered frequently. MMF: Minimal Marketable Features: Pieces that make sense to us, and to our users. Features: More granular than MMF.

Guiding goes better “feature by feature”

Tl;dr: Don’t do waterfall. This chapter was not particularly useful since a lot of the modern teams follow the practices described here by default.

Two types of “Guiding”

1. Activity based phases (Waterfall):

   1. Analysis —> Design —> Coding —> Testing.

   2. You end up planning for features your might not get to or which are not useful.

2. Feature by feature:

   1. More visibility as your project progresses, since you start seeing the software. You will not know how your project is going until you start seeing value.

   2. Sometimes you might even change features etc. based on the information you are receiving from the features you already released

Teams decide a deadline, and they want “everything” done by that deadline. Years of experience has taught us that this never happens. You end up either delivering lesser value, or deliver it late, or both. When we know this truth, why not work for it in a manner that makes it possible for us to deliver value early, and gives us better visibility over the progress (and not spring surprises at the last moment).

Organising

Feature teams >> Function teams

Planning feature by feature

Plans are useless, but planning is indispensable

- General Eisenhower

Visions are grand ideas, vague yet enticing. But how do you go from there to small detailed features? Planning shouldn’t involve a detailed list of what needs to happen and at what time. Planning should be:

1. Identify key feature that we will need to have early

2. Features we can’t live without

3. Defer implementing low-value ideas indefinitely. Let’s not waste time thinking about and keeping track of them.

Planning is important. We probably have to consider a lot of bad ideas to get a few good ones. Plan, but stay loose and ready for change.

Software people are terrible at estimating because humans are terrible at estimating. Let’s not try harder to estimate, let’s find a better way:

1. Set a time and money budget: Appetite (Shape Up)

2. Produce the most valuable features first

3. Keep the product ready to ship at any time

4. Stop when the clock runs out.

How do you get started on a project?

1. Get a team, do a time bound spike, and

2. figure out what we can do to provide value and about how long will it take.

Don’t just plan in the beginning, plan all the time. Work in sprints. And work with stories.

Important: Don’t take a big story and break it down into smaller technical items, called tasks. Using tasks mean Business people don’t have clear visibility over how things are going, and often don’t know how to help. Stick with stories. Break down stories into smaller stories, each making sense to business-side people, rather than technical steps.

How much should the team take on

1. The team should decide this

2. Yesterday’s Weather. You will probably get as much done this iteration as you did in the last iteration

Sprint planning

Discuss what all needs done, roughly discuss what will it take to get those things done. Estimate roughly (number of days?) on an aggregate level. Don’t estimate individual work pieces etc. Remember, the point isn’t to make good estimates, but to do good work at a consistent pace.

Fight off the desire to estimate things strongly. With estimates comes the need to “improve” them or compare them. Both these practices are pernicious.

On stretch goals: They are meant to be stretch goals, but people tend to try to meet them. Eager to please, they will unconsciously hurry, leave out a few tests, leave the code not quiet as clean as they might have etc. just to squeeze in one more feature. This creates pressure. Avoid it.

Generally speaking, estimates are likely to be wrong, and they focus our attention on the cost of things rather than on value. Consider de-emphasizing or eliminating cost estimates and steering to success by a focus on value.

Building the product, feature by feature

Each iteration is a full product development cycle. People giving the requirements (product managers, Business people) also need to learn how to break down their big ideas into smaller requirements. As you go, it means more product understanding, tightening what is a must have and what are nice to haves.

Build features and foundations in parallel

Ok, so you are building everything feature by feature. That’s cool. But how do you build the foundation feature by feature? By this I mean the overall system design.

You could build out the whole infra/design before hand and then build features, or build each feature completely, with its own infra one at a time. Both means that you will end up delivering lesser value in the given time.

Ron suggests that we create simple and functional features and designs to begin with, and keep improving it with every feature. This helps because you can now deliver the most value in the time + you are just learning about your product. There is no point over spending time on the foundation or a particular feature, when we might learn later that this is not what we wanted.

Build simple and functional versions first.

Of course this will take skill. For the PM to understand what mix of features do they want first. And for the devs, how much to design so that when needed, those designs can be changed or modified to handle other requirements. If the design needs to be scrapped, then it is desirable that you have sent less time on it etc. This is a skill that the team will need to learn.

Bug free

Write automated tests.

Assorted notes

Don’t whip the ponies, help them improve

• Don’t try make more progress by asking developers to “work harder”. “Work Smarter” instead, and that means you need to help the team work smarter. “Work smarter, not harder” means that we have to help them get smarter.

• Before focussing on individual productivity, work on team productivity. Analyse the sources of delays. Make sure the team has all the skills needed to do the work. Team members with key skills should be full time, not shared. Use specialists to increase other individual’s abilities. Value working together.

• Increase individual productivity by increasing capability, and not by urging people to work harder.

Twisty little passages: Technical Debt

• The time needed to build a feature comes from two main components: its inherent difficulty, and the accidental difficulty of putting it into whatever code already exists. Teams are good at estimating the inherent difficulty. What makes us erratic, what makes us slow down, is the accidental difficulty. We call this difficulty “bad code.”

• If we allow code quality to decline, some features go in easily, sailing right through. Others that seem similar get entangled in twisty little passages of bad code. Similar work starts taking radically different amounts of time.

• To keep progress steady, we have to avoid building twisty little passages, and when we do build then, we need to straighten them out.

Fin.

A completely finished “loop” is when you can provide the required input to your system, and it produces the desired output (or side effects, if that’s how you like it). The “Close the loop first” technique is about closing this loop as fast as possible by creating a barebones version of it first, providing all or some required inputs, and generating a partial form of the desired output. Once we have closed this barebones loop, we can then begin implementing behaviours from the inside out, so that with each new change our loop starts looking more like the actual system we want.

Sure, this is nothing new, right? We have all heard of this advice in various forms: build a proof of concept as quickly as possible; validate the unknowns first; if you want to deliver a car, deploy a skateboard first, etc. This is similar, but I am talking today purely from a “programming” point of view. In addition to helping you fail fast, “closing the loop” first also lets you build systems with more speed.

How exactly does it help? 🤔

Let’s look at what I am trying to say using a simple example.

You have some data in Comma Separated Value (CSV) format. You have to read the file and for each row, clean a few columns, check if the row exists in a database table. If it does, update it, otherwise create a new row in the database. We also want to create a new CSV file that has an extra column specifying if the row was created or updated in the database.

We know how we will roughly do it. Read the CSV file, and for each row:

1. Clean columns as necessary

2. Check if row is already in the database

   1. If it is already present, update it

   2. Else, create a new row

3. Add this row to a new CSV file with an extra column for creation/updation information.

We can go about writing our code in this exact order, or we can “close the loop” first, and then add capabilities to our code.

We start with an open loop

Step 1: Close the Loop!

Read the CSV and add each row to another CSV, with a new column “operation”

To be able to close the loop quickly, we will just add a static value "created" to the new "operation" column. Also add tests to check that each row in the input file is present in the output file, and that a new column exists.

import csv

def close_the_loop():
    input_file = open('data.csv', 'r')
    output_file = open('output.csv', 'w')

    input_csv = csv.DictReader(input_file)
    headers = input_csv.fieldnames + ['operation']
    output_csv = csv.DictWriter(output_file, headers)

    for row in input_csv:
        row['operation'] = 'created'
        output_csv.writerow(row)

    input_file.close(), output_file.close()

* I wanted to make the code more readable, for a blog, by having less indents. Otherwise a better way to read files in Python is by opening the file using with.

Aaaaaaand…the loop is closed!

Step 2: Add juicy bits

For each row, call a dummy method that will later implement the DB operation, but right now returns “updated/created”.

Update your test to check the correct value of the operation column based on what you mocked it with.

...

def update_or_create(row):
    return 'created'

def close_the_loop():
    ...

    for row in input_csv:
        row['operation'] = update_or_create(row)
        output_csv.writerow(row)

    ...

Actually implement the update_or_create method

Add a new unit test for update_or_create. By this point, you are almost done.

Add a method to do some data cleaning before writing to the output file.

Also update your tests to account for this change.

...

def clean_row(row):
    row['name'] = row['name'].strip

def close_the_loop():
    ...

    for row in input_csv:
        clean_row(row)
        row['operation'] = update_or_create(row)
        output_csv.writerow(row)

    ...

Aaaaaaaaand…work is done!

How does this work?

Even though it sounds like the same advice, I instantly visualised it when Kesha brought it up and said “let’s close the loop first”. I am a very visual person, and this created an image in my mind of a loop that needed closing. And when that loop gets closed, that’s another extremely satisfying image. For me, this image is important because we regularly hear about a lot of best practices and design patterns, but what really matters is how many of them can we remember to apply to our situation when we actually sit down to implement stuff. Visualising a concept helps me remember it for a longer time.

Photo by HUZAIFA SHEIKH on Unsplash

Other than allowing you to build you system quickly, piece by piece, there are other related benefits with this approach:

1. You get to “executing” code quickly, which helps you add more code faster and with more confidence (as you can quickly check if it is working or not by running your code, hopefully using the basic test you wrote).

2. Instead of getting 20 failures, across the entirety of those 2000 lines of code that you changed, you get fewer errors as you build your system bit by bit. Most of these errors would be around your latest changes and, thus, easier to debug and fix.

3. If you plan correctly, you can prioritise and selectively add capabilities to your system (maybe do the optimisations after initial deployment?) as you build. This helps you deliver basic capabilities as fast as possible, and sometimes that’s all what is needed (*cough* startups *cough*).

4. And of course, all the benefits of failing fast or uncovering the unknowns first are still valid: If your assumptions prove out to be wrong, your time investment is minimal at this point, and you can still figure out a way to work around this new-found information.

Caveats

Of course, this isn’t a standalone best practice that you can implement in isolation. You need to already have a clearer picture of what you are trying to build. And then, of course, some people might like it better to implement end to end in one go. And then there are things that are better implemented that way.

All this is, is a tool. Use it where you think it can be useful.

Ciao!

Update September 2020: The world has changed a lot ever since I interviewed with Intercom. In order to compensate for this new normal, the interview process has been changed a little. The rounds, by themselves, will still be a lot similar to what I have mentioned below, but their arrangement might have changed. Intercom might also have replaced/merged/broken down rounds to make it easier for candidates applying remotely.

Intercom’s was one of the most thorough interview processes that I have experienced/heard about, and it’s a good thing! It made me confident of the team I am going to be working with if all goes well. From what I could decipher from the rounds, Intercom is not only testing you technically, they also want to test you for your product skills and are judging if you will be a good addition to the team. They were also very responsive throughout the process, and the result of a round was shared within a week all the time (mostly within 3-4 days).

Intercom has also written a lot of resources to help you prepare for the interview. For starters, you should checkout this: How to prepare for software engineering interview questions. I will share links to Intercom’s blogs wherever relevant.

Remote Rounds

These rounds happen either asynchronously or over audio/video call (they used Google Hangouts for all my interviews).

Take Home Test

This was a standard filtering round. Intercom gives you a small problem (which they have mentioned in a blog post) that you need to submit the solution for. They have also written a blog post about what you should take care of, how do they assess the activity etc. here: How to prepare for engineering interview assignments.

HR Screener

This is just a call to dig into your background, motivations and interests etc. You can take this time out to also ask them about the interview process.

Technical Screener + Cultural Contribution

Two-thirds of the call is “technical” and consists of basic Algorithmic and Data Structure based questions. My suggestion here will be to cover as many ”Easy, Top Interview” questions as possible from Leetcode. The remaining call is focussed on “culture contribution” which was similar to other culture contribution rounds (discussed later).

Culture Contribution

Intercom has multiple rounds where they want to assess “What contribution will you make to Intercom’s culture”. I liked that they call this round “Culture Contribution” and not “Culture Fit” because doing the latter could lead to a lot of biased decisions as they explain in their blog post. They also do a good job of explaining what you can expect in the interview and how do they evaluate candidates.

My suggestion here is that do not try to “wing it” in this round. It gets difficult to extract helpful examples from your past to show, for example, how you respond to negative feedback, on the spot! You need to be prepared beforehand and go deeper into your past experience and come up with evidence/anecdotes which you think is going to be relevant for the interview.

Some questions that I prepared for, or encountered in the interviews were:

• Tell me about a time when you built a product and it failed. Why do you think you failed?

• Can you think of an instance where you made a mistake while working with a customer? How did you handle it?

• In busy times, how do you prioritise your customers and interactions?

• Tell me about a challenging project, lessons learned and what would you do differently next time

• Tell me about a time you went beyond what was expected

• Tell me about one time you improved something without being told to

• Tell me about a project that succeeded and why do you think it did

• Tell me about a disagreement you had with someone and how you handled it

• How do you handle conflict?

• A recent decision you made that you would like to change? Why?

• Tell me about a time you got some feedback that you didn’t agree with.

• What do you do when you get feedback that you don’t understand?

Onsite Rounds

If you clear all the previous rounds, Intercom invites you to their office in Dublin. This is a good opportunity to meet a lot of the team in-person, see the office. For people who haven’t been to Dublin before, it is also a great opportunity to see how you like the city (get the feelzzz).

Pair Programming: Minicom

This is a 1.5 hours long round in which you are given a small project (called Minicom) and you have to extend the codebase to add functionality. You are given 30 minutes to work on it by yourself, and then 30 minutes each to pair program with two Intercom developers. At the end, you are expected to do a little demo of the feature that you built and answer a few questions such as, “What other features would you add if you had more time”, etc.

The rounds are “Pair Programming” in the real sense. I am not that well versed with Javascript and there were a few frontend changes. The people who worked with me in the pair programming round helped me debug issues, and with Javascript syntax as well.

Intercom explains in depth about this round in their blog post.

Technical Design

This is a typical technical design round in which you are expected to design the backend for the Minicom task you worked on, go deeper into it and add a lot of features. You are expected to be able to communicate your design by creating rough diagrams on the board.

Problem Solving

This is a typical Data Structures and Algorithmic round, and involves solving 1-2 problems and writing code on a whiteboard in 45 minutes. The questions were still “Leetcode Easy” level, but they were more difficult than the ones asked in the technical screener at the beginning of the process.

Product Chat

The goal here is to figure out if you can think about the Product too and offer suggestions/opinions while you are working on Intercom. They try to figure this out by talking to you about a product that you like, and why do you like it.

There is no right or wrong answer in this round, I think they just want to hear you out, and understand how you think. Again, be prepared with a few products that you can talk about. You can get some followup questions, like what would you do if you were the CEO of that product? What do you think is the largest threat for them? Who are their competitors?

I will also suggest that you prepare for a few products that you are sure the interviewer will know about too. This makes the conversation interesting because the interviewer also gets a chance to share their opinions, and dive deeper.

Values Deep Dive

This is similar to the Culture Contribution round, but certainly more gruelling for me (could be because I was tired by this time 😅).

Role Chat

This isn’t an interview round, but a time you get to ask as many questions about the role, the company etc. that you want.

Role chat: not an interview, you get to ask questions, but by now I was too tired. Still, I liked everyone involved in the rounds. Helpful and nice people.

All in all, I had a great time interviewing with Intercom. I got to meet around 7-8 people, which really helped me in getting a better idea about the people I could be working with in the future. Everyone that I had a chance to meet with was extremely warm and helpful.

PS: I had completely forgotten, but it turns out I had tried applying for an Internship 4 years ago (some audacity!). Such a fun thing to find 😁 Destiny then?

Ciao!

Some basic tips:

1. If you have a large and/or a table which gets a lot of traffic, remember that a plain REINDEX command will take a lock on the table that won’t allow any write operations on the table till the command completes. Reindexing CONCURRENTLY is almost always a better option out. You can read more about it in the official docs: Building Indexes Concurrently.

2. Remember to set statement_timeout = 0; before running the reindex command since CREATE INDEX commands are also taken as statements by Postgres, and they will be killed if they go above a decided threshold.

3. To figure out indexes that a table has and the corresponding bloat percentage for each of them, you can use this query (we picked it up from PgHero’s codebase). We add a table_name = 'my_sweet_table' to the WHERE clause at the end of the query to only get the indexes for our table, but that is completely optional.

4. You can also use a simple query to get the definition of all the indexes for a table. These definitions can be used as is when we want to recreate them.

Recreating Indexes with Foreign Key constraints

Indexes that are not created for a constraint can be reindexed in the same way.

We have the definition of the original index, we can just replace the name with a temporary name and use CONCURRENTLY:

CREATE INDEX CONCURRENTLY
  new_idx
ON my_sweet_table USING
  btree (my_fk_column);

Now you can safely drop the original index. You can optionally use CONCURRENTLY here as well, read more about it in the docs: Drop Index.

DROP INDEX my_lovely_index;

You could also rename the new index to the original name (some frameworks, like Django, autogenerate index names using the table name, the app’s name, and a hash of both of these plus the columns of the model. You might want to preserve that name).

ALTER INDEX
  new_idx
RENAME TO
  my_lovely_index;

Also, if the index you are recreating is a unique index, you can add the keyword UNIQUE to the CREATE INDEX command.

Recreating Indexes with Unique constraints

Recreate the Index, with the keyword UNIQUE.

CREATE UNIQUE INDEX CONCURRENTLY
  new_uniq_idx
ON my_sweet_table USING
  btree (col_a, col_b);

Now, we want the constraint to use this new index. For that, we drop the original constraint, and add a new unique constraint that uses our new index. This is done in one atomic statement so that there is no time when there is no constraint on the table. We don’t have to rename the index this time as Postgres automatically renames it to the name of the constraint.

ALTER TABLE
  my_sweet_table
DROP CONSTRAINT
  uniq_constraint_777,
ADD CONSTRAINT
  uniq_constraint_777 UNIQUE
USING INDEX
  new_uniq_idx;

Recreating Indexes with Primary Key constraints

This is achieved in the same manner as we did for recreating the index for a unique constraint. The only difference is that this time the constraint that we add is a PRIMARY KEY constraint, of course :D

CREATE UNIQUE INDEX CONCURRENTLY
  new_pkey_idx
ON my_sweet_table USING
  btree (id);

ALTER TABLE
  my_sweet_table
DROP CONSTRAINT
  my_sweet_table_pkey,
ADD CONSTRAINT
  my_sweet_table_pkey PRIMARY KEY
USING INDEX
  new_pkey_idx;

That’s it! Tadaaaa.

Aside: I highly recommend reading “How Will You Measure Your Life?” and then “Power of Meaning”, in that order, if ever you come across such a problem in your life. I faced it recently, and these books helped me a lot.

Marginal Thinking and its Trap

Marginal Thinking is thinking in terms of a little bit more, or a little bit less. It talks about the “additional” or “incremental” cost of a decision, and it’s benefits.

Has any of you ever been in a situation where you were deciding whether or not to have another piece of pie at a buffet? Or whether to hit that snooze button on the alarm and get a little bit more sleep before getting up? …

The additional piece of pie is the marginal piece of pie. The additional half hour is the marginal half an hour of sleep.

Howard Baetjer (source - visit for a deeper introduction)

Marginal thinking is considered to be a good way of taking decision in Economics, but it is not as straightforward because it is easy to get the “cost” and the “benefits” of the “marginal whatever” wrong, and end up making a bad decision. For example, what is the marginal cost of eating that extra pie? No, it is not $4.

What’s the benefit of one more piece of pie? It tastes good. What’s the cost of it? I’m already overweight, I should be trimming back on my calories.

Howard Baetjer

We often consider the “marginal cost” to be the, almost always, tiny effort/investment needed to get the marginal benefit. We think that skipping a day in the gym “just” means not getting the day’s exercise, or sleeping for those extra 10 minutes “just” takes 10 minutes off our day, or spending that extra dime was “just” a dime over our planned budget, or scrolling through Instagram for those extra 10 minutes between our meetings were “only” some useless 10 minutes we utilised.

In reality, all these things are not “just” what they look like on the surface. They represent things like not meeting our savings target at the end of the month, and then having lesser capital when we need it the most, or finding that we have spent a total of 60 minutes scrolling Instagram because some meetings started late and some ended early, or continually missing breakfast and not being able to work effectively because we were feeling hungry.

In “How Will You Measure Your Life?”, the authors talk about how we end up taking morally wrong decisions by misjudging the marginal cost because those incremental decisions “felt” so harmless.

The marginal cost of doing something “just this once” always seems to be negligible, but the full cost will typically be much higher. Yet unconsciously, we will naturally employ the marginal-cost doctrine in our personal lives. A voice in our head says, “Look, I know that as a general rule, most people shouldn’t do this. But in this particular extenuating circumstance, just this once, it’s okay.” And the voice in our head seems to be right; the price of doing something wrong “just this once” usually appears alluringly low. It suckers you in, and you don’t see where that path is ultimately headed or the full cost that the choice entails.

How Will You Measure Your Life?

What they are referring to as the “full cost” is actually the “full marginal cost” that we glazed over while making the decision. That is how we end up breaking our habits too! It’s okay if I go have a few drinks today as well, right? It’s okay if I didn’t read for an hour today like I had planned to. What is it going to cost?

So, whenever you are going to break a habit …

Remember to live by the Rule of Rational Life

Every time you are going to break a habit because of marginal thinking, ask yourself what is the “full marginal cost” of the decision, and what are the marginal benefits? If the cost outweighs the benefit, is it worth it?

The rule of rational life, as shared by Jerry German, is that “As long as the marginal benefit exceeds the marginal cost, do it.” Here, as long as you are judging the “marginal cost” and “marginal benefit” correctly, you would do okay :)

And account for this additional cost every time

Other than the full marginal cost of making a habit-breaking decision, there is another cost that you are incurring. You are actually making it easier for yourself to break the habit again.

It’s easier to hold to your principles 100 percent of the time than it is to hold to them 98 percent of the time. The boundary—your personal moral line—is powerful, because you don’t cross it; if you have justified doing it once, there’s nothing to stop you doing it again.

How Will You Measure Your Life?

Even though the authors meant it for maintaining integrity, I think it is true for building good habits too: “100 percent of the time is easier than 98 percent of the time”.

Follow the discussion on Reddit: /r/getdisciplined

Bounded rationality means that people make quite reasonable decisions based on the information they have. But they don’t have perfect information, especially about more distant parts of the system.

Donella H. Meadows, Thinking in Systems

As a simple example, think of a fishing ecosystem. The fishermen benefit directly from how many fishes they catch every day. Ideally, the fish population that is lost by fishing or other causes (pollution, ageing etc.) gets replenished by natural reproduction. But the fishermen start overfishing to maximise their benefits, bringing the fish population below a critical number and natural reproduction alone is no more sufficient to replenish the fish population. The fishermen end up destroying their own livelihood.

Why couldn’t the fishermen foresee this? The more you fish, the lesser there will be left, isn’t that intuitive? No. Because the fishermen don’t know how much fish is there in the pond, or how many fishes will be caught by other fishermen the same day. They have a bounded view of the reality (the same example is also suitable for describing “Tragedy of the Commons”).

We would probably take the same decisions we criticize right now

However much we criticize the decisions made by someone else in a different position than ours, we tend to make the same decisions when put in their roles, subject to the same information flows, incentives, disincentives and goals as they are. As Donella Meadows notes, “We teach this point by playing games in which students are put into situations in which they experience the realistic, partial information streams seen by various actors in real systems. As simulated fishermen, they over fish. As ministers of simulated developing nations, they favour the needs of their industries over the needs of their people. As the upper class, they feather their own nests; as the lower class, they become apathetic or rebellious. So would you.”

But:

Seeing how individual decisions are rational within the bounds of the information available does not provide an excuse for narrow-minded behaviour. It provides an understanding of why that behaviour arises. Within the bounds of what a person in that part of the system can see and know, the behaviour is reasonable. Taking out one individual from a position of bounded rationality and putting in another person is not likely to make much difference. Blaming the individual rarely helps create a more desirable outcome.

Donella H. Meadows

In order to not get surprised by the outcomes of our/others’ “rational” decisions, I think it is valuble to:

1. Keep in mind that the decision you are going to take might have effects you don’t know of/want because you have a bounded view of the problem. Go out of your way then to understand what other knowledge exists and how can you incorporate that in your decision.

2. When criticizing any decision (that feature that someone built?), make sure you take a moment to think of what all information was available when the decision was made (I touch on taking and giving feedback in another blog post).

Create systems that function well despite bounded rationality

So does it mean that we will always end up making the wrong decisions? No. Some systems, like our liver/heart, are structured to function well despite bounded rationality. The right feedback gets to the right place at the right time. What if we could structure our teams/organisation in the same way? What if each of the stakeholders got at least all the information they need to make an “informed” decision? Doesn’t that also explain why we call for diversity in teams? Because diversity is one way of ensuring you get a lot of different perspectives/information flows that help you make better, more inclusive, decisions as a team.

The bounded rationality of each actor in a system may or may not lead to decisions that further the welfare of the system as a whole. If they do not, putting new actors into the same system will not improve the system’s performance. What makes a difference is redesigning the system to improve the information, incentives, disincentives, goals, stresses, and constraints that have an effect on specific actors.

Donella H. Meadows

In closing

The concept is simple, and we might also have an understanding of it on the subconscious level, but having a term for it in our vocabulary means we are more aware of its existence, and therefore much more enabled to account for it.

To paraphrase a common prayer: God grant us the serenity to exercise our bounded rationality freely in the systems that are structured appropriately, the courage to restructure the systems that aren’t, and the wisdom to know the difference!

Donella H. Meadows

Before we begin, this doesn’t mean I have perfected this. These are just some ideas I try to keep in mind to make sure I am not unnecessarily defending myself or my actions. All of us falter.

This: Avoid having an emotional reaction to feedback

You will continue to suffer if you have an emotional reaction to everything that is said to you. True power is sitting back and observing things with logic. True power is restraint. If words control you that means everyone else can control you. Breathe and allow things to pass.

Anonymous

Thank god for the internet, and impostor accounts! This quote has become a go-to for me whenever I feel that I am getting affected too much by what people have to say to me. Read it slooooooowly, and absoooooooorb it.

Based on this insight, I started giving myself time to absorb critical feedback. A very common scenario is receiving a code review. Sometimes when I find myself having an emotional reaction to a review, I will just go through all the comments and not respond. I will take some time (maybe a few hours) and then come back to the review. Having the time to absorb the feedback and getting my emotions out of the way, I find myself become more accepting of both objective and subjective suggestions, and just be a nicer person overall 😬

It is okay to be wrong

Phew. The pressure! I forget this often (maybe you too?). 

You proposed an idea, but the idea had loopholes. It is okay to propose bad ideas. You suggested a design, but the design will break for these edge cases. It is okay that you missed thinking of those edge cases. You acted a certain way, and it wasn’t the best that could have been done. It is okay.

It is completely okay to be wrong. Once I understand that, I calm down.

Let’s say I am leading a team. I might feel that if I am wrong, I will be respected less (not a hypothetical situation). But on the contrary, accepting that I might have been wrong lets people come closer. Because now there is no pressure on them as well to be always right, and then they can voice their opinions and ideas unfiltered, and that’s how good solutions are found, right?

Let go of the Ego

Another reason that stops us from accepting a mistake is Ego. Ego doesn’t let us say, “Oops, I totally missed that!“. The trick (I think), is to know that ego creeps up unannounced, and it is pertinent that you listen to the other person, without reacting to the words like it is an attack on you (“Don’t be the Alpha Geek”).

Ryan Holiday’s Ego is the Enemy is a good read for creating the awareness.

Hear out the other side

Another thing I do is, I try to see where the other person might be coming from. As in, why does the person think they are right or what they are suggesting is fair. If you understand the motive behind someone’s actions, it is also easy to reason with them, and/or understand their point (the idea was discussed in an unrelated book: Never Split the Difference). 

Related to this is the idea that if you want to buy something you already like, let’s say the iPhone, don’t search for “why should I buy the iPhone”. Rather, search for “why should I not buy the iPhone?” because that is a perspective you don’t have. Once you understand the arguments from both the sides, you can now make a more informed decision.

Learning from this, when someone disagrees with me, I try to think “What might be the reason that makes this person believe they are right? What information/insight do they have that I might be lacking?“. This also helps me ask better questions to clarify the opposing argument. This is much better than thinking “Why can’t they understand my argument?“.

I will be happy to know about what you do to take critical feedback well, and if you think I need to relearn a few things.

I recently came across the term “Alpha Geek” in “The Manager’s Path”.

The Alpha geek is usually an excellent, effective engineer who is driven to always have the right answers, and solve hard problems.

Alrighty! I want to be one! Where do I sign up?

He knows exactly why that thing you’re trying to do won’t work, and when it doesn’t, believe me, he’ll remind you how he told you so. If only you had listened to him and done things his way! At their worst, alpha geeks can’t let anyone else get any glory without claiming some of it for themselves. They are origin of any good ideas but had no part in creating the bad ideas, except that they knew they would fail.

Ummm… I don’t… think… I wanted… this?

The alpha geek can be very rigid about how things should be done and closed off to new ideas that he didn’t come up with. Alpha geeks can’t stand people complaining about systems they built or criticize their past technical decision… Are you eagerly seeking out the gotcha, hunting for mistakes, reluctant to admit that someone else has had a good idea or has written good code?

Ok, wait. Am I one? Do I come out as one?

I love design discussions, I love contesting solutions and have had some very livid arguments with my colleagues on subjective choices. Do I come out as rigid? Someone who takes offence if his mistakes are pointed out?

Revelation

After getting introduced to the concept, I am now actively thinking if I accept my mistakes or not. It is a fun exercise. It is not like I think I am an alpha geek, but I think I might be unconsciously exhibiting some of the bad characteristics of one. And the problem here is, you will never know you are becoming one until it is too late.

As my friend, and colleague, Priyank Verma pointed out:

We have this partial knowledge of the world, but we have formed very strong opinions based on this partial knowledge. We haven’t been here long enough to see how our decisions in the past have played out (has that multiple inheritance thingy we did last year, which looked super sexy at that time, made it a mess to change stuff in future?). And then we contest design and subjective decisions based on these strong opinions.

And he is right.

I do tend to defend my decisions around code and design and give excuses, where in reality I could have just said that a) I couldn’t think of it at that time, or b) I was wrong.

Although I am not that bad, trust me. If someone does prove objectively as to how this other way is better, I happily accept it. I tell the person he is right and the suggested solution is much better. Yes, let’s do this! But in the moments before getting proven wrong, do I behave rigidly like I am the all-knowing senior engineer who is often right?

Changing the way I go about discussions

Rigid. This is not who I am, and I don’t want to be coming off like this unconsciously as well. So now whenever I get into a discussion, I am actively evaluating my stance:

1. Am I right, or am I defending myself?

2. Am I right, or am I just scared to say I was wrong?

3. Am I right, or am I biased against the other person?

Another area where I have started doing this is Code Reviews. I realised that, more often than not, I took the review comments a little too personally, or was affected by them. Which isn’t needed or right. I give hard feedback, I will get it back too, there is nothing personal about it.

At their worst, alpha geeks are like a bad fish in the pond. The rotten apple that can damage the whole crop. You owe it to the people you work with to create and preserve a culture where everyone is open to new ideas and finding the best way out, together :)

Here’s to better designs and healthier discussions 🍻

(Originally posted on my Medium account)

While I have been working on large projects for over 2 years now, this was my first time taking on a more formal leadership role. A solver team lead is more a Project Manager than a People Manager, so I was not just managing, but also executing. This probably made the much-dreaded transition easier. I enjoyed and learnt a great deal these past few months and I totally sucked in the beginning. But I am glad to announce that by the end I was much less awful than I was at the start.

How did I become less awful? By learning from my own experience, but more so from the feedback that I got from the team. I recently collated these lessons and 1–1 notes during my quarterly retrospection (ignore the fact that it was my first time trying it, and stay impressed), and realised that it would make for a good read for people who are/were/will be in the same boat as me.

Here are some ideas I found useful:

Nurture an environment where all ideas are worth sharing

A team is a cocktail of all the different personalities that the members carry. Not everyone will be equally comfortable in sharing an idea they might have. Some people might not be confident about their ideas (which might not be a correct judgement on their part), some would avoid speaking unless directly spoken to, while some are not just used to speaking amongst that particular group of people yet (maybe it’s a new team). Despite all this, how do you get everyone to share ideas?

What I have learnt is that you can get people to share ideas that they have by asking the right questions. Questions that I frequently ask, when we have a problem at hand, are:

1. What do you suggest we do?

2. What do you think are some ways we can solve it?

3. Do you think there is a case where the solution will fail?

Ask these questions, and then pause. Give them time to think. More than that, let silence encourage the other person to speak. Chris Voss describes this as “effective pause” in his book “Never Split the Difference”. It is not in the same context, but I have seen it work because the premise remains the same. Effective pauses make people gather their thoughts and think. A typical conversation at the office goes like:

John Doe (hypothetical teammate): So, there is this issue. What do you think should be done?

Ketan Bhatt: Interesting… What do you think should be done?

JD: …?

KB: … :)

JD: …

KB: …

JD: Well… I guess we could try doing X?

KB: Makes sense, what do you think the situation would look like if we do X?

JD: There might be some issues with this particular thing, but that can be handled if we do a little tweak to X.

KB: Looks good, will the design hold when we also have to do this other thing we had in the backlog?

JD: Hmm, let me think more and get back to you?

KB: Sure!

Everyone who faces a problem also knows at least one way to solve it. Sometimes that way is all that is needed, sometimes it might need some tweaking. Otherwise, you can always suggest something you think might work and then discuss the solution further.

You do this enough number of times, people automatically start doing this on their own, refining their ideas, and then finally coming to you to share what they have got. And since now there are more people sharing their ideas, your solution bank just grew larger, and the quality of your team’s problem solving, as a unit, goes up. Woot woot!

Build an Event Loop

We have all been there. The project kicks off with meticulous plans, timelines, and metrics that we vow to keep track of. A few weeks into the quarter, the timelines and the metrics aren’t being tracked with the same punctuality, if at all. I fell victim to this as well.

We, as engineers, have an inherent inclination towards execution. It becomes easy to forget/procrastinate updating the timelines, backlog, skip stand-ups, planning for the coming weeks, etc. What follows is a loss of visibility:

1. Your team doesn’t know where it is heading, will we meet our goals, are we on the right track?

2. The other teams lose sight over what is going on with your team

3. You catch hold of issues, that can jeopardise the goals, too late. Course correction becomes harder.

Learning from mistakes and guidance from my lead, here is a suggestion: build yourself an Event Loop, and block your time explicitly for following it. We got this from a FirstRound Post by David Loftesness which is a must read if you are making the transition for a developer to a managerial role.

A lot of us follow something of this sort already, I found that making it explicit just works better for me.

Balance thy sprint well

My learnings are perfectly explained by Alan Page, Quality Director at Unity, in his Manager Readme. Specifically:

The ACM framework (Ambitious, Comfortable, Mundane). The idea is that if you look at the work you do over a week / sprint, some of that work is new, challenging, or ambitious, a big chunk of work is stuff that you’re just really good at (comfortable work), and you may end up with some work that you’re overqualified for, or is boring, but that just needs to get done (mundane).

We should work together to make sure you have enough ambitious work that you are challenged and growing, if you’re not learning something new every week, that’s something we should work on together. We also want to minimize your mundane work. Often, your mundane work may be someone else’s ambitious work.

Vikas Gulati, CTO at Squad and my mentor, often points to the following diagram, from this First Review post, for explaining roughly the same thing:

It is common knowledge that any good developer won’t be too happy if they don’t see themselves growing. But an overlooked fact is that growing is a continuous process, and not a badge that you get by working hard for a month. It is easy to burn out people by assigning ambitious tasks to them one after the another (with good intentions on your part), and it is often too late by the time one realises that some irreversible damage has been dealt. Another important aspect is gratification. Some tasks are challenging but provide gratification early, while for some it might come in much later.

Your sprint is like a dessert. Too much sugar? Thank you for the Diabetes. Too little? Is it even a dessert? And what about the nuts and cherries? Can’t have too much or too few of them as well. The perfect dessert is just the right amount of sweet, with nuts and cherries that randomly surprise you when you take a bite.

A sprint, much like a dessert, should be a good mix of challenging and comfortable work, with wins spread for good measure :)

These lessons might be just common sense to some (you have my respect people), but I felt that knowing these things explicitly just helped me channel the team’s strength in a better manner, and, thus, do justice to their talents.

(Originally posted on my Medium account)

In a bid to prepare ourselves for projected growth, we are at the moment trying to figure out what part of our system will break at what scale, and how. One step towards this was to also define strict timeouts for our database queries, and eliminate/fix bad queries in the process.

The problem

Our requirements were:

1. Be able to define different timeout values for different types of servers (app servers, analytics etc.)

2. The different limits should be well represented in the code so that they’re easy to discover, even by people who join our team in the future

3. It should be easy and quick to modify these limits

We identified multiple sources of our queries. Each of these might need a different query timeout. These sources are:

1. App servers: queries that run for our frontend facing APIs, like APIs that our Android app or clients use

2. Celery servers: queries made by our celery tasks that run asynchronously

3. Cron servers: queries made as part of crons

4. Alerts: we have a system that runs SQL queries at configured time intervals, and pushes the data (results of the queries) to relevant people (over Slack)

5. Analytics: queries that run as a part of our ETL (v0.1) system

We planned to incrementally reduce the timeouts because at every step/iteration, there will be queries which will not be able to run properly within the planned timeout. We will have to fix all those queries before we reduce the timeout. The incremental limits we defined for each iterations were:

This is a no-brainer if your requirements are simple. You can simply create roles in the database and set different timeouts for them.

Our backend is built using Django, and to accomplish this we would have to

1. Write a raw SQL migration to create the roles (if needed), and

2. Alter them to set the appropriate timeout

3. Set the database dictionary differently for different server in Django settings with the correct role and passwords

Why not just directly log in to the shell and do this? Because then this change isn’t represented in the code and creates gaps in knowledge over time. But, even though migrations are part of the code, they are just for change management, and rarely does someone go back to migrations to look for “logic” affecting your app’s behaviour.

Since we were planning on having multiple iterations, and there would be a lot of back and forth between the timeout limits while we are experimenting, it would become a hassle to write migrations and apply them every time something had to be changed. This solution was not for us.

The solution

We started thinking of a better way to accomplish what we had in mind.

We knew the “ease of configuration” would only come if we can set the timeouts from within Django somehow. Thinking more in this direction and connecting little tidbits we were aware about Django and Postgres, we realized that:

1. One can set a timeout using SET inside a Postgres session which is then adhered to until the end of the current session using: SET statement_timeout=<x>;.

2. Django publishes a connection_created signal every time a new database connection is created. This connection is then put in the Connection Pool from where it can be reused (governed by configuration parameters like CONN_MAX_AGE)

Aha! Can’t we just catch the connection as soon as it is created, and set the timeout to whatever we desire for the session? Yes we can :)

We also went ahead and set the timeout separately for each connection.alias. That gives us even more flexibility, we can now set multiple separate timeout values for the connections from the same server as well (for example: set a timeout of 5s for queries made for our Android app facing APIs, except for login API, for which we set the timeout to 1s. And then use <queryset>.using('<alias>') to use the timeout you want).

The benefit of this approach is:

1. Everything is in the code, you can just read and figure out what is happening

2. Easy to modify the timeouts

3. Since the logic is now in the application code, we can do more stuff with this, like setting a certain timeout only for some percentage of the connections.

4. Further, we can set different timeouts for different queries made from the same server

Caveats

1. Since we are running an additional query every time a connection is made, it has some implications. Even though Django’s documentation says that the effect is minor, it is worth checking out if it’s okay for your case

2. Since the timeout is set using Django’s signals, it means that wherever Django does not publish a signal, this will not work. One such case is when you are directly logging in to the Postgres shell (or by doing python manage.py dbshell).

The changes mentioned here gives us more control over our queries and we can selectively restrict our systems in case there is a 🔥 that needs strict actions to be taken to keep the more important parts of the app alive (not the best solution, but sometimes they don’t have to be 😃) .

(Originally posted on my Medium account)

UPDATE: Jun 2020

Haki Benita was kind enough to provide feedback on this article. Here is a link to what he suggested: Haki Benita’s comment on Medium. If you are learnt from or liked this article, you should definitely read Haki’s feedback as well.

Update 6th Mar, 2022: My friend and colleague Ayush Shanker recently published a follow-up post for most of the problems mentioned in this article. I recommend going through that post as well (or maybe just that post): ”Celery in production: Three more years of fixing bugs“.

As mentioned in an earlier post, we rely on Celery for publishing and consuming tasks to/from our RabbitMQ (RMQ) broker. We are very happy with the whole setup and it works reliably for us. But this wasn’t exactly the case up until four weeks back. We were plagued with a plethora of issues which we hadn’t got down to fix, mostly because first, the number of issues was small, and the occurrences less frequent.

But as the frequency and the types of issues increased, we started spending a lot of time in maintenance. I personally would have to keep an eye on the RMQ admin to see which queues weren’t being consumed properly, see if workers (we will use “workers” for celery workers or consumers) were still up or not stuck in an infinite restart loop. And every time I noticed something abnormal, I would have to restart the rogue worker/s manually.

The Issues

Here is a list of issues that we identified, and fixed:

1. Worker servers always had an unexplainably high RAM usage

2. Worker servers always had an unexplainably high CPU usage

3. Workers stayed idle, not consuming any tasks

4. Workers kept on restarting

5. Publishers did not adhere to task_publish_retry_policy

Let’s go over them one by one

Worker servers always had an unexplainably high RAM usage

Tl;dr: We had orphaned child processes still running. Use stopasgroup if you are using supervisord.

UPDATE: After a month of writing, this is still happening, albeit rarely and less severely. A final solution is to regularly (crons, maybe) kill the processes. This is also hinted at in Celery’s documentation.

We noticed our worker servers always having 100% RAM usage, and noticed that processes for old workers were still alive. This became apparent because we had very recently changed the configuration and we could see processes for workers with the old configuration (they were listening to inexistent queues). This was a red flag.

We use supervisor to control the celery workers and have to do a reread every time we change the worker config. On searching, we found out that when we reread and update or when we restart, supervisor doesn’t kill the old processes. This was also raised in an issue on supervisor. As suggested, we started doing stop and start instead of a restart (in hindsight this looks dumb now since all restart does is stop and start) and before rereads as well. Of course, this didn’t help things.

A ps auxf output (after removing unneeded info) from one of our servers clearly showed how there were old celery processes still hanging out.

USER        START   TIME COMMAND
ubuntu      03:38   0:57  \_ celery worker A
ubuntu      03:38   0:30  |   \_ celery worker A
ubuntu      03:38   0:32  |   \_ celery worker A
ubuntu      03:38   0:32  |   \_ celery worker A
ubuntu      03:38   0:31  |   \_ celery worker A
ubuntu      03:38   0:58  \_ celery worker B
ubuntu      03:38   0:31      \_ celery worker B
ubuntu      03:38   0:31      \_ celery worker B
ubuntu      03:38   0:31      \_ celery worker B
ubuntu      03:38   0:30      \_ celery worker B
ubuntu      Sep09   1:59  celery worker C
ubuntu      Sep10   3:04  celery worker D

It was clear because:

1. We had restarted all the celery workers quiet recently, as is visible by the START column, while the other two older processes are at least a day old.

2. How celery, roughly, works is that we start a parent process that starts more child processes (depending on the concurrency) and maintains a pool of these workers. This is consistent with how the newer workers are depicted, while the two older ones have no parent/child processes.

On further research, I came across an issue in celery that mentioned the same problem.

Turns out, celery parent processes don’t propagate the STOP signal to its child processes, leaving them orphaned (these are the old workers we saw in our ps output above). This got fixed by using stopasgroup in the supervisord config. As documented, with stopasgroup set to true, supervisor sends the STOP signal to the whole process group.

Worker servers always had an unexplainably high CPU usage

Tl;dr: Our --max-memory-per-child was set too low, we moved back to using --max-tasks-per-child.

We would be okay with this if the tasks were CPU intensive or the rate or volume of task consumption is high. They were not. It was just simple get and set on the database and the rate was about 17 tasks per second when we had ~12 workers spread over 4 servers.

To figure out what is going on, I sshed into a server with ~100% CPU usage, and did:

> ps aux -sort -%cpu

Found a worker process that was taking ~48% of the CPU. To check what was happening with that worker, I checked the logs generated by that worker. The logs showed that the worker was being killed after every 3–4 tasks because of reaching the memory limit (we were starting our workers with a --max-memory-per-child of some 200–300mb). Although it did look like a sane value to us, maybe we misunderstood how this limit works.

The logs were similar on other servers. This would explain the high CPU usage. Killing and creating processes would take CPU. And this would also explain the slow rate of execution. So, on a hunch, I removed this limit from the workers and moved back to using --max-tasks-per-child.

Instantly the rate increased to ~250 tasks per second (from 17) and the CPU usage also settled down. Huge win. Memory leaks are still covered because of the limit on the number of tasks.

Workers stayed idle, not consuming any tasks

Tl;dr: There was a deadlock because of a play between psycopg2 and ssl. Updating psycopg2 to 2.6.1 fixed the issue.

The queue had available tasks, and healthy consumers, but the tasks weren’t being consumed. This would happen every 2–3 hours for some particular workers. We had to manually restart the hanging workers and quickly became a headache.

One hypothesis was that a particular task was making the workers hang, since the RMQ admin showed picked but unacked tasks whenever the workers used to hang. Checking the logs for the hung workers confirmed that the workers had started a particular task but not completed it. Double checked this by looking at the active tasks for hung workers by doing:

> celery -A squadrun inspect active

Solution for this, as mentioned on the internet, was to periodically restart the workers. But this looked like a hack and not the solution to me. But we weren’t getting any further headway into this. On a hunch, again, we decided to use strace on a hung worker to actually see what a worker was stuck on. This turned out to be a good idea:

> sudo strace -p 27067
Process 27067 attached
futex(0x33372e8, FUTEX_WAIT_PRIVATE, 2, NULL ^C
Process 27067 detached

On searching about what futex and FUTEX_WAIT_PRIVATE means, and checking similar issues with celery, we stumbled upon a similar issue as ours.

From one of the comments on the issue:

Are you using Postgres in any way? Because that’s exactly what was happening with us. Postgres registers a locking callback with ssl, which all ssl requests use. Unfortunately Postgres unloads the callback after closing its connections, which is fine for Postgres, but any other ssl consumer (requests included) would be unable to release the lock, allowing a thread to deadlock itself if it tries to reacquire its own lock.

As mentioned, updating to psycopg2==2.6.1 (we were at 2.5.2) fixed the issue.

Workers kept on restarting

Tl;dr: celery==4.0 had a bug, which got fixed in v4.1.0. But we went ahead with our own fork and applied the patch to it.

Sometimes, we would have our workers continually restarting. This was a problem on our staging environment though, and, when nothing worked, emptying the RMQ queue once, and then publishing the tasks fixed the restarting worker.

To figure out what was going wrong, we again went back to the logs, and saw the following exception and traceback:

2017-09-09 16:48:44,188 [CRITICAL] celery.worker: Unrecoverable error: TypeError("'NoneType' object is not callable",)
Traceback (most recent call last):
  File "celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)

... ...

  File "billiard/pool.py", line 1487, in apply_async
    self._quick_put((TASK, (result._job, None, func, args, kwds)))
TypeError: 'NoneType' object is not callable

This one was the easiest problem to figure out. Looks like when there are tasks already in the queue, and a worker is consuming from multiple queues, this bug makes an appearance. They start consuming from the queue even before the queue is ready. This bug was fixed in celery==4.1.0. We were at v4.0. Before making the update though, we made sure there were no regressions or new bugs introduced in v4.1.0, we came across one major one. This bug made us less confident of making the update, because v4.1.0 was out fairly recently, and testing it thoroughly to make sure nothing else was broken would have taken a long time.

So, we forked celery and applied the patch that fixes the bug we were facing . with v4.0, and we were good to go!

Publishers did not adhere to “task publish retry policy”

Tl;dr: task_publish_retry_policy is broken in kombu==4.1.0, we downgraded to 4.0.2.

Just when we thought we were done with bugs and issues caused by celery and other related libraries, another made appearance. Somewhere amongst all this, we had also updated kombu to v4.1.0.

Now, for one of our APIs, we wanted the celery publisher to raise an exception as soon as the connection couldn’t be established while publishing. But it wouldn’t just happen. The worker will go to sleep(1) (as found out by the traceback when we killed the process manually) if a connection wasn’t established, and do this infinitely. We tried setting a stricter retry policy explicitly while publishing the task using apply_async but it didn’t work out.

Finally, we found out that task_publish_retry_policy was broken in kombu==4.1.0. We downgraded to 4.0.2 and things started working as expected.

Takeaways:

1. Logs made it easy to debug issues. Long live logs.

2. Test library updates before applying them to production, because great developers make mistakes too :D

A big thank you to Tarun Garg, my friend and colleague, for helping with the fixes :)

(Originally posted on my Medium account)

A lot of problems we solve at Squad deal with images. Images to be tagged, labeled, moderated, transcribed etc. And there is this thing about images, they are generally heavier than text, digitally. And if our contractors (users of our app) spend more money on internet packs/plans than what they earn, or if they have to wait longer for the image to download because they have a slow/bad connection, their ROI from using Squad is no more justified.

Once, there lived a simple program…

No points for guessing, we compress the images on our end before we send it out to our contractors.

The client sends us, what we call, a “Data Unit” (basically a JSON containing all the data, like text, images, etc. pertaining to one particular item). Each data unit can contain multiple images. And all of these need to be compressed. Our current architecture involves sending a task (using Celery) to our queue (RabbitMQ). These tasks get consumed by celery workers who are listening to the queue. Each worker picks up a task, which holds the ID of one data unit, loops over each URL, downloads it, compresses it, and then uploads it to a bucket in S3.

Sorted.

This simple solution served us well. We had 4 workers that were able to process 3-4 Data Units per second on average. Of course, if a Data Unit had more images, it would take more time for that particular Data Unit. This was really slow. And as we are looking at getting more data every second and every hour, this would just not scale. At the current rate, we would just be able to process 10k data units per hour. We set out to change this to 100k per hour, with the provision of increasing this rate to 1 million per hour with simple and minimal horizontal scaling.

The Upgrade

We figured the only way to do this was if:

1. the downloading, compressing and uploading to S3 could all happen somewhere outside of our servers,

2. asynchronously.

3. And something that could scale infinitely, without us having to manage the infrastructure.

The answer for us was AWS Lambda which has all these features, plus is dirt cheap.

We made a Lambda function that will take an URL and put the compressed version of it in S3 at a predefined location. But since we got multiple images per Data Unit, and we didn’t want to make HTTP calls in a loop (because they are costly as well), we created one more orchestrator Lambda function that took a list of URLs and for each URL hit the original Lambda function. And now each Data Unit for us is just one triggering of a lambda function. We hit rates of 30–40 Data Units per second, which is straight away a 10x growth.

And this is essentially with half the number of workers because we are using the other half workers to check if the compression happened and if the URL was not broken. We straight away hit our goal. And since we get more throughput from individual workers, adding even a single one will increase the rate by a lot. But there is another way this can scale. Right now each worker just sends images of one Data Unit at one time. Since all we are doing is trigger the lambda function with a list of URLs, we can club URLs from multiple Data Units and send them all at once. That directly means performance gain in the order of n, where n is the number of Data Units you club. This lets us scale it further to an insane rate!

Giveaways

It was straightforward to set up PIL for AWS Lambda, you just have to follow the provided docs really well. Fire up an EC2, with Amazon’s Linux image, and install Pillow, and then package this in a zip and download it. See?

No worries, we took the pain so you don’t have to. You can find the site packages zip along with the image compression and the orchestrator Lambda functions here in our repository. Break a leg!

(Originally posted on my Medium account)

For the lazy, here is a tl;dr:

1. I missed the team, and the hustle.

2. Platform team was facing acute bandwidth crunch.

3. It is the team that makes or breaks a company. And I am responsible for whatever happens to Squadrun.

4. <ambition>Bring me a problem, I will solve it better than anyone else<ambition\>. Squad has only ever helped me move towards this, can’t wait to learn more by doing.

For others, you are too free, but here it goes:

I left SquadRun on 2nd July 2017. On 10th August 2017, I am going to be back.

In reality though, I never actually left. I was in constant touch with the team here, talking to at least 1-2 different people daily (no exaggeration). I got constant updates on the good and the bad of whatever was happening in the team. I was contributing a little by writing a few articles here and there talking about our engineering team and culture.

It is because of these conversations I was having with the team that I started realising how much was I missing the hustle. On top of that, we already had a shortage of engineers in the team, and me leaving only made it worse. The team was, thus, facing an acute bandwidth crunch. Sometimes VG had to get down to fix things himself, which I believe shouldn’t be needed now. And I felt responsible. I started toying with the idea of getting back for a small period just till the time we hire more people and the bandwidth crunch gets sorted.

Taking a step back, during my notice period, I read “The Shoe Dog”. It made me realise that it is the team that makes a company great. This would not hold true a lot for an MNC or a startup that has grown to a large people scale (Zomato, Urbanclap etc), where you will just be a cog in the wheel (the cog size might differ based on a number of parameters). But at the stage at which we are right now, each individual contributor has a high impact. And we, as early employees, can not just ignore the responsibility of carrying SquadRun to great heights. I was already feeling that maybe my decision to leave wasn’t right, but wheels were already in motion and I wanted to take this plunge anyway. This also came back to me while I was toying with the idea of temporarily joining.

Also, I have come to realise that we overthink growth. We are thinking about it every week (at least I did), did I learn something new this week, how much did I learn, is my growth curve as steep as it was before? Sometimes we just need to also not take the growth thing so seriously, and lie back and enjoy the work we are doing. Don’t stress about it, you are getting better with every passing day.

1. growth isn’t exactly quantifiable, it is not a score you can just compare every week. Compare with where you were 6 months back, maybe?

2. you grow as the company grows, the challenges that are thrown at you become different, and with each problem solved, you learn something.

3. just learning a new framework or a language is not growth, it is also being able to solve problems more effectively, or being able to manage stress better, or multitask with more efficiency, or being able to be a better mentor. Growth comes in many different forms.

As Vikas told me when I was leaving, Squad has a lot of challenges for me still. But challenges or not, I am not letting anything happen to Squad. I have built a huge part of it, and I want to see, and not just see, but take, it to where it goes. Slowly, the idea of joining temporarily changed to an intense desire to be here long term. And here I am.

Now when I join back, I join back with the same vigour and passion I joined as an intern 2 years ago.

Let’s kick asses together, Game On!

FAQs

Q. Did Squad ask you to be back, or did you approach them? I did, completely my decision. VG supported the decision though.

Q. Kahi aur job nahi mila? (Didn’t find a job or what?) :P While I am fine with these jokes, I thought I will just clarify once. No, I didn’t apply to any place. I was still figuring out stuff and enjoying my break.

Q. Temporarily joined or here for good? Here for good. Until VG kicks me out.

When I joined SquadRun back in 2015, I was no more different than I am right now. And VG couldn’t be any more different than he is right now. And both of us couldn’t be any more different than each other than two people can ever be. This might not make sense, but it doesn’t have to. Feel it. Despite having these starkly different personality types, we clicked. He was the CTO of SquadRun, ever so serious. And I would pick him up (literally) and dance around with him (something that not a lot of people can boast of doing). But this is not about how well we clicked. This is about how Vikas goes on about his life solving hard problems every day, like a boss (please notice pun).

(L-R) Vikas, I and Shubham (another guy I love)

Vikas is empathetic and in all his dealings with people, he will always be fair. I have had my fair share of sad moments in life. And when they happened, VG would simply ask me to take a break if I was unable to focus. Once I made a sudden plan for a trip (before you judge me, Coldplay was in the country and I got free passes), VG happened to take one day to tell me it was fine for me to go away for 3-4 days. He paid the amount by which the flight price increased from the previous day, even though he really didn’t have to.

He is the type of guy who will let you take the love for something he helped you build that worked well, and _share_ the heat for things that didn’t work well (I know, I know that’s what great leaders do. But that’s the whole point! He is great!). Those times you did a mistake, sometimes major ones, you could see that he is not happy. And you would feel bad about it and despair that how are we going to fix it. And then the moment passes, and VG smiles at you and tells you it is okay, and that he thinks you can fix it. Your despair is gone, you scramble away to fix things, promising yourself you will never repeat the same mistake again. The understanding that mistakes happen, what is more important is that you learn from it and strive to not do it again, made sure we never had a terrible day at work.

As Squad grew, he magically started to handle all the new problems that come with people scale and in product development, in addition to what he was doing already. In true CTO style, he can break open-ended, large problems into smaller ones. He was involved in building hiring processes, managing product development, learning how to run a startup, learning how to make sure your engineers are happy and growing. He always knew the right thing to do. All this, and you will never see him go out and about beating his chest for something he did that solved a huge pain (I can’t say that about myself!).

If VG was in cricket, Rahul Dravid would have just been called the fence or something. Because VG is the wall. ~ Ketan Bhatt, giver of bad analogies

He doesn’t break. He is been at it for ~3 years. Constantly taking on new challenges, preparing/learning for them, anticipating some of them before they even come, constantly growing and constantly inspiring. 24*7. I haven’t met anyone who is so open to being a better version of himself every day. He is constantly experimenting. With routines, processes, books, food. What not!

And now, after I have left SquadRun, leaving the team with a bandwidth crunch, VG has again risen up to the occasion and is handling all the extra pressure with a smile. Even then, when I asked him to take a look at one of the companies I was interviewing with, he took out time to read up about the company and share what he felt. And that’s when I realised again, he is Gold.

I will just stop here because I am feeling better now. You praise us a lot for a small bug we could fix quickly, or a design we could think in and out of. But you need to be praised properly too. You have changed a lot of lives. Keep at it.

I know this reads like a fan-mail. Aha, good observation skills lovely reader. It is! I am a fan Vikas ❤️ and I know at least 10 more people who feel the same.

Sometimes when I tell my friends what we have built in a short span of time, I see there eyes going wide in disbelief. One candidate I was talking to about SquadRun thought we had 10–12 developers in the team, judging by what all we had built.

But we are not a team of 10–12. At the moment, we are just 5, really committed, developers.

And we get asked this often, how do we do it? Let me try and answer that. Two axes in the development plane have to be:

Quality

How and how well is the problem being solved.

We generally keep these things in mind while designing solutions for problems:

1. The problem should get solved, of course, taking the right tradeoffs

2. All practical failure cases are handled

3. Most importantly, the design “FEELS” right

Once the design is done and reviewed, we get down to implementing it. This means less time taken for the actual code to be written since the process is carefully thought of, and less time spent on fixing bugs.

Speed

Always be shipping.

You will never find a Squad member whiling away his time. The kind of people we hire, they get frustrated with themselves if they are not achieving the best that they can.

Some of the things that we have done to maintain speed are:

1. Our deployment method, although still far from being perfect, enables devs to deploy fast! 2 minutes average deployment time to all servers, with the longer ones taking around 6 minutes.

2. Encouraged to ship as soon as a story gets done and not wait for the deployments to get larger.

3. Pull requests are cleared at higher priority so code can be shipped ASAP.

4. Do the right thing. If a particular feature doesn’t need optimisations, don’t spend time on it. Premature optimisations are a no-no.

Speed and Quality are like position and momentum in Heisenberg’s Uncertainty Principle. You can’t achieve best levels of both at the same time (I know that is not exactly what the principle says, scientists). But SquadRun has managed to strike a healthy balance between the two.

A lot of this has been possible by giving more emphasis on why we need to build something, what to build and how to build (in that order), that is:

1. Why do we even need to solve this problem? Questioning with ‘why’ always helps ensure we are aligned with higher level goals. Best code is no code.

2. Out of N problems, what needs solving before others. (What has the most ‘leverage’).

3. What is the best way to solve the problem?

It would be wrong to say that we don’t make mistakes. But we do our best to not make the same one twice. We learn from them and are constantly experimenting with our development process, and division of work among the team members. We share feedback openly, call out what didn’t work (no matter whose suggestion it was), and do more of what worked.

But at the end, more than the process, a team of engineers who are passionate about growing and solving problems are the key to doing a lot with little.

We still have a long way to go and much to learn, if you’d like to join us in building a product which we believe is going to change the world, reach out to us!

(Originally posted on my Medium account)