Edward Huang

My Journey Through Burnout

Sat, 24 Sep 2022 17:50:00 +0000

Burnout doesn’t happen all at once. It happens slowly.

It is like an Obscurus - a parasite that forms in Book Fantastic Beast when a wizard or witch suppresses their magical ability. Just like Obscurus, it will eventually take over their host bodies. Burnout will take over one’s soul.

As a software engineer, Burnout happens when one is not recognized or satisfied after finishing the project. But it is also the anxiety and lack of self-confidence each time when trying to execute a project.

To prevent Burnout, a break from work is not good enough - the human mind will switch back to the old feelings after a long break and feel exhausted. Everyone has different ways of fighting Burnout. However, if you experience Burnout, I encourage you to take time off and consider what makes you burn out and which factors you control.

From A Personal Experience

I experienced Burnout when I worked for a startup.

I was initially excited to join a startup and make an impact. I have always thought about those crazy stories in the startup world that friends often discussed during family dinners. Working in a big company has become more and more political and repetitive. I felt like I needed to go for a smaller company so that I would be able to make a bigger impact and continue to advocate the benefit of functional programming through my colleagues.

I get to push code to production and enable the big feature on the first day of work. I was ecstatic.

I wanted to experience the culture of a startup, and I got to feel what it means to “put off fire all the time.” However, one important aspect of joining a startup is that I didn’t think about its mission.

I thought that if a startup is in a growth stage, you don’t have to care too much about the mission, as it found the product market fit. I never thought about how important the mission is within a startup.

The mission is what creates the culture. Believing in the mission is what motivates all of the employees to work long hours and on weekends. The mission is what prevents the company from shattering. If you don’t fully onboard with the mission, every day will be an uphill battle - the easier you will get Burnout.

It all Started with High Expectation

I was shipping great code, and my team was very impressed with the speed and the impact that I produced.

I was determined to be promoted to the next level. If I keep up with this speed of execution, they will promote me in a short amount of time. However, the feature that our team is responsible for wasn’t making enough impact on the overall organization, and we decided to pivot and disband the team.

It was normal to always pivot as a company. I was disappointed with the result of our hypothesis but am excited to move on to the other team.

The Moment when Feeling Turn South

I have a conflict with an influential staff engineer in the team.

Every PR was rejected because the approach wasn’t the way he wanted. Although he mentioned that it was not the correct approach to the problem, he gave vague suggestions. He expects me to guess what he thinks.

Instead of accepting various approaches to a problem, he has a full picture of what the approach should be. I spent a large portion of my time trying to guess what he thinks.

It delays the time to move the project forward.

Every standup meetings become like an interrogation session. I am trying to justify why I couldn’t finish the task at every meeting.

The X stages of Burnout

The X stages of Burnout started with wanting to prove my worth by working more hours. That leads to health, relationship, and financial deterioration in every aspect of my life. Then, you can’t handle it anymore by not caring about my work. Lastly, you tap out by losing purpose in my life.

I started trying to prove to the team that I could do this. However, the team didn’t recognize my effort because it didn’t impact any business metrics. The team denied my idea of introducing new tools to increase developer productivity because it doesn’t impact the business metrics.

Every action needs to be correlated to business metrics. Every action needs to go through that staff engineer’s approval.

Eventually, I felt exhausted. The setback made me question every action I took in shipping products - it destroyed my confidence.

I felt like I was a cog in a factory to help increase some important metrics of the company so they could raise more funding in the future.

I stopped caring about my health and my surroundings. I started to check my slack channel and PR when I woke up to know if I had hit the jackpot in predicting what he was thinking. I had high hope at the beginning of the day that I would be able to push my project forward. Yet, luck has never been in my favor.

I received comments on every PR from the staff engineer that the design approach is incorrect. I direct message him asking for more clarification, but he didn’t respond for more than an hour.

“There are only 24 hours in a day, and I couldn’t afford to keep waiting for him because I got some updates that I need to tell the team on standup tomorrow!” I thought.

While waiting for his response, I try to make some changes, hoping this assumption is correct. I pushed the changes and tried to ping him one more time, hoping this change would be it.

A couple of hours pass, and he eventually responds to my message - but he responds to them in the group chat.

I was astonished.

“This guy didn’t respond to my private message but tag me on a public group chat to show everyone how incapable I am in front of all the team members.”

I had another back-and-forth discussion in the group chat with him until the end of the day - realizing that I am still at the same progress as yesterday.

Anxiously, I wrote down my justification notes on reasons for tomorrow’s status update.

I went to sleep feeling guilty and ashamed.

I felt guilty because I couldn’t justify another reason why my PR is still under review. I felt ashamed because the team saw me as lacking the competency to unblock and move things forward.

I stopped caring about eating lunch or dinner. I spent most of my waking hours thinking/trying to move about my project. Even though I felt physically exhausted at the end of the day, I had difficulty sleeping because my mind kept feeling anxious and racing.

I stopped writing about technology blogs because my mind was exhausted.

I briefly discuss with my manager about my concern. However, he thinks that I lack confidence in writing code and that he told me that I need coaching.

Because of not being able to care hurts the performance review.

Finally, on my performance review, my manager discussed this issue where I have a problem moving the project forward, and he thinks this aspect is an orange flag. I will be in an improvement plan if I cannot improve on this aspect.

Enough is Enough

One influential colleague that is hard to work with can influence the performance and motivation of the team.

I thought before quitting and taking a break. I want to take another chance by exploring another opportunity within another team. Perhaps the root cause is not the company’s culture but the team’s culture. I searched if there were any opportunities elsewhere within the company. This feeling may go away if I don’t directly work with the staff engineer. There is an internal job posting available within the company, and I contacted the Hiring manager. The process went smoothly during the transition, and I mentioned to my manager that I would like to explore transferring to the other team.

A few weeks later, a new colleague discussed with me that he encountered the same issue with the same staff engineer. I thought it wasn’t me who encountered conflicts with that engineer.

I reciprocated my feelings, giving him some advice on handling such a situation. I had closure on the doubt myself for these past couple of months.

Conclusion

In my experience, Burnout started when you feel like you want to prove something because you are not valued. Over time, your mind gets exhausted with constant misalignment of output and expectation.

You started having difficulty caring for and focusing on your daily task. You quite quit, because you are mentally tapped out. If you don’t take further action, your mind will get more and more exhausted. Eventually, Burnout takes over your soul and destroys your physical and mental health.

At this moment, it may take months or even years to recover.

Burnout can be hard to quantify and talk about because it manifests so differently for everyone. For some folks, it stems from the organizational culture or workload.

For others, it can be the nature or intensity of the work or team dynamics.

Burnout appears in different shapes and sizes, and it can even be mitigated if it’s caught early enough and addressed effectively. But in more severe scenarios, the best (and maybe only) solution is to leave the workplace that was contributing to and causing your Burnout.

5 Anti Pattern for Writing Code in a Functional Programming Language

Mon, 18 Apr 2022 11:06:00 +0000

Functional programming languages have gained so much traction in these past few years.

Many people can see the benefit of writing code that contains features such as functions as a first-class citizen. They embrace immutability in a concurrent environment, running heavy computed tasks without worrying about some concurrency issue, and love to write generic code to be as DRY as possible.

I saw this as a good sign that functional programming language is becoming mainstream again. However, one of the hard parts of writing code in a functional programming language is its design pattern and anti-pattern, which are different from a regular programming language.

I often see engineers writing in a large codebase that I classified them anti-pattern. I had also committed to these anti-patterns when I initially wrote a production-ready application in a functional programming language. Since then, I have read many books about functional programming design patterns and books that have helped me create a more maintainable code.

Overly nested anonymous callback function.

An anonymous function can be good for code-reusability. However, too much anonymous function can hurt the eyes of those engineers who want to extend the functionality. Although DRY is the way to go, sometimes duplication is better than the wrong abstraction.

I have touched codebase where engineers will write a heavily concise and abstract method. The code is something like this:

Can you tell me what the definition of buildRunner is?

Then, this buildRunner is used in all action operations like authorize, capture, and void in the payment processor. I look at it for two days to finally understand what it is trying to do.

This creates an abstract to be as DRY as possible on all the functions you write. However, having a nested anonymous callback can be tough for regular engineers to build a new feature or maintain. Most of the engineers will need a couple of days to understand what buildRunner is trying to do.

The good thing about functional programming is that you can look at the function signature and know what it is trying to do right away. However, this function doesn’t explain much about what it does. It further confuses engineers trying to make a change in the codebase.

Therefore, a good rule of thumb is not to use an anonymous function is possible. Instead, use a higher-order function.

If you want to use an anonymous function, please put a type on the top to make it easier to read. http4s does this internally by wrapping its type-ins Kleisli. Kleisli in itself is an anonymous function that is A => F[B]. However, wrapping the anonymous function with a type definition on the top helps readability in your codebase.

Pattern Matching to the Extreme

The first thing that we learn about the benefit of writing code in functional programming is the pattern-matching feature - it eliminates the ugly if-else statement that we often use in the common programming language.

Pattern matching is nice if you only have a shortcode based. Things become more like a callback hell when you have more than two layers of pattern matching.

Often, engineers who are new to writing code with functional programming language don’t know how many built-in higher-order functions the language provides. Therefore, they default their function implementation through pattern matching and recursive function.

Having nested case expression and recursive implementation in your function implementation causes hard to read and understand code. It takes a long time in PR comments, and harder to find bugs if there is one in the implementation.

One solution to writing nested pattern-matching case statements is to only care about the success case and leave the error scenario outside the function implementation. Moreover, use the built-in higher-order function provided by the library or language, map and flatMap if possible. It gives better ergonomics to your codebase, and you can identify right away where that error is being handled.

The beauty of expressing your types in the function definition is that your function implementation no longer needs to handle all error case scenarios - the type system can propagate those to the caller - enforcing the function up the stack to handle those errors.

Using Monad Transformer on The Interface

Using Monad Transformer when encountering the nested effect is very useful. In the above scenario, a Monad Transformer is another solution to solve the heavily nested effect - it helps make your API composable. However, we should not expose Monad Transformer onto the interface because it makes our API tight to a specific Monad Transformer.

Let’s make a more concrete example. The below interface can be Future[Either[Throwable, String]] instead of EitherT[Future, Throwable, String].

All the function that wants to use someFunction as an API will also need to use EitherT.

What if it is a series of functions, and we see some function returns an OptionT?

We will need to call value a couple of times to return to our effect Future, unnecessary wrapping.

Alternatively, we should make someFunction returns a Future[Either[Throwable, String]] and let the effect detects what constraint you will need in your program.

In conclusion, having the purest form of effect is better than a monad transformer, as it doesn’t lock services that use the API to use a monad transformer.

Returning a Boolean Value On an API

Many APIs may return a single boolean value to indicate one logic or the other. The classic example, taken from Practical Fp in Scala, is the filter function.

What does’ filter’ actually do if you look at the function definition?

If the Predicate evaluates to true, it will discard the elements in the list. On the other hand, it can also mean if the Predicate evaluates to true, it will keep the elements in the list.

That is ambiguous.

There is also filterNot in Scala, which has the same function definition but has a different name. I often saw many bugs occur within these two functions because engineers oversee the difference between the two.

We can improve this by wrapping an ADT (Algebraic Data Type) around the Predicate with meaningful values.

This ADT helps us create a more specific function signature like this:

Whoever uses this function will understand if they want to either keep the element or discard them.

To solve this problem on the filter class, you can always create an extension method, filterBy from the scala List trait.

The key to avoiding confusion in boolean values is to use ADT to give meaningful values and extend those functions with the ADT. Nevertheless, this causes more boilerplate, but you get less confusion and bugs around constructing your application.

Wrapping all boolean returned API with an ADT may be overkill. Hence, you can wrap the boolean returned API with an ADT in the critical component and be flexible in the rest of your application. It is a matter of agreement with your team members.

Using Generic Data Structure in Your Trait

This statement may be controversial as, in regular software engineering practice, an interface should be as generic as possible to be extendable. It sounds great in theory but not in practice.

One example is Seq - a generic representation defined in the Scala standard library. It is very generic that List, Vector, and Stream all extend from Seq. This is a problem because each of these data structures behaves differently.

For example, we have a trait that returns a Future[Seq[String]]:

Some engineers will call the function fetchAll and convert the Seq into a List with the toList function.

How do you know that it is safe to call toList? The interpreter may define the Seq as a Stream, and in this case, it will have a different semantic, and it may throw an exception on the caller’s side.

Therefore, to decrease the number of surprises on the caller side, it is best to define a more specific type, such as List, Vector, Stream, depending on the application’s goal and performance.

Conclusion

The problem with these anti-patterns is that it is not an anti-pattern in a generic programming language.

For instance, we are taught that writing abstraction is good, and it keeps your codebase DRY. However, the overly nested anonymous callback function can be hard to read. The best way to solve this problem is to duplicate the code to increase readability. API that returns a boolean value may not be a problem, and it exists in a lot of the API designs and applications. Nevertheless, implementing an API that returns a boolean doesn’t give any clarity in terms of what those boolean mean. Furthermore, human minds often oversee small details in the documentation that can cause bugs in the implementation.

Pattern matching is a powerful feature in the functional programming language, but it is overly generic. If you can find a better higher-order function to implement the function, you should use that instead.

An overly generic data structure can increase ambiguity in the use of the API. Therefore, it is best to create a more specific type and make the function declaration as clear as possible to the call side.

I hope you avoid this anti-pattern when writing code in a functional programming language. Do you think these are anti-patterns? What is another anti-pattern you can think of when writing code in a functional programming language? Please comment them down below so I can also learn from them!

How to Avoid Read Inconsistency during a Transaction

Sun, 03 Apr 2022 14:05:00 +0000

Imagine when you are trying to transfer 100 dollars from account A to account B, and both accounts are in the same bank. After you initiate the transfer, you refresh your screen. However, when you refresh your screen, your total balance dip - that 100 dollars seem to vanish out of thin air. You see that account A is 100 dollars less. However, account B is not 100 dollars more. Then, you refresh the screen a couple of times to see that account B has gained 100 dollars.

This problem that you experience during a transaction is called read skew. An anomaly happens when you read the transaction at an unlucky time - during and after a written transaction.

It may be a bad user experience, but this will not cause any problem if you refresh the page after the successful transfer transaction.

However, read skew becomes a problem when doing a database backup or analytics query.

In database backup, we need to make a copy of the database. There may be a written request during the backup process coming in. If the read skew inconsistency happens, the backup result may be inconsistent. Some data are in the old version, and other data are in the new version. This inconsistent problem may become permanent with such an operation.

We need to scan over a large database in an analytics query and periodically check for data corruption. Read skew can cause the search and check to be inconsistent - often may have inconsistent results and fire off false alerts about data corruption.

Solving Read Skew

The problem with reading skew is that a read transaction is read one time in the old database version and another in the new database version.

The important point here is that the read transaction needs to be consistent - it doesn’t need to be the up-to-date version. It needs to be consistent from the beginning of the transaction until the end, so we need to keep the data version the same.

For instance, if Bob is running the read transaction at the data version 1, throughout that transaction, Bob should only be able to read the database data version 1. If in the middle of the transaction process, a new write transaction occurs, which will update the data in the database. Bob will not see that new version in his transaction.

Hence, we can make the transaction read from the consistent snapshot of the database - the transaction will see from all the data that other transactions committed in the database at the start of the transaction.

This feature is called snapshot isolation, and it is offered in a lot of relational databases, such as PostgreSQL and MySQL.

Implementing Snapshot Isolation

We need to keep various snapshot versions in the database to implement snapshot isolation. Each time the beginning of the transaction, the database will give the latest committed snapshot version to the transaction. Then, the database will keep track of each transaction with its corresponding snapshot version to keep the read consistent.

Each transaction has a transactionId, and the transactionId is retrieved from the database. Thus, the transactionId is always increasing. The database keeps track of each transactionId written to the database using createdAt and deletedAt values. The database created a tag on that operation with the transactionId from the transaction after the transaction is committed. The database further makes a snapshot of the new transaction and tags that snapshot with the latest transactionId. When a new transaction reads from the database, the database retrieves the latest committed transaction before the transaction with the following several rules:

Any transactionId that is currently not yet committed to the database will not be shown even if the subsequent transaction is committed.
Any aborted transaction will also not be shown.
The database will not show any transaction with the transactionId later (bigger) than the current transactionId.
The database will show any other transaction to other incoming transactions reading the database.

Let’s take a look at what happens in Bob’s scenario:

When Bob initiates its transfer transaction, it kicks off a background process to transfer 100 dollars from account A to account B. This transaction will first call the database or aid service to get an incremental transactionId, and initiate the transaction - let’s say the transaction is 1234.
The subsequent read transaction will need to do the same by getting an incremental transactionId and calling a read request to the database - let’s say the transactionId is 1345.
While the transfer is not yet finished, the database will not show Bob the data applied by transactionId 1234 (rule number 1).
If another write transaction was initiated after transactionId 1345, because that transaction has a bigger transactionId, the database will not show that transaction to transactionId 1345 (rule number 3).

During delete, instead of deleting the value in the field right away, the database will mark a tombstone on that field. One reason for not deleting the value right away is that those earlier transactions may still use that value. Thus, we can leverage garbage collection to check and delete the value asynchronously once all transactions use the value committed to their transactions.

Taking Snapshot Isolation to Distributed Environment

So far, we have explored how to solve read skew in a single node environment - we assume that databases are not distributed across multiple clusters.

How can we expand snapshot isolation in a distributed environment?

It will be hard to get a global, ever-increasing transactionId in a distributed environment. For one reason, each machine that potentially resides in a different database may have its UUID counter, and we need to have some coordination to ensure causality. If transaction B reads the value from transaction A, we want to ensure that transaction B will have a bigger transactionId than transaction A. How do we deal with a consistent snapshot in a replicated database?

Can we use the clock or time of the day as a transactionId to write to the database? Time-of-the-day clocks are unreliable, as NTP synchronization is based on unreliable networks. Therefore, some machines may have clock skew, arbitrarily moving backward in time. The time of one node may also be different from the time of the other node. However, if we can make the clock accurate enough, it can serve as a transactionId - the time later on the clock means that the events are produced later. How do we ensure that the clock is accurate enough for transactionId?

When retrieving the time-of-day values in each machine, we want it to return a confidence interval, [Tbegin, Tlast] instead of getting a single value. The confidence interval indicates that the clock has a standard deviation of plus or minus a range of Begin and Tlast. If two transaction, transactionX, transactionY coming in, [TbeginX, TlastX], [TbeginY, TlastY], and TlastX < TbeginY. We can ensure that transactionX is earlier than tranasctionY. However, if the value overlaps, that is where we cannot determine the ordering. This approach is used by Google Spanner to implement its snapshot isolation. Spanner will deliberately wait until it is over the previous transaction’s confidence interval not to overlap to commit the current transaction. Thus, they will need to keep the confidence time interval of each clock on the machine as small as possible to avoid latency. Google deploys an atomic clock or GPS server on each data center to allow clocks to be synchronized.

To ensure that the snapshot is the latest on each database replica, we can use quorum strategy to get all the latest transaction snapshot from all of its database clusters. Another strategy we can use is to ensure that the transaction always routes to the same database instance to have consistent snapshot results.

Wrapping Up

Read skew happens when you didn’t see a consistent result from reading the database data because another write transaction occurred in the background. A consistent snapshot is a solution to read skew in a single-node database.

A consistent snapshot is an isolation level that guarantees that each transaction will read from the database’s consistent snapshot, usually the most recent snapshot before the current initiated transaction.

Implementing snapshot isolation requires a monotonous increasing counter, transactionId, to determine which version to return to the transaction call. However, this can be tough when dealing with a distributed environment because coordination is required to generate causality. One solution to solve this is by using a time-of-day clock that returns a confidence interval to create an ever-increasing transactionId.

Lastly, to ensure each transaction gets a consistent snapshot, we can use the quorum strategy to always return the most recent snapshot from the current transaction returned by the majority of the node or to have session affinity on transaction calls and database instances.

How would you ensure read consistency in a distributed system? How would you solve the creating a global transactionId? Comment them down below!

Will the Newest Generation of Programmers End Up Taking Older Software Engineers Jobs since they are Younger and know the Latest Technology

Mon, 21 Mar 2022 21:49:00 +0000

“It’s great to be a software engineer nowadays.” One of my family members mentioned that at our family holiday party. I grinned and responded, “Yea, it is. However, there is a catch.” I explained, “Although the job market has been hot these days, we must keep up with the current trend. If we don’t stop learning in our field, we will soon be replaced by the younger ones.”

That was me preaching about the disadvantages of this career field right after I graduated from college. I was naive and was lost by the abundance of new frameworks and articles about the newest and hottest framework to learn next. I also heard from other more experienced colleagues back then that they needed to learn the newest technology and constantly study college-level algorithms throughout their careers to be competitive and keep up with this on-demand market.

“If you are not learning the newest language, you are afraid of getting unmarketable to the new company.” “If you didn’t get to use the newest framework in your company, you think that you may not be able to easily jump ship to the other company.”

As more and more younger generations, Bootcamp graduates are getting into this lucrative career field, you, as an experienced developer, should become wary that the new generation will have the newest and shiniest tools in their belt - with a much lower price to offer.

After working for several years, I can say that the newer generation will not take over the experience software engineer job.

Why?

The answer is that people who will have more experience have two things that the new generation doesn’t have - experience and domain expertise.

Experience

There is one thing that people will do ten plus years that new grad doesn’t: experience.

Experience can only be accomplished with time.

A new grad may know the new frontend framework, the new serverless technology that an experienced engineer may not know. However, in executing and scaling a real-world application, one requires experience.

You may think, why is experience required for writing programs? Overall, there are a lot of knowledge and tutorials on how to create a scalable system. As long as you follow either one of the famous tutorials out there, someone who knows how to type and code can easily write something that works.

Yes, anyone can code a chat application. However, leading a project and successfully creating a chat application by communicating through various team members requires experience.

Experience is what separates an engineer and a programmer.

During the project initiation, one needs to communicate why we cannot finish the project in one week to someone who is not technical. We need to understand if there are enough resources to create a scalable chat application successfully. Furthermore, we also need to account for other teams’ project roadmap and be able to sell them why this project is paramount for the company business OKR.

Experience also comes with soft skills. For instance, handling difficult situations with stakeholders when there is a greater push on the deadlines. Staff engineers will know who to talk to the right person for the right task, whereas a regular engineer may need a couple of trials and errors to reach the right person. Staff engineers will know when we should compromise between speed and scale. A staff engineer knows that the duplicate code is cheaper than the wrong abstraction.

They also know good enough methods and bureaucracy to make their idea heard.

For instance, I was trying to gather information about the system. There is no documentation. I pinged engineers who owned the system, and they were willing to explain to me and help me ask other engineers in their team for help. Nevertheless, I didn’t want to bother them and ping the other engineers myself. A seasoned engineer in my team advised me to leverage others if possible instead of bothering more people. Asking the wrong person may make a bad impression that we don’t know what we are doing, especially someone with a more senior title.

Domain Expertise

Domain expertise is an awareness of the target system’s environment. If you have been working on the marketing platform for a long time, you will know about the architecture, but you will also learn about the domain of marketing.

Gaining domain knowledge requires time and experience.

More experience leads to becoming domain experts.

Why is domain knowledge important?

Any engineer should have basic software engineering skills such as manual, automation tests, and design patterns. However, engineers that have a common sense of the domain can find most of the obvious bugs in the software. Thus, they can release products the right way that won’t cause any catastrophe.

Unfortunately, domain knowledge is not something that you can learn in college. You will need to be involved, create a real-world application, be on-call, and encounter various problems to gain that expertise.

For instance, scaling a company payment system may be tough if hiring an engineer who doesn’t know how payment works. An engineer who experiences outages, such as a third-party payment provider is down, has encountered a lot of the edge cases, such that they know how to test the system the right way.

Someone with less experience may know more technology and can quickly code up something. However, they may also break a lot of the things along the way.

Domain experts are more in demand than technical experts. This is because the advantage of domain expertise can decrease the amount of trial and error made in the system.

An engineer with great domain knowledge knows how to respond to an outage- they better understand different issues or scenarios. They can help deliver superior and faster applications, which helps improve the application’s marketability.

In other words, it requires years of experience to gain that knowledge.

Conclusion

Although engineers with ten plus years of experience may not be able to solve string manipulation Leetcode style questions faster than a new grad, they can lead the team on executing the project in the right way.

Moreover, domain knowledge is one of the reasons why companies are willing to pay big bucks for a software engineer with multiple years of experience. Business wants to have engineers who have done it before to produce the same or even better success.

For experienced engineers, don’t fear getting replaced. As long as you are eager to learn during your job, you will be saved. For engineers that just started on their journey, don’t get distracted with the newest technology, focus on the fundamentals, and be a sponge by constantly learning about the domain knowledge from your field.

Why do you think experienced engineers are in-demand right now?

Comment them down below!

How to Detect a Dead Node in a Distributed System

Thu, 17 Mar 2022 00:23:00 +0000

If AWS only provides a bare-metal server, and you will need to pay per request or per month on their node detection service. That will be harsh for them not to provide such service, but it will be very expensive for companies to have to pay for such feature and the instance. Detecting if a Node is dead sounds like a very simple process. However, it is actually a very hard process to do. We often granted third-party cloud services having this simple feature to help us monitor our node’s health.

To tolerate faults, we need to detect them. However, in this article, you will see how hard it is to detect node failure. We will also discuss a high-level architecture design on detecting a node failure detection with phi accrual.

Understanding How Latency happens

The slowness in the network is like traffic congestions in Disneyland. Imagine when you are waiting in line to ride the magic mountain. At the front of the queue, you see that the waiting time is 10 minutes. You may think to yourself, 10 minutes is not a long time. Hence, you wait in line. A few minutes pass by. You started to see that you are almost in the front of the queue - only realized that there is a longer section queue in front of you that you need to wait an additional 30 seconds. Latency work similar way.

When packets are sent from your machine to the destination machine, they travel through the network switch, and it will queue them up and feed them into the destination network link one by one.

Once it goes to the network link in the destination machine, if all CPU cores are currently busy, the incoming request from the network will be queued up by the operating system until the application is ready to handle it.

TCP performs flow control (backpressure) that limits the number of nodes sent across the network to alleviate the node it contains in the network link. Therefore, it has another layer of the queue for the packets in the network switch layer.

Why is it hard to detect node failures

Imagine if you are running a single program. Even though the program didn’t crash, it is slow and buggy. There is no stack trace in the program that mentions which part of the function or method is not working. This program will be much harder to detect failures than the previous fully failure scenario. This sort of failure is what is called partial failure.

If you are running a single program, if one part of the function is not working, the entire program will usually crash. By then, it showed up a stack trace that you can inspect further to understand why the system crashed.

Partial failures are much harder to detect because they are not either work or it doesn’t. There is numerous possibility why the program is “having a bad day.”

Since distributed systems don’t have a shared state, partial failure happens all the time.

If you didn’t get any response, that doesn’t mean that the node is dead. These are the possibilities of failures:

The message got sent to the network, but it was lost didn’t receive on the other side.

The message may be waiting in a queue and will be delivered later
The remote Node may have failed.
The remote Node may temporarily be stopped responding because of garbage collection.
The remote Node may have processed the request, but the response is lost in the network
The remote Node may have a process and responded, but the response has been delayed and will be delivered later.

If the network calls didn’t get a response back, it could never know the state of the remote node. However, you should expect no response back most of the time. What shall the load balancer or monitor service do?

Timeout

Usually, load balancers will constantly send health checks to check if the service is in good health. When a remote node is not responding, we can only guess that the packets are lost somewhere in the process.

The next action will be either retry or wait for a certain time until a timeout has elapsed. The retry option may be a little bit dangerous if the operations are not idempotent. Hence, timeout is a better way as doing any more action if you get no response may cause unwanted side-effects, such as double-billing.

If we want to make the timeout approach, how long should the timeout be?

If it is too long, you may leave the client waiting. Thus, having a bad experience on the web.

If you make the timeout too short, you may get a false positive, marking a perfectly healthy node dead. For example, if the node is alive, it has a longer time to process certain actions. Premature declaring the node dead and having other nodes take over may cause the operation to be executed twice.

Furthermore, once the node is declared dead, it needs to delegate all of its tasks to the other nodes, putting more load on the other node, causing cascading failure if the other node already has a lot of loads.

The right timeout period is based on application logic and business use cases.

A service can declare the operation timeout after an x amount of time if the users tolerate that time. For instance, the payment service can set 7 minutes as the timeout period if 7 minutes won’t cause a bad experience to the user. Many teams detect the timeout period experimentally through trial and error. In this scenario, the timeout that we set is usually constant. For instance, within 7 minutes, or 5 minutes…etc.

However, a smarter way to detect timeout is to not treat timeout as a constant value but consist of a distributed variance. If we measure the distribution of network round-trip times over an extended period and over many machines, we can determine the expected variability of delays.

We can gather all the data of the average response time and some variability (jitter) factor. The monitoring system can automatically adjust timeouts according to the observed response time distribution. This method of the failure detection algorithm is done with a Phi Accrual failure detector, which is used by Akka and Cassandra.

Phi Accrual failure detector using sampling fixed window size for each heartbeat to estimate the distribution of a signal. Each time a new it calls the heartbeat to the remote node, it will write the response time to the fixed window. The algorithm will use this fixed window to get the mean, the variance, and the standard deviation of the response time. There is a formula for detecting phi if you are interested here(link).

Thus, in the next section, we will briefly touch on the high-level design of the Node Failure Detection.

Designing a Node Failure Detection

We will use the A node failure detection component consisting of two things: the interpreter and the monitor.

The interpreter’s job is to interpret the suspicion level of the node. The monitor’s job is to receive the heartbeat of each node and delegate the heartbeat time to the interpreter.

The monitor will constantly do a heartbeat to each remote node. Each time it sends a health check to the remote nodes, it will receive a response within a time. It then sends the response time to the interpreter to detect the suspicion level of the node.

There are two ways of placing the interpreter: centralized vs. distributed.

The centralized way is to place the interpreter and the monitor as its own service, and the system interprets each node and sends the signal to other Nodes for further action. The result will be a boolean value, whether suspicion or not.

The distributed way is placing the interpreter in each application layer - letting the application have the freedom to configure the level of suspicion and the action it should take on each level of suspicion.

The advantage of the centralized way is that it is easier to manage nodes. However, the distributed approach may fine-tune or optimize each node to behave differently based on different suspicion levels.

We can use the Phi Accrual Failure algorithm for the interpreter that we discussed in the previous section. We set a threshold of what phi - if the phi result is higher than the threshold, we declare the remote node dead. If the phi result is lower than the threshold, the remote node is available.

While the monitor sends the request to the remote node, the interpreter starts timing the response time. If the remote node takes longer than the threshold to respond, the interpreter can stop the request and declare the node as suspicious.

Takeaways

We never think about detecting a node failure when designing an application because it is a build-in feature in each cloud provider. However, detecting a node is not an easy operation. One of the reasons is the no-shared state model in distributed systems. Engineers need to design a reliable system in an unreliable network.

Most of the time, companies do trial and error for detecting node failures. However, instead of treating whether a node is dead with a boolean value, we can approach them in variability - the distributed variance of when a node fails with a Phi Accrual failure detector and set up threshold level for timeouts.

Lastly, the high-level abstraction design for a node detection failure can consist of the monitoring component and the interpreter. The monitoring will constantly send a heartbeat to remote nodes and delegate the response time to the interpreter to analyze the suspicion level. If a node reaches a certain threshold of suspicion level, the interpreter returns a boolean value to the service that calls them to indicate additional action needed.

Do you have any other ideas in which you can detect a node failure in a distributed system? Comment them below!

This is How 1:1 should be Utilized To Maximize Career Growth

Wed, 09 Mar 2022 00:22:00 +0000

1:1 is an essential part of career growth, no matter which role you are in the company. However, 1:1 can be intimidating. It is scarce between two people talking about emotions, fears, and hope.

Sadly enough, many engineers wasted their 1:1 - you make a little agenda, status updates, or share some light feedback to one another during 1:1. Although these light conversations during 1:1 lighten your mood, you are not utilizing the full potential of 1:1 to make your career grow.

As someone who grew up in the Chinese Culture, I was often told to show respect to our elders or manager. That means we cannot criticize him, at least not directly and publicly. A senior manager in Chinese culture usually gives a very specific direction to their subordinates. Therefore, we are used to being told what to do and follow the rules. We assumed that if we did what we told, sooner or later, they would discover our potential and promote us sooner or later.

That is not the case in Corporate America. We only get what we negotiate or what we ask. Your manager will not notice what you want unless you ask them to help you. Your manager’s role is to assist and guide you to get what you want in your career. It acts more like a real estate agent instead of your elderly parent. Hence, the only way to express your career goals and receive feedback is 1:1.

In this article, I’d like to present three ways to help utilize your manager’s time and resources to help you grow during your 1:1.

Talk about Awkward Topics

There should not be status updates - the status updates or generic complement are much better done in team meetings. If your manager asks about status updates, you should loop them back to the awkward topic.

You should talk more about how you feel instead of what you say will make your manager feel.

"Very often, people waste most of the 1:1s potential. You might make a little agenda, and then give some updates, some light feedback, and share some complaints. It's helpful and valuable and nice. But, ask yourself: is the conversation hard? Are you a little nervous or unsure how to get out what you're trying to say? Is it awkward? Because if it's not a bit awkward, you're not talking about the real stuff."

- Mark Rabkin

If someone can overhear your conversation and not cringe, you should not be talking about it in 1:1.

People hate talking about awkward topics, and one reason is that awkward topic are uncomfortable.

When you start embracing the fact that 1:1 should be uncomfortable, you realize the benefit of these 1:1.

For instance, you discuss that you are a little bit burned out and start daydreaming about other jobs and why. You don’t really agree with the last feedback that your manager or coworker gave you in the previous performance review. You told your manager that the highlight of this week was the compliment they gave you on the project you do.

Although these topics are really cringy when you share them, it builds vulnerability and brings trust to the relationship between you and your manager.

Commit to saying one awkward thing every 1:1. You can tell your manager that you want to say awkward things about the team in 1:1 to show more vulnerability, and it comes easier and more naturally.

You will notice that you start talking about your mistake, what you feel about the current moment, and observing mistakes other coworkers made. You ended up building trust with your manager and help make the team more cohesive.

Write Down Action Items and Takeaways

Taking notes while doing 1:1 shows that you listen and care.

It also indicates to your manager that instead of taking a passive stance on your career, you are proactive.

It is not enough to have a couple topics in your head as you walk into the 1:1. You have to write them down consistently, in a shared space that you and your manager have access to.

This can also help your manager keep track of your career journey so that there can be a track record of all conversations upon the promotion cycle. It is a reliable way to track progress, a better way to recall achievements, and grow and change over time.

If your manager is not taking notes about the meeting, you should encourage them to take notes - say something like this, “This is important to me. If you want to take a minute to write this down, we can pause for a moment.” It helps them remember what is important for you and can help them do what they can to help you achieve your career goals.

It is a red flag if your manager cancels your 1:1 more than twice.

You can ask them to reschedule your 1:1 instead of canceling it, “I understand that you have something pressing, and these meetings are important to me. When can we reschedule our 1:1 meetings?” Having a doc can also signal to your manager that you want this 1:1.

Have an upcoming topic on the next 1:1 and write them down.

Once you take the notes, you follow up with your action items.

Ask your Manager how They feel

Managers are humans too.

Besides asking how you can get your manager to do a few more things that benefit you, you should also understand how to support them.

Why would you want to know how they are feeling? All in all, this 1:1 should be about you, not your manager.

For one thing, it builds rapport and trust, which can be invaluable when they’re considering who to promote or give responsibility to.

Showing empathy and accommodation to your manager makes them more likely to want to do the same with you.

You see, support and help are a two-way street. Your manager will be thankful and help you on your career journey if you can also make their life easier. In other words, you can help them to help you.

Sometimes, they want to do these things for you, but they are just swamped with all the projects that got thrown at them. They may have the best intention but are overwhelmed because they didn’t get any support and attention that they need. If they are swamped with many projects, you can take the lead on one of the projects to relieve some workload.

I usually will ask my manager on our 1:1 how they are feeling and what I can do to help alleviate their stress. I often ask how they feel about the project I am currently on and any concerns they have with the team.

1:1 is not only about you. It is sharing a conversation and feedback that keep stuff moving forward. It is not only beneficial to you but also to your manager. Therefore, I encourage managers to cherish their 1:1 with their direct report.

Takeaway

1:1 should be more than just a status update, and I encourage you to pursue the awkward topics. Moreover, taking notes or asking your manager to take notes in a shared google doc can help track progress and indicate that you value this precious time. Last but not least, treat your manager as a human and remember to also give support and earn trust with your manager.

By using these tips, you can utilize the full potential of your 1:1 that helps your career growth.

What other tips help you utilize your 1:1 with your manager to increase your career growth? Please comment on them below!

If you want to take a look more into effective 1:1, take a look at this resource:

Use This Mantra to Decide whether You Want to go to Big Tech or a Startup

Wed, 02 Mar 2022 08:35:00 +0000

When I left Disney Streaming Service to pursue another opportunity, my colleagues asked, “where are you going next?” I told him about a company that wasn’t as big as Disney. They responded with a joke saying, “I bet it is easier to solve the problem over there than it was here,” with a wink.

The truth is that it is not much easier.

It is just a different problem that I am solving. Therefore, these are three main differences between a startup and a Big Company that will help you decide where you want to pursue your next career journey.

Scalability vs. Product Market Fit

Many of the products are well-established when you work at big tech companies - everyone knows that the product has found the product-market fit.

So what happens when a company reaches this stage?

They start to scale.

They have started improving their infrastructure and want to create a better user experience, so they are investing a lot more resources to make the system reliable.

As an engineer, the company requires someone to dive deep into scalability issues—knowing the internal working on a library or infrastructure to make it better and faster. In other words, they are looking for a perfect solution for the system. Thus, they are also looking for people with a laser focus on scalability.

On the other hand, startups need to solve a different problem than Big Companies. Startups are always evaluating their product to see if they are solving the critical user pain points.

"Are we solving the right problem or not? And is there any traction to the problem that we solve"

Engineers in startups need to be close to the product and evaluate the product to make feature recommendations. They may think about mocking the backend and creating the UI for the sake of seeing if they should go all-in on the feature. They look for the right metrics to see if they solve the problems.

When you think about creating a Food Delivery App such as Doordash, you may think that there are a hundred orders a day, managing driver networks, scaling payment systems to accommodate multiple users.

As an engineer, you may start thinking of scaling the systems to accommodate a high amount of load. However, the way the founder got started, they delivered themselves. They code out some prototype, probably in some language they are most familiar with, such as Python or JavaScript.

They didn’t think about how network effect, managing multiple drivers efficiently, and payment systems for multiple users. They are grateful if users ARE using their app. If some miracle happens on the first day of launch, Tony Xu will go to the restaurant, pick up the food, and deliver it themselves. The first thing you don’t want to think about as a founding engineer in Doordash is Scalability.

They may copy and paste many things and work backward to test a simple prototype in the codebase. In other words, you will need to build a hacky system. Lots of the practice in the startup may frown upon other engineers. However, the last thing that they need to do is scalability.

On the other hand, engineers in Doordash now need to know how to scale a payment system to accommodate multiple users. They also need to make their system language agnostic as they scale internationally.

If you love scalability problems, you should go to Big Tech. They have enough resources to help you research the topic of making the system as available as possible. On the other hand, if you want to understand how the product decision is made and become closer to the product, you have a better chance to be in a Startup. Although you may need to sacrifice doing things in the right way, you gain the ability to use tech to find product-market fit.

Working on Specific Parts vs. Working on multiple parts

When I was in Disney Streaming Service, I got assigned to a specific part of the payment system - more specifically, a repository. They told me not to worry about other domains and focus on the Payment domain. However, there is not much time to slowly understand the domain in a startup, and I got to touch ten different repositories to build a feature that worked end to end.

If you work for a Big Tech Company, you may be assigned to a specific domain. Your role is to make the domain better and faster. Therefore, you may have known less about other domains and your neighbor services’ code base code because other teams maintain that domain.

If you want to change the contract between neighboring services or add new features to the current domain, you need to meet with other teams that interact with your service and discuss it. They will make changes on their end to not impact their service. These steps may include multiple meetings with the team, the PM of your team, and the PM before you can change the contract.

In a startup, you may touch on multiple codebases and be on many hats. For instance, you may be the one that needs to discuss with the vendor or find the right team to talk to for specific features that you are trying to test.

You may need to understand which repository is the right place for the new feature that you are trying to ship. Further, you need to communicate with the repository owner to make changes to the repository. You have to be able to articulate your thought on whether the product that PM was requesting is feasible and give insights on which part of the system needs to fix.

When the project is not going as expected, you will need to ping the right person or team to push the project further. In other words, the ability to chase after people because you are the person who gives ideas. Thus, you are also the person who is accountable for the project’s outcome.

Since everyone is busy with their roadmap, you have to communicate and ensure that your execution strategy is clear by clarifying all the questions to all stakeholders and PM as early as possible.

Although your title is a software engineer, you may do some project management, product management, and even a QE role sometimes because your responsibility is to ensure this feature is shipped properly.

Structured Organization vs. Flat Organization

This happens because of the notion of a startup. As a company grows and more and more people are hired, the organization will transform from a flat to a more structured organization.

In a big company, that means a lot more ceremony and team rituals.

When team A needs to do a contract change, team A PM may need to schedule time with team BPM to discuss. Team A PM needs to see if team B can squeeze these changes into their current roadmap. If all goes well, both teams discuss when this change should take effect, etc.

In a startup, engineers can message a director and ask questions regarding the domain they solve. The director will explain the domain for the engineers to further their research on the new feature they are pushing.

If there is an arrow on how the communication flows in an organization, in a big company, it will be PM to engineers or engineer manager to engineer. In a startup, it will be across all directions.

Conclusion

The problem you are solving in a startup is different from that of each employee working in a big company.

When I was in a big company, we solved scalability problems. There is already a product-market fit, and we need to figure out how our system doesn’t break and make it scalable and reliable.

In a startup, the problem that you are learning is cross-functional. You will be working on enabling certain features and seeing if that feature is deemed good enough even to go further.

You may be responsible for a single domain in a big company. Thus, you can be the subject matter expert in that domain. Thus, you only work on a couple of repositories, but your knowledge of a specific topic is deep.

In a startup, you need to understand how the user is interacting with your application and see if the feature you build can help increase conversion. You may be working from the front-end’s interaction to the payment service. Although you get to see how all services are wired together, you don’t go deep on that service.

The challenge in the startup is not about technology and scalability but more about how to ship your product successfully. You need to get everyone together to make this project going forward. In other words, a lot of project and product management work. You get to see if the product either works or fails.

This requires an engineer to be pragmatic and hacky.

Lastly, the organizational structure between a startup and a big company is also different. Your title is not clearly defined in a startup. Communicating in a big tech company may not be as flexible as a Startup.

If you want to keep pushing yourself and your career as an engineer to the edge, you should consider going to a Startup.

If you want to become a subject matter expert in a specific domain and specific field, go for Big Tech. If you want to start your company one day and be close to the company vision and product as much as possible, go for a startup.

Either big tech or a startup, you get to learn a different skill set and be a more well-rounded software engineer. As long as it pushes you forward to the ideal career you want, go for it!

Why Do Functional Programmers Prefer For-Comprehension Over Imperative Code Block

Tue, 30 Nov 2021 22:07:00 +0000

“Why do you wrap everything into an effect and put a for over here?” a Senior engineer in another team commented on my second code review. I looked into the code, and the first thing that popped up on my mind was, “Because it is readable.”

Readability is very subjective. Someone from the world of imperative programming and used to the Java 8 API Stream-like syntax will say that sequencing your program with for-comprehension will make it harder to read. On the other hand, functional programmers will view Java 8 API Stream-like syntax harder to trace through the effect of your code. Hence, I try to find a more objective point of view on why it is readable.

Looks like a Pseudo Code

For-comprehension is very similar to Haskell monad comprehension.

For Comprehension is syntactic sugar for flatMap.

flatMap represents sequential computations, and it is the main trait of Monad. Hence, the designer Scala uses monads for collection operations.

Monad comprehension in Scala was designed to look like imperative for loops in some generic C/pseudo-code-like language. Why? So that it looks like imperative sequential side-effecting code block like a C-style language. The left-array for assignment is typical in imperative pseudo-code. Hence, it creates better readability.

However, Scala and Haskell differ within the comprehension syntax - it can do more than perform the two monadic operations join and bind (or map and flatMap).

In Scala, a for Comprehension without a yield translates into foreach. foreach is an imperative iteration that produces side-effect. A yield can have a guard, yield bar if baz, which translates into filtering elements.

That reason alone doesn’t give a good reason why you would prefer for Comprehension over generic Java code block. However, since for-comprehension is equivalent to monad comprehension, you can ensure that your code runs sequentially in that effect.

Enforces Sequential Operation

Unless you use IO, having multiple effects inside your functions is hard to debug without for-comprhension.

Let’s take an example if you have a function that sequence multiple API calls:

def computeSequentially = {
  val api1Future = Future {...}
  val api2Future = Future {...}
  val api3Future = Future {...}
  
  api1Future.flatMap {api1 => 
   api2Future.flatMap { api2 => 
    api3Future.map{api3 =>
     //do something here
    }
   }
  }
}

From the code above, you may see that you sequence multiple API calls with flatMap. However, you are making a parallel call instead of doing a sequential call.

Future is eager. That means once you instantiate the value, it will trigger a thread and execute what is inside of Future.

Compared to wrapping each effect within the for-comprehension, it enforces any effect to execute sequentially.

def computeSequentially = for {
  api1 <- Future { ... }
  api2 <- Future { ... }
  api3 <- Future { ... }
} yield( /* do something */ )

Another benefit of using for-comprehension with effects is that another developer can easily take a quick look at the codebase and understand the intent of your program without understanding how Future works. You don’t need to guess if this block is running sequentially vs. in parallel.

The above code looks very simple because we are just doing simple API fetch calls. However, what happens if we have a nested effect? Such as calling an API that will return Future[Option[Int]] or Future[Future[Future[_]]]?

Increase Readability with Nested Effect

Without for-comprehension, you see something like this:

def fetchAPi2(someValue: Int): Future[Option[Int]]
def fetchApi3(someOtherValue: Int): Either[Throwable, Result]

def computeApi = {
  fetchApi2(1).flatMap{someValueMaybe => 
    someValueMaybe.map{
      fetchApi3(someOtherValue) match {
        case Left(throwable) => ???
        case Right(result) => ???
      }
    }.getOrElse(0)
  }

}

Then, if there are even more sequential calls, the line of the above code will go towards the right side - like a callback hell.

With for-comprehension, it helps flatten the amount of nested call-on effects.

import cats.data._

type FutureOption[A] = OptionT[Future, A]
type FutureOptionEither[A] = EitherT[FutureOption, Throwable, A]

def computeApi = {
  val futureEitherOption = for {
    someValue <- FutureOptionEither(fetchAp2(1))
    someOtherValue <- FutureOptionEither(fetchApi3(someValue))
   } yield ()
   
   futureEitherOption.value match {
     case Left(throwable) =>  ???
     case Right(result) => ???
   }
 }

We use the cats.data collection on monad transformer and use for-comprehension to compute multiple nested affect calls. The for-comprehension helps decrease the number of nested calls and make it look like a sequential call.

This rail-way-oriented programming. With rail-way-oriented programming, you can separate all error handling on all effects in one place - keeping the main logic like a pseudo code.

It is all based on Your Style Guide

Convincing the benefit of for-comprehension vs. regular code-block in that PR as it is all based on your team’s style guide.

If all developers believe that writing a regular code-block is more readable than for-comprehension, writing code-block is preferable.

In my case, another developer also commented, “plus 1” on his comments. However, I don’t fully agree with the explanation. The team prefers to write code in an imperative code block style instead of for Comprehension.

I ended up making changes to the preferred style guide.

Source:

programming languages - Why does Scala name monadic composition as “for comprehension”? - Software Engineering Stack Exchange

Does Seniority Title Really Matters in Tech

Tue, 21 Sep 2021 00:16:00 +0000

I remember that day when I gave two weeks’ notice to my Manager at Macy’s Tech in San Francisco. A few minutes later, the director in our department called me to schedule a one-on-one.

“Why do you want to leave?” he said. “Is it because of the money?”

It was partly because of the compensation. It is also the odd phenomenon where many people have resigned and moved on to their next career. I foreshadow something was coming, some big event, like a layoff in the San Francisco office. However, I just graduated from college and wanted to find a job to challenge myself daily.

“It wasn’t about the money. It was about the experience that led me to make the decision.”, I nervously said.

He told me if I was looking for a raise, he could provide or match the other company’s salary. He gave me career advice on that one on one that had me pondering why most people leave a company.

He said that he would switch jobs if he found a jump in his title or his salary. However, out of the two, a title bump trumps a salary bump.

This has me thinking - does the title matter? On the other end, other Software Engineers say that title doesn’t matter. What matters is your responsibility, and the other company will translate your responsibility to their level.

The Fear of Downlevel

Switching jobs in tech come with financial upside or title upside. However, it doesn’t come with both. Many of the engineers from a startup are down-leveled to bigger tech companies, such as FANG. For instance, a Senior Software Engineer into an SWE 2, a VP in a non-tech-focus company into a Senior Engineering Manager. In most of these cases, they will less likely take the position and keep on interviewing elsewhere.

On the other hand, they get a salary bump to the same level as their previous manager’s salary. Perhaps a promotion in their current company cannot top the level that they received in their offer.

What Title Means in a Company

Before you think if the title matters, we have to look at what the title means in the company. Why do we have the title in the company instead of having all one at the same level?

Responsibility

The company uses the title as the responsibility that of each layer of their project pipeline. For instance, a senior software engineer will have a different responsibility than a Software Engineer 1.

Senior Software Engineers are masters of craftsmanship. They know what they don’t know. They can independently handle all aspects of moderately complex problems based on what the stakeholder wants. They try to leverage and collaborate with other engineers to drive the project forward. Their responsibility may be to design features from end to end in a big company and to be well-versed in the company’s technology stack. That means process, architecture, design, implementation, test, and sustaining engineering. One example of a project that they do is to onboard a new payment processing flow in its system. During the code review, they usually are the ones that help mentor other engineers and giving them guidance on how to navigate the project the right way. They also know how to handle sticky situations where their work got block by some other team.

A Software Engineer’s responsibility is to deliver the right features. Often, they may need some guidance in delivering the project. It may be a technical problem and blockers on another team while completing the project. They are responsible for investigating their assigned problems and managed to design a solution. However, someone is responsible for API between components or sign off before they implement.

Pay Range

With bigger responsibility, the rewards will also be higher. Each title band will have its pay range, for instance, according to Levels. A software engineer at Facebook will have an average total salary of $267000, while a Senior Software Engineer will have an average total salary of $390000.

Seniority

In some companies, seniority comes into play. From a non-tech perspective, the title plays with seniority. Suppose you look at the job description for a Software Engineer vs. a Senior Software Engineer. A Software Engineer will require 0 - 3 years, and a typical senior software engineer will require at least five years. Some high-growth startup company who have grown in a short time sometimes put their initial engineers in a higher title than the engineer that joined later.

Although total years of experience doesn’t necessarily mean seniority, on overage, a software engineer typically requires a couple of years of experience to reach the master of craftsmanship.

A Software engineer that has been in a company for a longer time will also tend to understand more about that company’s domain and technology stack. Thus, as years went by, the company also trusted him to execute a higher responsibility that caused an increase in the title.

When does title matter?

Knowing what the title means, it is not standardized across companies. A senior title in company A doesn’t necessarily translate to a senior title in company B because a senior title in company B has a different responsibility than a senior title in company A.

However, there are a couple of cases where the title does matter.

When you want to gain more influence in your company and have a saying

A title represents the seniority in the company. Therefore, it also projects the credibility that you have in the current scope of the project.

There is a sense of respect if you have seniority in the company. Other engineers and stakeholders may take you seriously, and they are less likely to challenge you because of your title.

"For those in an underrepresented group, titles make a world of difference in establishing credibility. A principal engineer from an underrepresented group working at a medium-sized tech company said: "Once I got promoted to senior, then to principal, things got better each time. Still, even with a principal engineer title, I regularly get challenged on whether I am technical. Imagine what would happen if I did not have the title and the signal it carries."

- The Pragmatic Engineer.

Having a bigger title can impact the project’s current scope, budget allocation, etc. For instance, a senior director can persuade the higher-ups to focus on tech initiatives instead of product initiatives.

When you want to switch jobs

Oh yes, non-engineers tend to look at titles more sometimes to reach out to you because they believe your responsibility is somewhat similar to the same title in the company.

A recruiter will not reach out to you for a manager position if you have stated that your software engineer title. They will reach out to someone who has the title of either a tech lead or a manager to demonstrate that they have acquired the responsibility.

There has been a saying in the Blind community that the best time to switch jobs is when you just got promoted. One of the reasons is a title change is equivalent to a change in responsibility. Most companies will not promote engineers to the next level if they have not yet done that responsibility. Therefore, if you have been promoted to the next level, you probably are “ready” and have been doing the role of the next level. However, promotions within the company may not be as high of a salary bump as going to the next level of the next company. Other companies want to onboard engineers on the same level because they have been “doing that responsibility” on the previous company.

Takeaway

There are two groups of people who advocate whether the title is important or not. As the old sayings, great power comes with great responsibility. Companies create a title to differentiate responsibilities within their organization. Engineers have a higher responsibility results in more credibility in the eyes of the company. In some companies, the title often reflects the seniority you have in the company.

The title does matter when you want to establish your credibility and look to switch jobs. The title establishes the message on how much impact you create in the company.

To avoid being downlevel when switching domains, try to switch track within your company when you have the opportunity. On top of the title, understanding the responsibility and expectations can help you up for success. It is always good to over-deliver then under-deliver. Thus, taking too much responsibility might result in stress and less success in the long run. Nonetheless, too little responsibility may cause you to feel bored and frustrated.

Thanks for reading! These are my thoughts about the title in tech. Do you think the title matters? Please comment on them below share your thoughts.

Source

An Interview Question that Truly Tests your Experience as a Software Engineer

Tue, 14 Sep 2021 09:46:00 +0000

Leetcode has always been the holy grail to cracking the coding interview. Many engineers have bribe that the technical interview pipeline is broken. Three years ago, a technology company would not have told you to solve two Leetcode hard questions in a single setting. A lot of the technology company right now, such as Facebook, expects you to finish two medium-hard questions - or else you may not go to the next round of the interview.

This interviewing style becomes the race of who can recognize the most Leetcode style questions in one sitting and it doesn’t test on one’s ability and experience to problem solve.

Many companies also realized that having a regular algorithm from Leetcode is not an effective way to examine candidates’ success ability in the workplace. Therefore, some other kinds of interview questions have been created to test a candidate’s problem-solving ability and gauge the experience of candidates.

The Question

Imagine you have a service that needs to do some work and call a third-party API - let’s name it API A. API A takes a long time to compute. Hence, your service should only call API A once.

How would you ensure not to fetch API A?

One solution that most candidates will mention is the use of caching.

Let’s imagine you want to create a cache. The cache interface looks like this:

The cache has a get and a compareAndSet operations. For get, if the key doesn’t exist, it will return a null value. compareAndSet will compare put the prevValue, the previous value, of the value in the cache. If the prevValue is the one we mentioned, the currValue will override that prevValue. Suppose the prevValue inserted into the parameter is not the last value in the cache. In that case, the currValue will not be inserted into the cache. compareAndSet returns a boolean if the operation has succeeded insert the currValue.

You have to implement getAPI that will incorporate the cache interface:

What they Test You

This question doesn’t just ask you about the trivia things such as the cache invalidation strategy, the cache eviction policy, or the purpose of a cache.

Most candidates have heard of a cache, and the purpose of creating one is to increase your application’s latency. Moreover, a simple Google search can answer those questions.

This question about implementing cache is to test and gauge the candidates’ knowledge of whether or not they have experience writing services and APIs. You can assess whether or not they know about error handling. You can also see if they can write code by implementing a solution engineers use in a real-world scenario.

Most of the rudimentary answers for implementing the getAPI is using an if statement.

Intuition

Call get with the given key. If the operation encounters a cache-miss, call fetchAPIA. Once finish calling fetchAPIA, evoke the compareAndSet operation to refresh the cache with the given key.

The above initial implementation looks good. However, this is fine if you only have one request and only one user. If there are multiple concurrent calls to the cache and they all hit cache miss, the third-party service may get bombarded with multiple concurrent calls. With a high traffic spike on the service all at once, the service may encounter an outage. This phenomenon is often called the cache stampede.

How do you code your function so that it prevents cache stampede?

Candidates who encounter this issue will know how to solve it. However, if they haven’t seen this issue, you can also test their ability to problem solve by seeing how they will solve such an issue.

Furthermore, you can test candidates’ ability to handle the error. What happen if compareAndSet returns a false? Do you do a retry, or do you return the value with an error message?

These questions can sometimes touch different ways from a simple coding exercise. Experienced candidates will know some types of errors you want to recover and some types of errors you want to throw them gracefully.

Solution

The underlying problem with cache stampede is a classic problem in concurrent programming. If multiple threads are trying to access a memory location, how do you ensure that only one value gets access to that memory location? In other words, how do you ensure that only one thread at a time accessing the application? The solution is to use locking.

When it’s time for one of the workers to do some computation on the cache, it will acquire the resources - stopping other workers from computing that memory location. Any other worker with a stale or missing cache will need to wait until the lock is released. Then, they will retry the cache read. The primary data source, therefore, is not overload with the request.

Therefore, any method to avoid cache stampede limits the number of requests to the primary resources. There are multiple ways to solve this problem. Two simple ways are the most versatile method to avoid cache stampede.

Debouncing the request

The first one comes from the common JavaScript practice of preventing duplicate events from firing and causing noise in the system. This is something that frontend-engineer use frequently called Debouncing. What debounce function does is that it delays the invoking function after 100 milliseconds have elapsed since the last time system evoked the debounced function. Doordash did it in their backend using Kotlin Coroutine. However, if you don’t use a specific language with this mechanism, you must implement your debouncing method. Therefore, it is not maybe quite tough to do. Moreover, this solution will require another service or rely on the caller to have that mechanism, which increases dependency within the service.

Compare and Set

The compareAndSet is key in this question because it is the only way that has some synchronization inside.

Insert some keywords in the cache to indicate that there is already one worker doing the fetching. Therefore, another worker can either wait and retry again. For instance, once a worker call cache, and it gets a cache-miss, it will write a value with the key to the cache, such as “in-progress,” to imply to other work that the current key is a cache miss and that there is already a worker fetching the primary source. The catch here is that the calling operation needs to compare and set the process with atomic primitive. Another worker can wait and recursively call the function again to check if the “in-progress” has to turn into something else.

Are we done yet? Did you notice any other possible failures that we need to handle?

There is a scenario regarding the initial compareAndSet returns a false. There is also another scenario where the last compareAndSet returns a false.

The above method didn’t implement what happens when the get(key) returns an in-progress. Should they wait and retry, or should they return a specialized message to the caller saying they are “waiting”?

Closing

This new breed of questions tests candidates’ ability to problem solve and their experience in software development. Experience candidates may know when is good to retry and when is not good to retry. A problem with a mixed implementation of real-world system design is a better question for the tech interview than “how to traverse a BST.”

If that candidate was you, how would you solve the above problem? How should you handle the error above? Are there any other potential scenarios that you need to handle? Do you have any other good interview questions that examine candidates’ problem-solving skills and their experience? Please comment down below and share your thoughts!