What is enlightenment ?

It is possible to alter the state of our consciousness with meditation and psychedelic drugs. We have this illusion of self which seems as if it is directing the things we are doing. It is the thing which identifies with the thoughts and actions we do. But if you see clearly and question who this self is- Is it the thing in our head ? Is it our body ? If you think carefully it is not any of these but it is an artifical construct created by the brain to have an easier understanding of the world. So if you medidate, you can see this clearly that there is nothing different between us and the trees and the people infront of us. Everyone is part of the same consciousness that is part of this world. So that means there is nothing to worry about. You suffer only if you identify with the self and the ego. Once you realize that it is all just a story that your brain has created, you will not identify with your thoughts. Thoughts arise and go on their own. If you realize that the real you is different from your thoughts and feelings and there is nothing different between you and others, there is no expectations and no suffering. This is enlightenment.

But for this truth to be evident and be present, we have to practice meditation.

Also once you are enlightened nothing changes. You still have to work for living and have to be with people and do most of the things that you have been doing. The only difference is that now the real you is not caught up in the illusory self that identifies itself with the passing thoughts and feelings and suffers. You will have that perspective and hence you will respond to things more thoughtfully. Also when we know this, you will realize that there is no race to get anywhere. What is needed is all here and you will do what you have to do with full presence without rushing. And you will be free as you not bonded to your thoughts, feelings or your body!

Then what is the point of this life?  It is just this it just is. There is no specific point to it. Still you can't argue that the life of human is better with modern technology than without. It means you can still have some intention and work towards that, but understand that we are not bound to that intention and just work on it without judgement. But if you don't want to do anything and have the means to be that way, that is fine too.

https://samharris.org/the-illusion-of-the-self2/

Presence brings peace

If you are working on something and you are constantly distracted with something else, you will still have nagging doubts as to whether you missed something in the work you're doing. You will have to check and double check to make sure you did what you set out to do. Even after all this you will still feel that you were not as productive as you could be. You will still feel that you and your work is inferior to someone's who is focused when even though that feeling may or may not be true.

Now contrast that working with presence. Instead of feeling you're missing something, since you're fully focused on the work you're doing and you have already taken everything into account, you feel more confident of your work. Since you're fully present with your work you have considered everything without rushing, there is no need to check it multiple times. To top it all you know that you were as productive as you could be and that there is no anxiety as to whether a focused person could have done it faster or better, because - you have been focused. 

This my friends is why presence brings peace 

How to develop a practicing mind

When you are learning something, if you cant seem to learn well, blame it on your mind. The mind seems to be perfectly built to be always in the past or in the future. It compulsively tries to avoid staying in the present, but being there itself is the key to a fulfilling life says Thomas Sterner in his book - The Practicing Mind. 

In this book he forcefully argues that there is a right method of practicing anything. The expectation of achieving the goal makes us always think of the goal and hurry along to reach the goal. But the real thing to think about is not the goal but the journey.

Most of the anxiety we experience in life comes from our feeling that there is an end point of perfection in everything that we involve ourselves with.

He argues that there is no endpoint where everything is perfect but that we are always perfect. He brings forth the point eloquently with the question about when  is a flower  perfect. And this is the thing that spoke to me.

It is always perfect. It is perfect at being wherever it is and at whatever stage of growth it is in at that moment. It is perfect at being a seed, when it is placed into the ground. At that moment in time, it is exactly what it is supposed to be: a seed. Just because it does not have brightly colored blooms doesn’t mean it is not a good flower seed. When it first sprouts through the ground, it is not imperfect because it displays only the color green. 

When you develop a present-minded approach to every activity you are involved in and, like the flower, realize that at whatever level you are performing, you are perfect at that point in time, you experience a tremendous relief from the fictitious, self-imposed pressures and expectations that only slow your progress.

When you let go of your attachment to the object you desire and make your desire the experience of staying focused on working toward your goal, you are fulfilling your desire in every minute and you are patient with the circumstance. There is no reason not to be. There is no effort or “trying to be patient” here. It is just a natural response to your perspective. This shift in perspective is very small and subtle on the one hand, but it has enormous freeing power. No task seems too large to undertake. Your confidence goes way up as does your patience with yourself. You are always achieving your goal and there are no mistakes or time limits to create stress.

Another important thing he talks about is to not judge anything as good or bad, but be like a good instructor, who observes the student and if he is going in the wrong path, gently guides him on to the right path. And that is what he asks us to do in his DOC method. Do what you want to do, Observe how it is going, and if something is not to your liking or there is something you need to improve, just Correct it without judgement and emotional involvement in that thing. That is the path to progress.

In summary, creating the practicing mind comes down to a few simple rules:

       •  Keep yourself process-oriented.
       •  Stay in the present.
   .  Make the process the goal and use the overall goal as a rudder to steer your efforts.
       •  Be deliberate, have an intention about what you want to accomplish, and remain aware of that intention.

 
Some other quotes that have stayed with me.

Experiencing impatience is one of the first symptoms of not being in the present moment, not doing what you are doing, and not staying process-oriented

Nature knows what works because it does not have an ego to deal with. It is our ego that makes us create false ideas of what perfect is and whether we have reached it.We realize that, like the flower, we are just fine or, rather, that we are perfect when we are where we are and absorbed in what we are doing right at that moment.We realize that, like the flower, we are just fine or, rather, that we are perfect when we are where we are and absorbed in what we are doing right at that moment.

The second step in creating patience is understanding and accepting that there is no such thing as reaching a point of perfection in anything. True perfection is both always evolving and always present within you, just like the flower. What you perceive as perfect is always relative to where you are in any area of your life

If you think There is some place other than where I actually am now that I need to be. Only then will I be happy.” This is totally untrue and counterproductive. To the contrary, you are exactly where you should be right now. You are a flower.

Another excellent book that has something for anyone who wants to learn without feeling rushed and be happy in the process. The Practicing Mind - Thomas Sterner

Great advice on practice

If you're trying to master something difficult and the practice is very difficult, how will go about doing that ?

To a person's question on how to handle formula and proofs in maths, a person answered it the following way in mathexchange and it is a advice in mastering anything. Read it.

Source : https://math.stackexchange.com/questions/33656/whats-better-strategy-to-handle-tons-of-formulas-definitions
Answer link : https://math.stackexchange.com/a/33987/338003

You can read every book ever written on chess, but if you never play you will still be, at best, a middling player. Even if you memorize every rule in every book on chess you still won't become a particularly good player. You must play!
The same is true of math. You must solve problems!
I was always a natural with math, and I almost always grasped concepts the first time my professors covered them. For a long time I believed that this was enough. But as the topics became more and more abstract and more and more complex, I started to fall behind. It didn't feel like I was so good any more. I had developed the belief that practice was somehow beneath me. But practice is exactlyhow you get good at math.
Understanding is enough at a basic level. You can hold everything in mind and, if you understand it, you'll see the solution. But math continuously builds on itself. As layer upon layer of complexity is added, nobody in the world can hold all the pieces in their mind at once. Nobody! If you practice enough though, you no longer need to think about it. If you've practiced every layer below the one you're working on to the point that it's pure instinct, you don't have to hold any of the lower level stuff in mind anymore. You can focus all your attention on the high level content.
That's exactly like chess. You may understand basic tactics. You may have memorized all of them. But if you still need to look for forks and pins you have a long way to go. Put the books down and go play a few hundred games. Eventually, seeing the basic tactical elements will become as natural as breathing. Now you're ready to begin seeing the deeper elements of the game.
You asked whether you should focus on memorization or understanding. I'm saying neither. You didn't memorize your native language. You don't need to memorize math. Immerse yourself in it. The remembering will happen automatically. In terms of understanding, unless you completely master each step by practicing it until it's instinct (you really just need to get 90% of the way there; as you reuse the concepts down the road you'll go the last 10%), you'll never see the deeper elements of the game. The understanding you get from reading the text is shallow. The understanding you get from practice is deep. It's fluency.
Do not be seduced by the apparent superiority of problems over exercises (if you know what tools you're going to use from the outset, it's an exercise; if you have no idea where to start and need to puzzle it out, it's a problem). Problems are great, and, ultimately, you should definitely test your knowledge on them. But exercises should be the bread and butter of your training. Sure you know how to do them. They almost seem demeaning. But if you regularly work your muscles on these seemingly trivial tasks (like jogging or weight lifting), the challenging, novel, exciting tasks (like climbing a mountain) will get easier and easier.
Do not read the chapter a second time until you've attempted most of the problems at the end. If you solve several of them you'll find that the chapter makes way more sense the second time around. Go back and solve the rest of the problems and if you read it a third time it'll seem painfully obvious. If the book has all problems (at the freshman level all of your books are probably chock full of exercises, but in a year or two you'll start seeing books like this) and no exercises you must find as many problems with solutions as you can from other sources. Many high level texts have a handful of challenging high level problems at the end of chapters (frequently without solutions). Each problem will be unique and be solved in a different way. The lack of repetition means that it's very difficult to attain the "instinct" level. Find more problems elsewhere and solve them. As much as possible, only work on problems you have a solution to (the feedback is essential).
If you can't get additional problems. Just solve the ones you've got over and over (this works good for proofs, just make a list of proofs you want to know and work through it once every night or two with a blank stack of paper). In fact, if you couldn't solve a problem the first time, always re-solve it after you've seen the solution. Keep coming back to it until you can solve it without even a peek at the chapter or the solution. If you get stuck on a new problem for an hour or two, go back and solve similar easier problems for a little while and come back to it.
This may sound like rote memorization, and, beyond that, like a heck of a lot of work. It's not about memorization. Just try it for a while and I guarantee you'll find that your understanding goes through the roof (even if you think it's pretty darn good to begin with). And, well, yeah, it is a lot of work. But maybe less than you think. Doing one-hundred problems is not ten times as much work as doing ten. Problems eleven to thirty probably take about as much time and effort as the first ten. So, probably, do the last fifty. At the beginning new tasks are often unpleasant and frustrating, but with practice they become, if not fun, at least satisfying. Just like jogging. Most people stop practicing just when the learning curve is getting steep (that's the good part, even though it sounds like the bad part).
That's probably more than you expected. I've made my way through a lot of math though, and this is what I've learned. Meta-learned would be more accurate I suppose. If somebody had explained this to me clearly when I was a freshman, I'd probably have gotten quite a bit more out of my education.

Excellent advice.  

Breaking the addiction with the mind

What is addiction with the mind. Whenever a thought comes in the mind, we run away with the thought. we are addicted to each thought that the mind brings forth. Without knowing we are flying like a kite in the wind or a flotsam on the sea. This is addiction with the mind.

What if you're doing something important and you have to focus? Is there a way to break this addiction to the mind? There is a way to focus here. You can always build a refuge for focus in your breath.

Whenever you want to focus, start by sitting comfortably and breathing in and out and counting 1. Do this till 8 and then back to 1. In between if you notice that the mind has wandered bring the attention back to breathing and start from 1. By the time you're done you will have gathered your attention and then you can start focusing.

In between thoughts come, you can always tell your mind, not now and keep focusing on the breath. You will see that cravings will rise and fall, thoughts will come and go, but just keep focusing on the breath. This way you can always come back to your breath for focus whenever you feel distracted.

One thing to remember here is that you cannot always ignore the thoughts, but you can temporarily say, not now and continue with what you're doing. Later when you're free and have some spare time, you can ride the urgent thought waves you have been putting away for later and be mindful of what they're trying to tell. That way during the times of focus, they will not make their presence felt urgently.

This might seem like something difficult or unrealistic but it is very true. To create a habit with these, you can always use the habit formation rules of make it obvious, make it attractive, make it easy and make it rewarding. Also remember any habit change is a change of identity. So give enough votes to this new identity of meditator, so that the identity is solidified.

Applying laws of good habits

Continuing with James Clear's book Atomic Habits - I want to zoom in on the habit loop.

the habit loop is cue-craving-response-reward.

That gives the laws of behavior change

1. make it obvious
2. Make it attractive
3. Make it easy
4. Make is satisfying

 I will apply this to something I need to do e.g. say solving coding problems.

Make it obvious - It is already scheduled in my calendar and the time is also set. So it is obvious so much so that I never forget it.

Make it attractive -
1. Using temptation bundling, if I solve at least 3 problems daily, or totally at least 21 problems, then I get to watch a movie.
2. Join a group who solve coding problems. We can join coding problem solving sites like IB or hacker rank or other where you work on solving the problems.
3. Do something enjoyable immediately after a difficult habit. push my code to Github or read an interesting article both of which give satisfaction

Make it easy :-
1. Using the 2- minute rule. It is successful if I just start and read the first problem. I need not solve it also. At a minimum, I need to read the problem and start. That's it.
2.  I have all my laptop, coding editor, note book , pen all ready
3. I have no other distractions, I have eaten and am not thirsty

Make it satisfying:-
1. Reward if I solve 3 problems. Probably read an article.
2. Create a checkin chain in Github
3. Never miss twice : Remember what Vaman Sharma garu told us - ద్వితీయ విఘ్నము పనికి రాదు

You also use these four rules to create habits that stick. All the best and thanks to James Clear for such an amazing book!

Book reco - Atomic Habits

James clear has written an excellent book about habits. Why you should care about your habits. How you can create good habits and how you can break bad ones. I have read this and it is dripping with actionable advice. A must read book I would say.

My top five takeways from this book

Habit change is identify change: Your habits influence your identity and your identity influences your habits.
once your habits are established, they seem to stick around forever—especially the unwanted ones. Despite our best intentions, unhealthy habits like eating junk food, watching too much television, procrastinating, and smoking can feel impossible to break.This is because our identity is wrapped up in the habit. You can make the change easy by shifting your identity. For a smoker tyring to quit, instead of saying - I'm trying to quit smoking, try saying I'm not a smoker.
True behavior change is identity change. You might start a habit because of motivation, but the only reason you’ll stick with one is that it becomes part of your identity.

Every action you take is a vote for the type of person you wish to become. No single instance will transform your beliefs, but as the votes build up, so does the evidence of your new identity. This is one reason why meaningful change does not require radical change. Small habits can make a meaningful difference by providing evidence of a new identity.
The good news is that you don’t need to be perfect. In any election, there are going to be votes for both sides. You don’t need a unanimous vote to win an election; you just need a majority. It doesn’t matter if you cast a few votes for a bad behavior or an unproductive habit. Your goal is simply to win the majority of the time.
It is a simple two-step process:
Decide the type of person you want to be.
Prove it to yourself with small wins.

Environment is extremely important for habits: Habits have four steps
Cue, Craving, Action, Reward
Eliminate the cue and your habit will never start. Reduce the craving and you won’t experience enough motivation to act. Make the behavior difficult and you won’t be able to do it. And if the reward fails to satisfy your desire, then you’ll have no reason to do it again in the future.
Add more friction to things you want to avoid and make things easy for those you want to add.


The four laws of habit change: 

Creating Good Habits:

The 1st law (Cue): Make it obvious.
The 2nd law (Craving): Make it attractive.
The 3rd law (Response): Make it easy.
The 4th law (Reward): Make it satisfying.

Breaking bad habits:

Inversion of the 1st law (Cue): Make it invisible.
Inversion of the 2nd law (Craving): Make it unattractive.
Inversion of the 3rd law (Response): Make it difficult.
Inversion of the 4th law (Reward): Make it unsatisfying.

Why modern technology is both a strength and a weakness 
This is one reason why the versatility of modern technology is both a strength and a weakness. You can use your phone for all sorts of tasks, which makes it a powerful device. But when you can use your phone to do nearly anything, it becomes hard to associate it with one task. You want to be productive, but you’re also conditioned to browse social media, check email, and play video games whenever you open your phone. It’s a mishmash of cues.

How to be disciplined: 

“disciplined” people are better at structuring their lives in a way that does not require heroic willpower and self-control. In other words, they spend less time in tempting situations. Setup your environment to do the right thing easily and the wrong thing not without jumping through various hoops.

I cannot recommend this book highly enough. Just go read the whole thing and have it with you for reference always. Now this is not one book which has few insights that we will read and remember. It is packed so the apt name is - Habits - The complete reference. Get it asap.

Some Questions about being a better software engineer (And Answers)

1. How can I improve my coding speed and my ability to solve problems better ?
Practice practice and more focused practice. Time yourself solving problems.See where you are falling down. pick up and go again. The more you care, the faster you get better.
https://medium.com/always-be-coding/four-steps-to-google-without-a-degree-8f381aa6bd5e
https://medium.com/always-be-coding/abc-always-be-coding-d5f8051afce2

2. How much more time do I need to become good ?
Don't know but since time is the only variable, it just makes sense to  ABC (always be coding)

The Gifts of Imperfection - Brene Brown - Book Summary

This is a new book I started reading. It is written by Brene Brown a researcher about whole hearted living.  It is about how to live more authentically by embracing our imperfections. I really liked the book. I am not that emotional or not that vocal with emotions. So this book which talks about to work through difficult emotions and express them is really helpful.

This is my summary of the book till now...

Wholehearted living:
Cultivating the courage, compassion and connection to wake up in the morning and think, No matter what gets done and how much is left undone, I am enough.

Journey:
Courage compassion and connection can seem like big lofty ideals, but in reality they are daily practices that when exercised daily  become the gifts in our lives. And the good news is that our vulnerabilities are what force us to call upon these amazing tools. Because we are imperfect, we get to practice using our tools on a daily basis.

1. How courage compassion and connection are tools for developing worthiness
2 More questions on love, belonging and worthiness
3. What are the obstacles to this journey


Men and women who live Wholeheartedly do indeed
DIG Deep. They just do it in a different way. When they’re exhausted and overwhelmed, they get
Deliberate in their thoughts and behaviors through prayer, meditation, or simply setting their intentions;
Inspired to make new and different choices;
Going. They take action


Owning our story can be hard but not nearly as difficult as spending our lives running from it. Embracing our vulnerabilities is risky but not nearly as dangerous as giving up on love and belonging and joy—the experiences that make us the most vulnerable. Only when we are brave enough to explore the darkness will we discover the infinite power of our light.

Practicing courage compassion and connection in our daily lives is how we cultivate our worthiness.

Courage:
We need support and we should be courageous to ask for it and acknowledge when we are feeling down or shameful. When we share it takes the pain away.
It’s only been in the last few years that I’ve learned that playing down the exciting stuff doesn’t take the pain away when it doesn’t happen. It does, however, minimize the joy when it does happen.

Every time we choose courage, we make everyone around us a little better and the world a little braver. And our world could stand to be a little kinder and braver.

Compassion:

In cultivating compassion we draw from the wholeness of our experience—our suffering, our empathy, as well as our cruelty and terror. It has to be this way. Compassion is not a relationship between the healer and the wounded. It’s a relationship between equals. Only when we know our own darkness well can we be present with the darkness of others. Compassion becomes real when we recognize our shared humanity.

One of the greatest (and least discussed) barriers to compassion practice is the fear of setting boundaries and holding people accountable.
We have to stay away from convincing ourselves that we hate someone or that they deserve to feel bad so that we can feel better about holding them accountable. That’s where we get into trouble. When we talk ourselves into disliking someone so we’re more comfortable holding them accountable, we’re priming ourselves for the shame and blame game.

When we fail to set boundaries and hold people accountable, we feel used and mistreated. This is why we sometimes attack who they are, which is far more hurtful than addressing a behavior or a choice. For our own sake, we need to understand that it’s dangerous to our relationships and our well-being to get mired in shame and blame, or to be full of self-righteous anger. It’s also impossible to practice compassion from a place of resentment. If we’re going to practice acceptance and compassion, we need boundaries and accountability.

Connection:
connection as the energy that exists between people when they feel seen, heard, and valued; when they can give and receive without judgment;
and when they derive sustenance and strength from the relationship.

Until we can receive with an open heart, we are never really giving with an open heart.When we attach judgment to receiving help, we knowingly or unknowingly attach judgment to giving help.

The Wholehearted journey is not the path of least resistance. It’s a path of consciousness and choice. And, to be honest, it’s a little counterculture. The willingness to tell our stories, feel the pain of others, and stay genuinely connected in this disconnected world is not something we can do halfheartedly.
To practice courage, compassion, and connection is to look at life and the people around us, and say, “I’m all in.”

Power of love and belonging

Love is the most important thing in our lives, a passion for which we would fight or die, and yet we’re reluctant to linger over its names. Without a supple vocabulary, we can’t even talk or think about it directly.
— DIANE ACKERMAN

If we want to fully experience love and belonging, we must believe that we are worthy of love and belonging.
When we spend a lifetime trying to distance ourselves from the parts of our lives that don’t fit with who we think we’re supposed to be, we stand outside of our story and hustle for our worthiness by constantly performing, perfecting, pleasing, and proving. Our sense of worthiness—that critically important piece that gives us access to love and belonging—lives inside of our story.

The greatest challenge for most of us is believing that we are worthy now, right this minute.

learn that fitting in and belonging are not the same thing, and, in fact, fitting in gets in the way of belonging. Fitting in is about assessing a situation and becoming who you need to be to be accepted. Belonging, on the other hand, doesn’t require us to change who we are; it requires us to be who we are.

Love

We cultivate love when we allow our most vulnerable and powerful selves to be deeply seen and known, and when we honor the spiritual connection that grows from that offering with trust, respect, kindness, and affection.
Love is not something we give or get; it is something that we nurture and grow, a connection that can only be cultivated between two people when it exists within each one of them—we can only love others as much as we love ourselves.
Shame, blame, disrespect, betrayal, and the withholding of affection damage the roots from which love grows. Love can only survive these injuries if they are acknowledged, healed, and rare.

Belonging is the innate human desire to be part of something larger than us. Because this yearning is so primal, we often try to acquire it by fitting in and by seeking approval, which are not only hollow substitutes for belonging, but often barriers to it. Because true
belonging only happens when we present our authentic, imperfect selves to the world, our sense of belonging can never be greater than our level of self-acceptance.

Sometimes, as I turned to the data to craft definitions like the ones above, I would cry. I didn’t want my level of self-love to limit how much I can love my children or my husband. Why? Because loving them and accepting their imperfections is much easier than turning that light of loving-kindness on myself.

Practicing self-love means learning how to trust ourselves, to treat ourselves with respect, and to be kind and affectionate toward ourselves.

Practicing Loving and Belonging

To begin by always thinking of love as an action rather than a feeling is one way in which anyone using the word in this manner automatically assumes accountability and responsibility. - Bell Hooks

If we want to live and love with our whole hearts, and if we want to engage with the world from a place of worthiness, we have to talk about the things that get in the way — especially shame, fear, and vulnerability.

Things that get in the way
Shame
when we struggle to believe in our worthiness, we hustle for it.

Here are the first three things that you need to know about shame:
1. We all have it. Shame is universal and one of the most primitive human emotions that we experience. The only people who don’t experience shame lack the capacity for empathy and human connection.
2. We’re all afraid to talk about shame.
3. The less we talk about shame, the more control it has over our lives.

Shame is the intensely painful feeling or experience of believing that we are flawed and therefore unworthy of love and belonging.

While it feels as if shame hides in our darkest corners, it actually tends to lurk in all of the familiar places, including appearance and
body image, family, parenting, money and work, health, addiction, sex, aging, and religion. To feel shame is to be human.

If we’ve worked hard to make sure everything looks “just right” on the outside, the stakes are high when it comes to truth-telling. This is why shame loves perfectionists— it’s so easy to keep us quiet.

Shame needs three things to grow out of control in our lives: secrecy, silence,and judgment. When something shaming happens and we keep it locked up, it festers and grows. It consumes us.

I found that men and women with high levels of shame resilience share these four elements:
1. They understand shame and recognize what messages and expectations trigger shame for them.
2. They practice critical awareness by reality-checking the messages and expectations that tell us that being imperfect means being inadequate.
3. They reach out and share their stories with people they trust.
4. They speak shame—they use the word shame, they talk about how they’re feeling, and they ask for what they need.

Difference between guilt and shame
Guilt = I did something bad.
Shame = I am bad.

According to Dr. Hartling, in order to deal with shame, some of us move away by withdrawing, hiding, silencing ourselves, and keeping secrets. Some of us move toward by seeking to appease and please. And, some of us move against by trying to gain power over others, by being aggressive, and by using shame to fight shame

The four elements of shame resilience: Name it. Talk about it. Own your story. Tell the story.

When we’re in shame, we’re not fit for human consumption. We need to get back on our emotional feet before we do, say, e-mail, or text something that we’ll regret. I know that it will take me ten to fifteen minutes to pull myself together and that I will definitely cry before I’m ready. I’ll also need to pray. Knowing this is such a gift.

If you want to kick-start your shame resilience and story-claiming, start with
these questions. Figuring out the answers can change your life:
1. Who do you become when you’re backed into that shame corner?
2. How do you protect yourself?
3. Who do you call to work through the mean-nasties or the cry-n-hides or the people-pleasing?
4. What’s the most courageous thing you could do for yourself when you feel small and hurt ?


Our stories are not meant for everyone. Hearing them is a privilege, and we should always ask ourselves this before we share: “Who has earned the right to hear my story?” If we have one or two people in our lives who can sit with us and hold space for our shame stories, and love us for our strengths and struggles, we are incredibly lucky. If we have a friend, or a small group of friends, or family who embraces our imperfections, vulnerabilities, and power, and fills us with a sense of belonging, we are incredible lucky.

Guidepost #1 - Cultivating Authenticity 

Often people attempt to live their lives backwards: they try to have more things, or more money, in order to do more of what they want so that they will be happier. The way it actually works is the reverse. You must first be who you really are, then do what you really need to do, in order to have what you want. — Margaret Young

Authenticity is a collection of choices that we have to make every day. It’s
about the choice to show up and be real. The choice to be honest. The choice to
let our true selves be seen.

Authenticity is the daily practice of letting go of who we think we’re supposed to be and
embracing who we are.
Choosing authenticity means
cultivating the courage to be imperfect, to set boundaries, and to allow ourselves to
be vulnerable;
exercising the compassion that comes from knowing that we are all made of
strength and struggle; and
nurturing the connection and sense of belonging that can only happen when we
believe that we are enough.
Authenticity demands Wholehearted living and loving—even when it’s hard, even when
we’re wrestling with the shame and fear of not being good enough, and especially when
the joy is so intense that we’re afraid to let ourselves feel it.

Mindfully practicing authenticity during our most soul-searching struggles is how we
invite grace, joy, and gratitude into our lives.


It’s not so much the act of authenticity that challenges the status quo—I think
of it as the audacity of authenticity.

Caution: If you trade in your authenticity for safety, you may experience the following:
anxiety, depression, eating disorders, addiction, rage, blame, resentment, and
inexplicable grief.

Dig Deep:

Whenever I’m faced with a vulnerable situation, I get
deliberate with my intentions by repeating this to myself: “Don’t shrink. Don’t
puff up. Stand on your sacred ground.” I think there’s something deeply spiritual
about standing your ground.

Whenever I’m faced with a vulnerable situation, I get deliberate with my intentions by repeating this to myself: “Don’t shrink. Don’t puff up. Stand on your sacred ground.” I think there’s something deeply spiritual about standing your ground.

I try to make authenticity my number one goal when I go into a situation where I’m feeling vulnerable. If authenticity is my goal and I keep it real, I never regret it. I might get my feelings hurt, but I rarely feel shame.

Guidepost #2 Cultivating Self-Compassion

The thing that is really hard, and really amazing, is giving up on being perfect and
beginning the work of becoming yourself.
— ANNA QUINDLEN

Perfectionism is not the same thing as striving to be your best. Healthy striving is self-focused—How can I improve? Perfectionism is other-focused—What will they think?

Life-paralysis refers to all of the opportunities we miss because we’re too afraid to put anything out in the world that could be imperfect. It’s also all of the dreams that we don’t follow because of our deep fear of failing, making mistakes, and disappointing others. It’s terrifying to risk when you’re a perfectionist; your self-worth is on the line.

Perfectionism is a self-destructive and addictive belief system that fuels
this primary thought: If I look perfect, live perfectly, and do everything
perfectly, I can avoid or minimize the painful feelings of shame, judgment,
and blame.
Perfectionism is self-destructive simply because there is no such thing as perfect. Perfection is an unattainable goal. Additionally, perfectionism is more about perception—we want to be perceived as
perfect. Again, this is unattainable—there is no way to control perception, regardless of how much time and energy we spend trying. Perfectionism is addictive because when we invariably do experience shame, judgment, and blame, we often believe it’s because we weren’t perfect enough. So rather than questioning the faulty logic of perfectionism, we become even more entrenched in our quest to live, look, and do everything just right.
Feeling shamed, judged, and blamed (and the fear of these feelings) are realities of the human experience. Perfectionism actually increases the odds that we’ll experience these painful emotions and often leads to self blame: It’s my fault. I’m feeling this way because “I’m not good enough.”

To overcome perfectionism, we need to be able to acknowledge our vulnerabilities to the universal experiences of shame, judgment, and blame; develop shame resilience; and practice self-compassion. When we become more loving and compassionate with ourselves and we begin to practice shame
resilience, we can embrace our imperfections. It is in the process of embracing our imperfections that we find our truest gifts: courage, compassion, and connection.

They spoke about their imperfections in a tender and honest way, and without shame and fear. Second, they were slow to judge themselves and others.

A moment of self-compassion can change your entire day.

A string of such moments can change the course of your life.

— CHRISTOPHER K.GERMER

According to Neff, self-compassion has three elements: self-kindness, common humanity, and mindfulness.

Self-kindness: Being warm and understanding toward ourselves when we

suffer, fail, or feel inadequate, rather than ignoring our pain or flagellating

ourselves with self-criticism.

Common humanity: Common humanity recognizes that suffering and

feelings of personal inadequacy are part of the shared human experience—

something we all go through rather than something that happens to “me”

alone.

Mindfulness: Taking a balanced approach to negative emotions so that

feelings are neither suppressed nor exaggerated. We cannot ignore our pain

and feel compassion for it at the same time. Mindfulness requires that we

not “over-identify” with thoughts and feelings, so that we are caught up and

swept away by negativity.

Dig Deep: 

Dr. Neff’s Self-Compassion Scale.

It’s a short test that measures the elements of self-compassion (self-kindness, common humanity, and

mindfulness) and the things that get in the way (self-judgment, isolation, and

overidentification). www.self-compassion.org

“There is a crack in everything. That’s how the light gets in. It reminds me that our

imperfections are not inadequacies; they are reminders that we’re all in this together. Imperfectly, but together. “Today, I’m going to believe that showing up is enough

# Guidepost 3 - Cultivating a resilient spirit. 

If you look at the current research, here are five of the most common factors of

resilient people:

1. They are resourceful and have good problem-solving skills.

2. They are more likely to seek help.

3. They hold the belief that they can do something that will help them to

manage their feelings and to cope.

4. They have social support available to them.

5. They are connected with others, such as family or friends

Spirituality is recognizing and celebrating that we are all inextricably connected to each

other by a power greater than all of us, and that our connection to that power and to one

another is grounded in love and compassion. Practicing spirituality brings a sense of

perspective, meaning, and purpose to our lives.

From this foundation of spirituality, three other significant patterns emerged

as being essential to resilience:

1. Cultivating hope

2. Practicing critical awareness

3. Letting go of numbing and taking the edge off vulnerability, discomfort, and

pain

hope is a combination of setting goals, having the tenacity and perseverance to pursue them, and believing in our own abilities.Hopeful self-talk sounds more like, This is tough, but I can do it.

Tolerance for disappointment, determination, and a belief in self are the

heart of hope.Entitlement is “I deserve this just because I want it” and agency is “I know I can do this.”

Practicing critical awareness is about reality-checking the messages and

expectations that drive the “never good enough” gremlins.

practicing spirituality brings perspective, meaning, and purpose to our lives. When we allow ourselves to become culturally conditioned to believe that we are not enough and that we don’t make enough or have enough, it damages our soul. This is why I think practicing critical awareness and reality-checking is as much about spirituality as it is about critical thinking.

Most of us engage in behaviors (consciously or not) that help us to numb

and take the edge of off vulnerability, pain, and discomfort.

2. Addiction can be described as chronically and compulsively numbing and

taking the edge off of feelings.

3. We cannot selectively numb emotions. When we numb the painful

emotions, we also numb the positive emotions.

For many of us, our first response to vulnerability and pain of these sharp points is not to lean into the discomfort and feel our way through but rather to make it go away. We do that by numbing and taking the edge off the pain with whatever provides the quickest relief. We can anesthetize

with a whole bunch of stuff, including alcohol, drugs, food, sex, relationships, money, work, caretaking, gambling, staying busy, affairs, chaos, shopping, planning, perfectionism, constant change, and the Internet.

after years of research, I’m convinced that we all numb and take the edge off. The question is, does our _______________ (eating, drinking,spending, gambling, saving the world, incessant gossiping, perfectionism, sixty hour workweek) get in the way of our authenticity? Does it stop us from being emotionally honest and setting boundaries and feeling like we’re enough? Does it keep us from staying out of judgment and from feeling connected? Are we using _____________ to hide or escape from the reality of our lives?

While I was “taking the edge off” of the pain and vulnerability, I was also unintentionally dulling my

experiences of good feelings, like joy. Now I can lean into joy, even when it makes me feel tender and vulnerable. Joy is as thorny and sharp as any of the dark emotions. To love someone

fiercely, to believe in something with your whole heart, to celebrate a fleeting moment in time, to fully engage in a life that doesn’t come with guarantees— these are risks that involve vulnerability and often pain

Feelings of hopelessness, fear, blame, pain, discomfort, vulnerability, and disconnection sabotage resilience. The only experience that seems broad and fierce enough to combat a list like that is the belief that we’re all in this together and that something greater than us has the capacity to bring love and compassion into our lives.

Whether we’re overcoming adversity, surviving trauma, or dealing with stress and anxiety, having a sense of purpose, meaning, and perspective in our lives allows us to develop understanding and move forward. The heart of spirituality is connection. When we believe in that inextricable connection, we don’t feel alone.

Dig Deep: 

12 step vowel check 

A = Have I been Abstinent today? (However you define that—I find it a little more

challenging when it comes to things like food, work, and the computer.)

E = Have I Exercised today?

I = What have I done for myself today?

O = What have I done for Others today?

U = Am I holding on to Unexpressed emotions today?

Y = Yeah! What is something good that’s happened today?

People are like stained-glass windows. They sparkle and shine

when the sun is out, but when the darkness sets in, their beauty is revealed only

if there is a light from within.”

# Guidepost 4 - Cultivating Gratitude & Joy 

When it comes to gratitude, the word that jumped out at me throughout thisresearch process is practice. I don’t necessarily think another researcher would have been so taken aback, but as someone who thought that knowledge was more important than practice, I found these words to be a call to action.

Twinkle lights are the perfect metaphor for joy. Joy is not a constant. It comes to us in
moments—often ordinary moments. Sometimes we miss out on the bursts of joy because
we’re too busy chasing down extraordinary moments. Other times we’re so afraid of the
dark that we don’t dare let ourselves enjoy the light.
A joyful life is not a floodlight of joy. That would eventually become unbearable.
I believe a joyful life is made up of joyful moments gracefully strung together by trust,
gratitude, inspiration, and faith.

The dark does not destroy the light; it defines it. It’s our fear of the dark that casts our joy
into the shadows.

For me, and for many of us, our first waking thought of the day is “I didn’t get enough sleep.”
The next one is “I don’t have enough time.” Whether true or not, that thought of not enough
occurs to us automatically before we even think to question or examine it. We spend most of
the hours and the days of our lives hearing, explaining, complaining, or worrying about what
we don’t have enough of … We don’t have enough exercise. We don’t have enough work. We
don’t have enough profits. We don’t have enough power. We don’t have enough wilderness.
We don’t have enough weekends. Of course, we don’t have enough money—ever.
We’re not thin enough, we’re not smart enough, we’re not pretty enough or fit enough or
educated or successful enough, or rich enough—ever. Before we even sit up in bed, before
our feet touch the floor, we’re already inadequate, already behind, already losing, already
lacking something. And by the time we go to bed at night, our minds race with a litany of what
we didn’t get, or didn’t get done, that day. We go to sleep burdened by those thoughts and
wake up to the reverie of lack … What begins as a simple expression of the hurried life, or
even the challenged life, grows into the great justification for an unfulfilled life

We each have the choice in any setting to step back and let go of the mindset of scarcity.
Once we let go of scarcity, we discover the surprising truth of sufficiency. By sufficiency, I
don’t mean a quantity of anything. Sufficiency isn’t two steps up from poverty or one step
short of abundance. It isn’t a measure of barely enough or more than enough. Sufficiency
isn’t an amount at all. It is an experience, a context we generate, a declaration, a knowing that
there is enough, and that we are enough.
Sufficiency resides inside of each of us, and we can call it forward. It is a consciousness,
an attention, an intentional choosing of the way we think about our circumstances.

Dig deep:
We each have the choice in any setting to step back and let go of the mindset of scarcity.
Once we let go of scarcity, we discover the surprising truth of sufficiency. By sufficiency, I
don’t mean a quantity of anything. Sufficiency isn’t two steps up from poverty or one step
short of abundance. It isn’t a measure of barely enough or more than enough. Sufficiency
isn’t an amount at all. It is an experience, a context we generate, a declaration, a knowing that
there is enough, and that we are enough.
Sufficiency resides inside of each of us, and we can call it forward. It is a consciousness,
an attention, an intentional choosing of the way we think about our circumstances.

From taking turns being thankful during grace to more creative
projects like creating a jar to keep gratitude notes in, we’re making
Wholeheartedness a family affair

Fall into fall

Nature is one of the things that inspires us all. When we go and spend some time in the woods, it makes us feel connected to this earth. It makes us feel very peaceful. I personally feel a sense of connection to the past and future. Nature is never constant it keeps changing bringing about the change in seasons. Any change of season is another chance to go keep looking at the hills and mountains, streams and rivers, leaves and flowers and feel inspired and connected.

So today I want to leave you with just that message - go out into the nature and take a look at the beautiful fall colors. It is an inspiring message of letting go things which are unnecessary and get ready to face the harsh challenges ahead. Even in shedding the things that are unnecessary it puts up such a beautiful picture inspiring us all.

Consistency is the key

There is awesome power in compounding. Anything worthwhile takes time and it is a great advantage if you're able to wait for it a little more time than you think you can.

Just keep doing the work and keep improving and keep using it and it will definitely improve. The more consistent you are the faster you reach where you want to go.

Make the thing you want to do so easy and so ingrained that your mind thinks that thing to be the default path and it will start doing it first before anything else. And it is possible. How you ask - just keep doing the thing at same time daily with complete focus. And when done rest. Repeat it day after day. It seems so simple, so much so that it feels as if it doesn't work at all. But there in lies the secret, it is so simple not to work. It took me so much time to discover that, but that seems the secret to acting and action begets more action and in no time, there is an avalanche of results. So that is the mantra ... action consistently at the same time daily no matter what, whatever it takes.
I'm off to my work... !

When feeling doubtful after starting something, put in the work first and then reevaluate...

When we are working on something difficult, we know it will take time. But when it seems out of our reach, it is natural to question whether what we are doing is worth it or if it makes sense to abandon it. But working through this feeling is essential to doing anything worthwhile. But does this mean we should keep ploughing through without ever seeing the light?  In these cases it might be worthwhile to keep ploughing but keep a point to review the decision based on how far we have come and then take the decision whether it makes sense to proceed or to give up. This thought shouldn't bother us when we are working on something except at the predetermined decision point.

And that decision can be ideally decided in the beginning. But ideally since anything difficult takes atleast double the time budgeted for it initially, it seems the ideal decision point is after we have spent double the time initially expected on the difficult thing attempted.

To summarize,

- When doing anything difficult it is natural to feel doubtful of that.
- When you start feeling that, decide that you will do that difficult thing for x amount of time.
-  Then work hard on it for 2x time
- If it is achieved then there is nothing to do.
- If not achieved yet, review based on the progress whether to sink more time to finish it or to bail out.

So basically, when feeling doubtful  after starting something, put in the work first and then reevaluate... 

Some things I want to remember

Build routines

We have great ideas and goals. When we make a goal, distractions arise. So we cannot trust our will power. It is better to develop routines and habits which make us easy to do what we decided to do. 

Even great writers have routines where their muses visit them exactly at the same time of the day. The more scheduled time you have, the more peaceful and guilt free your unstructured time is.  

Compartmentalize our days and weeks so that we always have a chance to reflect and recommit to what we want to do. 

Do what you do with full focus for the day. After that have fun and sleep 

Fully Commit & Back Yourself to Win

Really commit to what you're doing. That moves mountains and when you burn your boats, you have no chance but to fight. 

Once you decide the whole world conspires to make it happen, provided you're ready to put in the work it requires. 

If you have less balls in your hand, you are more likely to care about making them work. 

 Have environment which enhances your positive traits 

Everyone has many thoughts - some positive, some negative. 

There are some people around with whom you always feel negative. And there are others who spread an air of positivity around them.... 

There are some articles which when you read makes you feel alive, positive and work to overcome your problems where are there are some which only cater to your base thoughts and make you addicted but really not helping you any way.. 

There are some places which by being noisy and distracting always make you lose your concentration.. 

Lot of internet products and websites are built to be addictive. There might be some articles which are good for you but most of them are addictive. And the worst part is they don't give the control to you choose what you want. It is better leave them than trying to look out for the positive. 

 It is a time of great opportunity 

It is a great time of development in technology and I'm really excited about the advances in software and AI to automate the boring parts of our work day. 

It is an exciting time to work towards shaping the future in the vision we have. It is a chance to work on problems you care about solving and make the lives of not only our people but all people better. 

 You cannot do it alone 

But this is not something you can do alone. By being generous with our ideas, being kind with our deeds, clear with our thoughts, we can work with like minded people to work together - enriching each others lives and achieving big goals. 

  Be patient & kind  

Hopes are high but that doesn't mean the results are also as fast. So it takes time and requires solving hard challenges. Sometimes you might doubt your own intentions.

 Be patient. Let the compounding magic work. Anything worthwhile takes a long time but with the magic of doing things consistently, you will reach it sooner rather than later. So just stay put. 

But during this time, be kind with people and more than anyone, be kind with yourself. 

Be Grateful & and let them know 

In this journey, you will come across many people. You will have been better for having known some people. Lot of people would have helped you with their gifts. You're where you are because of their generosity with their time, work and help. So take the time to count your blessings and let them know whenever you get a chance. Actually take the chance.  I take this chance to thank all of them who have helped me be where I am. 

Design of Data Intensive Systems - Part 5 Consistency & Consensus

Consistency Guarantees

Linearizability

make the system appear as if there is only one copy of data and all operations on it are atomic. It means all clients must be able to read the latest data once it is written. It is a recency guarantee

Linearizability is useful in

Locking and leader election
constraints and uniqueness guarantees (single up-to-date values which all nodes agree on)
When there are cross channel timing dependencies.
Implementing linearlizable systems

Single leader replication can be made linearlizable. But there could be stale replicas and split brain scenarios with the leaders and hence not completely linearlizable.

Multiple leader replication cannot be made linearizable as it needs conflict resolution which means there is no single copy of data

Leaderless replication can be made serializable if the reader performs read repair before returning and writer must read the status of a quorum of nodes before sending the writes. But in practice most of the leaderless system do not provide linearizability.

If a application requires linearizability it will not be available to some replicas during a network partition. It was work with the old replicas but then it is not linearlizable. And linearlizable systems are not that performant as the read and write timings are atleast proportional to the network delays.

Consensus alogorithms are linearizable. They have a way to avoid the split brain and stale replicas.

Linearizable systems are slow always not just during network faults. So latency sensitive systems might make some tradeoffs

Ordering Guarantees

Ordering is important to provide causality. In a single leader replication, all the writes go through the leader to determine the order of writes. In a serializable transaction, the writes are processed in a serial order. Timestamps and clocks are also an attempt to order the writes.

If a happens before b, then a could have caused b. So the causal order is a -> b. If a system obeys the causal order, it is said to be causally consistent. If you read something from database, if you see a piece of data, you should be able to see any data that precedes it.

Linearizable system maintains total order as it behaves as if there is only one piece of data

Two operations are consistent if they happen at the same time. If two events are ordered if they are causally related, but they are incomparable if they are concurrent. So causality defines a partial order not a total order

Which means there are no concurrent operations in a linearlizable datastore.

Generally, the causal ordering is sufficient for most applications. So to provide causal ordering, we can use monotonically increasing sequence numbers to order the events. When it is single leader it is easy, but when there are multiple nodes processing, there are ways to do the sequence, but that sequence doesn’t provide causality.

One way to generate sequence numbers that provide causual ordering is Lamport Timestamps. Here each node will generate sequence numbers and in addition to that it will also have the node id It will be a pair of (counter, node id). It can provide causal order in the following way. Each client will always send the maximum counter it saw among all nodes in each request. So say a node 1 has a counter of 5 and node 2 has max counter of 1. A client got a request from node 1 (with counter 5) and it sent a request to node 2 ( with counter 1), node 2 will see that and increment its counter to 6. That way it provides total ordering of all the operations .

But these timestamps cannot solve the problem of unique constraints as we will know the total order after the fact. When a system is trying to generate a unique username, then it needs to know if it is unique or not and if other systems are already processing that username. In additional to knowing total order, we also need to know when it is finalized. This idea of saying when a total order is finalized is called total order broadcast

Total order broadcast can be used to solve the uniqueness problem.

Write a username request into the log using a compare and set operation.
Read the log
If your request was the first request when you read, then you can commit the transaction. If some other’s node request was the first request in the log, then you abort the transaction as the username is already taken.
The implementation of the atomic compare and set operation requires consensus if multiple nodes request the same username.

If we have a linearlizable increment and get register, then it is possible to have total order broadcast. Each message that needs to be sent via total order broadcast has to get a sequence number from the linearizable register and attach it to the message and send it to all nodes. If a node delivers a message with sequence 4 and it gets a message with sequence 6, it knows that it has to wait for sequence 5 before it can deliver 6. For lamport timestamps, the counter may have gaps for e.g. if a node sees 6, for the next message it will send 7, but it will not wait for 5, but with total order broadcast, it has to wait for 5.

Implementing linearizable compare and set or total order broadcast both require consensus.

Consensus is required for

leader election to avoid multiple leaders and split brain. Consensus is required to select and agree on one leader.
Atomic commit — In a db supporting transactions on multiple nodes, if it fails on some nodes, then all the nodes should agree to commit or rollback on all nodes to maintain the atomicity of the transaction.

2 Phase Commit

For a transaction over multiple nodes, it is not possible to have an atomic transaction if each node can do its own commit/rollback process. It needs to go through a process of 2 phase commit. In this case, there is a coordinator to coordinate the transaction.

The application can send the changes across multiple nodes. Once the transaction is ready to commit, the coordinator starts phase 1 called prepare, where each node will prepare to commit by writing it in the log and when both the nodes are ok to commit, the coordinator will start phase 2, write the transaction in its logs and send the ok to both nodes. The both nodes then commit the transaction. If any of the nodes didn’t say an ok to the transaction in phase 1, then the coordinator aborts the transaction.

If any of the nodes or network fails during transaction, the coordinator aborts.

If the coordinator fails before confirming the phase 2, both the nodes will be waiting for coordinator to send the ok. Otherwise they cannot abort or commit because doing anything unilaterally will violate the atomicity. So it will block till the coordinator recovers and issues the commit or rollback.

There is something called 3PC (three phase commit) which assumes a system with bounded network delays and bounded response times. Basically if the crash of a node can be detected correctly, then 3PC can provide non-blocking transactions. But that cannot be assumed in general.

In practice 2 phase commit is implemented by distributed transactions XA (which is a coordinator api with which multiple applications interface). As discussed above during coordinator failure, there is a possibility for the locks to be held indefinitely. And either admin users have to resolve the locks or applications can use heuristics to unilaterally abort or commit.

XA can work with different applications, database and a message queue etc. If a transaction is within a homogeneous distributed system like a database, it can work well. but with heterogeneous distributed systems, are more challenging as they have to work with different technologies and cannot be optimized.

Disadvantages of 2PC

If a coordinator is not replicated and runs on single machine it become single point of failure
The application becomes stateful because it has to maintain the state in the coordinator log
Since it needs to work with disparate systems, it has only basic functions. I cannot detect deadlocks and doesn’t work with SSI (Serializable snapshot isolation)
For distributed systems, 2PC will amplify failures instead of helping to build fault tolerant systems.

Consensus

In consensus, one or mode nodes propose values and the algorithm decides a value. More formally the following properties must be satisfied

No two nodes decide differently
No node decides twice for same decision
If a node decides value v, then value v was proposed by some node.
Every node that doesn’t crash eventually decides a value.
Best known consensus algorithms are

View stamped replication VSR
Paxos
Raft
Zab
These algorithms decide a sequence of values which makes them total order broadcast algorithms. In each round the nodes propose what value they want to send next and decide the next message in total order.

These algorithms expect to work when a quorum of nodes are available.

To select a leader, we need consensus. When a leader is down, they start an election process which has an epoch number. When say an old leader from a previous epoch comes and tries to write something, the nodes which elected leader in a new epoch will not agree to it and hence the write fails. Basically it is a two step process. First a leader will be elected by consensus and when the leader wants to write something, again they need to agree and one of the node that participated in the leader election has to agree to this write. If any of the node knows of a different leader, it will not agree and hence quorum cannot happen and the write fails.

Limitations of Consensus:

The election process is like synchronous replication
It requires strict majority to operate
Since they rely on timeouts to detect failed nodes, transient network errors can cause frequent leader elections and terrible performance
Raft is particularly sensitive to network problems. If there is a weak network link, the leadership continually bounces and the system doesn’t work.

Membership and Coordination services:

Zookeeper and etcd are designed to hold small amount of data in memory. This data is replicated across multiple nodes through a fault tolerant total order broadcast algorithm

It has the following features

Linearlizable atomic operations
Using an atomic compare and set operation, a lock can be implemented. If multiple nodes try to set the value, only one will succeed. Distributed locks are implemented as leases which expires and eventually gets released if the client fails.

Total ordering of operations
When a resource is protected by a lock or lease, it needs a fencing token to prevent clients from conflicting with each other in case of a process pause. Zookeeperr implements this by totally ordering all operations and giving each operation a monotonically increasing transaction id called zxid and a version cversion

Failure detection
Clients maintain long lived session on Zookeeper servers. If there is a temporary failure or a zookeeper node goes down, the session is active. But if the heart beat fails for a more than a configured timeout, Zookeeper will expire the session and can be configured to release all the locks and leases held by that client. [These are called ephemeral nodes in zookeeper ]

Change Notifications
Clients can read the locks and values set by other clients and also watch for changes to them. Thus a client can find out when a new node joined the cluster and also when a node leaves the cluster (loss of ephemeral nodes). So the client can get these without polling by just subscribing to the changes.

Usecases

Allocation work to nodes - To identify a leader in a database scenario and also in a job scheduler scenario.
If there are lot of nodes, the Zookeeper doesn’t need to do leader election on all nodes, it can just work with 3 or 5 nodes, but can serve all the clients.

Zookeeper is useful to save only data which doesn’t change often. If you need to replicate application state, use another application called Apache BookKeeper.

2. Service discovery - In a cloud datacenter env, when a new service comes up, it can register with zookeeper so that its endpoints can be found by other services. It doesn’t need consensus but it can be useful for other services to find the leader since ZK already knows it.

3. Membership Services - A membership service determines which nodes are alive and are part of a cluster. A failure detection mechanism with a consensus mechanism can help determine which nodes are part of a cluster and which are not.

Source : https://books.google.com/books/about/Designing_Data_Intensive_Applications.html?id=zFheDgAAQBAJ

Design of Data Intensive applications - 4 Problems of Distributed Systems

Faults & Partial Failures
On a single machine if there is any internal fault, we expect the system to crash instead of giving wrong values.

But in a distributed system, there are always systems that are broken while others are working as expected. This is called a partial failure. Partial failures are non-deterministic. Also we don’t even know if something has succeeded or failed because the time taken for message travelling across a network is non-deterministic. So we don’t know which systems have failed in between also we don’t know if a system has failed or not. That is why working with distributed systems is difficult .

Unreliable Networks
Distributed systems are shared-nothing systems. They communicate only via asynchronous message passing through the network. But the network is unreliable.

If a request is sent many things could go wrong

Request may have been lost (unplugged network cable)
Request may have been queued
Remote node has crashed and is unavailable.
Remote node has paused temporarily (for garbage collection)
Remote node has processed the request, but response got lost in the network.
Response is delayed because our network is overloaded.
In all these scenarios, it is difficult to tell why we didn’t get a response. We usually handle these with a timeout.

When part of a network is cut off from another part due to a network fault, it is called a network partition

Timeout is only way to detect a fault, but there are trade-offs

If timeout is too long, then during this time the system is unavailable and users might see error messages

If timeout is too short, and the node is actually alive and busy with some action, and another node takes over, the action can happen twice. When a node is declared dead, then its responsibilities are transferred to other nodes. If the system is busy, then removing a machine might exaggerate the problem. It could cause all nodes to declare dead and cause cascading system failure.

Most of the network delay is due to queuing. Since all the requests are sharing the same pipe, there will be some queuing when the traffic is high. Different places where queues are possible

If several nodes send data to the same destination, the network switch must queue them up and feed them into the destination network link one by one. On a busy network link, the data has to wait in the queue till it can get a slot. If the switch queue fills up, some packets may be dropped.
When data reaches the destination machine, if all CPUs are currently busy, then the OS queues up the request until application is ready to handle it. Here also the wait time is not bounded.
In virtualized environments, the running OS is paused for 10s of microseconds when the virtual machine wants to use the CPU code. In this case VM cannot consume any data from the network, so incoming data is queued by the VM monitor increasing the variability of network delays
TCP performs backpressure in which a node limits its own rate of sending to avoid overloading network links or destination node.
If some packets are lost TCP will retransmit them which adds to the delay
In cloud environments, the machines are shared. So based on the workload of neighbors, the network delays are variable.
So the best way to set timeouts is based on experience. Also a system could monitor the delays and change the timeouts accordingly.

Unreliable Clocks
In distributed systems, time handling is tricky because of variable network delays in communication.

Each machine has its own hardware clock, which has minor calibration differences compared to others and can be slightly faster or slower compared to other machines. The clocks can be synchronized using NTP (Network Time Protocol) which allows the time to be adjusted according to the time reported by a group of servers. These servers are in turn get their time from a more accurate source like GPS recievers.

Computers have two types of clocks

Time of day clock — Returns the current wall clock time. These clocks are adjusted using NTP. So they should ideally be same on all machines, but if a machine’s local clock is too fast, the time might be reset, so the time can appear to be going backwards.So these cannot be used to calculate a duration.

System.currentTimeMillis() or clock_getTime(CLOCK_REALTIME) (Linux) return wall clock time.

Monotonic Clock — These clocks have time that always increases or grows. So these can be used to calculate the duration. System.nanoTime in Java or clock_getTime(CLOCK_MONOTONIC) in Linux does this.

Multiple CPUs on a machine means it could be using multiple timers, but the system presents a uniform monotonic time.
When NTP needs to reset the time, it will fasten or slow the timer based on how much faster it is going, but it will never make the time go backward.
NTP synchronization issues

The local computer quartz might drift
If local time differs a lot from the NTP, then the system might refuse to sync or it might be forcibly reset which causes the time to go backwards
If a node is firewalled from internet, then it is difficult to detect the clock is out of sync
If there is large network delay, then the synchronization might not work correctly
NTP services might be reporting wrong time
Leap seconds (59 or 61 seconds in a minute)can cause problems with systems which are not designed with leap seconds in mind.
In virtual machines, the clock is virtualized. If a VM is paused for some time, then the application sees the clock jumps forward by those many seconds or minutes.
If the device is not controlled by the application, then it is difficult to assume the time accurate. User can change the time on the device.
So if your application depends on clock synchronization, you have monitor and take action if a clock drifts too far from the others.

If you use timestamp to determine the ordering, you might lose some database writes due to clock skew. Use version vectors for ordering

Clocks can have a confidence interval based on the precise clocks and transactions can wait for some time to be confident that transactions are linearizable. Google Spanners True Time API uses confidence intervals. In order to ensure that transaction timestamps reflect causality, Spanner deliberately waits for the length of the confidence interval before committing a read-write transaction. By doing so, it ensures that any transaction that may read the data is at a sufficiently later time, so their confidence intervals do not overlap. In order to keep thewait time as short as possible, Spanner needs to keep the clock uncertainty as small as possible; for this purpose, Google deploys a GPS receiver or atomic clock in each datacenter, allowing clocks to be synchronized to within about 7 ms

Process Pauses

Even if everything is synchronized properly, it is difficult to rely on time because of arbitrary pauses

It could be stop the world GC
It could be unexpected IO for e.g. a JVM will lazily load a classfile when first used
A virtual machine may be suspended and resumed
User may close the laptop and open it
CPU switches to different thread
swapping to disk because of paging
Unix process SIGSTOPed and resumed later with SIGCONT
In all these cases the paused code will start running when execution continues without knowing that time has elapsed.

Ideas to overcome process pauses

Treat a pause like a outage. If it is possible to know before that a process is going to need GC, the app can stop sending requests to the node and other nodes can take the work.
Rolling restart frequently so that long term memory is never GCed which will not cause the stop the world pause.
Knowledge, Truth & Lies
Truth is determined by a majority

A node cannot trust itself completely as there could have been a GC pause and all the threads were oblivious to it. There could be a network breakage where a node is able to recieve messages and send but other nodes are not able to recieve it.

So the decision is made by majority ( half or more nodes )

So in case of leader locks, when a lock is held by a node and a gc pause occurs and when the node comes back up after the pause, its lock lease might have expired. In that case other node might have become the leader and it might have lock. So when the old node tries to write the db should check if the token is still valid. It can be implemented with fencing tokens (monotonically increasing token numbers )

If the nodes arbitrarily lie by saying it didn’t recieve a response when it did, then it is called a Byzantine fault and the consensus problem is called byzantine Generals problem.

Dealing with weak forms of lying

If there are corrupted packets, then protocols like TCP and UDP use checksums to determine and fix or retry them.

Sanitize inputs from users

Configure a client with multiple NTP server addresses so that one of them being bad can be tolerated.

System Models

To assume what assumptions an apply for each type of systems, we can consider the following systems

Synchronous: This model assumes bounded network delay, bounded process pauses and bounded clock error. This is not a realistic model for distributed systems because of unbounded network delay.

Partially-Synchronous: In this model, the system is synchronous most of the time, but sometimes there may be unbounded delays, or pauses or clock error. This is a good model for distributed systems as they work well most of the time except sometimes.

Asynchronous: Here there are no time assumptions. Some algorithms can work with asynchronous design but it is very restrictive.

Node failure models

Crash stop — In this model a node can go down by crashing and will never come back.

Crash Recovery — In this model a node can go down temporarily but will recover after some time and be available. The nodes have stable storage that is preserved across failures whereas the in-memory state is lost.

Arbitrary faults- Nodes may do absolutely anything including tricking other nodes.

For modelling distributed systems, partially synchronous system with crash-recovery is the best model.

Correctness Models

Properties of a distributed system that will indicate its correctness for a fencing tokens algorithm.

Uniqueness — No two tokens are same

Monotonic Sequence — If a request x was returned at tx and y was returned at ty and x completed before y, then tx
Availability — The node that requested a fencing token didn’t crash and eventually receives a response.

Safety and liveness

In the above example uniqueness and monotonicity are safety property, but availability is a liveness property. Safety is defined as nothing bad happens whereas liveness is defined as eventually something good happens. Safety properties should always hold, i.e never be violated in a system. But for liveness there could be some leeway, saying that the system will be available only if majority of the nodes are not crashed.

Theoretically, even if a system satisfies all the correctness properties, it is not guaranteed to run correctly in real life as the model is just a simplification and not necessarily holds always. For e.g. in a partially synchronous system, we assumed that stable storage survives restarts. What if it that storage becomes corrupt. In that case we still need to handle it. But the theoretical model becomes a good approximation to reason about.

Source : https://books.google.com/books/about/Designing_Data_Intensive_Applications.html?id=zFheDgAAQBAJ

Design of Data Intensive Application - Part 3 Partitioning

Why?

All data doesn’t fit in one machine. So different partitions are made on different machines and each piece of data will live in one partition. This is useful for scaling applications.

We have to see

How to distribute the data among partitions
How does indexes work with distributed partitions
How to rebalance the partition if some nodes go down or if one partition size increases more than limit.
Partitioning is combined with replication so that copies of each partition can be stored at multiple nodes. Even though one piece of data belongs to one node, it will be stored at multiple places for fault tolerance.

Skewed partitions and hotspots

Ideally, we would want the data to be distributed to partitions equally so that the load is evenly distributed. But if we don’t partition it correctly and all the data is on a single partition then it is called a skewed partition and it will cause a hotspot.

Random partitioning

To make sure data is partitioned equally, we can assign data to partitions randomly. But the drawback is that then when we need to search the data, we have to search in all the partitions which is wasteful.

Key Range Partitioning

Another way of partitioning is to assign a continuous range of keys to a partition like a encyclopedia. The key ranges don’t need to be evenly spaced, but can be changed based on the data. The partition boundaries need to adapt based on the data to avoid hotspots. The administrator can change the boundaries or the software can automatically change them.

If the keys are sorted, then range scans become very easy. For e.g. in the data is some sensor data, then if timestamp is the key for the data, it will be very easy to scan for temperatures in a week. But one disadvantage of that is that since all the data is together, when writing the data all the data will go into the same partition while all the other partitions sit idle.

Hash Partitioning

To overcome the drawbacks with the key range partitioning, the key can be passed through a hash function. Each partition can contain a range of hash values. The hash values are different even for closely related keys, so this will distribute the data evenly among different partitions.

The partition boundaries can be evenly spaced or chosen pseudo randomly (called consistent hashing — it uses randomly chosen partition boundaries to avoid the need for central control or distributed consensus. )

The disadvantage of hash partitioning is that we need to scan all the partitions for range queries.

To overcome this Cassandra supports a compound primary key, which contains two keys. In this the first key is used for hash partitioning and the second key can be used for range scans. This is useful because if a key is setup as (user_id, update_timestamp), then it is easy to get all the latest updates for a user.

Relieving hotspots:

In a social media application, any power user having lot of followers can cause hot keys. i.e if they do anything, it has to be written to all the followers. Since the key is the user’s id, it will cause lot of activity for the key and can cause a hotspot. To overcome this their key can be split further by adding a random two digits to the key for some of the records. This way the load can be split into different partitions. Then the application should also remember which keys were split as most of the data will be regular without splitting.

Secondary indexes:

We have seen how the primary key is partitioned, but most of the databases support secondary indexes. Secondary index allows us to query the data by a different key or term.

There are two ways to partition the secondary index.

each partition can have the secondary index based on the documents that are part of the partition called local index. The advantages are that writes are faster as the data will be updating only one partition but the reads are difficult because for a given term, all the data will not be in one partition but the query will have to be done in parallel on all partitions (scatter) and then the results assembled (gather)
Another option is for the seconday index to be seperate from the main partitions and it could be partitioned in such a way that all the documents for a single term stay at same place (global index). The advantage of this is that search, range scan will be easy as all the data for an index is at the same place, but writing will be slow as one document has to touch multiple replicas
Re balancing Indexes:

Rebalancing is needed when we add new machines or we remove old failed nodes.

when rebalancing is complete,

load should be shared fairly between nodes
while rebalancing is happening, db should continue accepting reads and writes
no more data than necessary should be moved between nodes.
Rebalancing Stratgies

Hash mod n is a bad strategy because if the n changes, most of the keys have to be moved.
Fixed number of partitions: Create a large number of partitions more than the number of nodes and assign for e.g. 1000 partitions for 10 nodes and assign 100 for each node. If a new node is added, then few of the partitions for each node can be assigned to the new node and removal can proceed in the opposite way. Also the partitions need not be assigned equally to all nodes, bigger machines can take higher load compared to the others
Dynamic Partitioning: For dbs with key range partitioning with fixed number of partitions, if we choose wrong boundaries then all the data will be in one partition. For these dbs it is possible to have dynamic partitions where initially there will one partition and as the data size increases and once it goes more than a threshold, the partitions are split and it continues on. The disadvantage is that initially all the requests will be served by one node only. So to overcome this, the db can start with a fixed number of partitions and then increase as required. It can also decrease if large number of keys are deleted. Here the number of partitions adapt to the total data volume.
Partitioning Proportionally to nodes: In fixed number of partitions, size of each partition is proportional to the size of the dataset whereas in dynamic partitions the number of partitions is proportional to the size of the dataset. A third option used by Cassandra is to have number of partitions proportional to the number of nodes. initially it will have fixed number of partitions per node and when a new node joins the cluster, new partitions are added, so the size of the partition decreases.
Rebalancing can be fully automatic or manual, but since it is an involved process and if something goes wrong it is better to have an operator in the loop to review the processing.

Request Routing:

Since the data is now split into different partitions how does the client know which node to connect to ? That is the service discovery problem and there are multiple tools for this

Approaches to solve this are

Client can connect to any node. If that node has the data it will handle the request. Otherwise it will query the right node, get the result and pass the data to the client.
Client can send the request to a routing tier. It will send the request to the right node and forwards accordingly.
Clients can be aware of the partitions. If they are aware they can send to the right node without the need of an intermediary.
When the data is rebalanced, the routing tier or the partionaware client has to know of the change. Generally this is a consensus problem and there are different solutions.

One solution is to use a coordination tool like Zookeeper to maintain the mapping of partitions to nodes. The routing tier can subscribe to the changes for these mappings.

Some dbs like Cassandra and Riak use a gossip protocol where cluster is aware of changes and clients can send requests to any node and the cluster will forward the request as required.

Source: https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321/