agile, Career, Product Management, Project

Changing the process from within

Would you agree with the statement below?

If you disagree with how something is done, you should give the process a try and drive changes from within.

Give yourself some time to think before reading the example below.

I was given an opportunity to drive some works that are normally done by Lead Developers/Team Leaders/Tech Leaders etc. I initially rejected it because I didn’t agree with that specific process. It is a discovery process where the engineer lays out options, considers the pros and cons, considers the use cases, proposes a solution, and gets a written agreement from the interested parties. Let’s dissect this process with a real example.

Example: Cross-account Kafka applications in AWS

  • Goal Summary: Source data from an application database in one AWS account to a MSK Kafka Cluster in another account.
  • Options: There are a few layers of decisions to be made. The publishing technologies could be custom publisher, Kafka Connect, generic in-house publisher. Then there is the cross account networking challenges. A proxy, direct connection, clone the DB etc. And then there are language choice, data flow design etc.
  • Decision: Kafka Connect with Network Proxy
  • Decision Maker/Interested Parties/Influencers: Product Manager from team A/Team B, C and X/ Team O, P and Q

This is an extremely over simplified version of the process. It took the team a month to finalise the proposal from trying out different options, to coordinating with different teams and many many discussions. But here is my interpretation of this process: we have a big challenge in front of us and we want to build some certainty before we step too deep into one direction. We also want to get an agreement between the teams that are involved, either in building them, might be using them, or potential maintenance team.

Why I disagree with this process

Retrospectively, I think I am too much of a purist when it comes to software development process. I have this ideal state of how Agile teams should be operating. Trust me, we are an Agile organisation but it is difficult to convey what that means nowadays.

First of all, the process is a waterfall in disguise. If it sounds familiar to you, the documentation that it produces, the contract model, the desire to be certain; they are very similar to the classic software specifications. For the record, there is nothing wrong about waterfall because it is still a great model for certain types of softwares. However, this process is so much leaner than a standard waterfall process. The document is in Git Issue and it was created with the understanding that things will change down the road.

Secondly, the reality is often not as good as the idea set out to be. The structure of the process is such that a single person or two mull the problem for a long period of time before presenting to the stakeholders. Although others are encouraged to challenge and be critical about the options and decision, there are a couple of problems. One being that not everyone will be on the same page or same level of technical understanding of the problem to challenge properly. Often people will just shrug it out and agree with whatever is decided. And then there is the biases of the individuals that will influence the decision.

And then there are two common outcomes at the end of the implementation of the decision. Teams tend to stick to the plan even if the road is bumpy and might not fit perfectly. This is sometimes accompanied with comments like “but we agreed to”, “we have invested in the decision”, and perhaps “the other teams are expecting this to be X”. The second outcome is that it changes and changes and the decision is not relevant anymore but the documentation and comms never really get updated. Although it is a preferred outcome nevertheless, why spend the time in the process anyway?

An iterative approach

Not sure if it s specific to Scrum or maybe it has a specific name to it, but let’s call it Agile for the simplicity sake. We know that the idea of Agile is to build small, try it out, showcase, learn and adapt to changes. Too many buzzwords, let’s use the same example above and see how we can iterate through it.

First we need to break the problem down into smaller more manageable problems. We can do this during refinement where the whole team will be there. Product Managers will focus on representing the customers and work with team to see what values can be delivered in the next week or two. There is no right answer for sure but the first goal can be to build a cross-account connection to first allow us and others to build applications outside of the cluster account. Let’s set that as a goal. The team discuss the options and agree to try on one.

A week of the team getting their hands dirty in the implementations, does it work? can we present something to the customers? If it works, that is great and move on to the next challenge. If it doesn’t, we learn and inform the next decision. A few iterations later, we have a living system that is not necessarily done or perfect but it is delivering value also we know that we are satisfying the requirements because we keep in touch with our customers.

This process does not guarantee the perfect solution at the end either. But what I value most here is the learning. Although we might have to do more actual work than the previous process, we do it as a team and we also learn how to adapt. In the Evolutionary Architecture principle, it is about supporting change and not about making the perfect decision. By involving in the change process, we learn how to change better.

We also build a team with thinkers and decision makers, and not just builders. I believe software development is a creative process and it often is not enough to just build things, but also to create things.

What do you think now

So we were here because my manager thinks that I should give the process a try to be able to understand the value from the inside and also drive changes if needed to. Do you agree?

For me, I am in a protest. I think that the process is against my ideals and I would not just do it so I can progress in the career ladder. This is definitely not looking good for my career progression but hey! it is about the journey, not just the destinations. I am enjoying my role and I am still learning and growing where I am as a senior developer. Although we know that there are pressure in climbing the ladder (perhaps a story for another time), I am not in a hurry to get anywhere just yet.


How I forced myself to spend time on a side project

I found it so much easier for me to be productive when I’m at office, forced to work from 9-5 every Monday to Friday than to working at home on a side project where I don’t really have a fixed working hour.

Why is it so hard?

Few differences that I observe to be different between office and home and how I fix them:

You are mostly working alone at home.

Most of the side projects I’ve done is a lone project. Somebody (or me) started it and a developer is needed and I volunteer and boom, you are all alone. It could be a good thing for some people, but I find it very demotivating.

Well, most projects failed badly and I blame the loneliness that I felt during the project. I felt that I needed approval from another peer. I needed somebody else to say that, “Hey, it’s awesome”. Or, “hey, you can do this better by doing this”. You have peers probably scrutinizing your work all the time at office. But when you don’t get the scrutiny, you don’t feel confident.

But hey, again, you are the master of everything now. It’s your own project and you know best about everything. I need to constantly remind myself that “I’m a good programmer” and “it’s OK to make mistakes”. It’s easier to say than done, of course. But, it’s really normal to fail. Failure is how we learn to be better.

Another thing that might help is the online community. Go to forums, stackoverflow, or github. Share your works, and get feedback or get some peers to help you do the things that you are working on. It’s kind of popular right now, isn’t it?

You don’t have a fixed time at home.

When you are at home, you start working when you feel like working. But there is always a problem to start working. I found that the problem with me is that I am lazy. And the reason of why I’m lazy is because the amount of time that I need to spend working is infinity. Unlike at office, you have a finite working hour everyday. You know that you just need to work until 5 o’clock and after that you are free. And that kind of make you happy a little bit.

Do the same thing at home. Set the amount of time you are willing to spend on your side project. For me, the easiest way to do so is by setting a timer. In fact, I have a timer running right now as I’m writing this. Set it to one hour each, don’t make it too long. Just enough for you to finish a small task.

So many distractions at home.

You got so much freedom at home that you probably won’t start doing anything. That’s certainly not true. I’m supposed to be working on my side project right now, but instead, I’m writing this article. Let it be. It’s something productive anyway. Don’t feel bad about yourself. Do things that you enjoy doing. If the project never got finished, at least you’ve got some fun time doing something else. Don’t to be too harsh on your life. There is time for playing, and there is time for working. When the time for working comes, you will find yourself working.

You don’t directly get paid when working on a side project.

Well, this is probably the hardest part. Money, either directly or indirectly, is probably the greatest motivation of all. It’s really hard to do something unless you are certain that it will give you something back in return. And that something is normally money. But, for personal project, you are normally getting the money at the very later stage of the project. Probably after you’ve spend thousands of hours and bloods (and money).

That’s why you need to swift your motivation a little bit. Start by thinking that money is not what you are aiming for. Let the project be a “just for fun” project without shutting its prospect to shine and become a gold digger. In other words, only do project that you enjoy doing unless you get paid upfront.