agile, Career, Product Management, Project

Changing the process from within

Would you agree with the statement below?

If you disagree with how something is done, you should give the process a try and drive changes from within.

Give yourself some time to think before reading the example below.

I was given an opportunity to drive some works that are normally done by Lead Developers/Team Leaders/Tech Leaders etc. I initially rejected it because I didn’t agree with that specific process. It is a discovery process where the engineer lays out options, considers the pros and cons, considers the use cases, proposes a solution, and gets a written agreement from the interested parties. Let’s dissect this process with a real example.

Example: Cross-account Kafka applications in AWS

  • Goal Summary: Source data from an application database in one AWS account to a MSK Kafka Cluster in another account.
  • Options: There are a few layers of decisions to be made. The publishing technologies could be custom publisher, Kafka Connect, generic in-house publisher. Then there is the cross account networking challenges. A proxy, direct connection, clone the DB etc. And then there are language choice, data flow design etc.
  • Decision: Kafka Connect with Network Proxy
  • Decision Maker/Interested Parties/Influencers: Product Manager from team A/Team B, C and X/ Team O, P and Q

This is an extremely over simplified version of the process. It took the team a month to finalise the proposal from trying out different options, to coordinating with different teams and many many discussions. But here is my interpretation of this process: we have a big challenge in front of us and we want to build some certainty before we step too deep into one direction. We also want to get an agreement between the teams that are involved, either in building them, might be using them, or potential maintenance team.

Why I disagree with this process

Retrospectively, I think I am too much of a purist when it comes to software development process. I have this ideal state of how Agile teams should be operating. Trust me, we are an Agile organisation but it is difficult to convey what that means nowadays.

First of all, the process is a waterfall in disguise. If it sounds familiar to you, the documentation that it produces, the contract model, the desire to be certain; they are very similar to the classic software specifications. For the record, there is nothing wrong about waterfall because it is still a great model for certain types of softwares. However, this process is so much leaner than a standard waterfall process. The document is in Git Issue and it was created with the understanding that things will change down the road.

Secondly, the reality is often not as good as the idea set out to be. The structure of the process is such that a single person or two mull the problem for a long period of time before presenting to the stakeholders. Although others are encouraged to challenge and be critical about the options and decision, there are a couple of problems. One being that not everyone will be on the same page or same level of technical understanding of the problem to challenge properly. Often people will just shrug it out and agree with whatever is decided. And then there is the biases of the individuals that will influence the decision.

And then there are two common outcomes at the end of the implementation of the decision. Teams tend to stick to the plan even if the road is bumpy and might not fit perfectly. This is sometimes accompanied with comments like “but we agreed to”, “we have invested in the decision”, and perhaps “the other teams are expecting this to be X”. The second outcome is that it changes and changes and the decision is not relevant anymore but the documentation and comms never really get updated. Although it is a preferred outcome nevertheless, why spend the time in the process anyway?

An iterative approach

Not sure if it s specific to Scrum or maybe it has a specific name to it, but let’s call it Agile for the simplicity sake. We know that the idea of Agile is to build small, try it out, showcase, learn and adapt to changes. Too many buzzwords, let’s use the same example above and see how we can iterate through it.

First we need to break the problem down into smaller more manageable problems. We can do this during refinement where the whole team will be there. Product Managers will focus on representing the customers and work with team to see what values can be delivered in the next week or two. There is no right answer for sure but the first goal can be to build a cross-account connection to first allow us and others to build applications outside of the cluster account. Let’s set that as a goal. The team discuss the options and agree to try on one.

A week of the team getting their hands dirty in the implementations, does it work? can we present something to the customers? If it works, that is great and move on to the next challenge. If it doesn’t, we learn and inform the next decision. A few iterations later, we have a living system that is not necessarily done or perfect but it is delivering value also we know that we are satisfying the requirements because we keep in touch with our customers.

This process does not guarantee the perfect solution at the end either. But what I value most here is the learning. Although we might have to do more actual work than the previous process, we do it as a team and we also learn how to adapt. In the Evolutionary Architecture principle, it is about supporting change and not about making the perfect decision. By involving in the change process, we learn how to change better.

We also build a team with thinkers and decision makers, and not just builders. I believe software development is a creative process and it often is not enough to just build things, but also to create things.

What do you think now

So we were here because my manager thinks that I should give the process a try to be able to understand the value from the inside and also drive changes if needed to. Do you agree?

For me, I am in a protest. I think that the process is against my ideals and I would not just do it so I can progress in the career ladder. This is definitely not looking good for my career progression but hey! it is about the journey, not just the destinations. I am enjoying my role and I am still learning and growing where I am as a senior developer. Although we know that there are pressure in climbing the ladder (perhaps a story for another time), I am not in a hurry to get anywhere just yet.

Standard