A microservice journey - part 4: Reliability

We live in a world which has come to expect everything to just work, all the time.
At the same time we started to move to the cloud?

Cloud is a completely different kind of beast to  self managed infrastucture.  Self managed mean control in all aspects of the infrastructure and total control when it comes to replacing hardware.
In the cloud, a separate organisation is managing infrastructure and for many more clients, not just you. And if a piece of hardware needs changing, they just take it out. Imagine if they had to give a warning to everyone and wait until everyone was ready for the hardware to be replaced. It would never happen!

A Shift in thinking


So we live in a world where people expect everything to be working all the time, yet we are moving to a platform which can be pulled from under us without any notice. Seems kind of counter intuitive. However this is the exact catalyst needed to ensure our application are purposefully build with resilience in mind right from the get go.  Even with this understanding in place, there are still times that solutions get deployed with little or no thought for resilience. We are used to thing being more stable, more inherently resilient.

Observability

The first step to being resilient is to know exactly what is happening in your system.
Along with making stuff resilient you also need to have enough information about what is going on in your solution to be able to understand what is happening and fix it. Observability is a critical aspect to any solution, and every single piece needs maticulous thought about what exception scenarios could happen, and what would that look like in our monitoring tool. You can not find every single problem all the time, but the more you find before you release to production, the less times you will be called out of bed at 3am in the morning to fix that broken something.

Recoverability

When something does do down, how long does it take to restore? Will the alternate stack work? Did you test it?
A common pattern companies try to employ is to have multiple data centres and run Active/Active. Google and AWS offer availability zones. Essencially this is active active, they both offer 3 data centres in a region. For Azure at the time of writing this unfortunately they do not support Availability zones. Azure offer multiple regions only and you have run Active/Active yourself.
But a simple question to ask is, when something goes down, will the system keep running? Does anybody need to do any manual steps to restore services?
If you build a microservices system, manual steps are the enemy. The more microservices, the more manual steps, the longer it takes to restore service.

Auto healing

A good way to make services resiliant to to have them heal them-selves automatically. What does this mean exactly? A service doesn't nessesarily need to restart itself or fix bugs automatically. But due to our reliance on fragile hardware, a self healing system can recover from component outages. Everything needs a back up plan, and a great way to achieve this is to run everything on a pub/sub backbone. The benefit of this model means messages are always on a queue and if a component is temporarily down, the messages will bank up and automatically continue when services are restored. Kuberneties also adds value here as it will natively watch out for unhealthy pods and create new instances which can replace services which are in trouble.




Comments

Post a Comment

Popular posts from this blog

What good looks like!!

Image
I think back to my favourite places to work, and wonder what made that place so memorable? What have I been pursuing every since then. Is it great new tech? is it working on interesting problems and solutions? Is it learning off great team member who think differently?  It is all of the above and more. Over the years I have worked at many places and had some hits and some real misses and realised what I really enjoy is working with people who are nice and make it a pleasure to come to work each day. This doesn't just somehow magically happen, this culture is specifically created by the leaders who foster it. I hear a lot about servant leader or empowering teams, but what does this mean exactly? As a leader, if your focus is on the success of the business. The solution comes first, must be the best, we can not compromise on quality, must be done quickly. You may get a good solution built, but you may also burn through people. If people leave, the vision is diluted and the new team m...
Read more

A microservice journey - part 2: what type of micro service are you?

Once we established we were on the microservice path, the question of just how would that look came up? We had established the problems and understood what we were trying to achieve, but what exactly is a microservice, and how do you go about building it? The importance of the first step can not be understated enough. If everyone understands the problem and really gets what we are trying to achieve, then microservices write themselves. Well not really, but by understanding what we are trying to achieve, the solution options can be assessed around a set of questions which would help drive the decisions being made. Can this service be deployed on its own, without affecting other parts of prod? If we had an influx of users using this feature, can we scale it out independent of the other components? If we did need to scale it out? do we need to scale out any other components as well? What happens if we had a network issue? would it still be able to do its job? If this service did ...
Read more

Validation Rules

Every business application must incorporate some kind of business logic and validate the response received from the user. There are a few ways to incorporate this logic through your application, which involve a mix of client and server side validations. This often means you will have some validation in your UI layer as Field validators and some logic in your business tier, where it is just not possible to bring to the UI such as verifying a date range does not overlap another set, or something which needs to access the database to verify against. The Validation Rules library attempt to unify this disparate approach to validation, and to automate some of the simpler validation logic such as required fields and field length validation. Instead of putting validation through the tiers, we can simply implement IValidation on the business object. This interface attaches a meta list of rules to the entity which we can build in the business layer and will propagate through down to th...
Read more