Category: Articles

Kubernetes basics for Docker users

The aim of this article is to build a high-level mind-map and understanding of concepts like Kubernetes, Helm and cloud-native applications. The assumption is that you have worked already with Docker.

Docker

So you know Docker. Docker container is like a virtual machine but lighter. Why lighter? It does not have operating system inside. It relies on host operating system. Docker adds only applications layer on top. You can run many Docker containers in a single server / virtual machine.

Docker Compose

You may have also heard about Docker Compose. For example when your application consists of .NET Core web server, MySQL database, and Redis cache – you can define 3 separate containers for it. To run all of them in a virtual network you define a docker-compose.yml file. Then 3 of them can be run with a single docker-compose up command.

Scaling the application for production

Now let’s imagine you want to scale your application. You introduce a load balancer and 2 additional web server containers. You are also adding a RabbitMq container and an instance of a background processing worker. There are also other requirements for production environment:

  •  containers need to be distributed across many servers
  • containers which do not need much resources can be run together in same server to use provisioned servers in a cost-effective way
  • when a container is not responding it should be restarted
  • when connectivity with container is lost it should be replaced with a new instance
  • number of containers should autoscale
  • number of servers should autoscale
  • new containers added to this environment should be auto-discovered
  • it should be possible to mount and share storage volumes in a flexible way

Things are getting complicated. To achieve all that requirements we must write a lot of code to monitor and manage infrastructure. This is called containers orchestration. Or… we can use Kubernetes (k8s) which has all that features and more built-in!

Kubernetes concepts

Pod

Abstraction of a single app. It can have one ore more containers. If containers are tightly coupled they may be placed in same pod. All containers inside a pod share storage volumes. A pod is an unit of deployment and scalability. Each pod has IP address assigned, so there is no need to care about port conflicts.

Node

This is how k8s names physical servers or virtual machines hosting the containers.

Cluster

Set of nodes available for k8s. Example cluster would be 4 nodes and 20 pods running on them, managed dynamically by k8s.

Namespace

All objects within a cluster can have a namespace. It allows to create many virtual clusters inside single cluster. It is useful for example to model many independent environments for staging in a single k8s environment.

Service

Since each pod has it’s own IP and pods can be started and shut down, it would be not easy for other pods to keep track of constantly changing IPs. That’s why we have Services in k8s. Service groups a set of pods by given labels. Pods may come and go, but as long as labels criteria are matching, all matching pods are automatically tracked. Service has a logical name assigned, so that other pods can use this name to communicate with the pods behind the service. Service routes and load-balances the traffic dynamically to relevant pods.

Services can be also used to point traffic to an endpoint outside k8s cluster. In this case instead of defining a service by providing pod labels selector, it is necessary to define an IP of the service backend.

Ingress

Ingress is used to expose services to outside word via http(s). It can also terminate SSL.

Deployment

Deployment specifies the pod and the number of its replicas that should be run. Deployments controller is responsible for rolling out updated pods (e.g. with updated container image). It starts new pods, shuts down old pods and then keeps monitoring them to make sure that desired number of replicas is run.

StatefulSet

StatefulSets are used to manage containers which contain data. Containers that have data cannot be just removed and replaced as we cannot loose its data. Pods in StatefulSets have sticky identities and persistence storage assigned. Persistent storage is not deleted when pod is deleted.

Worth to mention that in many scenarios managing persistence would be simpler outside Kubernetes cluster. Many cloud providers have sql an noSql as-a-service offerings which usually takes care about things like backups, availability and replication.

Monitoring containers

Each container has a liveness and readiness probe defined. Typically those are HTTP endpoints called by Kubernetes to check if container is healthy. K8s calls the endpoints periodically, e.g. every 10 seconds depending on configuration. When liveness probe fails container is restarted. When readiness probe fails traffic is not anymore routed to this instance. Health check endpoints must be implemented in every service. Simple liveness endpoint could be just returning status code 200. Readiness endpoint could additionally check things like database connection, cache readiness or amount of currently used resources to check if service is really ready to process new requests. When readiness endpoint detects a problem that could be solved by restarting the container, it could potentially switch a variable to force liveness endpoint to fail causing a restart.

Helm

An application targeting Kubernetes is configured in a set of yaml files for deployments, services etc. Those sets of yaml files can grow pretty complex. We also need some versioning tool for them. A common approach is to package all k8s files into a Helm package called a chart. What is being deployed to k8s cluster is a chart.

Cloud native applications

Kubernetes is an operating system for cloud native applications. Cloud native applications are usually designed in microservices architecture and aim to be cloud-agnostic – can run in many public clouds or in a private/hybrid cloud. There is already a lot of predefined Helm charts available in public repositories if you’d like to pull scalable k8s setup e.g. for Cassandra or Prometheus in your cloud-native setup.

Sources

What matters in creating software?

What are the most important things when building software? Is it architecture? Frameworks? Process? No. Those are only tools.

What really matters is the fact of delivering useful solution. Usefulness defined in terms of number of users or revenue. Useful solutions are not delivered too late or too early – they are delivered on time. Useful solutions are not too slow or too buggy – they have acceptable performance and are reliable.

What also matters is the satisfaction of the team after building the solution. It could be that kind of satisfaction which painter has when working on a painting – satisfaction during the process. There can be also some pain in the process but that would be another type of satisfaction – the one that athlete has after finishing a marathon. It was painful but feels so good after the finish. It could be also like a satisfaction of a cook who has just discovered new spices and learned something useful for the future.

Studying architecture or frameworks is also important, because it makes us efficient and more likely to deliver useful solutions. But the goal is not to learn, the goal is to apply the knowledge and make things happen.

Employee development plan

I was recently participating in managers workshop at my current company Mash. One of the learnings was about employee development plans. Please find below the key points that in my opinion should be always considered when crating development plans.

Not everyone needs a development plan

Many employees may feel good at they current position. It is not always appropriate to push for more. People may want to specialize at their current role and may be doing a great job in this role. Managers should not be pushing for “next” levels. Especially if it requires to change focus to other things. Not everyone who is a great engineer would be a great team leader or great designer. Adding new responsibilities may affect ability to continue the great work that given person is currently doing.

There are moments in career that people may actually want to change something and this is a proper moment to have a development plan which would support employee in reaching next personal goals. But it most cases it is far more important for employee to be able to fully engage in the current role. Development plans should be not a replacement for that.

Where the development happens?

If everything is working well in our workplace, here is where learning happens:

  • 70% of the learning comes from everyday tasks
  • 20% from colleagues (peers or mentors/leaders)
  • 10% from external sources like formal training, conferences or books – that are  great ways to get some initial knowledge but what makes an expert is practice and solving real problems, not reading articles.

This division should be also considered when designing development plan. Employee may spend many days at conferences but if everyday work is not giving opportunities to develop and use that knowledge – then development activities are far from being effective.

Who needs a development plan?

When preparing development plans, first think about poor performers. They may need extra attention to help them reaching expectations. Employees may perform below expectations because they did not have enough internal training and knowledge sharing to be able to carry on assigned tasks. If all the necessary knowledge sharing was done and there is still no improvement, then employee improvement plan should be discussed instead of development plan.

Also remember about your “rocks stars”. They usually also need extra attention to make sure that they have enough challenges to grow even bigger.

The most important is to listen to your team members to know what they really need. Development plans should be an outcome of real needs and expectations. As all other things in teamwork – it should not be based on assumptions.

DNA – week one

What is DNA?

DNA comes from Polish name “Droga Nowoczesnego Architekta” – it means: “The road of modern architect”. English abbreviation would be “TROMA”. This one of very little examples where Polish is simpler than English 😉

This is a 19-weeks course crated by 3 experienced architects who are also trainers. They are supported by popular blogger Maciej Aniserowicz who is the publisher of the course. He made the program well known thanks to the broad IT audience in Poland following his devstyle.pl blog and social media channels.

Course is dedicated to senior software developers and architects. The goal is to propagate modern patterns in software development. Presented theory is backed up by cases from real-life. It is meant to be the “killing feature” of the course. In addition to that there are also practical exercises after each week. The idea is  that participants get not only theory but also examples from real life and exercises to practice.

Visit droga.dev to learn more about DNA.

First impressions

I was following more ore less what was published by DNA mentors as “teasers” in recent weeks before full program became available. I was prepared to expect good content and I was not disappointed. The content is solid and also the way of presenting it is fully professional.

Good content was not a surprise for me after buying access to the course. I believed that those guys will do great stuff. I knew what I am buying. When you go to good restaurant you expect to get good food, no excitement here. But there was one thing that was a bonus that I was not expecting to be so meaningful: the community.

Joining DNA community on Slack was pure fresh excitement. This is a closed group for all course members. Joining this group was like joining tens of new teams in different companies in one day. People share and discuss how they approach various topics in their projects. This is great, tons of information and different points of view! You can read about an useful tool, online resource or interesting approach to specific problem. I expect that the value of content generated by the community may exceed the value of the course content itself as the time goes. Of course it’s Slack so no structure is present here and it will be impossible to find anything after couple of weeks 😉 But among all possible distractions you can have turned on, this Slack community is the beneficial one and can really broaden your mind

Eye opener #1

I like to search for analogies between constructions and software industry. In general there is a lot of analogy, but one of them seemed harmful to me: software developer is a creative role whereas constructions developer most often is doing a repetitive job based on detailed project. DNA explained that discrepancy – software developer should not be treated as construction worker but as constructor.

Developers write code which is a kind of design. In software the role of worker is taken by compiler and build pipeline. Having code written, we can then start many instances of running program created almost automatically. How beautiful it would be if we had “compilers” for building construction projects that would automatically execute design documents to create a real building?

Disclaimer to all construction workers who read that: if you don’t have specs from architect/designer/constructor, then you are also a creative worker creating something out of nothing 🙂

Looking forward for more eye openers when continuing with the course 🙂

Divide, eliminate and conquer

Ancient Romans were saying “divide and conquer”. I really like this rule both in system design and project management. To adjust it to contemporary times which are full of distractions I would also add “eliminate” phase. It helps to avoid conquering false targets and allows to focus only on biggest value goals.

Please find below some examples of what this rule means in practice.

Divide

  • Split complex problem into a number of smaller problems
  • Isolate application modules
  • Assign well defined responsibilities to teams and individuals
  • Break up project into phases
  • Define sub-tasks for stories to implement
  • Set clear SMART goals

Eliminate

  • Identify problems that can be ignored
  • Reduce scope
  • Remove distractions
  • Ask “why?”
  • Focus on goals, not procedures

Conquer

  • Execute goals with deep believe
  • Don’t give up
  • Improve
  • Know when is enough
  • Celebrate success

“Divide, eliminate and conquer” approach often helped me when I was feeling overwhelmed by a difficult problem or had to choose which path to go. I believe it is useful in wide range of situations starting from fulfilling New Year’s resolutions up to setting and executing enterprise strategy.

Alternative to traditional daily meeting

Most of the dev teams have daily meetings in the calendar. Daily meeting is a good opportunity to share with other team members what’s in our head currently. You may get help, recommendations or share your knowledge and  useful discoveries.

Daily meetings is also a good way to build the team spirit. Especially for distributed teams it is often the only chance during workday to see each others via video and hear each others voices.

But when following traditional meeting structure, we often can notice that team members are almost everyday saying something like: “Yesterday I was working, today I will be working, everything is OK”. Status is visible on board, others can see the contributions, why to waste time on going into details? Sometimes it’s indeed challenging and even uncomfortable for team members to say something more.

If this is your situation and dailies start to look as a waste of time and being uncomfortable for the team, your team may try a different approach: going through items in current sprint/board instead of going through people. You can find an interesting discussion about this approach here.

You may also mix both approaches by going by items for example on every 2nd day. The most important thing is continuous improvement and trying what works best for your team.

Pros and cons of microservices architecture

Cons

Microservices bring many problems that do not exist in monolithic applications or are much easier to solve. What challenges will you have?

  • Data consistency is difficult to achieve. Distributed transactions result in poor performance. The trade-off is eventual consistency but asynchronous communications is often not easy to accept
  • Error handling becomes more complicated because of distributed nature of the system. Failure in one service may require to trigger undo operation in another service
  • Synchronous dependencies between services reduce performance and availability. Calling service waits until called service will do its stuff. In case of synchronous communication SLA of the system is multiplication of its micro services (0.99 SLA in service one x 0.99 SLA in service 2 = 0.9801 SLA of the system composed of 2 services)
  • Remote calls between services are over network, usually HTTP, so data serialization is required what reduces performance
  • Bigger effort must be put into integration tests
  • Fault tolerance needs a lot of attention
  • Fully automated build and release pipelines are a must because of large number of deployment units. Manual management would be too time-consuming and too error-prone
  • Application monitoring, metrics, alerts and self-healing mechanisms must be implemented to successfully manage production environments. It is absolutely counter-productive to manually search logs in tens or hundreds of running application nodes

Pros

Why to take the burden then? When it is beneficial to choose micro-services over monolithic app?

  • Divide and conquer principle. System split into smaller highly independent parts is easier to manage and maintain. It is like splitting large complicated problem into many smaller and easier problems
  • Teams can be smaller and work more independently. Ownership is bigger because teams feel responsible for their parts. In large monolithic app responsibility is often diffused
  • Failures affect smaller parts of the system, other components may be still functional
  • Each system part can be planned, developed and deployed quite independenty.
  • Different parts of the system may use different technology what allows to update technology stack more often and use different tools for different problems
  • Each part of the system can scale independently. It is easier to identify bottlenecks
  • With mature dev ops culture it is easy to spin up a new service and provision new computing resources. It is often much more effective and less risky than modifying existing service
  • Microservices can be a medicine pill for tired teams. Sometimes teams just need to start from the scratch, leave old buggy system aside for a while and do something smaller that they can be proud of and have fun with

Summary

Microservices are technically harder to develop but in some situations it pays off to take that extra effort. This architecture would be a more natural direction for larger organizations with proven business model and mature teams who are committed to devops culture. For smaller products and start-ups which are still discovering their business model the extra burden brought with microservices may not help but be counter-productive.

What drives architecture decisions?

There is no single best architecture that fits all problems. Architecture decisions must be based on business and technical constraints. We often call those constraints as “non-functional requirements”.

What are the example drivers?

  • time – when project has to be delivered? is there a deadline like upcoming changes in law, scheduled event or a business opportunity that exist only for a while?
  • scope – how much there is to be done? is it a small project or long-term investment?
  • skills in the team – what programming languages, frameworks and tools the team is most productive with?
  • scalability – how many users will be using the software? how fast the number of users will grow? will there be any peaks in traffic?
  • performance – are the operations happening in real-time or can be done in background with acceptable delay?
  • security – is there a risk of data breach or system does not deal with private data?
  • maintainability – will the system be developed in long-term? will it be evolving and changing often? should regression tests be automated or it is acceptable to perform manual tests once and codebase will be frozen afterwards?
  • availability – what happens in case of system failure? is the plane crashing so that system has to be up 100% of time, or maybe it is a background data integration and it is acceptable when system will be down for a couple of hours?
  • business process automation – is it required to automate all the steps and cases in the process or it is acceptable to leave some manual work to be done for example by customer care department
  • platform – on which devices will the software be used? Smartphones, laboratory devices, cars, servers, watches, payment terminals or many of them?
  • usability – is it a B2C/C2C software intended to be used by end-consumers or only trained stuff will be using it?
  • already possessed resources – do employees already work on MacBooks and all DevOps tools are built for AWS? Then .NET 4.7 and Azure are not the best choice
  • budget – Last but not least. Are we investing in a product that already brings revenue? How much will we save or earn thanks to that project? Is it a new business idea to validate at lowest possible cost?

It is important to list constraints at the beginning of the project and make it transparent for both stakeholders and development team. Having common awareness of constraints allows to make more confident, accurate and quicker decisions.

It is also important to note that drivers may change when business environment will change. Good architecture will adopt without putting too much effort to prepare for that at early stage. Example: you may built the search functionality based on simple SQL “like” query, but abstract the search functionality so that it is decoupled and will be possible to be replaced by more scalable technology in the future without changing other parts of the system.

When the system has good logical structure it will be much easier to adopt it in the future for larger amount of users, even if the architecture at the beginning is a simple monolithic app. But if the logic is mixed-up since the beginning and code is a mess, no matter what fancy architecture and tools will be used – the project will fail.

Summary

  • Know and communicate your drivers.
  • Estimate how likely it is for each driver to change
  • Have a high-level plan of adopting to likely changes

That rules lead to better decisions when building software.

A story about running a software house

The beginning

In 2013 I’ve co-founded Epicode software house. Before starting the company we were doing after-hours projects with my business partner. He was handling sales and design. I was handling development. This is also how we split the duties in Epicode. He became CEO and I was CTO. At the beginning, it was quite an easy role in a company of 2 employees.

Working as we liked

We would probably not start Epicode as a full-time job if not the big customer open for cooperation: Grupa Pracuj where we both worked before. Grupa Pracuj was just starting a new startup: emplo.com and needed to build a development team for it. They become our biggest customer and shareholder. It was a good deal. We had financial security and independence. We could do the software business as we liked. Our customers including Grupa Pracuj cared only about results.

Quick start and early success

I’ve relocated back from Ireland to Poland to start working full time at Epicode. In July 2013 there was me, my partner, emplo.com project starting and 3 other small ecommerce customers we had as a result of our part-time moonlight work before Epicode. Good start to grow. Soon we started hiring and rented an office. After 2 years we were over 20 people: developers, testers and designers.

Growing challenges

Next 3 years has shown us exactly the challenges of growing company over 30 employees. For that you need to delegate and to have middle-management. We were not ready for that. As local “superheros” we felt that we must be directly involved in every project to be sure that it will succeed. We were focusing on project-related work, not on growing the company. There was also a constant feeling that if we hire more people we are not sure if we will manage to get enough customers. On the one hand we always had a big workload, but we were never sure if this workload is stable enough to hire more people.

When running a software house, the biggest challenge is to have proper balance between new projects coming-in and number of hired people. The only cost that matters are salaries. If you hire too many people you can quickly produce lost.  Crucial part is to have predictable sales pipeline so that you can plan at least couple of months ahead.

Learning sales

We did an effort to hire a sales person but it has only shown how unprepared we were to scale. That person reached out to over 500 potential customers but didn’t even manage to setup a single meeting. It turned out that sales person did not understand what we were really selling. For us it was obvious: we were listening what problems customer wants to solve and then we were proposing how we would approach that by providing appropriate software solution. It usually worked. Our sales meetings were not about sales, it was about free consultancy after which the customer simply wanted to work with us. But this approach did not scale without proper staff training, marketing content and well-prepared case studies. With an ad-hoc sales person concentrated solely on rates that we offer and deadlines that we can meet, the effect was terrible. We were treated like spammers.

Goal bigger than sales – do what you like doing

But the reality was that we were not convinced if we would really like to focus on marketing, sales and growing the company.  Once we have got a project we were sinking into delivering it, forgetting about searching the next thing to do after current project will be done. When I was working on non-technical stuff I had a feeling of loosing time and energy on something that is not really my pair of shoes. Using social media for promotion? Blah… We are hackers, developers, creators, not marketers. If somebody does not want to work with us, it’s them who should regret. I was not realising how arrogant that way of thinking was.

Our company started to drift into games direction and developing our own game. Many people we had onboard wanted to work on own product which would give much more freedom on how to work. Not to relay solely on B2B sales, contracts and timelines, especially that we were not mastering that processes. Developing a game had a chance to become a high-margin business – with higher risk but also with higher gratitude if it will turn out to be successful.

But for me personally it was not something I wanted to do. I was excited about solving real-life problems that companies or end-users have. I didn’t want to close myself in a dark room coding a virtual world. My preference was working with people and business processes.

There is no progress without change

After 5 years at Epicode I decided to go my own path. We had great time at Epicode with plenty of success stories and delivered projects. But I needed a fresh context and new fuel. Now I am excited to be a part of scale-up process at Mash.com.  Scaling is something that failed for us in Epicode, that’s why I am so enthusiastic about being a part of it at Mash.