Scrum Part 1 – Agile Manifesto

Standard

agile-manifesto

Scrum is an iterative and incremental agile software development methodology for managing product (software) development. We cannot discuss about Scrum without understanding Agile. In fact, many people could not run Scrum successfully because they do not have a clear fundamental understanding about Agile manifesto. When we understand the idea of Agile, all the prescriptive activities / ceremonies in Scrum would make better sense. First let’s look the 4 manifesto for Agile software development:

  1. Individuals and interactions over processes and tools
  2. Working software over comprehensive documentation
  3. Customer collaboration over contract negotiation
  4. Responding to change over following a plan

Individuals and interactions over processes and tools

Individuals and interactions over processes and tools

In development team, Agile encourages team members to interact with each other such as talking face to face, instant messaging, or email communication – instead of heavily depend on project management tool such as Trello, Asana, JIRA, etc. For example, if a team member notice there is a defect in the code written by another team member, he is encouraged to talk the other team member directly (maybe to discuss the root cause and possible solutions) instead of creating a ticket, assign to him in JIRA and call it a day.

joke

When individual interaction take precedence, team members build the bonding to cover each other back. When processes and tools take precedence, team members would only look at their own plate and teamwork rarely happens. Having said that, it does not mean processes are not important. They are important to facilitate the operation but processes and tools should not be the central of development team. It is the team members who are building the product (software), the processes and tools are merely there to facilitate them to get their job done.

 

Working software over comprehensive documentation

working-software-over-comprehensive-documentation

Logically speaking, would you rather to have a working software, or a pile of comprehensive documentation? Theoretically it is best to have both. Unfortunately projects often lacks the most precious commodity – time. If the team has to choose either one, it should always be working software (duh?). First, documentations are often outdated and team members are unconsciously trained to not rely on documentations because the perception is documentations are inaccurate anyway. Second, it is much easier to navigate a working software than to navigate a 200 pages functional specification document. Documentations are often the poor fella sitting at the shared drive that no one looks at. Always choose to build a working software over writing comprehensive documentation when facing the challenge of limited time. After all, what is the purpose of documentation if the software is not working?

Customer collaboration over contract negotiation

Customer collaboration over contract negotiation

This might not be directly relevant to the team members who are actively building the product (software). It is a mindset the business decision makers should have. When business negotiates a contract with customer (regardless internal or external), a set of objective or functional requirement is often agreed upon. However with today’s speed of change in market, circumstances tend to change faster than business could anticipate. It is impractically for business to re-negotiate a new contract whenever a change take place. Both the development team and business must understand that collaboration is what makes great software. Customer has the understanding of the market to describe the requirements; while development team has the skill to transform requirements into features.

Customer-collaboration-over-contract-negotiation-joke

Collaboration with customers is one of the fastest ways to make customers happy. There are some customers from hell but those are exceptions. Generally customers do find greater satisfaction if customers are aware that development team put their business success in their best interested over making money from them with a contract.

Responding to change over following a plan

Responding to change over following a plan

If Plan A doesn’t work, don’t worry, the alphabet has 25 more letters. The ability to respond to change is the core of Agile. Thanks to today’s speed of change, the only constant in the market is change. If internal speed of change is slower than external, the business is doom to fail. So if things tend to change so often, why bother to have a plan? A plan is made base on best knowledge and known facts at that point of time. It provides a guideline to keep team members on track. Team members might learn higher quality knowledge and identify invalid facts along the way. The team would make use of the new knowledge and more accurate facts to deal with the situation more efficiently.

Responding to change over following a plan.joke

Managers should be the leaders to head the change response but ironically it is also the managers who are often very reluctant to change and claim “We have a plan, we need to follow our plan”. This could happen for various reasons and the justification could be valid or otherwise. Not all changes happen for a better outcome however if the team could identify a change is happening (or has happened) and an appropriate response is required, the team should respond by introducing a new plan. There is little value in protecting the ego and refusing to change. Responding to change might be painful in the beginning, but it always yield better return for the project on the long term.

This is a quick introduction on the Agile. On Part 2, we will discuss more on the prescriptive activities in Scrum.

Dependency Injection for Web Service

Standard

Dependency Injection allows developers to cleanly inject a portion of concrete code implementation into the bigger scope of the system base on certain logical condition. Dependency Injection has been a preferred way of implementing code. In fact, ASP.NET 5 (Visual Studio 2015) supports Dependency Injection as 1st class citizen. Dependency Injection removes the need of having tons of if-else statement in code implementation and keep the codes clean. Some developers implement Dependency Injection even there isn’t such need at the moment as a standard practice because they might be needed in the future. Personally I’d prefer not to implement Dependency Injection unless there is a “good reason” for implementing it.

One of the “good reasons” is while designing a web service. Web service often serve multiple clients having various needs base on certain logical condition. It is web service responsibility to handle the various logic implementation. For example a web service is serving clients from various countries on the same endpoint might need to execute different codes to produce localized result depending on who is triggering the end point.

Consider this simple scenario:

We need to design a WCF web service that serve multiple countries on an eCommerce platform. There is an endpoint to accept a Product Id and the endpoint will return the formatted local price. The local price calculation depend on various factors such as tax, shipping, marketing promotion, and other business consideration to lower or higher the price for each country through custom discount mechanism.

requirement

Here is a good scenario to create a WCF web service that implements Dependency Injection to separate different calculation formula for respective country.

In the sample code, the Dependency Injection library that we are using is WCF (C#.NET) with Autofac.

Create a BaseService class. In BaseService class, we define a static container to register and store a list of logic. In this example, IProduct is an interface that get registered with different logic classes (UsProduct, UkProduct, MyProduct) depending on the logical condition.

We create a BaseService class for this purpose so that all the services in WCF could inherit BaseService class and access to BuildContainer method, which will be common among all services. In the following example, all public API method would be required to build the container to initialize the logic classes.

Base service class

Catalog service class

  1. Create a new Catalog service in WCF.
  2. Add a method GetProductPrice. It has UriTemplate as following which means Country and ProductId are parameters to be constructed in the Url endpoint.
  3. Inherits BaseService. Implements IProduct.
  4. BuildContainer is a method defined in parent class.

Define the respective logic class for each country…

United State product logic class

UsProduct class is also the parent class for rest of the country class. For example, the method GetProductBasePriceById is applicable for all countries implementation, hence this method stays at the parent class (UsProduct) so that it could be accessed by the child classes (UkProduct & MyProduct).

United Kingdom product logic class

UkProduct class inherits UsProduct and implements IProduct. GetProductBasePriceById could be accessed by UkProduct as UsProduct is the parent class.

Malaysia product logic class

MyProduct class inherits UsProduct and implements IProduct. GetProductBasePriceById could be accessed by UkProduct as UsProduct is the parent class.

All the product logic classes implement IProduct interface so that they all could be registered into Autofac container builder. (The following code is part of BaseService class shown earlier)

By registering the interface with the appropriate concrete logic class (base on the logical condition of country), Autofac would build a static container that allows the whole application know which concrete logic implementation to call.

If we put this into test using Postman, we would get the following result

Product price for United State

Country is specified by replacing {Country} parameter to “Us”. Result is returned base on UsProduct logic class implementation.

Us-test

Product price for United Kingdom

Country is specified by replacing {Country} parameter to “Uk”. Result is returned base on UkProduct logic class implementation.

Uk-test

Product price for Malaysia

Country is specified by replacing {Country} parameter to “My”. Result is returned base on MyProduct logic class implementation.

My-test

Some consideration…

  1. Performance for building a container and inject dependency on runtime instead of direct initialization of concrete class. From the above 3 examples, we could see each request completed within 15-16ms. I ran a few more test switching between countries, most of the request completed in less than 20ms, which shows there isn’t any major overhead for builder the container.
  2. What are some other reasons to implement Dependency Injection? Answer is unit test. With Dependency Injection in place, the codes would be much testable. (If you currently have codes that is not testable, consider using Shim under Microsoft.Fakes. However this reason is arguable as we do not necessary need Dependency Injection to test our code. All we need is appropriate interfaces in the codes.

One last thought…

Dependency Injection is a clean way of separating codes implementation which conforms to a standard set of Interface. With Dependency Injection developer no longer need to separate which line to execute using complex if-else statement. It makes the code base much cleaner and easier to implement different codes base on logical condition. The drawback is Dependency Injection makes debugging more complicated as the developer need to first figure out which class has been injected during runtime. It is a recommended approach if you have a standard set of interface but requires different codes to be executed base on logical condition.

Multi-Country Address Database Design

Standard

Database design is one of the most critical elements in software development. Realizing that

  1. New programming design patterns and frameworks are introduced from time to time, codes get refactored and system get revamped;
  2. New UI frameworks are introduced from time to time, presentation layer get thrown away and redesigned;
  3. However database design stays the same for as long as possible because business simply do not want to risk valuable production data and development team do not want to deal with legacy codes to accomodate the new design.

A good database design can stand against the agility of business needs. The development team is not expected to frequenty adding or modifying database (for example adding new tables and altering relationship between tables) unless it is “totally necessary”.

While “totally necessary” could be subjective depending on how well a database is designed to handle the changes, a good database design is expected to handle various changes in business requirements without major redesign of the existing database.

Case Study: Handling addresses in multi-country database

I was working for a car classified MNC that have businesses in 3 countries. We were rolling out CRM for Malaysia, then Indonesia and finally Thailand. Each country has a different address format.

Malaysia

[FLOOR] [,][BUILDING]
[STREET_NUMBER,] STREET_TYPE STREET_NAME [HOUSE_NUMBER]
[SECTION] [CITY_DISTRICT|VILLAGE]
POSTAL_CODE LOCALITY
[STATE]
MALAYSIA

Indonesia

[BUILDING]
[SUBBUILDING]
[HOUSE_NUMBER] STREET_NAME
[CITY] [TOWN] [LOCALITY] POSTALCODE
INDONESIA

Thailand

[PLOT] [HOUSE_NUMBER, VILLAGE]
[ROAD]
[SUBDISTRICT, DISTRICT]
[PROVINCE] [POSTAL CODE]
THAILAND

Following Agile principle of Just Enough and Just In Time (and a typical product manager would choose to deal with the other countries in a later time rather than taking the overhead now), development team would design a database schema to cater only for Malaysia user address format.

The following database schema design is efficient to get the job done. Address is the child of Customer. Section, District, Postcode, Locality, Province, and Country are the referencing tables for Address table. Among different location levels, the parent-child relationships have been defined to indicate the hierachy of the location levels. The database is fully normalized and has strong data integrity by defining appropriate foreign keys:

ERD

The development team proceed with loading data to the respective reference tables (Section, District, Locality, Postcode, Province, Country) and launched the CRM for Malaysia users. Soon production data start flooding into Customer table and Address table. Everyone is happy with the system.

2 months later business informed the development team, “Guys, it’s time to let our Indonesia users enjoy our amazing system!”, “Sure!” comes the reply.

When development team looked at Indonesia address format, it is quiet different from Malaysia address format. Leaving AddressLine1, AddressLine2 and CountryId field aside, the remaining fields are different.

Country-Address-Format

The existing database schema cannot fit Indonesia data in any elegant manner due to the parent-child relationship between location level and obviously it is not a good idea to have a Postcode table filled with Malaysia Postcode data, Indonesia Locality data and Thailand Province data (refers to row #3).

Not only Indonesia, even Thailand has a different format compare to Malaysia and Indonesia. The mapping will only get more complicated when more countries join the party.

In order to support Indonesia, the existing database schema need to go through moderate design change to cater for Indonesia address format. Later when system need to support Thailand address, another database schema change would take place.

Instead of continue maintaning a structured table design to map one field in a table to one business entity property, while making changes to support Indonesia (or even better when designing database schema in the first place), a hierarchical relationship could be defined for maintaining the dynamic relationship of address location levels.

Simply put, a hierarchical relationship is self referencing mechanism to dynamically define n-level of parent-child relationship.

The most common usage for hierarchical relationship is defining Employee and Manager relationship. A database could have an Employee table. Within the Employee table there is a ManagerId field and the ManagerId field has a foreign key to the same Employee table Id field. The record could keep referencing within the table itself until hitting a record where his ManagerId is null, which indicate that record is the highest level in the hierarchy. In this case, most likely the record is the CEO record.

To put it into context of Address, take a look at the following database schema design:

hierarchical-relationship

In this design:

  1. Tables are much lesser and relationship is much cleaner
  2. Development team can be assured that this design could handle any address structure in any other countries.

The above database design schema shows:

  1. There is a Customer table.
  2. There is an Address table that could have multiple records for a particular Customer (one to many relationship). The Address table has a foreign key to Location table to tell which is the smallest level of location the Customer belong to.
  3. There is a Location table that has a hierachical relationship with itself. Base on the ParentId field, you can “figure out” the full set of address.

So how do we “figure out” the hierachy?

Let’s take a look at a sample set of data in Location table:

Location

By using the following SQL query with Id = 27:

WITH LocationTree
AS ( SELECT Name, ParentId
FROM Location
WHERE Id = 27
UNION ALL
SELECT Location.Name, Location.ParentId
FROM Location
INNER JOIN LocationTree ON Location.Id = LocationTree.ParentId)
SELECT Name
FROM LocationTree
You would get result as following:
Id-27
This result shows the child parent relationship between the location. Cheras is the parent of 56100 (postcode), Selangor is the parent of Cheras and Malaysia is the parent of Selangor. In address context, Postcode 56100 is inside Cheras, Cheras is inside Selangor and Selangor is inside Malaysia.
The Id = 27 is set in Address table LocationId field to represent the lowest level of location the application could identify for the particular user.
Now if we modify the query to Id = 28, you would get a different set of location hierachy:
Id-28
Not only the hierachical relationship would work for single country, but also other countries with lesser location level. Assuming that we have a customer from Indonesia and the smallest location level we could identify is City, say Id = 22, where the Id = 22 is set in the Address record belongs the Indonesia user. By running the same query, we would get the following:
Id-22
The mechanism for hierachical relationship is simple in the address context. Howeer at the application level, the code need to set the smallest location level into LocationId in Address table. At the application level the input would need to be in the form of selection rather than free text input (else it would defeat the normalized database implementation). For example:
location-selection
Regardless of how many location level we have, or what is the sequence of the location level, a hierachical relationship will allow us to have a normalized yet flexible database schema that is resilient against business changes like supporting more countries.

An application developer that handles the programming codes for multiple countries might complain he could not map the list of address result hierachy into his object property. Imagine he has 4 levels for Malaysia, 3 levels for Indonesia and unknown level for Thailand. How would he design his codes to map what does each level means or represent for each country without having 300 lines of if-else statement? A trick for handling such complexity is through dependancy injection per country. After all, it is the application code job to give context to the data while the database job is to provide storage for the data.

One last thought…

Having a hierachical relationship would increase the flexibility of database schema, but it increases complexity in application codes and would not provide a clear one to one mapping on table field to object property. Such implementation is fundamentally different from how a typical development team would design the Address table. Such complexity is not required if a development team is designing a system to support only 1 country address format. However hierachical relationship should be given consideration if the database needs to handle multiple-country address with various combination of location level relationship.

GitFlow Branching Model

Standard

Code branching could help large development teams to manage code base efficiently and to avoid unnecessary dependency of feature launches. When code branching is done right, team leads and development managers could efficiently manage the development of features and respond to issues reported promptly. However when code branching is poorly designed, not only development team cannot build features, the deployment manager would also not able to make a deployment.

GitFlow Branching Model is a branching strategy not too complicated to comprehend yet it could cover most development teams need.

gitflow branching model

Master Branch

Master branch consists of the updated set of source code as per production. This is the most critical branch and should be guarded against any unreviewed commit into the branch. No one should commit directly into Master Branch. Master Branch should only be used for branching out for production hot fix and not any other purposes. There is one exception during the early phase of the project, Master Branch will be branched out as the base for Development Branch. Once a system has gone live, the primary purpose of Master Branch is to serve as a foundation for any production hot fix. Master Branch is especially useful when there are multiple development teams making multiple production releases within short period of time. Master Branch will serve as the most updated branch for any development team to work on a production hotfix.

Development Branch

Development team branch out a Feature Branch using Development Branch as base. Once a Feature Branch is finished worked on, the code in Feature Branch will be merged into Development Branch. Development Branch could also be seen as an integration branch. If there are multiple develoment teams working on multiple features parallely, here is where multiple features get merged. Any conflicts between feature branches must be resolved and a merged version of the source code in Development Branch is expected to be stable. Ideally, developer should not commit directly into Development Branch, however there could be minor changes to the code base that it would be too much of overhead to create a new Feature Branch for such change, for example an improvement of an utility method or an update of a configuration value.

Feature Branch

Feature Branch is branched out from Development Branch for specific team of developer or individual developer to work on a particular feature. This is the branch where most check in happens for developers. Multiple developers would check in to the same branch if they are working on the same feature. Once a feature is finished worked on, developers would merge their code back into Development Branch. However, prior to merging a Feature Branch into Development Branch, it is highly recommended for developers to merge the latest code in Development Branch into the Feature Branch. If possible, do it from time to time. This process is sometimes called Forward Integration and sometimes called Rebase. The purpose of doing a Forward Integration is to reduce the risk of a big-bang merge directly into Development Branch. Once a Forward Integration is performed in Feature Branch, it would be relatively easy to merge code from Feature Branch into Development Branch. Additional note for Feature Branch is it should not be running for too long. If for whatever reason (such as stakeholders started changing their mind on the requirement after development started), Forward Integration need to take place from time to time to reduce risk on a big-bang merge into Development Branch.

Release Branch

A Release Branch is cut when there is a planned deployment to production. When the deployment manager deploy code into Staging or Production environment, the source code would come from a Release Branch. QA would be testing against the code in Release Branch and developers would fix bugs and check in the fixes into Release Branch directly. On the next deployment to Staging, QA would be testing the latest bug fix from Release Branch until QA is satisfied with the quality of the feature (or the quality of the system as a whole). Then, deployment manager will get the code in the same Release Branch and deploy them into production. Once the deployment to production is completed, the source code in Release Branch need to be merged back into Development Branch for future development and Master Branch for any hot fixes.

Hotfix Branch

A Hotfix Branch is only required when there is a production hotfix that need to take place very quickly and get deployed back into production. A Hotfix Branch is branched out directly from Master Branch to ensure the Hotfix Branch consist all of the latest stable features that have been release to the users. Development team would work on a Hotfix Branch usually in a relatively short period of time, then get the code tested and deploy the code back into production. Once the code is deployed into production, the changes in Hotfix Branch have to be merged into Development Branch for future development and Master Branch for any hot fixes.

Visual Studio (Team Foundation Server)

In .NET development using Visual Studio (TFS), code branching and merging functionality is rather limited compared to the rest of the open-source world hence most .NET developers use simple 1 level branching model.

For GitFlow Branching Model to work in Visual Studio development, the development team need to make a conscious effort to follow the branching and merging guideline.

One last thought…

Code branching is an overhead if the development team is not large enough or the system is not complex enough to require branching. If you have a small team (let’s say 2-3 developers) or a relatively small system, I suggest you simply use a source control for repository purpose and do not over complicate your development process with unnecessary code branching overhead.