The importance of business value tools in medium and large-scale E-Commerce projects

image from: https://unsplash.com/photos/lRoX0shwjUQ

I wrote this article some years ago (2015) when I was working at VentureOak. Today I was having a brainstorming session and this topic came out! I thought, let’s bring it here and share it with my followers.

Big e-commerce projects can bring big problems to business value, especially related with databases. With the growth of an e-commerce project it is certain that you will have a database and, apart of technical point of view, it’s important that the reality of the business matches with the reality stored in the database.

Bugs and features can create in our “business reality” some incorrect “state machines”, which until discovered, will provide, to reading processes, a wrong view about business. So, since the first time that a functionality started to save bad data in the database, your project will work with that information.

After getting some “business data” problems, project team has to proceed with its fix. That is possible by solving problem at root cause, but team can’t forget that database needs to be fixed too accordingly the reality. This database tasks can be done securely with routines fixing each record of each affected entity, or even directly at database file assuming at our own the risk. At the end both ways are a “data fix”.

In a short term both presented solutions can be acceptable, but not at medium or large term. Business value of a project is directly threatened by inconsistent databases because the stored reality is not reliable. Also, it’s too expensive to have a unique person monitoring all databases, searching for this kind of problems at the database and reporting wrong situations.

This problem made me think about a way to help development team in a business value point of view. The solution must be around a tool, which apply validations around resulting data and creates readable files notifying all bad cases for all functionalities of our project. I’ll investigate a several possibilities and explain which can be the best to solve this threat to business value.

The main motivation to think about a big problem.

Databases as a threat to the business value of a project

How to protect Business Value from Project IT Team point of view

Technical factors such as scalability, used technologies and the production process are in a direct way, the impulsion channel of the application business value. Through them, the customer evaluates the application / product on its value and trust.

In the production process, regardless of the development teams, projects are provided with elements that ensure the successful integration of new features and bug fixes, including QA and “testers”. Whether in the development’s progress, either in controlled “staging” environment, there are manual and automatic tests searching failures both technical and/or business side. With this is not intended to exclude the use of unit testing techniques, which directly affect the outcome of programmers, and which have proven a huge advantage in the final product quality / functionality.

With regard to human factors, the team management, the planning of new features of the project and the treatment of application problems lead, in a roundabout way, to the project business value. The interaction between the various elements of the teams, regardless of their duties, is equally important. It is obvious that the higher the “know how” of each individual and the greater its interaction with the other, the better the analysis made by the managers. On the other hand, the more effective the integration planning for a given need, the greater the business value, which grows securely.

Considering an e-commerce project, logistics, management or others, when it assumes a medium or large scale, it is common that, after a certain period of time, it happens collisions between the actions of features, or even between the own business rules. Usually, the planning team along the customer take the necessary decisions to resolve the collision. The time required to solve the problem, since the analysis and decision making, to the development and implementation, can be enough to generate wrong information in existing databases, plus the impossibility of disable that application area as a way of prevention. In a situation more severe, the application database can even get wrong business logic for lack of knowledge of integrated development anomalies, until they are identified.

The process

We are then in front of a clear threat to the application’s business value. The database represents the business reality from a virtual perspective, and whether it is misleading, data inconsistency problems could cause the project failure. Other complications can arise when we deal with inconsistent data: see the example of an e-commerce store that presents stock of non-existent products, physically, in the warehouse.

As a security measure, it can be used business intelligence elements. These elements, using data warehouses, allow the strategic management layer to be provided with treated and extracted information in readable formats of what is the reality of the business. The problem here is that the tasks are performed, mostly, in order to help increase the Business Value through reports and KPI indicators of how the application is returning the investment.

It becomes evident the need for an active monitoring of the generated results by the application. The data inconsistency, at the time of reading, whether for purposes of planning, whether for project consumption, directly puts into question the value that the business can achieve, under the penalty of imminent failure.

Short-term Solutions

After appearing some specific situations and, as already described, proceed with the identification of all possible cases, it comes the data fixing tasks. These tasks can be developed in a reusable way, through recurrent and localized procedures to a specific data range, or through unique routines for each case. It is up to project management layer to decide if this approach is the most appropriate. This decision may be based on a deep analysis of the process model and used database, to establish the current level of trust in the application. However, those actions should exist only on a short-term level. The allocation of a resource for this kind of tasks affects the development of new features, especially if they are related to the data errors found.

Another kind of action can be through the cooperation of B.I. teams. They provide reports tailored to each validation to be performed, which may even create a quantitative error estimations associated to its calculations. Again, in the medium and long term, this approach can be problematic, because it is subject to the whole process and workflow implemented in extraction techniques and data processing.

Suggested Solution

Considering a development environment like “local — staging — production”, the tool structure might start with some basics:

  1. At “local”, the database should be always a recent snapshot: analyzing the generated information means that the database should have always the most current data. Thus, according to an appropriate schedule, there should be a copy of the database to a development environment (“local”), where the tasks can be performed regardless of the applied cost performance without the production environment be affected in the analysis operations. Considering we are dealing with an online store, it is not correct that order data is validated when users trigger the creation of more records.
  2. The data integrity component should not be “deployed” to “production” or even to “staging”: depending on the entire tree of elements and positions organization of a project, it makes sense that the component is available only in “local “even if shared with the entire IT team.
  3. The tool should contain a “user-friendly” GUI: the interaction with this tool should have a graphical interface. Any employee of the team should be able to run new checks and get their results.
  4. The tool should be multitasking: the tasks concurrency is a necessity, processing at the same time different user requests, data validation and result reports.

Upon getting done the basic principles, it will be the development team set the other needs to implement this data integrity component.

After all the analysis, requirements and needs are clear, the choice of technology will be required to support the application needs. I think the best step is from a development framework that provides the largest number of features, so the focus centers on the construction of the validation tool, not ignoring factors such as performance, cross-platform, interactive media and the language in which it is built. In the proposed solution were considered frameworks like Laravel (php), Node.js (javascript), Zend Framework2 (php) and Sails.js (javascript). Laravel framework was the choice, considering its performance (it is a very lightweight framework), the ability to interact with Queues (possibility of managing tasks performed in the background, either locally, or in connection with API Queue services) and the quite short learning curve.

The approach about the development of data integrity component started from the implementation of an “engine” of job management, successfully implemented, using the setup of a “Beanstalkd” queue system and Laravel Queue feature. This way it was possible to keep active processes on a webserver that are listening for new instructions created by the use of the tool.

After finishing the implementation of basic communication between the application and the queuing system, it was implemented the jobs structure that asynchronously, query the database and processes the information in accordance with procedures defined according to business logic. Giving no details about the technical implementation, that actually focus on the entire structure of classes and design patterns used to implement, the tool proved to be able to gather and analyze, with the databases, records that should be analyzed (by selecting a specific type of job and receiving the respective time interval attributes from user inputs), and therefore apply all the necessary validation rules defined in configuration files.

Up to this point, the project is provided with a tool that validates information, denouncing all the wrong cases and likely the reason for the respective “state machine” that define it wrong. Considering an example of a logistics management system, in the case of an order tagged as delivered, but that the sold item is stored in the information system as “in stock”, this error will be recorded.

Pros and cons of proposed solution

However, there were negative aspects in the implementation. First, and with greater impact, with the implementation of the validations in multiple databases, the need for storage has become a problem. To accomplish the basic principle of maintaining a copy of a local environment (even a server on the IT network side), when analysed various databases and also the fact that its size increased at a great rate, it became necessary investment in support equipment.

Another point to highlight is the availability of the IT team to allocate resources for maintaining analysis tool. The technical aspects at infrastructure level, as well as the business value aspects regarding the maintenance of validation rules, and the creation of new validation tasks, can become costly for the project.

Conclusion

The project value is determined by all components adjacent to itself. The contribution of all staff members and all the tools available, make this value vary.

The possibility of developing a tool, under the Business Value scope, is reflected in a direct view of reality that is the operational system, and how it behaves. Being able to find the flaws, getting alerts, or even create sufficient maturity for automatic resolution of these conflicts, helps a lot in the reduction of threats that, in an extreme scenario, can cause injury at many levels to the business.

When the concern of the whole team is focused on the development and implementation of new features and the resolution of performance issues and errors, the concern for the business value can be lost. The technical vision of a project must also be proportional to the business value vision, not forgetting that should be shared and accepted the importance of each action by all project so after that it will grow itself and grow its Business Value.

Software Engineering Manager and Software Engineer | Server Side Trainer | Human stuff as a hobby.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store