There are many benefits that AWS services have to offer, including solutions that simplify the process of extracting data from different sources and injecting them into a new destination. The event bus architectural pattern, one of many architectural patterns, can be divided into three major components: The event source, the event listener and the bus channel. 

The event source is where the information will come from, and this will normally trigger any process that should start. The event listener is tightly integrated with the event source and is going to be in charge of processing whatever information comes from the event source. Lastly, the bus channel is in charge of transferring information between multiple parties, where there can be various implementations adopting different data structures. 

An ETL, or “extract transform and load,” is a procedure of copying data from one or more sources into a destination system that represents the data differently from the source. Extracting information could involve multiple steps like encoding, sorting, doing aggregations, changing formats and combining information. The information could also come from different sources, which means we can combine different options like microservices, text files, databases and much more.  This can be done using tools like SQL Server integration services or C-data sync. 

As an example, let’s imagine working on a complete project rewrite of an existing system with multiple data sources like web services, XML feeds or text files. It’s important that the data being used for development is as close to the original type as possible, as it will provide a better picture of how the application is functioning in comparison to the original.

As is the industry standard, this project would be a phased-approach release, and an agile methodology like scrum would be implemented in the development process. This would allow for a better process of pulling information from the source, as needed, and means that the first couple of sprints would let the team focus on only a couple of modules at a time.

An advantage of creating a process like the one described above is that this structure helps to manipulate data for many different purposes. One of the main core concepts of an ETL is to transform information, so if there is a limitation on what information the developers can see, you can use this process to change some values for testing purposes.

Now that you know what kind of environment you will work in, it’s time to design the ETL to pull information. This can be accomplished by selecting the trigger that will execute the entire process, which could be an action done by a user or cron expression. Next, you will define what is going to be the route and which stops this route will have.  A stop can be a new data source from which to pull information, that means new passengers in the bus, or a destination data source, where the information will be dropped. You will then need to define what will be the “passengers,”  meaning which data points will be fetched at the starting point, which ones will be dropped at a specific stop, or if new passengers will be picked up along the way.

The final step is more oriented toward the design of the system, and important considerations that should be taken care of in this process. These include:

• Data integrity: It’s important to map what data points will need to be processed first in order to maintain a sequence of events. The whole process can be understood as a database transaction. With each step/stop, we assume that the previous ones have been executed successfully, but then if a difficult problem occurs, we get an exception and need to be able to handle that gracefully. There are multiple strategies, ranging from a complex rollback to a simple retry policy.
  
• Detailed logging:  Since you are moving to a serverless architecture with microservices, it’s important to have logging information that can provide as many meaningful details that will help developers to debug any problem in the future. And here is where Cloudwatch steps into since that’s the service that will become one of your allies to solve any problem.
  
• Concurrency problems: This is possibly the most difficult part of the whole process because you will need to think ahead of which scenarios can happen where the same resource can be accessed by multiple data points or multiple stops within the execution. For instance, if we have a microservice that uploads images to S3 and serves them to a CDN-like cloud front, it might be possible that invalidations will be required where there are special limits.
  
• Access restrictions: This is a general consideration in every software development process, but it is important to validate that the system has the right permissions to access all of the information that is needed.

Now that the ETL process has been defined, here is what AWS provides for us through three popular services:

• The first service that we will highlight is SQS, which is a message queuing service that enables you to decouple and scale microservices, distributed systems and serverless applications.
• The second service is AWS Lambda,  which lets you run code without provisioning or managing servers. This means that you pay only for the compute time you consume.
• The third and final service is Cloudwatch, which provides you with data and actionable insights to monitor your applications, optimize resources and get a unified view of the overall health of your systems.

Now, combining the concept of a stop that we described before with the services above, this will be the composition of a stop.

To recap, you will have the data coming from the event message queue, then the code handler that will be in charge of processing the data and finally, you will have all the outside resources that there will be interaction with. These could be the databases where we put information, or the different sources from where we pull information, including microservices.

This entire process helps to facilitate the interaction of the data between all of the microservices by having a mapped route of how the information is managed, and then there is a clear understanding of how to interact with the services between them. 

AWS provides a variety of services that can help to satisfy various requirements, in this case, leveraging their flexibility and scalability with their services to create a system that speeds up the process to extract information for your projects. 

When implementing changes to a monolithic architecture, you can uncover unexpected behaviors. For instance, modifying a portion of code in one class can cause another to fail. The number and intensity of these unexpected behaviors will depend on how well the monolith was created in the first place.

While maintaining a monolithic architecture you should not only be concerned with performance but you should also be concerned with the monolith’s ability to scale up or down. In addition, training staff on how a monolith is architected and where to make changes properly takes time, slowing down the team’s velocity in order to build new features. Even if the team understands the complete application and can implement a minor fix without affecting anything else, an entire deployment is required and can delay deployment times.

Approaches for Moving to Microservices

Moving to microservices provides faster scalability, as having more minor, smaller services means they will be quicker to start. Team productivity will increase as a result of limiting the context of each service to a more narrow set of business rules. There could be an improvement in performance; however, this also relies on how well the services are implemented (if you have poor code, you will have poor performance). Finally, a hidden advantage is that you are not limited in the languages or technologies that you can use. For example, let’s say you have a service in .NET and another in Python. If both use JSON to return information, then they can communicate with each other or with the front ends.

There are a number of approaches that can be used to transform a monolith to microservices:

The big bang release – Getting rid of the monolith and starting from scratch. 

Splitting the development – This means that there will be new microservices living together with the existing monolith. The idea is to work on actively developed applications so everything that is a new feature, as long as it can be accomplished technically and move the required dependencies, will be extracted to a new microservice.

API proxy or substitute layer – Creating a middle layer that will receive all the calls. This layer can aggregate information from multiple services before responding, or even just pass through the request but log some details.

Divide and conquer approach – Used when combining “macroservices” with “miniservices.” Conceptually, they are two different services; macroservices can be as big as a monolith, while miniservices are focused on a single feature or goal. 

Strategies to Handle the Challenges Along the Way

There are some challenges that may be encountered during the transformation process, but there are strategies that can help to overcome these obstacles. 

Lack of Training

The first issue that can come up occurs when there is not enough proper training, either because the business did not provide a properly analyzed business model or the developers did not follow the proper rules when working with a decoupled approach. If, in the end, you maintain the same database (and perhaps add additional Docker containers), you will still have the same bottleneck. In this scenario, splitting the work and moving new functionality to new services will allow you to slowly move the monolith to a new architecture. While this is a solid approach, it comes with a possible drawback. The team may begin creating several small services without the proper documentation instructing others on where to find functionality that they need in the sea of services that have been created.

Lack of Tests

Another one of the challenges can be interesting and exhausting at the same time; there are no unit tests, no integration tests or any performance metrics. In this situation, you would have a difficult time measuring whether any improvements are being made or if you are getting the same results. A solution to this problem is to create the substitute layer (or Proxy). The main purpose of this Proxy layer is to log results and performance metrics. Once there is enough information, you can start moving the microservices and replacing the calls within the substitute layer from the monolith to the new microservices. This also works as aggregator services, where information that is retrieved from multiple services can be combined, whereas there was only one call before.

No Business Commitment

An additional challenge is when there is no business commitment. This can be obvious when there is resistance to change, and the most important thing is to prove that there is progress as fast as possible. In these scenarios, you will be evaluated on how quickly you can provide something of value to the business. You can again split the development and apply the “divide and conquer” approach. Whenever there are minor fixes like sending an email or creating a PDF based on a template, these are ideal scenarios where the strategy will show its results since there are key functionalities that are easy to start separating. Dealing with resistance to change also requires a few more steps: scoping out the required work as accurately as possible, making the services as self-contained as they can be and juggling with the number of resources you have available, as allocating time to the developers can be difficult. 

Big Bangs

The last category of challenges is when you are working with a complete rewrite (which could be seen instead as an opportunity) and working with scheduled releases on critical operations. Oftentimes, the “big bang releases” can be prevalent. This happens when there is a dedicated team and no new features are included in the application, as the main objective is to prepare for a future where change may occur and the work will start to increase.

Techniques for Success

Below is an outline of solutions and implementations to create new services in a more productive way.

Creating miniservices – These are used to accomplish generic tasks such as sending emails, uploading files to a file repository like S3, generating PDFs based on a template or validating XML with XSLT files. 

Creating shared libraries – This is also important because it will allow you to reuse logic that almost never changes, regardless of the client. Examples include creating TSLint configurations or logger configurations, and hooks to run after every commit (like using Husky – if we are talking about a JavaScript environment). Those are the types of shared libraries that will save time later.

Creating NPM packages for your organization to generate a bootstrap of code – For example, create-react-app is something used a lot in the industry, but at First Factory we have created a CLI that starts a new project with create-react-app but also configures the React Router, installs some of the shared libraries and configurations and integrates with a Cognito layer that we use for internal authentication. Every time we need to create a new application that relies on the infrastructure that we have defined, it removes all the repetitive work. In other organizations, similar projects can be created where the stack and the installed libraries may vary, but overall the process is to reduce the time needed to create a new service.

Create a process to migrate data automatically – Our Director of Engineering created a base ETL that relies on code handlers, so whenever we need to do a data migration we analyze the data sources and, for each data source, a handler is generated that extracts the information in a particular order, transforms it and then loads that into a database. Then, the only objective of the team is to create the handlers that will deal with the custom logic for each project, but this base template manages the overall wrapper.

A microservices architecture provides the greatest flexibility and resiliency for modern applications, allowing for high scalability and quicker delivery of new functionality. The key to success when migrating from a monolithic to microservice-based architecture comes down to the traditional core foundations of software development: due diligence in planning, risk mitigation and having the focus and patience to follow a comprehensive plan closely.

The right development methodology will completely transform your team. When you find the right one, you’ll likely notice that your employees are more motivated, working well together, and are getting things done faster than ever before.

Reaping these rewards does take time, whether ramping up a new development team or changing methodologies within an existing group. Before you pick an approach, you’ll have to reconsider the way that your development team operates.

One thing is for certain, though: You should avoid chasing waterfall software development methodologies, as they have fallen out of favor in recent years and may no longer be the most efficient option for your team.

Today, we’re going to look at why the waterfall methodology is rather outdated and explore other methodologies to replace it with.

Waterfall Methodologies Might Not Be Right for You

The waterfall approach to software development has been the default method for building digital products for decades. However, new technologies and capabilities have created other options that are challenging the traditional waterfall model.

What is the waterfall methodology? Let’s take a quick look at the steps involved.

Requirements

Obtain all of the requirements from the client or from the internal team requesting the software. The goal of this approach is to acquire every piece of information upfront so that the client doesn’t have to be spoken with until the product is completed.

Design

Although it is sometimes broken up into two stages, the design stage is the time in which the UI is completed and tested.

Implementation

This is the phase where the programmers take the design and other requirements and turn it into a working product.

Verification

The complete product is released to the customer, who then verifies that it is up to their specifications.

Maintenance

During this phase, the team fixes bugs as necessary with small updates.

It is easy to see why this holistic approach to software development, similar to that of traditional product development such as automobiles, was dominant for decades. Computer software was often shipped on disks or CDs and needed to be complete, with all bells and whistles, prior to a consumer being able to use it. Technology also moved at a slower pace and there were fewer programming languages and developer tools that contributed to a software build. The pace of change is only accelerating and the web has changed the way software is distributed and consumed. Software as a Service and cloud-based applications are the current models and everyone is deploying new features rapidly.

Other Software Development Methodologies to Explore

While the waterfall methodology is sensible, several notable software development methodologies have emerged in recent years that are worthy challengers. Let’s go through them.

Agile

The agile development methodology has been around for over two decades, yet it’s still being treated as a radical new way to develop software. In 2001, 17 different technology experts released the “Agile Manifesto.” Among other things, the manifesto was focused on four major principles for developing software:

1. Individuals and interactions over processes and tools
2. Working software over comprehensive documentation
3. Customer collaboration over contract negotiation
4. Responding to change over following a plan

As we can see, these four principles are an intentional challenge to the waterfall methodology. During its two decades of growth and adoption, it’s become the standard methodology for anyone who wants to release software quickly and then continue to develop it over time.

Just like JavaScript has several different frameworks that are used in various situations, many different frameworks have evolved around the agile methodology. The following methodologies are frameworks built upon the above four principles.

SAFe

Short for “Scaled Agile Framework,” SAFe is an enterprise-level framework based on the agile methodology. This framework is designed to empower large-scale businesses to continuously deliver new software or updates to existing software. With a “top-down” approach, this methodology starts with an organization that must first identify the type of value stream needed, as well as any Agile Release Trains, also known as ARTs. The Agile Core Values are then approached – alignment, built-in quality, transparency and then program execution. 

The entire goal of this framework is to create a faster time-to-market and catalyze productivity from teams. With the average teams consisting of 5-9 employees, and a program increment duration of 8-12 weeks, SAFe is a common choice for many fast-paced industries. 

This framework was created by Scaled Agile, Inc., which sells software that aims to help enterprises adopt their specific version of agile development. This is one of the key software development methodologies preferred by large-scale businesses.

Scrum

Scrum is a new take on the agile methodology that focuses on “sprints.” A sprint is a predetermined amount of time for completing a designated goal. 

Scrum generally observes the ceremonies of sprint planning, daily Scrum, sprint review, and sprint retrospective. Each sprint starts off with a planning session for all team members where the work of the sprint is defined, agreed upon and estimated. The team works on their tasks throughout the sprint, perhaps checking in for quick daily stand-ups to ensure blockers and unplanned priority tasks are addressed. Key stakeholders are brought in for a sprint review to see the work done and after all is complete, the team may conduct a retrospective to evaluate efficacy. 

It is best practice for a sprint to last less than one month (typically 3-4 weeks), if not much shorter. There are to be no changes made during a sprint that could adversely affect the overall goal, and the time to plan a sprint should only take less than a couple of hours (up to a full work-day, if necessary). 

Adherents of the Scrum methodology say that it’s great for complex projects, as it unites the entire workforce under one specific goal per sprint.

Kanban

Kanban is another take on the agile methodology that aims to make the most of what’s available. The actual practice of Kanban dates back to the 1950s when Toyota pioneered a method of having workers track what new materials were needed on a card, which was then passed to the warehouse.

The older Kanban methodology has merged with the modern agile approach to create a new take on software development. The goal of this methodology is to match the number of works in progress to the team’s total capacity.

At the core of the Kanban methodology is the Kanban board. It’s a board that contains a simple three-step workflow:  To Do, In Progress, and Done. The entire team can see the board, so they know what needs to be done or what needs to be reviewed. This is a great approach for production support channels or when juggling multiple smaller projects at once

How to Adopt a New Methodology

Changing your software development methodology will not happen overnight, and it might not even happen quickly. It requires an intentional adoption of the core principles of an approach by your entire team. It will most likely call for using new collaborative tools, as well.

The most important aspect of changing methodologies is ongoing training for your development team. You might discover that some developers are so entrenched in the waterfall methodology that they need regular reminders about new workflows. Eventually, though, the new methodology will be internalized and you’ll start to thoroughly enjoy the perks of your new methodology.  

The First Factory team is a leader in software development. When you want to take your business to the next level, let us help. With a proven combination of technologies and software development methodologies, we can bring your vision to life. Contact us today!