Forge for Extensibility-First Architecture

The past couple of years, I have been fortunate enough to have dug into many scaling tech organisations, across e-Commerce, enterprise and “prosumer” SaaS, marketplaces and even consumer products. I’ve seen many poor practices, I paid attention to the patterns, common issues and bottlenecks for scaling product engineering.

I’ve also seen some best practices and where they have had a measurable and distinct impact on the ability of a company to scale. I say company, because this isn’t just about the ability to ship new features. This is about unlocking new market opportunities, so critical when a tech company hits a certain scale.

One of the most critical aspects I have seen, one which intersects and overlaps multiple drivers for scale and growth, is Readiness for Extensibility.

Extensibility Explained

In software engineering, extensibility is defined as “ the quality of being designed to allow the addition of new capabilities or functionality .” It is a measure of the ability to extend a system and the level of effort required to implement the extensions.

“No single platform can provide everything out-of-the-box to meet business needs. To drive product and quality excellence, the extensibility of the chosen software platform is critical.“

An extensible software platform should be:

  • flexible
  • configurable
  • customizable
  • upgradeable
  • accessible,
  • collaborative.

What can be extensible?

  • Data model - The existing data model can be extended with new data types, including custom data types.

  • Processes - It is essential to automate the mundane, repetitive tasks to allow the end-user to focus on the high-value task and minimize room for errors. The processes should be highly adaptable and supported by tools that non-programmers can easily use.

  • User Interface - For an application to be successful within an enterprise, user interface (UI) is key. UI should be intuitive, highly configurable, performant, and mobile.

  • Integration - Business processes span multiple enterprise applications and external web services. Data moves in between several downstream and upstream systems. These processes should be well integrated to provide a cohesive and seamless experience for the end-users.

For many use-cases, there will be a need to combine multiple areas together to realise a fully-fledged feature through extensibility.

Setting the scene - Organisation Personas

Extensibility will mean different things to different people at different companies depending on many organisational dynamics. To help frame the problems and identify areas where extensibility can be. impactful, we can use some characterisations and particular attributes of the company through personas.

Exploration Startup
When a team is small, typically the dynamic of an earlier stage company or startup, devs are all chipping away at the same “monolithic” codebase, driven by the need to realise a founder’s vision and sell the idea to an early stage investor while all the time, getting feedback from the all important ‘potential’ customers.

This is generally the approach recommended for such as stage, as it is more efficient to develop and maintain in an environment of complete uncertainty.

The process is highly iterative, subject to frequent and often significant change (the Pivot) for the Exploration Startup. The team may just be 1 - 5 devs at this stage, perhaps resourced through contractors or freelancers.

Traction Startup
A more mature startup, let’s say a few years old with solid traction, has developed an understanding of the core proposition and solution they are providing. While there will still be significant R&D work underway, this will likely be around the periphery of a stable domain model and core feature set.

The dev team has probably grown to 5 - 10 members at this stage, but the business is demanding greater velocity to realise the longer term product vision in its features. The company is running out of cash, and trying to build enough buzz and excitement around the vision with early stage customers and using this to simultaneously pitch VCs to raise their Seed or early Series A financing.

The challenge faced by the Traction Startup at this stage is that the need for a combination of a stable core service combined with rapid development of features, each which add significant technical debt due to the inevitable shortcuts taken to prioritise customer impact versus technical and architectural maturity, will seem appropriate now but very expensive to pay back later.

Scaleup Startup
At this stage, the market has really validated that the company meets its core needs and the market is big enough (and perhaps growing in itself) - so much so that the company has raised anywhere between $20M and $150M in Series A / Series B funding. The company will have revenues that are growing at least at a steady rate with solid levels of customer retention.

The organisation may be growing significantly, both across Sales, Marketing and centralised operational functions - but also in Product, Design and Engineering - fuelled by a healthy round of injected capital. The engineering function may be growing from 10 - 30 people to 100, 150 or as much as 400 people covering software engineering, devOps, QA, secOps and more besides.

Yet, now there is competition - new startups ave recognised the opportunity in the market and are releasing new products with much fanfare. Large enterprises have pivoted a product area to address the same problem area and take advantage of the opportunity by filling gaps in their existing product offerings.

The pressure on the company is intense - while simultaneously adjusting practices and process to accommodate way more people, and facing the pressure of new competition and a race to dominate a category - the importance of shipping new features that keep them ahead has never been greater. It has also never been harder - teams feel like they are much more ineffective than when they were smaller, more nimble. New layers of management, new processes, politics and bureaucracy creep in. Just when you think everything should be going smoothly, you realise it has never been harder.

More than anything, it can start to feel like a different company and the culture may have changed - resulting in employee turnover and attrition while adding new bodies coming from other mature enterprises and successful startups. There maybe leadership changes, as founders now realise that they weren’t the right fit to take the company into the next stages.

How Does this relate to Extensibility?

I have been exploring how a readiness for extensibility can serve to address many of the challenges faced by companies at their relevant stages of growth and development.

I am excited to say, that what I have observed, tested and found has been enlightening.

An “Exploration Startup” can leverage extensibility to rapidly iterate prototypical ideas quickly, often crudely, in order to demonstrate a key hypothesis of their vision. This beats a powerpoint presentation hands down, every time.

A “Traction Startup” can defend the solid and stable core domain model, APIs and resources at a time when ordinarily, huge swathes of technical debt is absorbed by the systems. A small core team responsible for the core services can be augmented with community, contract, freelance or on-demand resources that are hired to consume extensibility interfaces and bring new experimental features to life quickly so that they can be tested with customers. If they don’t work or need significant change, they can be discarded without long term burden on the core systems. If they are validated, then they can continue to develop independently or by converted to native core features at any time.

A “Scaleup Startup” can unlock new Platform-enabled capabilities to support the immense organisational growth that has become stuck, creating faster and safer environments to develop independently with clear ownership and domain boundaries. Internal Platform capabilities can be externalised to unlock new go to market propositions - such as Developer-facing propositions through APIs, SDKs and marketplaces for extensions and addons built by the community. Strategic partnership programmes and integrations can be accelerated through self-serve and assisted models - key for distribution and growth.


In this thread, I will open up thoughts, explorations and examples of extensibility in practice. In particular, I will demonstrate how our very own Forge platform can hep you achieve an extensibility-first architecture at whatever scale or stage of development your company is at.

Extensibility Architecture

I am the sort of person that often starts with a visual in mind. I like to think about things in terms of systems and flows. So let’s start with a high level concept of an extensible architecture in relation to a “Traction Startup”. I’ve picked this example to start with as it clearly identifies core/native aspects working alongside extensibility-enabled components.

Before Extensibility

It’s pretty likely that, in its simplest terms, the application architecture would look something like this with a separate client app frontend (for web, mobile or multiple) consuming services in the form of APIs from a single monolithic backend application.

When developing features for these components, developers are likely to contribute to the repository for each, perhaps hosted on a platform like Github, Gitlab or Bitbucket. There is no real distinction between “stable” or “core” features, UIs, services or data models compared with more experimental features or experiments.

Not such a big deal with a handful of individual contributors, but at scale this becomes are difficult operational challenge. Particularly if you already have years of technical debt crammed into the code base as legacy.

Extensibility Enabled Approach

With some provisions made for extensibility we still have a common setup, a largely monolithic application architecture with abstracted front-end consuming API services from the backend application.

However, our core services provided by that main application are augmented with further extracted Extensions (or Plugins or Addons) that are developed independently, consume the provided services in the form of:

  • REST / GraphQL APIs
  • Client SDK
  • Webhooks
  • Real-time notifications / Websockets events

An Extension is easy to build using the provided core interfaces, without exposing the entire system, IT accesses and codebase to plugin developers - ideal for leveraging flexible resourcing models such as community-driven projects, global freelance or contractors or campaign / project / spike style projects.

Extensions can be implemented in a traditional project cycle within teams, but are also much more accessible as they could be developed in a Hackathon type event, or Design Sprint, or any other way which may stimulate imagination and creativity towards solving real customer problems.

Importantly, if the Extension concept doesn’t work, the experiment fails, or requires a different approach, it is fast and easy to throw away and start over without polluting the codebase and giving other teams years of legacy and maintenance to contend with.

Also critically, this is a secure approach as teams can only consume services and interfaces provided by the core team, where the full security and compliance governance model alongside baked-in security practices (authorisations, scopes, validations, application firewalls, rate limits etc.) is applied. Teams that are able to fully access application code are more susceptible to introducing security vulnerabilities as they seek to ship their feature among the backdrop of complexity.

Anatomy of an Extension

In this post I will dig a bit deeper into a term that I’ve throwing around: Extension

Extensions are the pieces of code that are a product of extensibility. Other names for Extensions commonly used are “Plugins”, “AddOns”, “Modules”, which are all essentially variants of the same thing.

Some platforms even brand the concept, such as Trello’s “PowerUps”. These are also extensions.

Types of Extensions

There are a couple of typical types of extensions that we see (and can expect there to be also many new types and grey areas, but for simplicity sake, I’ll start with these two…).

  • Programmatic Extensions
  • Custom UI Extensions

Programmatic Extensions

These are commonly functional extensions, that are designed to act upon some triggered event in the host product. For example, say I wanted to create an Extension that would add up all the story-points in a Kanban board whenever I add or update a Kanban card and send that information to a Google Sheet.

Well, there’s no UI planned for my extension, just some logic and functionality that would consume the SDK interface to initialise and register the extension, fetch the cards data, total up some of the card properties, construct an API call to my Google sheet and send it.

These types of functions are great for working with data, manipulating and transforming data and extending workflows.

Example

Our plugin will look something like this:

kanbanAppSDK.addListener("BOARD_LOADED", function() {
  console.log("Board and Board Content loaded");
  const cards = kanbanAppSDK.board.getCards()
  const storyPointsTotal = cards.reduce((total, obj) => obj.storyPoints + total,0)
// Some stuff for sending data to a Google sheet that we will ignore for now
// POST https://sheets.googleapis.com/v4/spreadsheets/{spreadsheetId}/values/{range}:append

});

After loading the KanbanAppSDK we will set up an event listener that will fire when the Kanban Board is fully loaded. From here, we can use another SDK method to fetch all of the Cards from the loaded board and run our function to calculate the storypoints total. From there, we can do whatever we need with that data. I’ve not implemented the Google Sheets example as that’s beyond the scope of this write up.

Custom UI Extensions

These are often used to build fully fledged in-app features and new experiences, combining programmatic access points with the ability to inject UI elements into the product experience.

The UI could be heavily contrived, declarative formats which are controlled by the host platform or sandboxed and fully flexible areas in which to put a complete custom interface. It really depends on the intended developer experience and product strategy.

Many platforms provide some mix of approaches, as they ultimately cater for different needs and use-cases aligned to different audiences making up the “developer” or “maker” personas in their target audience.

In order to have control, or at least, to guide the makers towards creating interfaces that fit within the host product (in terms of quality, experience and style) additional utility tools may be provided such as Design Systems, Design Guideline documentation and stylised Component Frameworks.

Example

In our case, let’s say that instead of an API call to to a Google Sheet, we will instead leverage an existing UI extensibility point that allows us to control the value of a Label in the header of our Kanban board.

kanbanAppSDK.init({
  extensionPoints: {
    headerLabel: [{
      value: `${storyPointsTotal},
      onLoad: () => {
        const cards = kanbanAppSDK.board.getCards()
        const storyPointsTotal = cards.reduce((total, obj) => obj.storyPoints + total,0)
      } 
    }] 
  }
})

Extensions Structure

In order to load our extension we need to create, at the very least, our own index.html page. This will be responsible for:

  • Loading the KanbanAppSDK.js client
  • Loading our javascript code that consumes KanbanAppSDK.js methods, such as subscribing to event listeners, initialising extensibility points etc.
  • Loading our custom javascript logic
  • Loading our custom presentation / views

index.html simple example

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width,user-scalable=no" />
    <title>Kanban App SDK - Board Loaded Event</title>
  </head>
  <body>
    <script src="./kanbanAppSDK.js" type="application/javascript"></script>
    <script>
      kanbanAppSDK.addListener("BOARD_LOADED", function() {
        alert("All Objects Loaded");
      });
    </script>
  </body>
</html>

All we would need to do, is host this file somewhere suitable and provide the hosted URL to the host platform as part of whichever “extension” submission process they have adopted.

During development, we may be able to serve this file from localhost, a local dev server (such as Anvil) or serve via tunnel such as Ngrok.

For production, it will depend on the host platform. Some platforms require you to self-host your plugins. Others will require you to upload your code to their systems and they will host plugins for you.

Plugin hosting, deployments, workflows, security and compliance will be a big topic to discuss in future.

Here’s a quick sketch I created to illustrate the dynamics of working through the different stages of a tech company lifecycle, with and without extensibility-driven architecture.

At an early stage, there’s not too much difference. Small team, working fast to ship an MVP to early test users and a few customers (if we’re lucky). Take that validation and learning back to the concept to reinforce to potential investors that this is real, momentum is building.

Perhaps not depicted here is that our hypothesis is that if an extensibility framework didn’t carry with it significant investment and was available from day zero, then many of the “hacks”, “PoCs”, tests and experiments could be done in an extensible way and allowing to get started with clean architecture from the early stages.

In the traction phase, we see real impact. While a growing team deals with a growing monolithic application which is bloating with technical and feature debt. We’re seeing the struggle to maintain forward and upward momentum.

In the extensible approach, you can see that we are still carrying the monolithic core (no micro-services re-architectures here!!) but allowing new features at different sizes, scopes, fidelity, quality to be brought into the product context in small and manageable chunks in relative isolation. Basic platform capabilities are forming to support this, by “protecting the core” and enabling the experimental simultaneously.

As the company starts to scaleup rapidly, teams form. Those teams form sub teams. People are either treading on each other (in code terms) or completely stuck by dependencies.

In the Extensible version, the monolith still exists (it’s now the mountain), enabled with facilities to support independent development of cross-functional teams features and roadmaps with clear interfaces, governance and utilities provided to ensure safe and reliable contributions.