Continuous Delivery in regulated, safety-critical or high-risk products

I work as a passionate Continuous Delivery expert with more than 20 years experience in the field. If you are working in a regulated, safety-critical or high-risk product and your Software Development Life Cycle needs optimization — or just a second opinion — I can help

About me
I hold a M.Sc. in Computer Science and Communication theories from my youth.
   Later in my career I also pursued a Master in Business Coaching — leadership through conversation.
   I founded and led Praqma in 2007; a consulting bureau specialized in Continuos Delivery and DevOps.
   I sold the company in 2018. At that time we were ~50 specialized consultants in three countries (DK,N,S).

Throughout my entire career I have dedicated my focus on organizing software developer routines and chores in a way that fosters software with quality built-in — often GxP grade (life science), safety-critical (manufacturing industry) and high-trust (finance) and often with the added complexity of hardware-in-the-loop. At the core in achieving this I build a continuous flow in software deliveries, through a high degree of autonomation and automation; anything-as-code.

I have a passion — and profound knowledge — in utilizing and adopting DevOps and DevX principles in software development processes, but most importantly I adhere to lean principles. I have many years of experience in adapting both lean manufacturing principles and lean management principles to lean software development.

The catch-all term for this approach is Continuous Delivery.

Despite my general focus on advisory, leadership and strategies — I still write code every day and I contribute actively to the tactical implementation. I believe in leading by example.

Quality-by-Design

a systematic approach to development that begins with predefined objectives and emphasizes product and process understanding based on sound science and quality risk management.

Quality-by-Design (QbD) is a software development discipline that is widely adapted in regulated software productions — and even in safety-critical and high-risk areas that may not be strictly regulated by an external body, but nevertheless work by the same self-imposed strict quality measures. Some of the more commonly known trademarks of Quality-by-Design in everyday software development workflow manifests as:

• V-model — for verification and validation
• Traceability — all changes must trace to a specific requirement — and to a specific verification
• Segregation of duties — a general four-eye principle
• Code classification — not all code changes impose the same level of risk
• Releases management — Strict control over the deployment process, only authorized versions reach the production environment

In many ways QbD is the direct opposite paradigm to the emerging vibe coding trend. In Quality-by-Design we’re not hacking to explore exciting what-if scenarios. Instead we strive to prove that the code is working as intended, solving the specific problems that we analyzed in forehand.

Unfortunately, in many cases these Quality-by-Design trademarks also end up getting a lot of beating and are often accused of creating a rigid process.

The criticism I hear: — Waterfall processes, Big Design Up Front, bureaucratic over-registrations, unfree procedures, long-lived feature branches, fragmented complex dependency maps, long wait states, rubber stamp peer-reviews, way too many hand-overs, release procedures that requires intense heroic over-time work with countless of loop-back, tedious paperwork and coordinations between too many parties which lead to more hand-overs and more idle time.

Throughout my career as Continuous Delivery consultant I’ve seen many software development teams that suffer from some — or all — of the dysfunctions above and at the same time excuse it with:

“due to the the regulated process we are forced to work like this”.

It is my own observation that regulated environments and other strict QbD implementations usually suffer way harder than other environments from bad performance in all of the four core DORA metrics:

• Change lead times
• Deployment frequency
• Change failure (and rework) rate
• Failed deployment recovery time

But I also find this to be quite a paradox. On paper Quality-by-Design has a huge potential to become a DORA high-performer; observability and process understanding is precisely what fuels Elite DevOps.

Quality-by-Design aligns naturally with lean manufacturing concepts like a hand in a glove; Lean manufacturing has several concepts specifically designed to build quality in at the source, as opposed to glue it on as an afterthought.

Jidoka — “Autonomation” a made-up word that implies automation with autonomy as a human.

A one-piece-flow that is the incarnation of minimize-work-in-progress, each product item is started and continued until it’s completely done - like a conveyer belt. The one-piece-flow is specifically designed to make problems surface early, so they can be dealt with at the source.

The andon cord a line that runs throughout the entire factory, when a defect is detected the andon is pulled which brings the entire process to a stop until it’s fixed. No process can pass a defect on to the next.

S.M.E.D Single-minute-exchange-of-die — a concept used in lean manufacturing methodology that references a true iconic story that drastically reduce the time it takes to complete equipment changeovers or production setups. Due to the extremely high (positive) impact on the outcome. S.M.E.D. is referring to the general principle that even large endeavors are considered justified despite extensive effort and cost, if the long-term system outcome is proportional.

In lean software development:

• Jidoka implies a lot of automation at all levels of the V-model, simply shift-left.
• The one-piece-flow and the andon cord becomes the fabled CI/CD pipeline.
• The S.M.E.D. principles becomes a host of tools, scripts, plugins, extensions, templates that we develop specifically to serve and standardize our own process.

I have made it my hallmark to take the postulated rigidness out of configuration management, traceability, thorough testing, release management and quality-by-design principles and maintaining all these virtues by combining them with Continuous Delivery principles to create a constant flow of software deliveries with high throughput and stability.

The end-goal is to enable deployment-on-demand with all quality, stability and QbD compliancy intact.

Continuous Delivery in QbD context

Companies I aided
In my capacity as a Continuous Delivery consultant I have assisted the following companies (the time span is 20+ years):

Ericsson AB, Sony-Ericsson, Nordea, ATP, Novo Nordisk, Grundfos, Danfoss Drives, Rational Software, Volvo Trucks, Novelda, Radiometer, Alm Brand, TDC, Everllence (MAN es), Tetra pak, Oticon, Tieto Finance, William Demant, Kamstrup, LM Ericsson, GitHub, Siemens Gemesa, Programming Research QA, AP Pension, Microchip (Atmel), Terma, Brüel & Kjær Sound and Vibration, Arla foods, BankData, Santander Consumer Bank, Styrelsen for IT & Læring, 3Shape, Schlumberger, Universal Robots, Cryptera, Danmarks Radio, BK Ultrasound, Yxlon International

As an incarnated Continuous Delivery expert I am passionate about creating flow and automate — everything. Here’s an array of stuff that I have automated, and turned into silent background processes over time:

Task management, branching strategies, semantic versioning, telemetry, metrics, visualizations, release notes, packaging, static code analysis, product dependency maps, unit tests, integrations tests, end-2-end tests, fuzzy tests, performance tests, regression tests, test data synthesis, production-like environments, developer environments setup, containerization, onboarding, offboarding, digital twins, fast-forward merge to main, peer-reviews, infrastructure, builds, security scans, deployments, releases, documentation. — I could go on.

When scrutinized, some of these steps may at first seem quite controversial to try to automate in a traditional Quality-by-Design process. A few examples; I often get that push back that “keeping the main integration branch in an always shippable state” is impossible in a regulated process and that “removing the costly rubber stamp peer-review” is not possible due to QA policies and compliance with regulations.

While I do admit and agree that some of these automation efforts can approach S.M.E.D sized endeavors in most regulated environments, I have nevertheless proven at several occasion that even hard-to-crack nuts like these can be overcome.

I dare you:

• Keeping the integration branch in an always shippable state — does not imply that everyone is hacking directly on main.
• Removing the ritual rubber stamp peer-review — does not imply that you unconditionally sacrifice the four-eye principle.
• Skipping the Big Design Up Front and allowing end-users to get a strong voice in a Participatory Design process — does not imply that the V-model is cast aside.
• Utilizing AI and prompt engineering in your process — does not imply that you are merely vibe coding and hence scope creeping away from requirements.
• Utilizing Open Source and maybe even releasing it yourself - does not imply that your Intellectual Property is jeopardized or that you expose yourself to a security risk.
• Dropping hard deadlines and adhering to #NoEstimates — does not imply that everything is up in the air and all delivery plans are void.
• Firing red-tape heavy weight tools like Jira and Azure DevOps and swap to a DevX driven developer-first stack - does not imply that you are violating neither GDPR, NIS2 nor QbD principles.
• Bringing the deploy frequency to on-demand — does not imply that you loose strict control over the deployment process.

In this post I argue why GitHub's Pull Request are flawed in a strict configuration managemnt context, making it impossible to maintain the required traceability - and what to do about it

I know that many developers who are used to work in regulated environments might instinctively feel an urge to call me out and claim that I’m off. Or miscredit me for not knowing what I’m talking about. But I do. I’m definitely not advocating abandoning QbD principles. I’m merely showing an optimized path to walk to the same goal.

I have written a good handful of blog posts diving into the nitty gritty details on these topics, even presented at conferences on quite a few of the topics as well. But more importantly I have actually implemented all of these concepts in QbD processes. If you want to dive into any of these specific topics then please reach out to me; I’d be happy to elaborate, share and demonstrate.

My point is simply, that it pays out to leave the traditional construction paradigm in software development and enter in to a lean software as a factory¹ paradigm instead. Especially in regulated software development environments, where lean concepts are usually already very well understood and accepted. I’ve seen the epiphany happen in real time several times:

“Aaah! So we’re essentially making software the same way we make hardware!”.

“Yes!” — It makes good senses in general, but especially when there is hardware-in-the-loop. Aligning to lean processes helps build mutual understanding across organizational silos.

Flow

In this post I argue why GitHub's Pull Request are flawed in a strict configuration managemnt context, making it impossible to maintain the required traceability - and what to do about it

It is generally recognized within lean methodology, that mura (unevenness) is the cause for most of the muri (overburdenness) which drives muda (wastefullness).

Therefore creating flow — optimizing the value stream – is the most important condition to establish. This is true within lean software development too.

As DevOps Research and Assessment institute’s (DORA) annual “State of DevOps” report has established several times:

“Throughput beats Stability²”

To optimizing the value stream is the essence of long-term system thinking — nothing is more important.

Optimizing the value stream eliminates most problems, not just in the software product itself, which even gets higher quality and stability as a side effect, but it also has a positive effect in the team where flow alleviate cognitive load and burnout. As we progress, more and more processes becomes automated and hereby standardized. And slowly but surely they are converted to non-disturbing background processes, which enables the software developers to keep the eye on the ball; to analyze problems and to solve problems.

As Mary & Tom Poppendieck poetically describes in “Lean Software Development” in 2002:

“Either you in process of analyzing and understanding a problem
— Or you are in process of implementing and solving a problem
— Or you are in process of wasting your time”

My preferred way of working, when optimizing workflows and developing software products is based on kanban upstream and downstream boards, which are held continuously prioritized by risk and effect. To minimize work in progress, to establish Participatory Design and short feedback loops with end-users, to create a collaborative Open Source like sentiment adhering to #NoEstimates and XP principles — and then to compensate the business for lack of deadlines by maintaining ambitious but realistic delivery plans - often in the form of roadmap or upstream kanban boards.

Transformational Leadership

My general position when it comes to exercising leadership can be summarized in the concept Transformational Leadership; With a focus that the team should be led from within, by a playing coach or a role model, who lives the same values as the team strive to achieve in general. The tools in achieving this are centered around motivation and stimulation. Often anchored in lean management principles and problem oriented didactics. We must learn and develop while we work.

How do we get started

Maybe you already have a concrete situation in mind where you see me as a possible fit. Great — don’t hesitate to reach out.

Book a call

During my time in Praqma I developed the Continuous Delivery Metric Model; A tool used to help companies assess current state of their SDLC Value Stream and come up with a road map for improvements

Below I have sketched a three-step approach I have used at several clients over the time, think of it as inspiration — I can adapt to your organization’s need in any way you wish. An alternative, even less formal first encounter, could be that I come by you and your team and offer a free inspirational workshop.

1. Genchi Genbutsu (observation, analysis, research, interviews, demo, workshops, value stream mapping)
2. Reporting and roadmap
3. Kaizen kultur

Genchi Genbutsu

I suggest an initial phase (Genchi Genbutsu) which typically runs for a week or two, dependant on the complexity of your current as-is work process.

Genchi Genbutsu is a lean term; “go-see-for-yourself” which is the foundation of lean problem solving; you go to the Gemba “where the work is done” and systematically observe and analyze with the purpose of suggesting and balancing possible improvements.

In this phase, it is also important to establish relations to those in your organization who shall own the results and outcome. As a consultant or guest in your organizations it’s imperative that the formal ownership is placed with someone in your own organization, with sufficient level of decision power and seniority. The person of persons who has ownership will work with me in a close relation in this part of the assignment, and contribute in the planning, observing, analysis, research, interviews etc.

Reporting and roadmap

The initial phase is concluded by a summary report and recommendation to a roadmap or an upstream kanban board. It’s a short phase, just one or two meetings with stakeholders, with the primary purpose to mark the transition from Genchi Genbutsu to the continuous improvement phase.

Kaizen Culture

Immediately hereafter an informal but facilitated kaizen (Continuous Improvement) culture is established. In this culture, changes are introduced as deliberately small continuous improvement to existing everyday chores and processes, carefully designed to be non-intrusive way that does not create anxious big-bag situations or raises the cognitive load on any team members.

Kaizen means “good change” in the implied meaning “small change”. It is about changing the wheels on the bud — while it’s rolling.

Focus is on an ongoing prioritization (upstream kanban board) of possible tasks and improvement with the aim to gain the largest possible effect with the smallest possible change. And then keep a focused implementation strategy through minimizing work in progress in the kanban downstream board.

In the beginning of this phase I will personally co-work with team members on the individual improvements, and keep focus on implementations not creating disturbance but improvements of existing processes. This focus may require some adjustment and getting-used-to. I will take dedicated responsibility for the implementation and the outcomes until the culture is understood and adopted.

It is likely that changes will touch on and involve several teams or departments. I will strive to work as a catalyst. I will start and finish one task at a time, but shift frequently between teams so the kaizen culture will be built and balanced between the teams.

We will coordinate and evaluate regularly among the teams in joint events like retrospectives, pulse checks but most importantly by gathering metrics and telemetry on actual progress. Measuring on the four code DORA metrics is usually a good place to start.

Book a call

Practicalities

My engagement can range from just a few inspirational workshops to year-long engagements. And I have had several clients where I continued in an advisory role after my day-to-day engagement ceased.

After more than 20 years as a Continuous Delivery expert, and with many of them in a capacity as employer, I have a very large network of professional like-minded and I do have the capability to offer to take lead and responsibility and scale up on S-M.E.D. sized endeavors with curated freelancers, collaboration partners or even student workers for grunt work or Open Source contributions if necessary.

I run a tight ship with a lean cost structure, hence my pricing is attractive and easily 20-60% less than I charged when working in the same capacity in a Management Consulting Bureau. My compensation is adapted to the extend of the contract we make.

I strive to be easy to be onboarded …and offboarded.

---

1. LinkedIn post on the relevance of the Toyota Way in software ↩

2. An interesting proof from DORA, that Throughput beast Stability ↩