Some software projects teach you that you didn’t know what you didn’t know — and keep teaching you for a decade.
This one started with a hand-drawn sketch on a single sheet of paper, scanned and emailed as a project brief. It survived a complete architectural rebuild, a regulatory inspection by a major global authority, and a full decommissioning cycle. Today, it is the single source of truth for manufacturing control limits at twelve production sites. It supports a top-tier pharmaceutical company.
I’ve been the technical lead on this engagement since the beginning. Here is what we learned.
The Problem
A large drug manufacturer tracked batch production data across global sites. They used paper records and manual spreadsheet exports. They monitored CQAs, yield, and mixing settings. A prior software solution had already failed. One table grew to seven million records. Overnight ETL jobs did not finish. Data availability was unpredictable. A classic case of OLTP architecture being pushed into analytical workloads it was never designed for.
The brief: build something that works at scale. It must integrate data from SAP, MES, and LIMS. It must meet FDA 21 CFR Part 11 and GMP requirements. What was less clear was how many layers that brief contained.
Why GMP Software Is a Different Category
Non-functional requirements often do not show up in early lists. These can include 21 CFR Part 11-compliant audit trails. They can also include session controls. They can include disaster recovery cycles. They can include data retention schedules. Users don’t ask for them. Development teams often don’t flag them.
In a standard application, a missing audit trail is a missed feature. In a GMP-validated system, it is a compliance finding that can surface during an FDA or EMA inspection.
GAMP 5 provides the Computer System Validation framework, but it is a set of recommendations, not a checklist. Interpreting it correctly requires domain expertise — knowing what a regulator will actually examine, not just what the guidance says.
We learned this the hard way. An internal audit after the first version went live produced a twenty-four-page findings document. We fixed every item. Then the platform scaled to eight sites and hit the same architectural wall as its predecessor. We started again.
I didn’t know what a non-functional requirement was when this project began. That list now runs to ten pages and gets handed to every new team member on day one.
The Rebuild
The second version replaced a virtualized database layer we could not control with a custom data warehouse. It was tuned for analytics queries on time-series batch data. This solved the processing bottleneck that had killed two systems.
Tech stack
|
Layer |
Technology |
|
Application framework |
.NET with DevExpress components |
|
Databases |
MSSQL, PostgreSQL |
|
Data processing |
C# Windows Service (decoupled from UI) |
|
Statistical analytics |
R (Cpk indices, SPC control charts) |
|
Integration sources |
SAP, MES, LIMS, QMS (~50 active sources) |
Specification limits are retrieved dynamically from a connected product specification system and overlaid onto incoming process data at query time — making deviation detection data-driven rather than hardcoded.
Two decisions proved critical in hindsight:
Own the full data pipeline. In version one, the client provided data and we visualised it. When their processing broke, our application broke with it. In the rebuild, we owned ingestion to presentation.
Design for CSV from the start. The first version was built and then validated. Requirements discovered during Computer System Validation are expensive to retrofit. The second was architected with IQ/OQ/PQ environment structure, testing requirements, and change control procedures already embedded.
What Validated Software Actually Requires
These are the requirements that consistently surprise development teams entering regulated environments for the first time.
|
Requirement |
What it actually means |
Consequence of getting it wrong |
|
Audit trail |
Every data creation, modification, and deletion logged with timestamp, user, and previous value — per 21 CFR Part 11 |
Compliance finding on first inspection |
|
Testing coverage |
Criticality-based — batch release systems require substantially more than standard ~10% coverage, with evidence retained as part of the OQ record |
Test gaps become findings; retroactive documentation is not accepted |
|
Disaster recovery |
The plan must be periodically executed — drop the database, restore from backup, verify record counts match — not just written |
Having the plan without execution evidence is a finding |
|
Infrastructure changes |
Any OS, server, or security patch triggers requalification across all environments (IQ → OQ → PQ) before production is touched |
Skipping revalidation invalidates the system’s validated state |
|
Separation of roles |
The author of a validation document cannot be its approver — system owner, validation expert, and QA reviewer must be distinct |
Cannot achieve go-live sign-off with a single person covering multiple roles |
How the Platform Grew
The platform launched as a Process and Product Monitoring (PPM) solution. It implemented Continuous Process Verification (CPV). Batch data was ingested. Specification limits were applied dynamically. Deviations were surfaced automatically. The R statistical layer calculated Cpk values and SPC control charts. This helped engineers spot drift before it became a documented CAPA.
Expansion followed user needs, not a roadmap:
- Version 1 Read-only PPM. Deviation visibility across sites.
- Version 2 QMS integration added. Engineers could see CAPA context alongside batch data.
- Version 3 Bidirectional API. Deviations raised directly into the QMS from the platform; CAPAs returned in context.
- Current Method Monitoring module added for analytical method performance. ~50 active data sources across 12 sites.
That is how validated platform software grows in large organisations: not through feature selling, but through demonstrated trust.
The Regulatory Inspection
During the decommissioning of the first version — still active as a live GMP system — a regulatory inspection occurred. One early question: why was a particular development tool selected? Without a documented rationale, there is no answer.
“I don’t know” is never acceptable. I learned that at one in the morning on a call I wasn’t expecting.
A finding was raised on disaster recovery: the plan existed but had never been executed and evidenced. Resolution took weeks, navigating approval chains, running the full recovery procedure, documenting everything. There were no critical findings.
Validation expertise is not project overhead. It is the mechanism that prevents findings that cost far more to close than they would have cost to avoid.
What Changed
User acceptance with real data. Wireframes are not enough. We let you try features in the sandbox. You can do more than see the final result in TEST. You can brainstorm and improve it in an agile way.
Combined URS and functional specification. We merged the traditional URS and FS documents into one Requirement Specification with a Functional Risk Assessment. This supports parallel development while keeping the risk-based rationale required by GAMP 5. It delivers the same audit-ready output with a faster cycle.
Full-stack ownership. Business analysis, development, QA, validation, system ownership, and three-tier support — all BGO. The client’s product owner defines requirements. Everything else is ours. Accountability is unambiguous.
The Short Version
In a GMP environment, software and its surrounding documentation are treated as a single entity.
- An undocumented decision didn’t happen.
- An untested disaster recovery plan isn’t a plan.
- A test without a record wasn’t done.
The technical challenges are solvable. Knowing what regulated environments require is often the hardest part before you start.
It also takes expertise to design for it from day one, instead of retrofitting under audit pressure.
Harry is a GMP Validated Systems Champion and solution architect for BGO Software’s Validated Systems portfolio. With nearly ten years of experience in GxP processes and more than 15 in software development, including work with leading pharmaceutical companies, Harry is the ideal choice for learning about Good Manufacturing Practices, processes, and the technology that goes with it.
link to the author’s linkedin profile
