Beyond PoC - Lessons from Implementing AI in a Mid-Sized Biopharma Company
Let's Talk session
14th October 2026
Moderator: Stefan Thys
Meeting Slides
Meeting summary:
Scaling AI Beyond Proof Of Concept: Stefan explained how a mid-sized biopharmaceutical company moved approximately ten AI projects beyond proof of concept over two years by combining platform capabilities, conventional IT delivery, governance, validation and business involvement.
Platform Foundation: Stefan said successful proofs of concept exposed the need for a shared platform before wider deployment. Initial projects were selected as non-GxP, non-critical use cases with potentially high impact, including question-and-answer tools for statements of work and policy documents. Platform decisions included hosting location, SaaS versus local deployment, ownership, software choices and alignment with the organisation’s existing technology ecosystem. The company selected Microsoft and Azure because it was already a Microsoft environment; Microsoft also recommended a supplier that became a delivery partner.
Supporting Technology: Stefan noted that supporting functionality can account for approximately 80% of project expenditure, rather than the AI model itself. The architecture may require document or data downloads, cleaning, version management, intermediate tables, vector databases, document conversion, monitoring, traceability, translation and secure presentation or storage of results. Documents were converted to PDF where useful for standardisation and extraction of graphics and footnotes, while spreadsheets required different handling; text chunking was relatively straightforward, but charts, images and graphs remained more complex.
Process Analysis: Stefan advised starting with traditional IT and process analysis rather than deciding in advance that a problem must be solved with AI. Teams should understand the current process and confirm that AI adds value before implementation. He also advised retaining conventional approaches where they are simpler or cheaper, such as using SQL and intermediate tables instead of asking an AI system to navigate unnecessarily complex source structures.
Evolutionary Architecture: The team moved from a basic retrieval-augmented architecture to more advanced hybrid and multi-agent approaches, but Stefan stressed that the technology remains immature and should be reviewed within roughly six to twelve months. Teams should use the least complex and least expensive architecture that meets user needs, avoid upgrading models merely because newer versions exist, and select models according to their strengths; for example, Mistral performed better than ChatGPT for some difficult scanned documents. Prompts and agents should be kept as atomic as possible, with examples included where appropriate.
AI Security Validation And Observability: Stefan described AI as non-deterministic software requiring continuous monitoring, repeated testing, access controls and
specialist security and compliance involvement, while Chris and Alpesh questioned how data validation and model replacement should work in regulated or changing environments.
Model Drift: Stefan explained that conventional software normally gives the same result for the same valid input after release, whereas AI systems are non-deterministic and can drift in quality over time. Observability is therefore needed to check whether agents and retrieval systems remain within defined quality parameters. Test questions should be rerun regularly, potentially every few days or at least monthly rather than being left unchecked for long periods.
Data Validation: In response to Chris, Stefan said that data-cleaning steps in a validated or GxP system must be described, understood and validated. He recommended deterministic, conventional code for cleaning rather than AI unless the data are exceptional. Data-transfer steps also need documented lineage, metadata and validation so the organisation understands what data are present and how they move from source to destination.
Access Controls: The implementation copied Veeva security settings and checked access at document-chunk level. Consequently, two users could receive different answers because the system only used material each person was authorised to access. Stefan said this behaviour must be tested carefully. AI services should also use separate accounts rather than operating under individual user accounts, with appropriate logging and additional security controls.
Continuous Testing: For a non-deterministic system, Stefan recommended substantial testing at release rather than relying on one or two examples. The team used statistical analysis based on Cochran’s formula to estimate test-set size; as an illustration, approximately 385 tests could be needed for 95% certainty under particular assumptions. The team was working towards repeating these tests monthly and reviewing changes in results over time.
Model Replacement: In response to Alpesh, Stefan said organisations should assume that an LLM may need to be replaced because suppliers change models or availability can be affected by external decisions. Changing the LLM requires prompt review and retesting because the replacement may interpret prompts differently and produce different results. The team reduced some continuity risk by using established or European models and not attempting to follow every new model release.
Governance And Regulatory Delivery: Stefan described an integrated project-management, validation and legal process in which compliance, quality, legal, security, infrastructure, procurement, finance and users participate from the outset and approval processes keep pace with AI development.
Legal Involvement: Stefan advised involving legal specialists familiar with both the EU AI Act and the organisation’s business as soon as a project receives an identifier. Legal should also be informed if a project stops or restarts. The team developed approval standards so that defined combinations of data and LLMs could be handled through notification or automatic approval rather than repeated individual approvals.
Validation Methodology: The company created an integrated project-management, validation and legal methodology based on milestones and required documents. The first version was deliberately strict and cumbersome; people were trained, changes initially required approval, and the process was progressively adapted based on experience. Stefan said shortcuts should not be taken, but teams should be allowed to propose controlled simplifications once the rationale for the controls is understood.
Compliance Readiness: Stefan recommended that compliance and quality teams understand the specific AI tools, objectives and operating environment, particularly in validated settings. Several projects had previously been stopped because compliance had not been involved or did not understand the implementation. He also recommended using established computer-system-validation and AI guidance, such as relevant professional documents, rather than creating unsupported practices that could fail during inspection.
Organisational Controls: AI projects can expose weaknesses in existing security, data catalogues, infrastructure and approval processes, creating related improvement work. Stefan said this should not prevent teams from starting, because the AI implementation can later be reviewed and integrated with improved foundations. However, approval processes must operate quickly enough to match the pace at which AI solutions become outdated.
People Adoption And Operating Model: Stefan and Chris discussed the importance of aligning business and IT, involving users, providing focused training and professionally reworking successful grassroots experiments before they become important company systems.
User Awareness: Chris asked how users should choose between different AI models and levels. Stefan distinguished between personal use of general AI tools and applications that use AI in the background. In an embedded application, users may not need to know which model is used, but they should be told enough to understand that outputs are non-deterministic and require judgement. For individual use, the organisation limited the available tools and provided training on prompting and appropriate use cases.
Adoption Factors: When Chris asked whether this approach achieved adoption, Stefan said adoption depended on factors including training and individual attitudes. People who viewed AI as a partner adopted it more readily, while technically capable people could still choose not to engage. He therefore recommended involving users rather than excluding them from IT-led delivery.
Professionalisation: Stefan compared employee-built AI solutions with complex spreadsheets created outside IT: such experiments can be creative and useful, but once they become important or critical to the company, they should be taken into professional development, security, validation and support processes.
Business And IT Alignment: Chris raised the recurring problem of business and IT wanting different outcomes before a proof of concept begins. Stefan said AI does not remove this longstanding problem; it can make it worse when stakeholders insist that an undefined problem must be solved with AI. Teams should establish shared vocabulary, communicate sufficiently, verify requirements and retain standard project-delivery fundamentals.
Training Scope: Stefan recommended internal training for technical teams, infrastructure, operations, compliance, quality and other stakeholders. Training should cover not only AI capabilities but also the specific tools being used, technical risks, legal requirements and the organisation’s objectives. External suppliers should train their own staff, but organisations must retain sufficient internal knowledge to govern and operate the solutions.
Data Architecture And Information Access: Stefan explained that AI can often begin with operational data rather than waiting for a data warehouse, provided access, structure, security and lineage are managed appropriately.
Operational Data: In response to Chris’s question about data products and warehouses, Stefan said modern AI can increasingly work directly with the operational data used by people. A warehouse may still be needed for other reasons, but lack of one should not automatically delay useful AI delivery. In his example, the organisation delivered useful AI applications first and reworked them later when a data warehouse became available.
Data Preparation: Stefan said data are rarely perfectly clean, but if people can work with them, they may already be sufficiently usable for an AI application. Clinical and production data often have one or more human cleaning steps already applied. He recommended starting without additional cleaning unless testing shows that it is necessary, while applying documented deterministic cleaning where validation requires it.
Security Inheritance: The platform inherited source-system permissions so that responses were limited to material accessible to the requesting user. Stefan also warned that direct AI access to systems such as Veeva, Benchling or accounting applications could allow unintended data changes, making application-to-application communication, logging and permission boundaries important design concerns.
Investment And Return On Investment: Pascal questioned how organisations can justify AI investment when token costs vary and projects may fail to scale, while Stefan distinguished an initial capability-building phase from a later phase in which ROI becomes a formal requirement.
Scaling Economics: Pascal highlighted that an AI solution can appear valuable in team testing but lose its economic case when expanded across the organisation. They also noted that variable token usage makes costs difficult to forecast and that value may include quality improvements as well as direct financial savings. Stefan acknowledged the difficulty and said that project value should be reassessed as scope and scale change.
Initial Investment: Stefan said the early stage should allow the organisation to gain practical experience with the technology through carefully selected projects without making ROI the only immediate criterion. He described this as an initial investment in building capability and understanding how AI works in the organisation.
Maturity Transition: Stefan acknowledged that this approach eventually has to change. In his experience, a later restructuring of the IT department marked a more mature phase in which ROI became important. He did not provide a specific ROI calculation, but separated exploratory capability-building from subsequent business cases and financial accountability.
Cost Baseline: Stefan gave an indicative figure of approximately EUR 200,000 for one policy-document application excluding the platform, including document downloads, cleaning and version management. He also warned that supporting technology and automated testing can represent substantial costs, particularly because repeated AI tests consume tokens.
Practical AI Selection Principles: Stefan concluded that AI should be used selectively, after process analysis and testing demonstrate a real need, with business users defining expected answers and conventional methods retained where they are more suitable.
Expected Outputs: Stefan said teams should provide prompts or agents with representative examples and define in advance what a correct answer should be. Business users are essential for judging whether actual outputs are consistent with the intended result, so testing cannot be left solely to technical teams.
Classic Alternatives: Stefan repeatedly advised checking whether a conventional solution is adequate before selecting AI. Simple calculations, SQL transformations or other deterministic approaches may be cheaper, easier to validate and more reliable than an AI component.
Project Selection: The recommended starting point was non-GxP, non-critical work with meaningful potential impact, such as document question-and-answer tools. As experience and governance mature, teams can consider more consequential applications with appropriate validation, security and legal controls.
Additional questions:
1. We’re a 20-person biotech with no internal IT function, and our two bioinformaticians are now using AI to build internal tools and UIs. Without IT or software engineering support to lean on, how are other small companies deciding which AI-built tools need formal validation, and what minimum controls have proved proportionate?
Stefan: I would recommend Devote consulting, located in Copenhagen. I have very good experience with them. They have supported us throughout the complete project and they learned together with us.
2. For UK start-ups, where would you recommend finding legal expertise on the EU AI Act, and since the field is still so new, how have you judged whether an adviser really has the depth you need?
Stefan: I would not limit myself to the UK legal councils. Let me find out who supported us on the projects. For a UK based company: Not sure if the EU AI act will have impact on them; this would depend if they are or plan to work in the EU (or if the UK joins the EU again). It all depends what data they are using. A quick search on “EU AI act, risk levels” in Google will give a description of the 4 risk levels the EU AI act describes and they can then see where they fit in. If indeed they run a risk, they can involve legal council.
Comments