Skip to main content
XenonStack

Enterprise AI

XenonStack Blog Human-Centred AI: Design Principles for Enterprises

and | Published | Updated

Key takeaway

  • Human-centred AI designs systems around people. Learn the core principles, NIST AI RMF and EU AI Act oversight points, and an enterprise adoption sequence.

Human-centred AI is the practice of designing AI systems around what people need, can understand and can control, and then checking that the technology and the business case support that. In an enterprise, it means named human owners, clear explanations, a way to override or stop the system, and care with personal data.

This guide gives general readers and enterprise leaders the principles, the human oversight points in the NIST AI Risk Management Framework and the EU AI Act, and a practical adoption sequence. For the wider governance picture, see our guide to ethical standards for AI governance.

What is human-centred AI?

Human-centred AI starts with people and asks what they want and what they need, then asks what is technically feasible and what is viable for the organisation. A product is not built just because the technology is interesting. The people who will use it, and the people it will affect, shape the design from the first step.

The idea has a research history. In 2019 Microsoft Research welcomed the launch of Stanford's Human-Centered AI Institute, and noted that the design of human-AI interaction experiences was called out as an important area requiring further study. Microsoft also publishes 18 guidelines for human-AI interaction, applied at four points: first use, during use, when the system is wrong, and over time.

Many people prefer the word augmentation to the word artificial. The aim is for AI to extend what people can do, with the person still accountable for the outcome.

What are the principles of human-centred AI when designing with data?

Eight principles are widely used when teams design with data. The table turns each into a design question and a practical step. The right-hand column points to the NIST characteristic it most relates to; the mapping is our own reading, not NIST wording.

PrincipleDesign questionPractical stepRelated NIST characteristic
Make lives betterDoes the system help the person, or only the process?Define the human benefit before the model metric.Valid and reliable
Better inform and guide organisationsDoes it help a decision-maker act with better evidence?Show the evidence and the confidence, not only the answer.Explainable and interpretable
Unlock new insightsCan it surface patterns people cannot review at scale?Have domain experts check each new pattern before it drives action.Valid and reliable
Make the previously hidden visibleDoes it reveal links people would miss?Keep the reasoning traceable so a reviewer can challenge it.Accountable and transparent
Support positive behaviour changeDoes it nudge with consent, and support the person?Pair data insight with human support, and let people opt out.Safe
Best use of time, effort and resourcesDoes it free people for judgement and creative work?Automate routine steps; keep people on the exceptions.Valid and reliable
Include vulnerable populationsWho is missing from the data, and who bears the risk?Test with the groups affected, and review for harmful bias.Fair with harmful bias managed
Protect privacyDo people know what is collected, and can they choose?Collect the minimum, state the purpose, record consent.Privacy-enhanced

Do not treat data as the truth. People create data, and people choose the data sets used to train models, so bias and blind spots can enter at every step. Review what a data set leaves out before relying on it.

What do NIST and the EU AI Act say about human roles?

The NIST AI Risk Management Framework was released on 26 January 2023 and is for voluntary use. NIST says it is being revised, so check the NIST page for the current version. Its guidance on trustworthy AI characteristics names seven: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed.

Two-column diagram mapping four human oversight roles (understand, guard against over-trust, override, stop) to the design features that enable them

On human roles, NIST says human judgement should be used when choosing the metrics for these characteristics, that deploying AI is the joint responsibility of all AI actors, and that trade-offs between characteristics should be resolved transparently and justifiably. It also says human intervention may be needed when a system cannot detect or correct its own errors.

Article 14 of the EU AI Act requires high-risk AI systems to be designed so that natural persons can effectively oversee them while in use. As appropriate and proportionate, overseers should be able to understand the system and monitor it, stay alert to over-reliance on its output, interpret that output, decide not to use it or override it, and intervene or stop it safely. Application dates for high-risk rules have been changing, so confirm the current timetable on the official EU pages.

Human roleWhat the design must provideSource
Understand and monitorDocumentation and live performance viewsEU AI Act Article 14
Guard against over-trustConfidence signals and prompts to check the outputEU AI Act Article 14
Override or reverseA clear way to reject or undo an outputEU AI Act Article 14
Stop or interveneA stop control or equivalent procedureEU AI Act Article 14; NIST (shut down, modify or intervene)
Set the metricsHuman judgement on thresholds for each characteristicNIST AI RMF

How do you balance what people want with what is feasible and viable?

Three questions need to be answered together. Is it desirable for the people involved? Is it feasible with the data and the technology? Is it viable for the business? A system can be technically possible and still fail on the first or the last. Because AI makes feasibility look easy, teams should also ask whether a problem is better fixed by a change to the process or the team rather than by a model.

How does human-centred AI apply to AI agents?

Agents act, not only advise, so the human controls must sit in the workflow. Useful patterns are recommend-only modes, approval for high-impact actions, least-privilege access and a decision record. Our guide to agentic AI risk management sets these out, and agentic AI governance and compliance covers the policy side. AgentGRC keeps autonomous agent actions inside policy, with the audit trail risk teams need, and ElixirData Context OS keeps governed context, versioned policy and decision traces in one decision layer above AI agents.

How should an enterprise adopt human-centred AI?

StepWhat to doOutput
1. Name the peopleList users, affected groups and the accountable ownerStakeholder map
2. Set the human benefitState the benefit and the harm to avoidSuccess and harm criteria
3. Review the dataCheck sources, gaps, consent and purposeData review record
4. Design oversightAdd explanation, override and stop controlsOversight design
5. Test with real usersInclude the groups most affectedTest findings
6. Monitor over timeTrack errors, complaints and drift; review on a scheduleReview cadence

Impact also arrives at scale. A routing app that sends many drivers through one quiet neighbourhood helps each driver and burdens the neighbours. Review effects on groups and communities, not only on individual users. Adoption problems of this kind are covered in AI adoption challenges; for a regulated sector, see ethical AI in healthcare, and for service teams, responsible AI in customer service.

When is human-centred design not enough?

Good design does not replace controls. A human in the loop who cannot see the evidence, has no time to review, or is rewarded for speed gives little real oversight. Design can also be sound for the intended user and harmful when data is reused for another purpose later, so state the purpose and review it when strategy or ownership changes. Consider the benevolent use and the non-benevolent use of the same data before you collect it.

To map one decision your AI should make, the human who owns it, and the policy it depends on, book a Context Workshop. See what happens in the workshop.

FAQ

What is human-centred AI?

Human-centred AI means designing AI systems around the needs, understanding and control of the people who use them or are affected by them, rather than building a product only because the technology is interesting.

What are the principles of human-centred AI when designing with data?

Eight principles: make lives better, better inform and guide organisations, unlock new insights, make the previously hidden visible, support positive behaviour change, make the best use of time, effort and resources, include vulnerable populations, and protect privacy.

Which framework should enterprises use for trustworthy AI?

The NIST AI Risk Management Framework is a voluntary framework. Its guidance names seven characteristics: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed. NIST says the framework is being revised, so check its page.

What does the EU AI Act say about human oversight?

Article 14 requires high-risk AI systems to be designed so people can effectively oversee them, including understanding outputs, guarding against over-reliance, overriding the system and stopping it. Confirm current application dates on the official EU pages.

How is human-centred AI different from keeping a human in the loop?

A human in the loop is one control. Human-centred AI covers the whole design, from the problem and the data to explanation, override, stop controls and review over time.

Follow XenonStack on LinkedIn: XenonStack on LinkedIn for new guides on agentic AI, data and platform engineering.

Ready to put AI agents to work? Explore ElixirClaw, the agentic OS platform to build, orchestrate and govern AI agents.

Table of Contents

navdeep-singh-gill

Navdeep Singh Gill

Global CEO and Founder of XenonStack

LinkedIn profile

Navdeep Singh Gill is serving as Chief Executive Officer and Product Architect at XenonStack. He holds expertise in building SaaS Platform for Decentralised Big Data management and Governance, AI Marketplace for Operationalising and Scaling. His incredible experience in AI Technologies and Big Data Engineering thrills him to write about different use cases and its approach to solutions.

Get the latest articles in your inbox

Subscribe Now

Interested in solving your challenges with the XenonStack team?
Get Started

Tell us about you and your priorities so we can prepare the right conversation. It takes about 2 minutes.

Please Select your Industry
Banking
Fintech
Payment Providers
Wealth Management
Discrete Manufacturing
Semiconductor
Machinery Manufacturing / Automation
Appliances / Electrical / Electronics
Elevator Manufacturing
Defense & Space Manufacturing
Computers & Electronics / Industrial Machinery
Motor Vehicle Manufacturing
Food and Beverages
Distillery & Wines
Beverages
Shipping
Logistics
Mobility (EV / Public Transport)
Energy & Utilities
Hospitality
Digital Gaming Platforms
SportsTech with AI
Public Safety - Explosives
Public Safety - Firefighting
Public Safety - Surveillance
Public Safety - Others
Media Platforms
City Operations
Airlines & Aviation
Defense Warfare & Drones
Robotics Engineering
Drones Manufacturing
AI Labs for Colleges
AI MSP / Quantum / AGI Institutes
Retail Apparel and Fashion

Continue