Designing Safe and Accountable GenAI as a Learning Companion with Women Banned from Formal Education

Researchers conducted a participatory design study with 20 Afghan women excluded from formal education to understand how generative AI can safely support their learning and career development. The study reveals that women view GenAI as a compensatory peer and mentor rather than an information source, while identifying critical needs around privacy protection, cultural safety, and pedagogically sound guidance.

This research addresses a critical gap in GenAI development by centering the needs of marginalized populations facing severe educational barriers. The study demonstrates that GenAI tools, when designed with accountability and safety at their core, can serve transformative purposes beyond information retrieval—functioning as mentorship proxies in contexts where human mentoring networks are inaccessible. The participatory design approach itself yielded significant psychological benefits, with participants showing measurable increases in aspirations, perceived agency, and perceived pathways to employment after envisioning GenAI applications tailored to their circumstances.

The research contextualizes a broader challenge in AI development: most GenAI systems are optimized for users with stable access to connectivity, privacy, and formal institutional support structures. Women in gender-restrictive environments face compounding constraints—surveillance risks from household members or state actors, limited offline resources, competing domestic responsibilities, and cultural contexts where certain educational paths are deemed inappropriate. These constraints make generic GenAI solutions inadequate or potentially harmful.

The identified design priorities—safety-first interactions, user control mechanisms, context-grounded support under resource constraints, and pedagogically aligned assistance—establish a framework applicable beyond this specific population. The finding that direct-answer interactions undermine genuine learning has implications for how educational institutions and AI developers approach responsible AI deployment. Organizations building AI tools for educational purposes should prioritize these design directions to ensure their products genuinely enable learning rather than creating false progress markers. The research suggests that accountable GenAI design can simultaneously reduce harm and expand opportunity for vulnerable populations.

• →GenAI functions as a compensatory peer and mentor for women excluded from formal education, requiring safety and cultural-context design considerations.
• →Participatory design processes themselves generated measurable increases in participants' aspirations, agency, and perceived employment pathways.
• →Privacy and surveillance risks represent critical constraints that generic GenAI systems fail to address for users in restrictive environments.
• →Pedagogically sound assistance that supports genuine learning outperforms direct-answer interactions that create illusions of progress.
• →Safety-first design frameworks with user control mechanisms are essential for responsible GenAI deployment in marginalized communities.