21 articles tagged with #human-in-the-loop. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Syll is an open-source, self-hosted AI agent framework that enables personal automation across multiple interfaces—APIs, CLIs, web browsers, and desktop applications. The system allows users to teach agents through direct demonstration, compiling actions into reusable skills while maintaining transparency through multimodal logging and local artifact storage for inspection and control.
A comprehensive study of deployed LLM-based agents across 26 domains reveals that production systems rely on simple, human-centered approaches rather than complex automation. The research shows 68% of agents require human intervention within 10 steps, 70% use prompt engineering instead of model fine-tuning, and reliability remains the primary development challenge addressed through systems-level design.
Researchers introduce the NOVA framework, which models AI knowledge discovery as an adaptive sampling process and identifies fundamental scaling limitations. The analysis reveals a contamination trap where false positives accumulate faster than genuine discoveries as knowledge becomes scarce, with cumulative generation costs following a Zipf-distributed scaling law demonstrating asymptotic diminishing returns.
Researchers present an empirical study revealing that Large Language Models struggle with cyber threat intelligence (CTI) tasks due to domain-specific vulnerabilities rather than generic AI failures. The study identifies three failure modes—spurious correlations, contradictory knowledge, and constrained generalization—and proposes targeted defenses to improve LLM reliability in security operations.
Researchers introduce flow control, a technique that enables real-time steering of vision-language-action (VLA) models through simple user inputs like keyboards without requiring model retraining. The method allows users to guide robot actions toward their intent while maintaining high-quality outputs aligned with the model's learned expert distribution, improving task success rates and completion times.
Researchers introduced DeskCraft, a new benchmark for evaluating AI desktop agents on complex, long-horizon professional workflows in creative and engineering software. The study reveals significant performance gaps, with GPT-4 achieving only 31.6% accuracy on standard tasks and 27.6% on interactive tasks requiring human collaboration, highlighting challenges in multi-step automation and proactive agent communication.
Researchers developed a hybrid system combining formal symbolic planning with large language models to improve capability-based planning in industrial automation. The system integrates natural-language interaction, explainability, and human-approved knowledge model adaptation, achieving high accuracy across planning and query tasks while maintaining formal correctness guarantees.
SmartIterator is a visual analytics framework that helps data scientists systematically evaluate and choose between multiple unsupervised learning results across parameter sweeps. The approach operationalizes structured six-phase workflows for three clustering and topic-modeling method families, enabling informed decision-making by visualizing data grouping quality, stability, membership confidence, and domain context simultaneously.
Researchers propose an algorithm that uses large language models to generate portfolios of optimization models rather than single outputs, addressing the reliability gap in LLM-generated solutions. The method leverages LLMs in dual roles—as generative and evaluative components—with theoretical guarantees that high-quality candidates appear in the portfolio as long as either role aligns with human preferences.
Researchers have developed an AI Teaching & Learning Assistant, a Moodle plugin using Retrieval-Augmented Generation (RAG) to provide students with Socratic tutoring while enabling educators to supervise content generation. The system grounds LLM responses in teacher-provided materials to minimize hallucinations and misinformation, achieving high faithfulness scores (0.97) and strong user satisfaction (4.00/5.00 rating).
Researchers propose a trust framework for AI agent skills—reusable code packages that extend language models—treating them as untrusted by default until verified. The approach introduces verification levels, capability gates, and correctness criteria to enable sustainable human-in-the-loop oversight without operational bottlenecks.
Researchers introduce a lightweight LLM agent architecture that uses first- and second-order state dynamics to model gradual clinical concern escalation rather than abrupt threshold-based responses. The approach makes AI decision-making more transparent by revealing sustained risk signals before escalation, enabling better human oversight in clinical settings.
Researchers propose a reactor-model-of-computation approach using the Lingua Franca framework to address nondeterminism challenges in AI-powered human-in-the-loop cyber-physical systems. The study uses an agentic driving coach as a case study to demonstrate how foundation models like LLMs can be deployed in safety-critical applications while maintaining deterministic behavior despite unpredictable human and environmental variables.
Researchers developed a human-in-the-loop LLM system for grading handwritten mathematics assessments that reduces grading time by 23% while maintaining accuracy comparable to manual grading. The system combines automated scanning, multi-pass LLM scoring, consistency checks, and mandatory human verification to handle pen-and-paper tests at scale.
Researchers introduce DexHiL, a human-in-the-loop framework for improving Vision-Language-Action models in robotic dexterous manipulation tasks. The system allows real-time human corrections during robot execution and demonstrates 25% better success rates compared to standard offline training methods.
Researchers introduce PONTE, a human-in-the-loop framework that creates personalized, trustworthy AI explanations by combining user preference modeling with verification modules. The system addresses the challenge of one-size-fits-all AI explanations by adapting to individual user expertise and cognitive needs while maintaining faithfulness and reducing hallucinations.
Researchers developed a framework for analyzing AI diagnostic systems in clinical settings by preserving original AI inferences and comparing them with physician corrections. The study of 21 dermatological cases showed 71.4% exact agreement between AI and physicians, with 100% comprehensive concordance when using structured analysis methods.
MuTSE is an interactive web application designed to evaluate Large Language Model outputs for text simplification tasks across multiple prompting strategies and proficiency levels. The tool addresses a methodological gap in NLP research by providing researchers and educators with a structured, visual framework for comparing prompt-model combinations in real-time.
Researchers developed CODE-GEN, a human-in-the-loop AI system that uses retrieval-augmented generation to create multiple-choice programming questions for educational purposes. The system achieved 79.9% to 98.6% success rates across seven pedagogical dimensions when evaluated by subject-matter experts, demonstrating strong performance in computational verification tasks while still requiring human expertise for complex instructional design.
Research comparing human-in-the-loop versus automated chain-of-thought prompting for behavioral interview evaluation found that human involvement significantly outperforms automated methods. The human approach required 5x fewer iterations, achieved 100% success rate versus 84% for automated methods, and showed substantial improvements in confidence and authenticity scores.
Researchers conducted a qualitative study analyzing Human-in-the-Loop (HITL) themes in AI application development through diary studies and expert interviews. The study identified four key themes around AI governance, iterative refinement, system lifecycle constraints, and human-AI collaboration to guide future HITL framework design.
