y0news
AnalyticsDigestsSourcesTopicsRSSAICrypto

#token-level News & Analysis

2 articles tagged with #token-level. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles
AIBullisharXiv โ€“ CS AI ยท Mar 37/106
๐Ÿง 

Token-level Data Selection for Safe LLM Fine-tuning

Researchers have developed TOSS, a new framework for safely fine-tuning large language models that operates at the token level rather than sample level. The method identifies and removes unsafe tokens while preserving task-specific information, demonstrating superior performance compared to existing sample-level defense methods in maintaining both safety and utility.

AIBullishHugging Face Blog ยท Oct 16/107
๐Ÿง 

Introducing RTEB: A New Standard for Retrieval Evaluation

The article introduces RTEB (Retrieval-augmented generation with Token-level Evaluation Benchmark), a new standard for evaluating retrieval systems in AI applications. This benchmark aims to provide more granular and accurate assessment of how well retrieval systems perform at the token level rather than traditional document-level metrics.