DataTrove

https://github.com/huggingface/datatrove

DataTrove is a library to process, filter and deduplicate text data at large scale, with prebuilt...

Tokens:13,022
Snippets:176
Trust Score:9.6
License:Apache-2.0
Update:2 weeks ago
Tokens:
Raw