A Rust SentencePiece tokenizer implementation

Tokenization into words or sub-word units is a key component of Natural Language Processing pipeline. Modern approaches such as Byte Pair Encoding (Sennrich et al., 2015), WordPiece or SentencePiece (Kudo et al., 2018) segment rare words into sub-tokens i… Read more

Read full article

Similar

MS: Here’s why we love programming language Rust and kicked off Project Verona

Mozilla-created programming language Rust could one day help Microsoft kill a large chunk of its worst security bugs. (more…)

I write a Rust client library without 3rd party libraries recently

Redis client for Rust. Contribute to ltoddy/redis-rs development by creating an account on GitHub. (more…)

Making slow Rust code fast

Performance tuning using Criterion.rs and flamegraphs... (more…)

My rust adventure continues as I have been furiously working on Rust/WinRT for the last five months or so. I am looking forward to opening it up to the community as soon as possible. Even then, it … (more…)

Learning Rust in 2020

Educational blog posts for Rust beginners. Contribute to pretzelhammer/rust-blog development by creating an account on GitHub. (more…)

Latest News

The “it” in AI models is the dataset

CMA seeks views on AI partnerships and other arrangements

Py2app – Create standalone Mac OS X applications with Python