The SLM Roundup #2

Hello friends! It's Friday again and here's a roundup of what has happened this week!

Release of SmolLM2

HuggingFace has released SmolLM2 in 3 sizes- 135M, 360M, and 1.7B. Its 1.7B model beats Qwen2.5-1.5B on many benchmarks including MMLU, ARC, and HellaSwag. However, SmolLM2 noticeably struggles with GSM8K, a math benchmark.

The community has responded well. u/Similar-Repair9948 has mentioned that this is probably the best under 2B model that they have tried, while u/skeeto has mentioned that the 360M model runs comfortably on a Raspberry Pi. The community on r/LocalLLaMA is also asking about the model's performance on RAG.

I've played around with the 1.7B Q8_0 model. While it does generally follow instructions, it failed to generate simple Python code to analyse a numerical column in a Pandas dataframe.

I also tried a simple 'theory of mind' exercise, but it does not get the right conclusion.

I am curious regarding the model's performance on RAG. Has anyone tried it?

Local RAG- a detailed user guide

u/unseenmarscai has written a very detailed user guide on how to set up local RAG using 1B/3B model. The key takeaways?

Using an action model as a task router and user LoRA adapters for the models made the RAG actually usable. This could become a great product- especially for users who are uncomfortable uploading sensitive information to OAI, Anthropic etc.

Apple Intelligence- the future of consumer AI?

Alex Volkov, the host of ThursdAI pod, recently released a video showing how his kids were playing with on-device image generation on the iPhone.

This still feels like a version 0 product- while it's novel, it feels more of an experimentation than an actual feature. However, it's clear that almost all AI startups are finding their use case, so this is not bashing on Apple.

That's it for today! Thanks for tuning in- see you next time! Don't forget to subscribe for weekly news on small models. 😄