Large Language Models with MLX

Provenance

Independent

Scope

Early experiment, kept for reference

Artifacts

Source code

An early experiment: a command-line chat that runs 7B Mistral and Llama 2 models locally on Apple Silicon with MLX, with persona prompts and weight conversion to MLX format. No benchmarks or evaluation.

Experiment, not a case study. I built this in late 2023, shortly after Apple released MLX, to see what running a 7B chat model locally on a laptop involved. It is kept here for reference. It has no tests, benchmarks, or evaluation, so it is not evidence of anything beyond “it runs”.

What it is

A small Python package (about 1,000 lines) with a command-line chat interface for local inference on Apple Silicon:

• Models: a decoder-only transformer written against MLX, loading Mistral-7B-Instruct (v0.1 and v0.2), Llama-2-7B-chat, and OpenHermes-2.5-Mistral-7B, with one chat template per model family.
• Weight conversion: utilities that convert raw PyTorch or safetensors checkpoints, or Hugging Face checkpoints, into MLX’s .npz format.
• Personas: a few-shot system prompt per character, plus example dialogues, selected with a CLI flag.

python llm_w_mlx/cli/llm.py --personality dwight --model Mistral-7B-Instruct-v0.1 \
  --weights path_to_weights --tokenizer path_to_tokenizer --max_tokens 500

What it does not include

• No quantization. The models run at the precision the converted weights were saved in.
• No throughput or memory measurements, and no comparison against llama.cpp or PyTorch MPS.
• No tests and no evaluation of response quality.

What would make it evidence

Turning this into evidence would mean measuring tokens per second and peak memory against a reference runtime at fixed prompts, and checking output parity against the reference implementation. I have not done that, and the repository has been inactive since 2024.