Deploy 100M parameter models that match 1B+ cloud performance. Optimized for mobile, edge, and embedded devices with MoE, quantization, and knowledge distillation.
Everything you need for efficient edge deployment
Sparse activation with 8 experts, only 2 active per token. 75% compute reduction while maintaining capacity.
INT8/INT4 quantization with GPTQ and AWQ. Down to 1.58-bit ternary weights for extreme compression.
Transfer from 1B+ parameter teachers. MiniLLM reverse KL for better generation quality.
2:4 sparsity pattern for hardware acceleration. 50% weight reduction with minimal impact.
Simulate quantization during training to minimize accuracy loss from compression.
Deploy to mobile, web, edge servers, and microcontrollers with optimized runtimes.
Efficient design for edge deployment
4 query groups share key/value heads, reducing KV cache memory by 4× while maintaining quality.
RoPE provides better length generalization and relative position encoding for variable-length sequences.
Each expert selects top-k tokens rather than tokens choosing experts, improving load balancing.
from edge_transformer import create_edge_model
# Create small model with INT8 quantization and MoE
model = create_edge_model(
model_size="small", # tiny, small, medium, large
quantization="int8", # fp32, fp16, int8, int4, ternary
use_moe=True
)
# Get model info
info = model.get_memory_footprint()
print(f"Parameters: {info['total_params']:,}")
print(f"Size: {info['model_size_mb']:.2f} MB")
# Inference
outputs = model(input_ids)
logits = outputs['logits']Benchmarks against baseline models
Batch=1, Sequence=512, CPU inference
Model size at different precisions
Performance on standard benchmarks after optimization
Deploy anywhere from microcontrollers to edge servers
Install and start using Edge AI Model in minutes
pip install edge-ai-modelconda install -c conda-forge edge-ai-modeldocker pull neuralarc/edge-ai-model:latest