Agentic AI Motivator for Entrepreneurs

Agentic Voice AI Motivator for Entrepreneurs Overview A 2-week solo sprint researching, architecting, and building a fully local, agentic AI mentor on Apple Silicon. Processes voice/text queries to deliver dynamic, personalized entrepreneurial guidance via streamed audio. Key Outcomes Tri-Layered Stateful RAG: Engineered a LanceDB vector database featuring three distinct data layers (mentor_knowledge, assistant_memory, user_memory) that intelligently updates memory state after every turn. Hybrid Search & Retrieval: Scaled embeddings to 700+ dimensions (nomic-embed-text-v1.5 via fastembed). Built a hybrid search pipeline combining semantic and full-text keyword matching, optimized with Reciprocal Rank Fusion (RRF). Local Audio & Quantization: Integrated zero-shot voice cloning using Chatterbox Turbo (fp16, selected over 4-bit for audio fidelity) and OpenAI Whisper STT. Performance & Memory Optimization: Solved Apple Silicon memory overloads via strict process isolation and model garbage collection. Slashed Time-to-First-Audio (TTFA) from 5–6 minutes to ~42s using local model caching, function call restructuring, and audio streaming. AI-Assisted Dev & Manual RLHF: Directed Gemini for schema design and search upgrades. Applied intensive end-to-end RLHF auditing to refine agent responses and ensure reliable, high-utility interactions. Acquired Skills & Tech Stack AI/ML: Agentic Systems, RAG, Local LLMs (Qwen 2.5 7B), Zero-Shot TTS, Whisper STT, Model Quantization, Apple MLX Framework. Data Engineering: LanceDB Vector DB, Hybrid Search, Reciprocal Rank Fusion (RRF), High-Dimensional Embeddings (fastembed). Core Stack & Systems: Python, Rust, JavaScript, HTML, Memory Management (Garbage Collection, Process Isolation), Latency Reduction, Manual RLHF. ------ I learnt key terminology for agentic AI. I researched how I can build a local voice AI agent where I could clone a sample voice file and implement RAG. I built the voice AI agent step-by-step. I used Gemini models for coding & learning. I directed Gemini many times. For example, I directed it to write code and design a vector database that had three types of data: mentor_knowledge, assistant_memory, user_memory. The AI agent is powered by python for the AI part, javascript, html, and rust for the app. The app supports both voice and text queries. The app initially used Kokoto TTS, which turned out to not have voice cloning. I switched to another zero-shot TTS which includes high-fidelity voice cloning. My app is able to transcribe a given video, convert it and a given article to a 700+ dimension vector. The app initially had 300+ dimensions vectors, in v1. I changed to 700+ dimensions vectors in v2, by changing from all-MiniLM-L6 which used sentence-transformers to nomic-embed-text-v1.5 with fastembed. The latency of the app dropped significantly. Implemented a hybrid search model after initially having a simple semantic search: combining semantic search + full text search to enable precise keyword matches as well. Directed Gemini to upgrade the software's search logic: learnt about rerankers, and implemented a so-called Reciprocal Rank Fusion algorithm to choose the best candidate chunks from the hybrid search. Learnt how RAG works. Implemented a local vector database using LanceDB. Implemented a stateful user_memory.json which is intelligently updated by the agent after every turn. Implemented RLHF (reinforcement learning from human feedback) to fix errors and ensure a useful interaction. Learnt debugging by hands-on building the software, by using print statements in different locations of the software to understand the flow of the app. Manually changed function calls* to enable faster first time to audio (for example the mentor AI started to speak around +- 40 seconds after submitting a (text) query). * such as those that insert new memory to the vector database. Hands-on directed Gemini meticulously to implement MLX (Apple's deep learning framework) for the different tools, after I discovered that it gave bad advice in the form of tools that were not suitable for Apple Silicon. Released two final versions in a private repository, where v2 is a fully functional complex AI voice agent. I tested zero-shot voice cloning of the 4-bit variant of chatterbox turbo, and it was very low quality the cloning ability, so I switched back to fp16. Directed numerous times to debug, optimize, and upgrade different parts of the software. Learnt how to view a vector database using Data Wrangler in VS Code, and export .csv to manually view the (latest) entries. Tech stack also includes the MLX variants of Qwen 2.5 7b 4-bit (open-source LLM), Whisper (open-source from OpenAI) STT (speech-to-text), and Chatterbox Turbo (open-source TTS). Future upgrades could be parent-child retrieval technology for advanced RAG.