Vector Databases

The knowledge assistant's pilot searched its 50,000 chunks in the simplest way there is: multiply the question's vector against every stored vector and keep the eight highest scores. Once the full corpus is loaded, the same few lines of numpy compare each question with 1.04 million vectors of 1,536 dimensions, which is 1.6 billion multiply-adds and 6.39 GB of memory read per question. On the four-vCPU cloud machine used to measure this chapter's numbers, one such scan took between 0.6 and 0.8 seconds. Chapter 1's latency budget allows dense and keyword search together 80 milliseconds at the 95th percentile, so the scan alone overran it eight to ten times over, before the model had seen anything.