We were talking about AI in #wetfish the other day and I remarked that it would be useful if there was some distributive computing scheme for AI related computation. Each participant would donate the spare cycles on their computer/phone/smart refrigerator/etc to train models and/or infer queries.
Hence an AI for the people!
From each computing to their ability, to each computing to their needs!
Users of the world, unite! You have nothing to lose but your surveillance!
Seize the memes of production!
Etc.
Sounds nice right? But there are enormous challenges involved. Fortunately, wetfish has decades of experience creating software and working in the community. I thought an informal discussion around the problems involved might be interesting to see if the project is even feasible. I have included a lightly formatted transcript of people's initial thoughts.
Luke
back in the day, they had distributed computing projects to analyze SETI signals and protein folding. I wonder if some kind of project like that could work for FOSS LLMs. "The People's AI"
goos
You can offload to cpu memory but the performance hit is massive on nearly all platforms
You can split across gpus and accept a perf hit on operations that have to cross gpus
Doing shit over the network is still a rdma level question at this point
The peoples ai would have to be shitloads of small models all being coordinated somehow
Luke
> goos
> The peoples ai would have to be shitloads of small models all being coordinated somehow
My suggestion comes from a sheer lack of ignorance or the underlying technologies, but could you use a block chain (on whoever's machine, let's put that question aside for now) to co-ordinate the process?
Mozai
LLMs on the blockchain! for your smarthome! install it on your Zune!
blubb
in the metaverse!
Mozai
whoop right I skipped one, thank you
rachel
> Luke
>> goos
>> The peoples ai would have to be shitloads of small models all being coordinated somehow
> My suggestion comes from a sheer lack of ignorance or the underlying technologies, but could you use a block chain (on whoever's machine, let's put that question aside for now) to co-ordinate the process?
A block chain is like a global lottery - nobody is really coordinating except by agreeing that a new block is valid.
Distributed AI compute would be far more complicated. You'd have to be able to split the task into multiple chunks that can all be processed in parallel and each individual instance would be computing a smaller part of the whole problem
It's more akin to folding@home than a blockchain
Mozai
i wonder how you'd split up the matrix-math among different threads, nevermind different remote cores. Would the slowdown caused by network i/o be insufferable?
Luke
folding@home was the project I was thinking of.
Mozai
deepfakes@home
Luke
> Mozai
> i wonder how you'd split up the matrix-math among different threads, nevermind different remote cores. Would the slowdown caused by network i/o be insufferable?
I assume so, but I feel like there is a clever solution to that, that only works for distributed computing
goos
Mozai your typical “rack ‘o gpus” setup has 8x800gigE interfaces on each host for rdma between hosts - one per gpu
And then another 2x400 or something for frontend network
The slowdown by network is Still a massive pain point to design around at that scale
Mozai
the "folding@home" analogy suggests many cheap machines connected by 10Mbit/s pipes already clogged up with unrelated traffic, so more like 5Mbit/s or worse.
goos
Folding@home is much more “dispatch and chew and report back” vs an ai workload
Mozai
aye
works if you can cleanly chunk-up the work
goos
Exactly
Thats why my picture of the citizens ai is a lot of small models with some outside coordination of some sort
Think about how such a platform would be used and it starts to make sense
Luke
if I'm running a client, my machine could run the coordination for my own asks and reciprocally, donate its extra cycles and such to other requests
goos
yes
Mozai
that's economics, babee! How to rearrange resources so they do the most good!
goos
Good timeslicing and pooling would be pillars of that design
but at a more macro scale i sense
Than what those terms typically mean in gpu space
Maybe you could use a computationally cheap blockchain tech as the ledger of like…. Whatever currency you design as the “i did your work so now you do mine” method
But i sense theres better ways to achieve the goal that that fufills
Luke
> goos
> Maybe you could use a computationally cheap blockchain tech as the ledger of like…. Whatever currency you design as the “i did your work so now you do mine” method
I was thinking more like a ledger of what requests have been fulfilled and which ones need more work, but if you're own machine did the coordination then you wouldn't need it
goos
yeah idk what that looks like yet
but the whole idea seems somewhat viable