SCF Round 5 Nominations Postmortem
First off I want to thank everyone for their participation and care around the SCF. This is truly a community fund and I want to always foster that core value. We are also a fund however and giving away any money but especially a lot of money comes with a pretty large set of challenges. Regardless of how you feel right now though the SDF and this community has done a good job with the SBC and SCF. Could we do better? Absolutely. Never be duped though into thinking we don’t care, aren’t intentional or are just throwing money away carelessly. That is not true and never has been. Good projects have come out of the SBC and SCF. Tools and apps we use every day, people you know, talk to and appreciate. For some, like me, it has actually changed our lives entirely. The community funding programs the SDF has put on have been phenomenal. It’s incredibly difficult to give money away well in a public or decentralized way but our efforts have not gone unrewarded. Please don’t lose the forest for the few trees which didn’t make it. Every SCF has gotten better and SCF 2.0 will continue those improvements in a more concerted and exponential way. I’m here for you if you all will continue to be there for each other and this fund. Keep showing up, voicing your opinions respectfully and pushing this program into better and better places. We are a growing and maturing community, remember this is not just about you and your project, it’s bigger than that.
With that said, in the spirit of transparency I would like to take this opportunity to pull back the curtain on the numbers and criteria of the SCF nomination voting process. Let’s kick things off with a graph:
The first thing to notice is that nomination participation in this round has nearly tripled compared to previous rounds. That’s a huge jump. The next obvious thing to notice is that vote removal is on a very aggressive exponential curve upwards while the leftover valid votes have remained relatively static. One thing this chart doesn’t show is that our criteria for removing votes has always stayed the same throughout all of these rounds. Up until now we’ve chosen not to publish those criteria as that would make circumventing those criteria much easier. I’ve had a hunch for a while that a day would come where our criteria and the verification process we’ve been using would crack and we’d see the exponential curve we saw this round. This is why SCF 2.0 is so essential for the survival of the SCF. A panel of judges is necessary for the nomination phase and better voting mechanics is essential to help counteract, not just catch bad actors. Suffice it to say we’re not going to be using Keybase verification anymore, it’s entirely insufficient for distinguishing between malicious and new voters. It’s relatively high friction for participation yet that friction doesn’t assist us much in helping to distinguish between legitimate and illegitimate voters.
Our criteria for voter disqualification for every round has been:
- Duplicate IPs with sufficient account profile evidence to suggest they could be the same person
- All accounts created on or after the round announcement date with zero proofs, no bio info, no additional devices and which only voted for 3 projects
- All accounts created during the nomination week
These criteria all have significant flaws but they are right for Keybase as a verification method. Duplicate IPs could just be a roommate voting for his buddies app which would be fine, but they could also be indicative of a click farm or the same fellow once on his laptop and again on his phone or tablet. I'm fine with the roommates, but click farms and multiple devices from the same person is cheating. The issue is I can't know for certain, and that's the rub, you can rarely know for certain, you can only give it your best guess and lean on the most fair side. Disqualifying all accounts created on or after the round announcement date is hardly fair but if the account in question has added zero social proofs, no bio info, only has the single device and only voted for 3 projects, that’s a very strong signal they could have been paid, coerced or manipulated into voting for a specific project. Removing all new Keybase accounts during the nomination week doesn’t actually remove many votes, but it's pretty obvious if you're only now creating an account, you don't care about anything but voting for a single project which A) means two of your votes are meaningless throw away votes or B) could mean you've been paid, coerced or manipulated into voting for that specific project.
The final and primary glaring issue with these criteria is they get weaker and weaker over time as sock puppet accounts get older, add fake proofs, bios and devices. Accounts that were caught in previous rounds get harder and harder to detect appropriately and soon most votes won’t mean anything as the majority will have to be removed in order to ensure fairness leaving only a select handful of original “verified” accounts. Aka a panel of judges. What SCF 2.0 will do for the nomination phase is simply formalizing the inevitable. I want to include more voters, I want to lower the barrier to entry. I want friends and family who don’t care about Keybase and adding proofs and all this mumbo jumbo to be able to participate, and they will, just not in the nomination phase. Once the right projects are in, actually distributing the prize pool becomes much more straightforward and traditionally has seen orders of magnitude less vote removal anyway.
So in SCF 2.0 between a panel of judges for the nomination phase and SMS voter verification we’ll be in a much better place to ensure the most fair and still inclusive community fund.
I feel for those projects caught in the crossfire of this imperfect process but there is no perfect open online voting system. What we have has actually worked incredibly well, the top 8 actually usually doesn’t shift much beyond a few projects. I’m also not accusing anyone of manipulation and neither should you. Oftentimes we just fall victim to a bigger process which attempts to make the best of a tough situation. If it smells like a bot, acts like a bot and looks like a bot that doesn’t mean it is but we have no choice but to treat it like it is.
Finally if any entrant would like to know their own numbers in this round PM me and I’d be happy to share. I’m choosing not to publish those numbers here publicly in order to protect those who were more heavily affected by our cuts. I will say the before and after only changed the position of 3 projects and that is very common when we make our cuts, very little changes for the majority.
Thank you once again for your care and participation. I want to do right by this community. I’m confident together we can do it if we respect one another and push for bigger goals than just our own fistful of cash.
