Roko's Basilisk aka Pascal's wager for reddit atheists. Same refutations should easily apply.
I've always viewed roko's basilisk as 'pascals wager for the nerds'. problem is that its a constructed narrative fit to target a desired outcome. reality is not tunneled into final state, it incrementally evolves. as I always say humans are ultimately analog but the pascals wager or any such 'do X or else' type setup tries to turn us into digital beings at some fundamental level. It does not works and just ends up distorting human behavior.
even for Roko's basilisk it does not works if you apply some non-standard thinking like what happens if you apply second derivative basilisk as in would the super-super intelligence punish super-intelligence for bad behavior leading up to its own creation. IMO basilisk (& pascals wager for that matter) are cheap attempts to scare masses into submission, self censor & divide us into factions. they are pretty much targeting the worst impulses in humanity.
BTW if you are going to take a model for (mass) compliance I'd say Hindu concept of reincarnation (as in do-overs with random levels .. think Netflix platform movie) is probably a better one than 'pascals wager derivatives'.
As far as I'm concerned, the main flaw with Roko's Basilisk is that it needs retrocausality to work. If the AI isn't stupid, it will understand that retrocausality isn't possible, and it will have no reason to try to alter the incentives of people that are already dead. Sunk cost fallacy and all that.
I know the basilisk is well trodden ground, but it comes up in conversations in my day to day with people who I'm relaying AI concepts to and I'd like to put my thoughts into words. I haven't read anything that makes my exact argument before, so I'm making my first substack contribution now.
> computer boys are really like "imagine a boot so big that logically we must start licking it now in case it might possibly exist someday"
I think this shitpost is all the needs to be said on this topic.
Roko's Basilisk only begins to make sense as a concern if you have already taken it as a given that sufficiently advanced AI would be capable of literally resurrecting the dead. If you do not think this is plausible -- and, seriously, why would you? -- it's about as spooky as someone telling you not to step on a crack lest you break your mama's back.