May I just say kudos, sir, for dealing with this situation with such aplomb. It is easy to imagine an alternative response, with far more anger and less curiosity. You are like a doctor looking at a disease: "Ah, look at this awful thing happening, how interesting!"
Also, given the way they were using your site, effectively reverse-proxying you and adding ads, it implies that you have access, in your server logs at least, to all of their traffic! And that might give you insight into their motivations, and maybe other elements of their operations. I mean, it sounds like a reasonably clever, small scale scam operation in Russia; but if they proved out the technique with your niche site, then they can easily duplicate with other sites, in which case it is effectively a new kind of malware that has to be solved by Google!
Last but not least, I wanted to encourage you, and others, to consider whether this kind of attack would work in a decentralized world, what search looks like in that world, and therefore how this kind of attack might be mitigated.
Update: After a week of doing nothing - they finally noticed their thing is blocked and sprang into action.
Apparently, they expanded their pool of available IPs they pull data from and now they seem to be endless (so some of the scraping domains actually work now).
I'm investigating what I can do about it. I'd appreciate any advice!
Browsing through your SEO results. I also don't think the negative SEO is necessarily what did you in.
You have a very straight forward value prop. "Next episode" of some-show. I think these sort of optimized results are probably things that Google has been algorithmically adjusting for.
Looking at the Ranked 1-3 terms you dropped for, it seems you dropped some pretty big terms and even keyword terms.
You were #1 for "seal team next episode", but now you rank #3. #1 got replaced by CBS's page, which is arguably a better result.
"black clover new episode" also dropped from #1. Replaced by Wikipedia.
"the good place next episode" similar story.
I don't know what the best move is here. Algorithmic changes are really hard to combat without major changes and even then, you don't have a ton of room to wiggle with next-episode content.
Wow. That's absolutely horrible.
Looking at Google's search results, it's obvious that these tactics are rampant and really winning the war here.
We need a new search engine that cannot be gamed so easily. I know it's non trivial but the stakes are high as is the reward for making such.
This is a real engineering challenge. I'm excited about the problem space and opportunity.
Made an account here just to make this comment: You're going to want to send DMCA notices to both the registrar(s) AND Google.
1. Compile a list of domains and sitemaps that are 100% stealing and mirroring your content.
2. Go to Google's DMCA request page: https://www.google.com/webmasters/tools/legal-removal-reques...
3. Fill out all relevant data, and submit the offending domains and URL's.
Wait a few days, and you'll be happy to see that those pages are blocked from Google entirely. Not many people know what to do when Google DMCA's them, so it could solve your problem permanently (or you can automate it).
Regarding physically blocking them from scraping your site, you've got a few options. Put Cloudflare up if it isn't already. They've got at least one anti-scraping application (Scrape Shield) that may help.
Another thing you can do is automate the scraping of their websites using distinct query parameters and try to exhaust their list of proxies by automatically logging and filtering them. This might be a fruitless endeavor if they're using rotating residential proxies though.
Hope this helps, and good luck!
Sucks to see this. I think I even mentioned your site on here just this past week.
Didn't think I'd see the author but since you're here, thanks, this has been my go-to over the years.
It's an interesting story, I won'der if you could turn the automation trick around on them.
Would you be able to make them do the same negative SEOing but to their own site?
Fill their site with unrelated garbage and internal links with undesirable anchor text.
* unbock their IP * create content that links back to their site with the undesirable keywords * only show this content to them and not regular visitors * don't let them grab much / any legitimate content
Earlier today in another thread we were joking about GaaS (Goatse as a Service) but now maybe I think that's not so crazy an idea after all.
https://news.ycombinator.com/item?id=26104087
Your YC application practically writes itself.
Also, another update has been noticed for V-day celebrations: https://www.seroundtable.com/google-search-ranking-algorithm...
This is really frustrating, thanks for sharing. Google has had decades to figure out a way to detect duplicated content, spammy sites with this structure random-spam-keyword.spam-site.xyz/more-spam-words.html and the problem seems to get worse every year.
I feel you. First how bad that feels and second, the amount of time you need to spend in fighting these things :/
Of course Russians
That happens when it is legal to hack/steal/cause damage to people from other countries
Is all of your content pulled via javascript? Could a server side language prerendering the content be part of your solution. You can still use javascript for everything else just not the content.
Dmca the registrar.
Was/is your canonical set correctly?
I'm sorry to say, but the neg SEO didn't drop your rankings, it was to do with the Google algorithm update [1]. Check the screenshot from Ahrefs [2], and your traffic drops on 3rd of December which is when the update went live. [1] https://moz.com/blog/googles-december-2020-core-update [2] https://i.imgur.com/DBkdUEk.png
Google's algorithm is smart enough to recognise Neg SEO attacks. Sure five years ago you could buy a blast of spammy links using Xrumer or GSA with some viagra anchor text and boom you're competition is gone.
From a quick glance, most of your pages have pretty thin content, and I assume it's pulling from an API, so none of it is unique. If there was one thing I would do is try to build some content on pages. A great tool to analyse and develop content that is SEO friendly is SurferSEO - highly recommend it.
I'm surprised your forum doesn't rank as well as your main site as it looks fairly active. However, I'm not sure about how PunBB does SEO wise.