What is Google Drive... for?
I see folks describe Google Drive errors / limits like this and people describe "a business critical operational system" and I wonder. Is Google Drive even supposed to do this thing?
Not pointing the finger back at them, if Google doesn't make it clear they probably should, but at least as an individual I always see Google Drive as a single user cloud light system that allows for some sharing and organizational functions... but still not an "industrial" type cloud service.
5M items sounds suspiciously like one machine has to keep in RAM the complete list of a users files for certain operations. User accounts with large numbers of objects were probably causing those machines to OOM. 5M items, at about 1000 bytes per item (name, metadata, a few uuids, etc) is 5Gbytes. 5Gbytes is about the amount of RAM most tasks will be given.
I could imagine that some services within Google haven't been carefully designed - for example, perhaps the quota service reads a complete file listing into RAM, adds up the sizes, and then writes the available quota.
This is probably the easy fix, rather than redesigning every service to be able to stream objects correctly.
While a 5 million file limit sounds reasonable at first glance, it isn't just the free tier that is affected. Google Drive also sells storage plans for up to 30 TB for an individual user, and there is no mention of such a limitation anywhere on any plan. If I bought and actually wanted to use that 30 TB (or heck even 5 TB), it isn't unreasonable to want to store a few million files. Regardless of the technical reasons behind it, not mentioning this restriction up front is before the sale is inexcusable.
As one of the comments points out, Google sells 30TB accounts. With a 5M file limit, you need an average file size of 6MB to fill your drive.
Rug pull. That's what this is.
The Google Drive landing page (https://www.google.com/intl/en-US/drive/) as of April 1, 2023 still doesn't mention the 5M per user maximum files count limit.
Is it an April fool's joke? /s.
Do better $GOOG.
The key is that every company is very happy to lose this kind of customers.
Sounds like they want to protect against resource exhaustion. Touch a file, it's empty with a size of zero bytes. But depending on the block size of the storage device, it's likely to be between 4k and up to even 1M.
It seems like there is a much lower limit for "download all". There is a 100 or so item directory that I want to download, and "download all" tries to wrap them in a zip file and fails. I think it wants to pre-scan all the files for viruses before zipping, but is unwilling to do that many. So I have to download the files one at a time.
They also have what looks like an arbitary limit on "Shared Drives"
The number of stored files: 400,000
Someone must've copied their node_modules folder into Google Drive by mistake...
Interesting that this was enabled with apparently no outreach. Maybe Google needed some of those employees that it recently laid off
Part of this seems like some of the wonky design that seems to be under the hood of Drive. Just looking around at how rclone has to do things, the difference between shared and individual drives, the fact I couldn't even originally find a way in the console to see my shared drive usage (even though I was the owner of it and it was on my account) it just seems like there is a very very wonky odd design to how the system works. Not like an object store under the hood with a layer of "document" logic applied, but rather what started as initially some API-esque thoughts on document storage that grew into a monstrosity. I wouldn't be shocked if there's just a ton of tech debt there.
I'm surprised it knows how to count that low.
So are there different limits for Google for Business/Workspace/whatever they are branding it this week? Or is this for all Google drive accounts?
Well this is scary, I wonder if all users/plans are impacted? I have north of 8.5million files in GDrive and I luckily have not hit an error yet.
Tangent: I was wrong to ever consider it sus that Google includes gsuite with their cloud revenue. Clearly it’s being used that way.
Maybe Google cloud using NTFS disks (Maximum number of files on disk: 4,294,967,295)? :))
Just checked, my C drive has a bit more than 1 million files. What are you doing that you have 5 million separate files in your Google Drive?
From reading the thread it sounds like this is 5M per user.
Google Drive appears to be targeted at "human" usage, i.e. people uploading or creating files. I would guess that this is also worked into the cost – that the assumptions of the amount of work a human can do are a part of the price formulation. The reporter of this bug seems to be using this as a storage backend for software though, which I don't believe is the intended use-case.
Looking at S3 pricing, just the storage is ~5x the cost of Google Drive, and then you need to add transfer and API calls on top of that.
I don't personally think that there are reasonable use-cases for human users with 5 million files. There may be some specialist software that produces data sets that a human might want to back up to Google Drive, but that software is unlikely to run happily on drive streamed files so even those would be unlikely to be stored directly on Drive.
(Disclaimer, I work at Google, not on Drive, this is my personal reading and interpretation of the public info, I don't have any inside info here)