8

We have a client who is looking to upload about 5G per year of files onto client records in CiviCRM.

We can obviously arrange for the server to manage that, but wondered if anyone has ever notice speed issues with uploading so many files? They need to keep records for 7 years, so will eventually have around 35G of files attached to activities.

6

the files are stored outside of civi database, the database itself contains only the file name, it shouldn't be a problem

However, did you think about using a "real" document management system (ie. one that offer full search on the documents and revision and whatelse) and integrate it with civi? given the quantities you expect, might be easier for the users

  • I've been fiddling with attachments recently. Might want to consider how it sanitises filenames too- CRM_Utils_File::makeFileName and cleanFileName strip anything non-alphanumeric (including spaces), you won't necessarily ever be able to get back the original filename. – DaveFF Apr 4 '16 at 14:59
1

As the files are stored on disk, their size is not a significant issue, as @Xavier mentions in his answer. However, retrieving the files becomes slow at the operating system level in linux if you have a very large number files in the same directory - see answers 2 and following of https://stackoverflow.com/questions/466521/how-many-files-can-i-put-in-a-directory. Choosing the type of file system for the disk drive you store the files on can have a dramatic impact on performance (https://unix.stackexchange.com/questions/28756/what-is-the-most-high-performance-linux-filesystem-for-storing-a-lot-of-small-fi). Probably easiest optimization is to pay for an SSD drive rather than a HDD.

Separately, you may want to search these files both by content and by contact name. Consider using https://github.com/civicrm/apachesolr_civiAttachments.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.