For our project, we have a large dataset, around 6 GB which we would like to include. We also have multiple pre-trained models totaling around 20 GB, and we were wondering how we should handle these large files as GitHub rejects the commits. An idea of ours was to host it on a google drive folder using our EPFL account. Would this be an acceptable solution?
You can use a piece of code that downloads the desired files when they are needed in your computation from your Google Drive folder (which must be public). This is the way we used to make sure we had everything we wanted.
How to handle really large files
Hello,
For our project, we have a large dataset, around 6 GB which we would like to include. We also have multiple pre-trained models totaling around 20 GB, and we were wondering how we should handle these large files as GitHub rejects the commits. An idea of ours was to host it on a google drive folder using our EPFL account. Would this be an acceptable solution?
Thanks!
Hi,
You can use a piece of code that downloads the desired files when they are needed in your computation from your Google Drive folder (which must be public). This is the way we used to make sure we had everything we wanted.
Add comment