Our group wants to do the reproducibility challenge as the second course project. We were looking at some papers published in the recent conferences, and there are roughly two types of papers:
Papers that use neural networks and deep learning models, which usually contain lots of experiments on lots of datasets.
Papers that use simpler models like regression models. Some examples are the papers suggesting sampling techniques to mitigate distribution shift or find a core-set of the training data. These papers usually have some theoretical contributions and some experiments to confirm their results.
Our main concern is the grading criteria, and we don't know what we are expected to deliver. If we want to redo the deep learning papers, we probably have to exclude some of their experiments from the larger datasets due to our limited computational power. On the other hand, the implementations of the papers in the second category are less complicated compared to the ones in the first category.
So we will appreciate any more explanation, particularly about what we are expected to deliver and what is considered a valuable outcome of this challenge.
there is no 'shortcut' here, as we will normalize by the difficulty of what you have done.
also check if the papers already have existing open source code, in which case your contribution needs to be explained in what it adds (or verifies) on top of that. (of course if a paper did not release existing code yet then this is not a concern)
Thank you very much for your reply. What about experiments of the deep learning papers? For example, if a paper includes experiments on both CIFAR and ImageNet and we only reproduce the CIFAR one because of the limited computational power?
yes this is fine in that case, we don't expect you to spend several GPU years.
once you already finished to reproduce smaller experiments, and think you would get additional valuable insights from larger scale experiments later feel free to contact me again later to see if we can use the EPFL IC cluster potentially
Questions about the reproducibility challenge
Hi,
Our group wants to do the reproducibility challenge as the second course project. We were looking at some papers published in the recent conferences, and there are roughly two types of papers:
Our main concern is the grading criteria, and we don't know what we are expected to deliver. If we want to redo the deep learning papers, we probably have to exclude some of their experiments from the larger datasets due to our limited computational power. On the other hand, the implementations of the papers in the second category are less complicated compared to the ones in the first category.
So we will appreciate any more explanation, particularly about what we are expected to deliver and what is considered a valuable outcome of this challenge.
3
there is no 'shortcut' here, as we will normalize by the difficulty of what you have done.
also check if the papers already have existing open source code, in which case your contribution needs to be explained in what it adds (or verifies) on top of that. (of course if a paper did not release existing code yet then this is not a concern)
Thank you very much for your reply. What about experiments of the deep learning papers? For example, if a paper includes experiments on both CIFAR and ImageNet and we only reproduce the CIFAR one because of the limited computational power?
yes this is fine in that case, we don't expect you to spend several GPU years.
once you already finished to reproduce smaller experiments, and think you would get additional valuable insights from larger scale experiments later feel free to contact me again later to see if we can use the EPFL IC cluster potentially
Add comment