Comment on The Pile: An 800GB dataset of diverse text for language modeling (2020)parentComments−CamperBob23yIs there a straightforward way to download that dataset, the way there was for the original RedPajama data? SlimPajama appears to have been released as 60,000 small files, which is ridiculous.
Comments
Is there a straightforward way to download that dataset, the way there was for the original RedPajama data? SlimPajama appears to have been released as 60,000 small files, which is ridiculous.