docs: what if we also want to separate metadata from their target (source?) of raw data?

[tianjianjiang]

Background

For replying to @timoschick 's questions (via Slack DM), my thought gets stuck with the question that may sounds similar to what @cccntu asked with #12: what is the expected interfaces for us to collaborate?

Three Sub-questions

1. mC4, one of my preferred datasets, comes with URLs and timestamps directly. Do we still want to have our own JSONL files?
2. Some datasets for https://github.com/bigscience-workshop/evaluation may also work for us, e.g., GEM (those originated from Wikipedia or WikiHow), CRD3, WikiANN, etc. The (1) situation appears again an may be a bit more complicated.
3. (1)+(2), suppose we also want to use some parts from https://github.com/bigscience-workshop/promptsource, especially the "applying templates" part, which I noticed that https://github.com/bigscience-workshop/evaluation has been duplicating and my gut tells me such duplication can be avoided, just not sure whether we can get helps from data-tooling.

Thoughts

• Again, like @cccntu aforementioned in (WIP) Add loading script for arxiv dataset #12, shall we go for quick solutions or not?
  • To my limited knowledge, fastai got some callback function based transformation;
  • Or we simply want to have our own copies of datasets.
• Perhaps we will have more than one Python packages.
  • The current one;
  • The one that may be shared with promptsource, evaluation, metadata, and probably more?

Side Notes

• The title of this issue follows https://github.com/angular/angular/blob/master/CONTRIBUTING.md#commit , just my habit;
  • Which sometimes overlaps with GitHub's default Labels, and yet in my opinion, labels should serve different purpose (e.g., search filters);
• The milestone and the project attached to this issue ticket are also just my habit... My apologies;
• I took liberty of assigning some of us, pardon me for the intrusion or the unintended exclusion;
• Perhaps we will also need some issue ticket templates and code owner file (and a PR review policy?).

[changjonathanc]

For context

We (w/ Timo and Lucile) have discussed on slack about #12 . From my understanding, we would prefer that everyone just convert the dataset and save it to our format in another script, before running code in this repo.

Personally, I think my approach in that PR is also fine, we can merge that to a "feature" branch for reproducibility purpose. (wandb even saves each run to an unnamed branch)

---

• mC4

Do you really plan to train on that? If so, it won't make sense to convert and save the dataset, and you should probably do something like what I did. And if most people need to do that, we can refactor the code.

Also want to note that the timestamps in C4 (probably mC4, too) are all in a one month range, meaning the crawl date, not the actual date the text is written.

• evaluation

Again, we have discussed on slack about evaluation, and we currently don't plan to add more evaluation during training, and we expect each experiment to do evaluation outside of our codebase, after training.

To my limited knowledge, fastai got some callback function based transformation;

What do you mean here? Are you suggesting allowing callback to map/preprocess the dataset?

[tianjianjiang]

@cccntu

Thank you for the reply!

• mC4

Do you really plan to train on that?
If so, it won't make sense to convert and save the dataset, and you should probably do something like what I did. And if most people need to do that, we can refactor the code.

No strong preference here.
Suppose (m)C4 is too big to process locally, I can also turn to smaller datasets from GEM, for example.

Also want to note that the timestamps in C4 (probably mC4, too) are all in a one month range, meaning the crawl date, not the actual date the text is written.

Indeed. One of reasons why I tried Common Crawl based corpora like OSCAR and (m)C4 is that I find it intriguing to compare timestamps with URLs containing dates. However, it is just a very minor point.

@timoschick and I are more interested in various traits of URLs, such that timestamp-related behavior just one of them. Although I reported something about dates of URL during our latest meeting, it was only because I wanted to have some progress and date is the only trait of URL that can be easily quantified.

• evaluation

Again, we have discussed on slack about evaluation, and we currently don't plan to add more evaluation during training, and we expect each experiment to do evaluation outside of our codebase, after training.

When I was talking about evaluation in this PR, the context is probably always reusing dataset/code from Evaluation and other WGs, not really about running additional evaluation during/after training.

To my limited knowledge, fastai got some callback function based transformation;

What do you mean here? Are you suggesting allowing callback to map/preprocess the dataset?

Yes, exactly!

[timoschick]

To my limited knowledge, fastai got some callback function based transformation;

What do you mean here? Are you suggesting allowing callback to map/preprocess the dataset?

Yes, exactly!

My thinking was that for our first experiments, it would be much easier to have a unified format jsonl format that everyone can work with so that you don't need to understand any internals of this codebase (and don't need to write a single line of code) to train and evaluate models. Of course, the underlying assumption is that for our initial experiments, we only use small datasets to continue pretraining our models (e.g., 100k randomly sampled documents from mC4).

I'm not sure whether it makes sense to add custom solutions for each dataset we want to consider in the current phase, given that in the end, we will be using only a single dataset to which data-tooling will provide access. However, if providing all data in a jsonl file does not work for you, something like a custom callback function (as used by fastai) works best in your case and you have the time to implement this solution, feel free to do so!

We can also discuss this with the rest of the group on Friday?

[SaulLu]

Super interesting all these suggestions @tianjianjiang ! And it's very nice to explain your coding practices!

I agree with @timoschick , I think it would be nice to put forward a training pipeline that works without adding a line of code for all the metadata so that someone joining the project can very quickly understand how he/she can run a training. For the moment, it seems to me that the jsonl format chosen was a good compromise to be able to represent all types of metadata.

It seems to me that the idea of the first experiments was to extend the pre-training on reasonable sizes of dataset and that it is thus for this reason that the pre-processing stage is made upstream of the training (thanks to the map method) and not in parallel of the training (as one could consider doing it with a collate_fn). When we discussed this point with @timoschick and @cccntu, we decided to keep this solution for the moment because it was the simplest. However, we were clear that these choices were by no means set in stone and that we could make further developments according to the needs of future experiments. <- I mention this because I don't think we had in mind to pre-process all of mC4 when we discussed it.

With the current pipeline, I think it will be clearer to separate the individual preprocessing of each dataset to conform to the chosen format from the core library bsmetadata.

That being said, I was planning to suggest on Friday that we add another folder at the root of the repository to store the end-to-end code - from a dataset to training and evaluation metrics - specific to each experiment so that this code is not "lost" and that each experiment can document its specificities. Ideally, this folder - different from bsmetadata - could really be identified as second reading for users of the library. For example, for HTML metadata I have to code a parser (which requires a bit of work), it might be a waste to lose it. Maybe this new folder would also be a good place if you ever want to use specific code for your experiments too (with monkey patching or more properly calbacks). EDIT: This last suggestion could also be conditioned by whether or not these experiments are carried out on the Jean Zay supercomputer which will add constraints.

I just want to raise a point, if we don't do any specific dataset pre-processing in the core of the library, maybe we would gain some clarity by removing the experiments folder currently in bsmetadata.

[timoschick]

That being said, I was planning to suggest on Friday that we add another folder at the root of the repository to store the end-to-end code - from a dataset to training and evaluation metrics - specific to each experiment so that this code is not "lost" and that each experiment can document its specificities.

I just want to raise a point, if we don't do any specific dataset pre-processing in the core of the library, maybe we would gain some clarity by removing the experiments folder currently in bsmetadata.

I fully agree with both suggestions!

[tianjianjiang]

@SaulLu @timoschick Thank you for suggestions. They totally make sense to me, too. I will keep this issue ticket open until our next meeting for the record.