`nemo_gym.hf_utils`#

Module Contents#

`create_huggingface_client`
`check_jsonl_format`	Check for the presence of the expected keys in the dataset
`download_hf_dataset_as_jsonl`	Download a HF dataset and save as JSONL. If `artifact_fpath` is provided, downloads that specific file using `hf_hub_download`. Otherwise, uses datasets.load_dataset() to handle structured datasets.
`upload_jsonl_dataset`

nemo_gym.hf_utils.create_huggingface_client(token: str) → huggingface_hub.HfApi[source]#

nemo_gym.hf_utils.check_jsonl_format(file_path: str) → bool[source]#: Check for the presence of the expected keys in the dataset

nemo_gym.hf_utils.download_hf_dataset_as_jsonl( config: nemo_gym.config_types.DownloadJsonlDatasetHuggingFaceConfig, ) → None[source]#: Download a HF dataset and save as JSONL. If artifact_fpath is provided, downloads that specific file using hf_hub_download. Otherwise, uses datasets.load_dataset() to handle structured datasets.

nemo_gym.hf_utils.upload_jsonl_dataset( config: nemo_gym.config_types.UploadJsonlDatasetHuggingFaceConfig, ) → None[source]#