Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add script sys path back to test #797

Closed
wants to merge 4 commits into from
Closed

Add script sys path back to test #797

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5487d37
Add script sys path back to test
aspfohl Dec 11, 2023
d97eeb6
Apply suggestions from code review
aspfohl Dec 11, 2023
6aeaf1f
Update tests/a_scripts/inference/test_convert_composer_to_hf.py
aspfohl Dec 11, 2023
40af72e
Merge branch 'main' into anna/syspath
dakinggg Dec 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 15 additions & 7 deletions tests/a_scripts/inference/test_convert_composer_to_hf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,26 +4,34 @@
import math
import os
import pathlib
import sys
from typing import Callable
aspfohl marked this conversation as resolved.
Show resolved Hide resolved
from unittest.mock import ANY, MagicMock, patch

from composer import Trainer
from composer.loggers import MLFlowLogger
from composer.utils import dist, get_device, using_torch_2
aspfohl marked this conversation as resolved.
Show resolved Hide resolved

from llmfoundry.callbacks import HuggingFaceCheckpointer
from llmfoundry.models.mpt.modeling_mpt import ComposerMPTCausalLM

# Add repo root to path so we can import scripts and test it
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How come this wasn't handled by the fixtures? / Why were these tests passing in CI?

repo_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
aspfohl marked this conversation as resolved.
Show resolved Hide resolved
sys.path.append(repo_dir)
import shutil
from argparse import Namespace
from typing import Callable, Optional, cast
from unittest.mock import ANY, MagicMock, patch
from typing import Optional, cast
aspfohl marked this conversation as resolved.
Show resolved Hide resolved

import pytest
import torch
import transformers
from composer import Trainer
from composer.loggers import MLFlowLogger
from composer.utils import dist, get_device
from omegaconf import DictConfig
from omegaconf import OmegaConf as om
from torch.utils.data import DataLoader
from transformers import PreTrainedModel, PreTrainedTokenizerBase

from llmfoundry import COMPOSER_MODEL_REGISTRY
from llmfoundry.callbacks import HuggingFaceCheckpointer
from llmfoundry.data.finetuning import build_finetuning_dataloader
from llmfoundry.models.mpt.modeling_mpt import ComposerMPTCausalLM
from llmfoundry.utils.builders import build_optimizer, build_tokenizer
from scripts.inference.convert_composer_to_hf import convert_composer_to_hf
from tests.data_utils import make_tiny_ft_dataset
Expand Down
Loading