{"id":13492913,"url":"https://github.com/google/model_search","last_synced_at":"2025-05-14T23:07:28.186Z","repository":{"id":37671085,"uuid":"331071860","full_name":"google/model_search","owner":"google","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-30T21:36:15.000Z","size":228,"stargazers_count":3259,"open_issues_count":52,"forks_count":526,"subscribers_count":90,"default_branch":"master","last_synced_at":"2025-04-19T22:27:29.466Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-19T18:26:34.000Z","updated_at":"2025-04-17T14:35:12.000Z","dependencies_parsed_at":"2024-11-20T06:01:59.307Z","dependency_job_id":null,"html_url":"https://github.com/google/model_search","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fmodel_search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fmodel_search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fmodel_search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fmodel_search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google","download_url":"https://codeload.github.com/google/model_search/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254243362,"owners_count":22038046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T19:01:10.415Z","updated_at":"2025-05-14T23:07:23.178Z","avatar_url":"https://github.com/google.png","language":"Python","readme":"# Model Search\n\n![header](https://raw.githubusercontent.com/google/model_search/master/model_search/images/model_search_logo.png)\n\nModel search (MS) is a framework that implements AutoML algorithms for model architecture search at scale. It\naims to help researchers speed up their exploration process for finding the right\nmodel architecture for their classification problems (i.e., DNNs with different types of layers).\n\nThe library enables you to:\n\n* Run many AutoML algorithms out of the box on your data - including automatically searching\nfor the right model architecture, the right ensemble of models\nand the best distilled models.\n\n* Compare many different models that are found during the search.\n\n* Create you own search space to customize the types of layers in your neural networks.\n\nThe technical description of the capabilities of this framework are found in\n[InterSpeech paper](https://pdfs.semanticscholar.org/1bca/d4cdfbc01fbb60a815660d034e561843d67a.pdf).\n\nWhile this framework can potentially be used for regression problems, the current\nversion supports classification problems only. Let's start by looking at some\nclassic classification problems and see how the framework can automatically find competitive\nmodel architectures.\n\n## Getting Started\nLet us start with the simplest case. You have a csv file where the features are numbers\nand you would like to run let AutoML find the best model architecture for you.\n\nBelow is a code snippet for doing so:\n\n```python\nimport model_search\nfrom model_search import constants\nfrom model_search import single_trainer\nfrom model_search.data import csv_data\n\ntrainer = single_trainer.SingleTrainer(\n    data=csv_data.Provider(\n        label_index=0,\n        logits_dimension=2,\n        record_defaults=[0, 0, 0, 0],\n        filename=\"model_search/data/testdata/csv_random_data.csv\"),\n    spec=constants.DEFAULT_DNN)\n\ntrainer.try_models(\n    number_models=200,\n    train_steps=1000,\n    eval_steps=100,\n    root_dir=\"/tmp/run_example\",\n    batch_size=32,\n    experiment_name=\"example\",\n    experiment_owner=\"model_search_user\")\n```\n\nThe above code will try 200 different models - all binary classification models,\nas the `logits_dimension` is 2. The root directory will have a subdirectory of all\nmodels, all of which will be already evaluated.\nYou can open the directory with tensorboard and see all the models with the\nevaluation metrics.\n\nThe search will be performed according to the default specification. That can be found in:\n`model_search/configs/dnn_config.pbtxt`.\n\nFor more details about the fields and if you want to create your own specification, you\ncan look at: `model_search/proto/phoenix_spec.proto`.\n\n### Image data example\nBelow is an example of binary classification for images.\n\n```python\nimport model_search\nfrom model_search import constants\nfrom model_search import single_trainer\nfrom model_search.data import image_data\n\ntrainer = single_trainer.SingleTrainer(\n    data=image_data.Provider(\n        input_dir=\"model_search/data/testdata/images\"\n        image_height=100,\n        image_width=100,\n        eval_fraction=0.2),\n    spec=constants.DEFAULT_CNN)\n\ntrainer.try_models(\n    number_models=200,\n    train_steps=1000,\n    eval_steps=100,\n    root_dir=\"/tmp/run_example\",\n    batch_size=32,\n    experiment_name=\"example\",\n    experiment_owner=\"model_search_user\")\n```\nThe api above follows the same input fields as `tf.keras.preprocessing.image_dataset_from_directory`.\n\nThe search will be performed according to the default specification. That can be found in:\n`model_search/configs/cnn_config.pbtxt`.\n\nNow, what if you don't have a csv with the features or images? The next section shows\nhow to run without a csv.\n\n## Non-csv, Non-image data\nTo run with non-csv data, you will have to implement a class inherited from the abstract\nclass `model_search.data.Provider`. This enables us to define our own\n`input_fn` and hence customize the feature columns and the task (i.e., the number\nof classes in the classification task).\n\n```python\nclass Provider(object, metaclass=abc.ABCMeta):\n  \"\"\"A data provider interface.\n\n  The Provider abstract class that defines three function for Estimator related\n  training that return the following:\n    * An input function for training and test input functions that return\n      features and label batch tensors. It is responsible for parsing the\n      dataset and buffering data.\n    * The feature_columns for this dataset.\n    * problem statement.\n  \"\"\"\n\n  def get_input_fn(self, hparams, mode, batch_size: int):\n    \"\"\"Returns an `input_fn` for train and evaluation.\n\n    Args:\n      hparams: tf.HParams for the experiment.\n      mode: Defines whether this is training or evaluation. See\n        `estimator.ModeKeys`.\n      batch_size: the batch size for training and eval.\n\n    Returns:\n      Returns an `input_fn` for train or evaluation.\n    \"\"\"\n\n  def get_serving_input_fn(self, hparams):\n    \"\"\"Returns an `input_fn` for serving in an exported SavedModel.\n\n    Args:\n      hparams: tf.HParams for the experiment.\n\n    Returns:\n      Returns an `input_fn` that takes no arguments and returns a\n        `ServingInputReceiver`.\n    \"\"\"\n\n  @abc.abstractmethod\n  def number_of_classes(self) -\u003e int:\n    \"\"\"Returns the number of classes. Logits dim for regression.\"\"\"\n\n  def get_feature_columns(\n      self\n  ) -\u003e List[Union[feature_column._FeatureColumn,\n                  feature_column_v2.FeatureColumn]]:\n    \"\"\"Returns a `List` of feature columns.\"\"\"\n```\n\nAn example of an implementation can be found in `model_search/data/csv_data.py`.\n\nOnce you have this class, you can pass it to\n`model_search.single_trainer.SingleTrainer` and your single trainer can now\nread your data.\n\n## Adding your models and architectures to a search space\nYou can use our platform to test your own existing models.\n\nOur system searches over what we call `blocks`. We have created an abstract API\nfor an object that resembles a layer in a DNN. All that needs to be implemented for this class is\ntwo functions:\n\n```python\nclass Block(object, metaclass=abc.ABCMeta):\n  \"\"\"Block api for creating a new block.\"\"\"\n\n  @abc.abstractmethod\n  def build(self, input_tensors, is_training, lengths=None):\n    \"\"\"Builds a block for phoenix.\n\n    Args:\n      input_tensors: A list of input tensors.\n      is_training: Whether we are training. Used for regularization.\n      lengths: The lengths of the input sequences in the batch.\n\n    Returns:\n      output_tensors: A list of the output tensors.\n    \"\"\"\n\n  @abc.abstractproperty\n  def is_input_order_important(self):\n    \"\"\"Is the order of the entries in the input tensor important.\n\n    Returns:\n      A bool specifying if the order of the entries in the input is important.\n      Examples where the order is important: Input for a cnn layer.\n      (e.g., pixels an image). Examples when the order is not important:\n      Input for a dense layer.\n    \"\"\"\n```\n\nOnce you have implemented your own blocks (i.e., layers), you need to register them with a \ndecorator. Example:\n\n```python\n@register_block(\n    lookup_name='AVERAGE_POOL_2X2', init_args={'kernel_size': 2}, enum_id=8)\n@register_block(\n    lookup_name='AVERAGE_POOL_4X4', init_args={'kernel_size': 4}, enum_id=9)\nclass AveragePoolBlock(Block):\n  \"\"\"Average Pooling layer.\"\"\"\n\n  def __init__(self, kernel_size=2):\n    self._kernel_size = kernel_size\n\n  def build(self, input_tensors, is_training, lengths=None):\n```\n\n(All code above can be found in `model_search/blocks.py`).\nOnce registered, you can tell the system to search over these blocks by\nsupplying them in `blocks_to_use` in `PhoenixSpec` in\n`model_search/proto/phoenix_spec.proto`. Namely, if you look at the default specification\nfor `dnn` found in `model_search/configs/dnn_config.pbtxt`, you can change the\nrepeated field `blocks_to_use` and add you own registered blocks.\n\nNote: Our system stacks blocks one on top of each other to create tower\narchitectures that are then going to be ensembled. You can set the minimal and\nmaximal depth allowed in the config to 1 which will change the system to search\nover which block perform best for the problem - I.e., your blocks can be now\nan implementation of full classifiers and the system will choose the best one.\n\n## Creating a training stand alone binary without writing a main\nNow, let's assume you have the data class, but you don't want to write a `main`\nfunction to run it.\n\nWe created a simple way to create a `main` that will just train a dataset and is\nconfigurable via flags.\n\nTo create it, you need to follow two steps:\n\n1. You need to register your data provider.\n\n2. You need to call a help function to create a build rule.\n\nExample:\nSuppose you have a provider, then you need to register it via a decorator we\ndefine it as follows:\n\n```python\n@data.register_provider(lookup_name='csv_data_provider', init_args={})\nclass Provider(data.Provider):\n  \"\"\"A csv data provider.\"\"\"\n\n  def __init__(self):\n```\n\nThe above code can be found in `model_search/data/csv_data_for_binary.py`.\n\nNext, once you have such library (data provider defined in a .py file and\nregistered), you can supply this library to a help build function an it will\ncreate a binary rule as follows:\n\n```build\nmodel_search_oss_binary(\n    name = \"csv_data_binary\",\n    dataset_dep = \":csv_data_for_binary\",\n)\n```\n\nYou can also add a test automatically to test integration of your provider with\nthe system as follows:\n\n```build\nmodel_search_oss_test(\n    name = \"csv_data_for_binary_test\",\n    dataset_dep = \":csv_data_for_binary\",\n    problem_type = \"dnn\",\n    extra_args = [\n        \"--filename=$${TEST_SRCDIR}/model_search/data/testdata/csv_random_data.csv\",\n    ],\n    test_data = [\n        \"//model_search/data/testdata:csv_random_data\",\n    ],\n)\n```\n\nThe above function will create a runable binary. The snippets are taken from the\nfollowing file: `model_search/data/BUILD`.\nThe binary is configurable by the flags in `model_search/oss_trainer_lib.py`.\n\n\n## Distributed Runs\nOur system can run a distributed search - I.e., run many search trainer in\nparallel.\n\nHow does it work?\n\nYou need to run your binary on multiple machines. Additionally, you need to\nmake one change to configure the bookkeeping of the search.\n\nOn a single machine, the bookkeeping is done via a file. For a distributed\nsystem however, we need a database.\n\nIn order to point our system to the database, you need to set the flags in the\nfile:\n\n`model_search/metadata/ml_metadata_db.py`\n\nto point to your database.\n\nOnce you have done so, the binaries created from the previous section will\nconnect to this database and an async search will begin.\n\n## Cloud AutoML\nWant to try higher performance AutoML without writing code? Try:\nhttps://cloud.google.com/automl-tables\n","funding_links":[],"categories":["Python","AutoML","Profiling","Deep Learning Framework","Scheduling","Tools and projects","其他_机器学习与深度学习","Libs With Online Books"],"sub_categories":["Profiling","Auto ML \u0026 Hyperparameter Optimization","LLM","Philosophy"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle%2Fmodel_search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle%2Fmodel_search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle%2Fmodel_search/lists"}