NASLib is a Neural Architecture Search (NAS) library for facilitating NAS research for the community by providing interfaces to several state-of-the-art NAS search spaces and optimizers.
License: Apache License 2.0
Hi, I'd like to use NASLib with my own dataset. Is there an option to do that?
The download script source scripts/bash_scripts/download_nbs_zero.sh all no longer works, and it seems like the drive folders do not exist anymore. Is this data hosted somewhere else? This is for the running the correlation results of the zerocost branch.
Dear Sir,
Thank you for your interesting and valuable work on NASBench.
I am currently studying zero-cost proxy methods and would greatly appreciate it if you could provide information on how to access the trained weights of the architecture networks within NASBench.
Your assistance would be much appreciated.
Thank you for your time and consideration.
Sincerely,
Hello,
I was trying to use NasBenchmark301 based on the example notebook here there are lots of dependency incompatibilities with the NASLib.
It also requires certain branch of Auto-PyTorch and Auto-Pytorch also has incompatibility issues with the NASLib.
Following the another reference in the NASLib's readme I reached the nas-bench-x11 repository and it has also some incompatibility issues.
My question is, what is the suggested and proper way to use NB301 benchmarks?
Based on that we are planning to develop a surrogate model for a task.
Thank you in advance,
In file NASLib/tree/Develop/naslib/predictors/zerocost_v1.py, it use the search space defined in NASLib/tree/Develop/naslib/predictors/utils/models. It basically defines some architectures of nasbench101 and nasbench201.
However, there is already a module called search_space. including many search spaces, and they are well-structured,
Why the zerocost_v1 don't use the model defined in search_space?
Hi,
How can I download nb201_cifar10_full_training.pickle, nb201_cifar100_full_training.pickle and nb201_ImageNet16_full_training.pickle files? I can't find these files. Can you help me?
Thanks
I currently use NASLib as a backend for defining my own search space for a benchmark. I noticed while sharing my code that the following imports introduce a number of dependencies that are not strictly necessary in use cases where only a subset of all the search spaces defined by NASLib are to be used. Auto-importing these search spaces enforces the constraint that the dependencies of each and every search space must be satisfied in order to use any single one of the search spaces. Is this truly necessary? For reference, I am currently using only the NASBench-201 search space.
NASLib/naslib/search_spaces/__init__.py
Lines 1 to 10 in afa98c3
Hi,
When I try to restart my training from the stored checkpoint it takes some random architecture instead of the one already stored. Is there any way to avoid this?
Currently I am running DrNAS, DARTS and GDAS optimizers for search and evaluation purpose with CIFAR 10 dataset.
In the tutorial, when for example, it is written, "with NB201() you can simply query the benchmark", you may provide some examples or precise documentation. It's not clear what you can fill in for the query(arguments exactly).
It seems like you would be able to do NB201.query('cifar10', Metric.TEST_ACCURACY)
Instead of listing the setup and call, you're guided to use a config file. It's better to give the config parameters in the documentation with the function calls. Then say, you can collect these things in config files.
In the video and tutorial, I find it difficult to see why the introductory example doesn't actually train any models and the one using a NAS predictor supposedly does. It seems as if the former happens simply because the NASBenchSearchSpace is loaded, but that stays the same through the second (XGBoost example), there you do have load_labeled=False in one of the calls. That sounds like it may prevent accuracy lookup, but it's more likely that it is in the configuration file that is loaded. I would make it explicit, either in the code or in the configuration file.
Hi, I come across this error after submitting it to Coda-Lab and I believed it should have been something wrong in the evaluation system.
Traceback (most recent call last):
File "/worker/worker.py", line 705, in run
put_blob(stdout_url, stdout_file)
File "/worker/worker.py", line 214, in put_blob
'x-ms-version': '2018-03-28',
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 99, in put
return request('put', url, data=data, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 438, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 324, in send
raise ConnectionError(sockerr)
ConnectionError: [Errno -3] Temporary failure in name resolution
Hi, dear organizer:
I come across some issues when coding on submission, it seems like we don't have an option for re-define the network, which means in-place ReLU will break the register_full_backward_hook by rising errors like:
RuntimeError: Output 0 of BackwardHookFunctionBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can remove this warning by cloning the output of the custom Function.
Hi,
I'm trying to sample few architectures from NAS-Bench-101, and then query the socre, and also get the torch-model according to the architecutre.
I wanna use the model to do some customer training or testing.
In core/graph.py, it shows we should us the following code to parse the model to pytorch module
**Use as pytorch module**
If you want to learn the weights of the operations or any
other parameters of the graph you have to parse it first.
>>> graph = getFancySearchSpace()
>>> graph.parse()
>>> logits = graph(data)
>>> optimizer.min(loss(logits, target))
But the graph.parse() will fail if i use NasBench101SearchSpace as graph instance ,
Hi everyone,
I have been trying to set up NASLib for a few days now and have been running into issues with the stated dependencies of the library. A number of dependencies have very precise version numbers, such as numpy==1.16.4 and pytorch==1.15.4, which are causing compatibility problems down the line for other libraries which don't have any version numbers specified at all. My estimate is that the latest versions of some other libraries deeper in the dependency tree, which are being pulled by conda/pip, are incompatible with the specific older versions specified in the requirements.txt. As of writing this issue, I have been unable to satisfy all the version dependencies in the requirements.txt with any of the python versions I tested, including 3.5 - which was specified as the minimum working version in the setup.py. Could you kindly provide some advice on this?
Setting up an environment that ignored the version-specific requirements but did include the requested libraries resulted in the unit tests failing with these errors
Hi everyone,
I was comparing the code for the search space in the original repo (as per the changelog) with the NASLib implementation since we were observing consistently worse performance (down by 5-6 percentage points) when trying to re-create the top-10 or even top-100 architectures' performance and noticed that, in the NASLib implementation here, when compared to the corresponding original code here, there is a missing BatchNorm + ReLU before the global adaptive pooling layer. Interestingly, I did not see any mention of these layers in the paper on arXiv. But introducing them by making the following change to the aboive-mentioned NASLib code instantly re-created the expected top-10 model performance:
# post-processing
self.edges[edge_names["postproc"]].set('op', ops.Sequential(
nn.BatchNorm2d(channels[-1]),
nn.ReLU(inplace=False),
nn.AdaptiveAvgPool2d(1),
nn.Flatten(),
nn.Linear(channels[-1], self.num_classes)
))Hi,
when should we expect the official release to be? And until then, which branch is the most updated yet reliable to use?
TIA.
A problem occurs in the Bananas optimizer, when an architecture outside of the search space is sampled, and the zero_cost_scores must be calculated on-the-fly using the ZeroCost
Exception has occurred: StopIteration
exception: no description
File "/home/jhaa/NASLib/naslib/predictors/utils/pruners/measures/model_stats.py", line 7, in get_model_stats
model_stats = tw.ModelStats(model, input_tensor_shape,
File "/home/jhaa/NASLib/naslib/predictors/utils/pruners/predictive.py", line 141, in find_measures
model_stats = get_model_stats(
File "/home/jhaa/NASLib/naslib/predictors/zerocost.py", line 31, in query
score = predictive.find_measures(
File "/home/jhaa/NASLib/naslib/optimizers/discrete/bananas/optimizer.py", line 93, in query_zc_scores
score = zc_method.query(arch, dataloader=zc_method.train_loader)
File "/home/jhaa/NASLib/naslib/optimizers/discrete/bananas/optimizer.py", line 241, in _get_best_candidates
model.zc_scores = self.query_zc_scores(model.arch)
File "/home/jhaa/NASLib/naslib/optimizers/discrete/bananas/optimizer.py", line 232, in new_epoch
self.next_batch = self._get_best_candidates(candidates, acq_fn)
File "/home/jhaa/NASLib/naslib/defaults/trainer.py", line 112, in search
self.optimizer.new_epoch(e)
File "/home/jhaa/NASLib/naslib/runners/bbo/runner.py", line 75, in <module>
trainer.search(resume_from="", summary_writer=writer, report_incumbent=False)
Currently, we are passing model.arch into the zc_method.query() function. This differs slightly from the Fall-School Tutorial, where a graph is being initialized, an architecture is being sampled, the graph is being parsed and then the graph object is being passed in.
Hi,
I have been trying to set up NASLib for a few days. I am trying to run naslib/benchmarks/darts/runner.py. But get_darts_api function is trying to import nasbench301 library. Requirements.txt of the nasbench301 and naslib is inconsistent. For example: nasbench301 tries to install torch=1.5.0 while naslib tries to install torch=1.9.0. That's why I get an error when I try to run it. Can you help me solve this dependency problem? How should I edit the requirements.txt file?
Hello everyone,
How to get the train loader to calculate zero-cost proxies for architectures in NAS-Bench-ASR?
Hi, thank you for sharing these benchmark tools!
Let me ask one question about a dataset provided by this repo.
When I looked at data in NAS-Bench-301 trained on CIFAR-10 whose URL is provided in README.md in this repo, the format of architecture is supposed to be encoded numerically unlike NAS-Bench-201 (see also the code and its output below).
import pickle
data = pickle.load(open("./nb301_full_training.pickle", "rb"))
print(list(data.keys())[0])(((0, 6), (1, 4), (0, 0), (1, 5), (1, 4), (3, 2), (0, 6), (3, 2)),
((0, 1), (1, 4), (0, 1), (1, 4), (2, 6), (3, 4), (1, 5), (2, 1)))I'm wondering if we could know the encoding rule somewhere. I tried to do so but I found it difficult because we have at least four related NAS-Bench-301 repositories as follows.
README.md in https://github.com/automl/NASLib to use NAS-Bench-301.the full code is now available here: https://github.com/automl/nas-bench-x11..Best regards,
I am running tutorial/zc_intro.py and it returns this error:
`[Errno 2] No such file or directory: '/home.../naslib/data/zc_nasbench201.json'
File "/home.../naslib/utils/get_dataset_api.py", line 24, in get_zc_benchmark_api
with open(datafile_path) as f:
File "/home.../tutorial/zc_intro.py", line 113, in
zc_api = get_zc_benchmark_api('nasbench201', 'cifar10')
FileNotFoundError: [Errno 2] No such file or directory: '/home.../naslib/data/zc_nasbench201.json'`
I have already checked the data for the benchmark but this file was not there to download.
The search space of nas benchmarks can be classified into two categories, one is NAS-Bench-101 which using DAG to represent search space, and the other is DARTS-like search space such as NAS-Bench-201, NAS-Bench-301, NATS-Bench etc.
Why there is no single path search space such as the one used in Sinlge Path One Shot, FairNAS,AngleNAS,RLNAS etc.
I have been attempting to use the library recently but I have noticed all the unit tests fail and have done for a for all the commits I have checked. I have also tried the tutorial and some of the methods but the same errors that the unit tests come up. Do you know what the last commit for the library is? I think the errors mainly originate in the graph construction when I try and debug.
Hi,
I'm currently working on predictor experiments with NASLib and encountered a snag. I was trying to use the script /NASLib/scripts/predictors/run_nb101.sh, but it seems to need access to a benchmarks folder. However, this folder appears to be missing in the source code, specifically in the path NASLib/naslib/benchmarks/create_configs.py.
Could you please provide some guidance on how to set up or find this benchmarks folder? Any advice or direction towards the right documentation would be very much appreciated.
Thanks for your help!
Best,
José Carreira
In the get_candidates function of the zerocost branch optimizers Bananas and Npenas there is a discrepancy between how candidates in the next_batch are stored.
If the acquisition function is being optimized via "random_sampling", then model is being stored:
...
candidates.append(model)Otherwise, if it is being optimized via "mutation", then model.arch is being stored:
...
candidate = arch
candidates.append(candidate)However, the function get_best_candidates (which is called directly after get_new_candidates) treats candidates as a list of models:
values = [acq_fn(model.arch, [{'zero_cost_scores' : model.zc_scores}]) for model in candidates]Does this imply that the optimization of the acquisition function via "mutation" is not used in the main loop of either Bananas or Npenas? If so, how and when should this option be used?
Hi,
Are there any plans to add and make a centralized api for differed zeroshot searching algorithms?
For example, "random search" from NAs w/o training paper; "prune-based search" from TE-NAS paper; "genetic search" from Zen-NAS; "reinforce" and etc.
We have included several new parameters for Bananas and Npenas in order to determine whether the zero cost proxies are used, and whether they come from the benchmark or the ZeroCost predictor:
zcuse_zc_apizc_onlyShould these be included as initialization parameters, or in the config files?
If they were to be included in the configs, create_config.py would need to be modified.
Hello,
Can you help me with downloading and loading Taskonomy dataset data?
Download script only fetches raw pixel pngs, "class_object" labels and "class_scene" labels. That only covers "autoencoder", "class_object", "class_scene" and "jigsaw" tasks. I've found that "normal" task labels can also be downloaded from downloads.cs.stanford.edu similarly to "class_object". That makes 5 out of 7 in total. How to get "room_layout" and "segmentsemantic" tasks?
When loading TaskonomyDataset it requires some json containing template paths. In loading configs (like here) they are marked as "final5K_splits".
How do i generate those files? I can not seem to find any scripts related to this.
I am having trouble downloading the taskonomy dataset using download_tnb.sh. I created a directory for each of the buildings and ran the file. Now, my files look like this (I will give one filename as example):
dataset/
----benevolence/
--------class_object/
------------point_0_view_0_domain_class_object.npy
--------class_scene/
------------point_0_view_0_domain_class_scene.npy
--------normal/
------------point_0_view_0_domain_normal.npy
--------rgb/
------------point_0_view_0_domain_rgb.png
----other_building/
etc
With this file structure, generate_splits.py looks at wrong files and my splits are empty, for example benevolence.json has "[]" an empty list as entry.
I have been struggling to find the right way to download the data with the correct naming and could really use some help.
This question is more about what the paper ("NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies") describes in Table 10 from Appendix D. It is still unclear to me whether the XGBoost was trained on ZC scores and/or encodings on one single benchmark and then tested on the rest of the benchmarks or XGBoost was trained on a subset of benchmark models and later tested on models from the same benchmark.
Hi,
I cannot open https://github.com/automl/NASLib/blob/master/examples/naslib_tutorial.ipynb either with my Chrome or after I download it and run locally. I suppose this file is broken. Could you help have a look?
Unreadable Notebook: NASLib-Develop/examples/naslib_tutorial.ipynb NotJSONError('Notebook does not appear to be JSON: \'{\\n "cells": [\\n {\\n ...',)
Thanks,
Jinning
When installed by pip or GitHub, with full data, test_benchmark_apis.py always fails.
is there anything help doc when installing NB301? or the problem is that the API in NB301 has changed now?
I downloaded zero-cost metrics using the script download_nbs_zero.sh. Can you explain how can I relate architecture encoding such as (4, 0, 3, 1, 4, 3) in nasbench201 to architecture string such as |nor_conv_1x1~0|+|avg_pool_3x3~0|avg_pool_3x3~1|+|avg_pool_3x3~0|nor_conv_3x3~1|avg_pool_3x3~2|? In addition, can you explain how you extract validation accuracy from NASBench201 api? Thank you!
I don't want CIFAR, Taskonomy or anything, I have my own data, but I can't get it to work. I couldn't replace CIFAR10 and I don't know the data format and it overwhelms me with errors. Can you help me with this?
Hope to get help. Some errors that occur when I execute the command :
coverage run -m unittest discover -v,
The test result is “Ran 81 tests in 247.720s，FAILED (failures=6, errors=18)”
======================================================================
ERROR: test_get_config_from_args_default_bbo_bs (test_utils.UtilsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/marui/NAS/NASLib-Develop/tests/test_utils.py", line 38, in test_get_config_from_args_default_bbo_bs
self._test_get_config_from_args_default(config_type="bbo-bs")
File "/data/marui/NAS/NASLib-Develop/tests/test_utils.py", line 71, in _test_get_config_from_args_default
config_child = utils.get_config_from_args(config_type=config_type)
File "/data/marui/NAS/NASLib-Develop/naslib/utils/__init__.py", line 205, in get_config_from_args
value) if arg in config else eval(value)
File "<string>", line 1, in <module>
NameError: name 'v' is not defined
======================================================================
FAIL: test_forward_pass_aux_head (test_nb301_search_space.NasBench301SearchSpaceTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/marui/NAS/NASLib-Develop/tests/test_nb301_search_space.py", line 120, in test_forward_pass_aux_head
self.assertEqual(aux_out.shape, (3, 512, 8, 8))
AssertionError: torch.Size([3, 256, 8, 8]) != (3, 512, 8, 8)
======================================================================
ERROR: test_feed_forward (test_hierarchical_search_space.HierarchicalDartsIntegrationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/marui/NAS/NASLib-Develop/tests/test_hierarchical_search_space.py", line 50, in test_feed_forward
logits = final_arch(data_train[0])
File "/data/marui/anaconda3/envs/naslib/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/marui/NAS/NASLib-Develop/naslib/search_spaces/core/graph.py", line 410, in forward
edge_output = edge_data.op.forward(x, edge_data=edge_data)
File "/data/marui/NAS/NASLib-Develop/naslib/search_spaces/core/primitives.py", line 388, in forward
return self.seq(x)
File "/data/marui/anaconda3/envs/naslib/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/marui/anaconda3/envs/naslib/lib/python3.7/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/data/marui/anaconda3/envs/naslib/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/data/marui/anaconda3/envs/naslib/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 457, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/data/marui/anaconda3/envs/naslib/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 454, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size [64, 64, 3, 3], expected input[2, 3, 32, 32] to have 64 channels, but got 3 channels instead
Are these errors normal ? Is there something wrong with my environment settings ? How should I resolve these errors ?
Looking forward to everyone's response, thanks!
I have run Nasbench101 in the zero-cost Naslib and got an error
Have anyone tackled this problem?
When running the pip install -e . command, set up fails. It seems tripped up on the numpy installation, but numpy is already installed (version consistent with that required).
See error message above. The result is consistent across three different systems: M1 Mac and two Linux. Although not making the entire set up script fail, torchaudio also cannot be installed with the command suggested under Set Up in the NASLib docs. Additionally, it seems many of the packages specified in requirements.txt seem to have conflicts.
Thank you for your fantastic work NAS-Bench-Zero-Suite.
I found a lot of zc_<name_of_nasbench>.json like zc_nasbench201.json, but I can't tell what tuples represent what actual architectures in the search space.
For example, I don't know (4, 0, 3, 1, 4, 3) in zc_nas-bench-201.json represents what architecture (like |avg_pool_3x3~0|+|skip_connect~0|nor_conv_1x1~1|+|none~0|avg_pool_3x3~1|nor_conv_1x1~2|).
I would be glad to know. Thank you.
For a hash, say
from naslib.search_spaces import nasbench301
hash = '([(0,6),(1,2),(1,4),(0,6),(0,6),(1,3),(4,0),(3,6)],[(0,6),(1,2),(1,4),(0,6),(0,6),(1,3),(4,0),(3,6)])'
nbg = nasbench301.graph.NasBench301SearchSpace()
nbg.set_spec(eval(hash))
nbg.prepare_evaluation()
nbg.parse()
print(nbg.adj)
nbg = nasbench301.graph.NasBench301SearchSpace()
hash = '([(1,3),(0,5),(0,4),(2,4),(1,4),(2,0),(2,5),(1,1)],[(1,3),(0,5),(0,4),(2,4),(1,4),(2,0),(2,5),(1,1)])'
nbg.set_spec(eval(hash))
nbg.prepare_evaluation()
nbg.parse()
print(nbg.adj)
The adjacency matrix and other properties of the model 'nbg' does not change.
How can I initialize a NASBench-301 model with NASLib such that it can be trained?
Also, is it possible to extract a pure PyTorch model from the nbg graph?
Hi, what would be the best way to recreate a subset of benchmarking accuracy data from the ZC json?
For ex: I would want to replicate the val accuracy obtained for the architecture 1-100 indexed from NASBench201. I understand that the val accuracies and corresponding ZC scores were open-sourced to remove the need for training, but I need to perform training for an internal evaluation. How do you suggest to go about this?
Thank you.
In the zerocost branch, Ensemble has been extended to the Zero Cost case and contains a single option for its base predictor, XGBoost. The XGBoost predictor has been adapted to the zero-cost case by implementing a set_pre_computations function, as well as modifying the BaseTree class. Currently, the Ensemble class supports only this single predictor:
trainable_predictors = {
"xgb": XGBoost(
ss_type=self.ss_type, zc=self.zc, encoding_type="adjacency_one_hot", zc_only=self.zc_only
)
}Should all other predictors be available in the merged Ensemble class and extended to the zero-cost case?
Hi, I tried to clone the automl-competition repo to the local machine and tried to set up an offline evaluation with scripts provided by the tutorial. It seems all datasets work fine except Imagenet-16(Hash check failed.).
Traceback (most recent call last): ..... File "//workspace/automl_naslib/naslib/utils/DownsampledImageNet.py", line 80, in __init__ entry = pickle.load(f, encoding="latin1") _pickle.UnpicklingError: invalid load key, '\x1d'. .....
Hi,
I'm trying to setup the package on my computer so I can develop using naslib. Since I'm on Mac M1, I decided to use a DevContainer so I could install python 3.7. However, I can't manage to complete the installation, even though I followed the tutorial from https://automl.github.io/NASLib/setup/. Attached are my Dockerfile and the error logs.
Could you replicate this issue or offer any solutions?
FROM python:3.7-slim
RUN apt-get update && apt-get install -y \
build-essential \
libblas-dev \
liblapack-dev \
gfortran \
gcc \
git \
sudo
COPY NASLib /NASLib
WORKDIR /NASLib
RUN pip install --upgrade pip setuptools wheel
RUN pip install -e .
https://drive.google.com/file/d/1u8SwecZAW3htEvAoAv_qwDdUt8io3EOl/view?usp=sharing
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for scipy
Building wheel for tensorwatch (setup.py): started
Building wheel for tensorwatch (setup.py): finished with status 'done'
Created wheel for tensorwatch: filename=tensorwatch-0.9.1-py3-none-any.whl size=214516 sha256=8a1ef03deddb8c6e4b02754b30cf5882c329fdaca38517095a97e174f050982b
Stored in directory: /root/.cache/pip/wheels/f9/59/17/bcd355d22c397b603c6beaa3762fa004a767511b7996743bec
Building wheel for iopath (setup.py): started
Building wheel for iopath (setup.py): finished with status 'done'
Created wheel for iopath: filename=iopath-0.1.10-py3-none-any.whl size=31532 sha256=e578ff6c0399820c2920c18d4cdc800860d74ac3af5245dd7839084b33c0bbaf
Stored in directory: /root/.cache/pip/wheels/aa/cc/ed/ca4e88beef656b01c84b9185196513ef2faf74a5a379b043a7
Building wheel for pydotz (setup.py): started
Building wheel for pydotz (setup.py): finished with status 'done'
Created wheel for pydotz: filename=pydotz-1.5.1-py2.py3-none-any.whl size=20160 sha256=4f8ddfbe209d9c11abbdf888f1e9057d744d7ebb00f1eb44bf258793aaf0b5c4
Stored in directory: /root/.cache/pip/wheels/c9/15/f2/f9f771c57101e0fe198670c3a35930c86c545e4155629ef619
Building wheel for autograd-gamma (setup.py): started
Building wheel for autograd-gamma (setup.py): finished with status 'done'
Created wheel for autograd-gamma: filename=autograd_gamma-0.5.0-py3-none-any.whl size=4031 sha256=434aa9fa69fd7d50bb0a97a73f9b18075a0cbe208929d5bf9aa43989f0a352fa
Stored in directory: /root/.cache/pip/wheels/9f/01/ee/1331593abb5725ff7d8c1333aee93a50a1c29d6ddda9665c9f
Successfully built fvcore grakel pybnn tensorwatch iopath pydotz autograd-gamma
Failed to build scipy
ERROR: Could not build wheels for scipy, which is required to install pyproject.toml-based projects
Are there any examples about how to train an architecture on TransNAS-Bench-101 search space? And where can I download the dataset to train the architecture?
Hi, I wanted to test a Zero-Cost Predictor on all 15,625 architectures for NASBench201. In the NAS-Bench-Suite-Zero paper you state that you did this for all the architectures -
For all NAS-Bench-201 and TransNAS-Bench-101 tasks, we evaluate all ZC proxy values and the respective runtimes, for all architectures
But I am running across some issues. In the predictor_config.yaml it only has an option for random sampling of architectures with the parameter uniform_random.
I wrote my own code to exhaustively sample all architectures using naslib.search_spaces.nasbench201.graph.get_arch_iterator(). However, PredictorEvaluator.load_dataset() then loads 15,625 graphs into a list, which around 6,000 architectures in, requires too much memory for my computer. I wanted to see if I was going about this the best way or if there was a better way.
Thanks.
In the zero-cost branch optimizers Npenas and Bananas, the validation accuracy of architectures is being queried from the zero-cost-benchmark as follows:
model.accuracy = self.zc_api[str(model.arch_hash)]['val_accuracy']The question is, whether this supports the case where the user wants to use the ZeroCost predictor because their dataset or search space is not supported by the zero-cost benchmark.
If this is a case that we want to support, one option would be to introduce a parameter use_zc_api and use it as follows:
if self.use_zc_api:
model.accuracy = self.zc_api[str(model.arch_hash)]['val_accuracy']
else:
model.accuracy = model.arch.query(
self.performance_metric, self.dataset, dataset_api=self.dataset_api
)Hi,
Looks like the notebook https://github.com/automl/NASLib/blob/Develop/examples/example_runner.ipynb is broken and there is an error there.
Hi, the download link of TransNAS-Bench-101 is invalid now. I wonder if anyone has the data file and would like to share it with me. Thanks for the help!
I am trying, without success, sampling random architectures from the defined Search Spaces such as DARTS and SimpleCell spaces.
Is it possible to sample an entire PyTorch object representing the whole architecture? If not, is it possible to sample all the possible cells from the search spaces? Can you provide some code examples for doing this?
Thanks.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.