Giter Site home page Giter Site logo

aimhubio / aim Goto Github PK

View Code? Open in Web Editor NEW
4.8K 4.8K 284.0 63.97 MB

Aim πŸ’« β€” An easy-to-use & supercharged open-source experiment tracker.

Home Page: https://aimstack.io

License: Apache License 2.0

Python 91.64% Mako 0.08% JavaScript 1.86% HTML 0.70% Shell 0.17% Cython 4.71% C 0.12% CSS 0.70% Dockerfile 0.01%
ai data-science data-visualization experiment-tracking machine-learning metadata metadata-tracking ml mlflow mlops prompt-engineering python pytorch tensorboard tensorflow visualization

aim's Introduction

Drop a star to support Aim ⭐ Join Aim discord community

An easy-to-use & supercharged open-source experiment tracker

Aim logs your training runs and any AI Metadata, enables a beautiful UI to compare, observe them and an API to query them programmatically.

Discord Server Twitter Follow Medium

Platform Support PyPI - Python Version PyPI Package License PyPI Downloads Issues



SEAMLESSLY INTEGRATES WITH:


TRUSTED BY ML TEAMS FROM:


AimStack offers enterprise support that's beyond core Aim. Contact via [email protected] e-mail.


About β€’ Demos β€’ Ecosystem β€’ Quick Start β€’ Examples β€’ Documentation β€’ Community β€’ Blog


ℹ️ About

Aim is an open-source, self-hosted ML experiment tracking tool designed to handle 10,000s of training runs.

Aim provides a performant and beautiful UI for exploring and comparing training runs. Additionally, its SDK enables programmatic access to tracked metadata β€” perfect for automations and Jupyter Notebook analysis.

Aim's mission is to democratize AI dev tools 🎯


Log Metadata Across Your ML Pipeline πŸ’Ύ Visualize & Compare Metadata via UI πŸ“Š
  • ML experiments and any metadata tracking
  • Integration with popular ML frameworks
  • Easy migration from other experiment trackers
  • Metadata visualization via Aim Explorers
  • Grouping and aggregation
  • Querying using Python expressions
Run ML Trainings Effectively ⚑ Organize Your Experiments πŸ—‚οΈ
  • System info and resource usage tracking
  • Real-time alerting on training progress
  • Logging and configurable notifications
  • Detailed run information for easy debugging
  • Centralized dashboard for holistic view
  • Runs grouping with tags and experiments

🎬 Demos

Check out live Aim demos NOW to see it in action.

Machine translation experiments lightweight-GAN experiments
Training logs of a neural translation model(from WMT'19 competition). Training logs of 'lightweight' GAN, proposed in ICLR 2021.
FastSpeech 2 experiments Simple MNIST
Training logs of Microsoft's "FastSpeech 2: Fast and High-Quality End-to-End Text to Speech". Simple MNIST training logs.

🌍 Ecosystem

Aim is not just an experiment tracker. It's a groundwork for an ecosystem. Check out the two most famous Aim-based tools.

aimlflow Aim-spaCy
aimlflow Aim-spaCy
Exploring MLflow experiments with a powerful UI an Aim-based spaCy experiment tracker

🏁 Quick start

Follow the steps below to get started with Aim.

1. Install Aim on your training environment

pip3 install aim

2. Integrate Aim with your code

from aim import Run

# Initialize a new run
run = Run()

# Log run parameters
run["hparams"] = {
    "learning_rate": 0.001,
    "batch_size": 32,
}

# Log metrics
for i in range(10):
    run.track(i, name='loss', step=i, context={ "subset":"train" })
    run.track(i, name='acc', step=i, context={ "subset":"train" })

See the full list of supported trackable objects(e.g. images, text, etc) here.

3. Run the training as usual and start Aim UI

aim up

Learn more

Migrate from other tools

Aim has built-in converters to easily migrate logs from other tools. These migrations cover the most common usage scenarios. In case of custom and complex scenarios you can use Aim SDK to implement your own conversion script.

Integrate Aim into an existing project

Aim easily integrates with a wide range of ML frameworks, providing built-in callbacks for most of them.

Query runs programmatically via SDK

Aim Python SDK empowers you to query and access any piece of tracked metadata with ease.

from aim import Repo

my_repo = Repo('/path/to/aim/repo')

query = "metric.name == 'loss'" # Example query

# Get collection of metrics
for run_metrics_collection in my_repo.query_metrics(query).iter_runs():
    for metric in run_metrics_collection:
        # Get run params
        params = metric.run[...]
        # Get metric values
        steps, metric_values = metric.values.sparse_numpy()
Set up a centralized tracking server

Aim remote tracking server allows running experiments in a multi-host environment and collect tracked data in a centralized location.

See the docs on how to set up the remote server.

Deploy Aim on kubernetes

Read the full documentation on aimstack.readthedocs.io πŸ“–

πŸ†š Comparisons to familiar tools

TensorBoard vs Aim

Training run comparison

Order of magnitude faster training run comparison with Aim

  • The tracked params are first class citizens at Aim. You can search, group, aggregate via params - deeply explore all the tracked data (metrics, params, images) on the UI.
  • With tensorboard the users are forced to record those parameters in the training run name to be able to search and compare. This causes a super-tedius comparison experience and usability issues on the UI when there are many experiments and params. TensorBoard doesn't have features to group, aggregate the metrics

Scalability

  • Aim is built to handle 1000s of training runs - both on the backend and on the UI.
  • TensorBoard becomes really slow and hard to use when a few hundred training runs are queried / compared.

Beloved TB visualizations to be added on Aim

  • Embedding projector.
  • Neural network visualization.
MLflow vs Aim

MLFlow is an end-to-end ML Lifecycle tool. Aim is focused on training tracking. The main differences of Aim and MLflow are around the UI scalability and run comparison features.

Aim and MLflow are a perfect match - check out the aimlflow - the tool that enables Aim superpowers on Mlflow.

Run comparison

  • Aim treats tracked parameters as first-class citizens. Users can query runs, metrics, images and filter using the params.
  • MLFlow does have a search by tracked config, but there are no grouping, aggregation, subplotting by hyparparams and other comparison features available.

UI Scalability

  • Aim UI can handle several thousands of metrics at the same time smoothly with 1000s of steps. It may get shaky when you explore 1000s of metrics with 10000s of steps each. But we are constantly optimizing!
  • MLflow UI becomes slow to use when there are a few hundreds of runs.
Weights and Biases vs Aim

Hosted vs self-hosted

  • Weights and Biases is a hosted closed-source MLOps platform.
  • Aim is self-hosted, free and open-source experiment tracking tool.

πŸ›£οΈ Roadmap

Detailed milestones

The Aim product roadmap ❇️

  • The Backlog contains the issues we are going to choose from and prioritize weekly
  • The issues are mainly prioritized by the highly-requested features

High-level roadmap

The high-level features we are going to work on the next few months:

In progress

  • Aim SDK low-level interface
  • Dashboards – customizable layouts with embedded explorers
  • Ergonomic UI kit
  • Text Explorer
Next-up

Aim UI

  • Runs management
    • Runs explorer – query and visualize runs data(images, audio, distributions, ...) in a central dashboard
  • Explorers
    • Distributions Explorer

SDK and Storage

  • Scalability
    • Smooth UI and SDK experience with over 10.000 runs
  • Runs management
    • CLI commands
      • Reporting - runs summary and run details in a CLI compatible format
      • Manipulations – copy, move, delete runs, params and sequences
  • Cloud storage support – store runs blob(e.g. images) data on the cloud
  • Artifact storage – store files, model checkpoints, and beyond

Integrations

  • ML Frameworks:
    • Shortlist: scikit-learn
  • Resource management tools
    • Shortlist: Kubeflow, Slurm
  • Workflow orchestration tools
Done
  • Live updates (Shipped: Oct 18 2021)
  • Images tracking and visualization (Start: Oct 18 2021, Shipped: Nov 19 2021)
  • Distributions tracking and visualization (Start: Nov 10 2021, Shipped: Dec 3 2021)
  • Jupyter integration (Start: Nov 18 2021, Shipped: Dec 3 2021)
  • Audio tracking and visualization (Start: Dec 6 2021, Shipped: Dec 17 2021)
  • Transcripts tracking and visualization (Start: Dec 6 2021, Shipped: Dec 17 2021)
  • Plotly integration (Start: Dec 1 2021, Shipped: Dec 17 2021)
  • Colab integration (Start: Nov 18 2021, Shipped: Dec 17 2021)
  • Centralized tracking server (Start: Oct 18 2021, Shipped: Jan 22 2022)
  • Tensorboard adaptor - visualize TensorBoard logs with Aim (Start: Dec 17 2021, Shipped: Feb 3 2022)
  • Track git info, env vars, CLI arguments, dependencies (Start: Jan 17 2022, Shipped: Feb 3 2022)
  • MLFlow adaptor (visualize MLflow logs with Aim) (Start: Feb 14 2022, Shipped: Feb 22 2022)
  • Activeloop Hub integration (Start: Feb 14 2022, Shipped: Feb 22 2022)
  • PyTorch-Ignite integration (Start: Feb 14 2022, Shipped: Feb 22 2022)
  • Run summary and overview info(system params, CLI args, git info, ...) (Start: Feb 14 2022, Shipped: Mar 9 2022)
  • Add DVC related metadata into aim run (Start: Mar 7 2022, Shipped: Mar 26 2022)
  • Ability to attach notes to Run from UI (Start: Mar 7 2022, Shipped: Apr 29 2022)
  • Fairseq integration (Start: Mar 27 2022, Shipped: Mar 29 2022)
  • LightGBM integration (Start: Apr 14 2022, Shipped: May 17 2022)
  • CatBoost integration (Start: Apr 20 2022, Shipped: May 17 2022)
  • Run execution details(display stdout/stderr logs) (Start: Apr 25 2022, Shipped: May 17 2022)
  • Long sequences(up to 5M of steps) support (Start: Apr 25 2022, Shipped: Jun 22 2022)
  • Figures Explorer (Start: Mar 1 2022, Shipped: Aug 21 2022)
  • Notify on stuck runs (Start: Jul 22 2022, Shipped: Aug 21 2022)
  • Integration with KerasTuner (Start: Aug 10 2022, Shipped: Aug 21 2022)
  • Integration with WandB (Start: Aug 15 2022, Shipped: Aug 21 2022)
  • Stable remote tracking server (Start: Jun 15 2022, Shipped: Aug 21 2022)
  • Integration with fast.ai (Start: Aug 22 2022, Shipped: Oct 6 2022)
  • Integration with MXNet (Start: Sep 20 2022, Shipped: Oct 6 2022)
  • Project overview page (Start: Sep 1 2022, Shipped: Oct 6 2022)
  • Remote tracking server scaling (Start: Sep 11 2022, Shipped: Nov 26 2022)
  • Integration with PaddlePaddle (Start: Oct 2 2022, Shipped: Nov 26 2022)
  • Integration with Optuna (Start: Oct 2 2022, Shipped: Nov 26 2022)
  • Audios Explorer (Start: Oct 30 2022, Shipped: Nov 26 2022)
  • Experiment page (Start: Nov 9 2022, Shipped: Nov 26 2022)
  • HuggingFace datasets (Start: Dec 29 2022, Feb 3 2023)

πŸ‘₯ Community

Aim README badge

Add Aim badge to your README, if you've enjoyed using Aim in your work:

Aim

[![Aim](https://img.shields.io/badge/powered%20by-Aim-%231473E6)](https://github.com/aimhubio/aim)

Cite Aim in your papers

In case you've found Aim helpful in your research journey, we'd be thrilled if you could acknowledge Aim's contribution:

@software{Arakelyan_Aim_2020,
  author = {Arakelyan, Gor and Soghomonyan, Gevorg and {The Aim team}},
  doi = {10.5281/zenodo.6536395},
  license = {Apache-2.0},
  month = {6},
  title = {{Aim}},
  url = {https://github.com/aimhubio/aim},
  version = {3.9.3},
  year = {2020}
}

Contributing to Aim

Considering contibuting to Aim? To get started, please take a moment to read the CONTRIBUTING.md guide.

Join Aim contributors by submitting your first pull request. Happy coding! 😊

Made with contrib.rocks.

More questions?

  1. Read the docs
  2. Open a feature request or report a bug
  3. Join Discord community server

aim's People

Contributors

alansaul avatar alberttorosyan avatar aramaim avatar arsengit avatar bhaveshk658 avatar dependabot[bot] avatar devfox-se avatar fetz236 avatar geeeekexplorer avatar gorarakelyan avatar hamik25 avatar jamesj-jiao avatar jangop avatar jialin-wu-02 avatar karomourad avatar khazhak avatar mahnerak avatar martenlienen avatar mihran113 avatar mike1808 avatar n-gao avatar nzw0301 avatar roubkar avatar rubenaprikyan avatar sgevorg avatar sharathmk99 avatar tamohannes avatar tatevv avatar vkohov avatar yodaembedding avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aim's Issues

Artifact name validation is too strict

I see artifact names are validated to only include alphanumeric characters by the follow regex: re.search(r'^[^\d\W]\w*\Z', str(name))

This does not allow for artifact names such as total_loss or fast_rcnn/cls_accuracy. Is there any particular reason why artifact names cannot contain special characters?

Aim can't find docker

When I run aim up in conda environment, Aim can't find docker which is already installed. Could you please let me know how to resolve this?

How to log hyperparameters with Aim and Lighting

I want aim to store the hyperparameter configuration that was used for the experiment. I have tried the following, but it does not seems to work

model = Training_Models.ED_Training() lighting_data = datamodule.DataModule(data, conf) trainer = pl.Trainer( gpus=-1, logger=AimLogger(experiment="Test"), max_epochs=10, ) model.hparams.update(conf.hyperparameters)

How can I achieve that?

Add Tensorboard logdir Integration to docs

There is an ability at Aim to read Tensorboard logdir and search through the experiments recorded by TensorBoard.

Add section in docs to explain how to run Aim on Tensorboard logdir.

Add Aim query language information/examples popover in explore screen

Problem

New users who are not familiar with Aim and Aim query language don't know how to search/retrieve tracked experiments and how to make search queries in explore screen.

Solution

Show examples of Aim QL and provide links to QL documentation and detailed specs.

image

Details

Inside "IF" input component of explore screen add hint icon. When clicking on that, open popover with following information:

  • 2 code examples wrapped in markdown with python syntax highlighting;
  • a footer section with links to docs and specs(GitHub);

Code examples to show:

  • context.subset in ("train", "val") and hparams.learning_rate >= 0.01
  • experiment == "XLM-syntok-dynamic-full-titles-avg" and hparams.align == "XLM" and hparams.dataset in ("qqp", "mrpc")

Footer content:
"Aim Query Language is pythonic and fairly easy to get used to. If having issues, please refer to the docs for more examples or the detailed spec."
docs href: https://github.com/aimhubio/aim#searching-experiments
spec href: https://github.com/aimhubio/aim/wiki/Aim-Query-Language

ModuleNotFoundError: No module named 'aim.stack_grps'

Trying to get aim up and running. From the user account it reports docker not installed (not true, but I'm assuming it requires root). Fresh conda install as root, conda create -n keras python=3.8 pip

then pip install aim
aim up

  File "/root/anaconda3/envs/keras/bin/aim", line 5, in <module>
    from aim.commands.cli import cli
  File "/root/anaconda3/envs/keras/lib/python3.8/site-packages/aim/commands/cli.py", line 3, in <module>
    from aim.commands.cmd_provision import provision_command
  File "/root/anaconda3/envs/keras/lib/python3.8/site-packages/aim/commands/cmd_provision.py", line 4, in <module>
    from aim.commands.helpers import controller_args, aim_home_option, init_aim_home_option, pass_aim_context, \
  File "/root/anaconda3/envs/keras/lib/python3.8/site-packages/aim/commands/helpers.py", line 4, in <module>
    from aim.config.aim_context import AimContext, AccountContext
  File "/root/anaconda3/envs/keras/lib/python3.8/site-packages/aim/config/aim_context.py", line 3, in <module>
    import aim.controllers
  File "/root/anaconda3/envs/keras/lib/python3.8/site-packages/aim/controllers/__init__.py", line 1, in <module>
    from aim.controllers.ctl_network_environment import NetEnvController
  File "/root/anaconda3/envs/keras/lib/python3.8/site-packages/aim/controllers/ctl_network_environment.py", line 10, in <module>
    from aim.stack_grps.grp_application import ApplicationStackGroup
ModuleNotFoundError: No module named 'aim.stack_grps'```

Any ideas?

AttributeError: 'TrackMetricsCallback' object has no attribute '_implements_train_batch_hooks'

Also having an issue with the keras tracking callback:

  File "GeneFindingTestingTrainedTransformer.py", line 276, in <module>
    callbacks=[csvlog, AimTracker.metrics([aobj])])
  File "/home/josephguhlin/miniconda3/envs/keras/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py",
line 108, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/home/josephguhlin/miniconda3/envs/keras/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py",
line 1074, in fit
    steps=data_handler.inferred_steps)
  File "/home/josephguhlin/miniconda3/envs/keras/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py", line 2
35, in __init__
    cb._implements_train_batch_hooks() for cb in self.callbacks)
  File "/home/josephguhlin/miniconda3/envs/keras/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py", line 2
35, in <genexpr>
    cb._implements_train_batch_hooks() for cb in self.callbacks)
AttributeError: 'TrackMetricsCallback' object has no attribute '_implements_train_batch_hooks'```

Could aim record iteration information?

I look at the docs and find that track func only accept epoch as input.
In fact, in modern deep learning, we always use iteration as the points in time axis.
Although I can use iteration as epoch, but I also want to record some metrics which only corresponding to epoch.
So, can you have any solutions?

Should aim run in docker?

I install docker in Ubuntu, then I start it and my docker service is active and add the my user in docker group also.

Then I run aim up in my anaconda environment. But it says Oops I don’t have docker installed.

Could u please specify how to run theaim up?

hyperparameters

Hi, thanks for your fantastic work!! But i hava a question: i use pytorch and argparse, when i use like these:

`
args = parser.parse_args()

timestamp = time.strftime("%m-%d_%H-%M", time.localtime())

aim.set_params(args, name='hparams')  # Passing name argument is optional

`
the name of my runs will automaticly be 'default' ? how can i specify this ?
and i can not see my args in runs?

thank you!!

Address is not accessible

Hi,

I am sorry for another maybe basic question which I cannot find doc or explanation from the README.
When I tried aim up --tf_logs <my_location> --host localhost --port 10008
It directly returned me

Failed to run Aim UI.
    Please check if address localhost:10008 is accessible.

Is it related to firewall? Or some setting I must do before launching it?

I originally thought it was functioned similarly as tensorboard and I am trying the same way.
My log files are located in a remote server, I tried to run aim at the server for some port, and I will forward it to my local machine's local port so that I can open it in a browser.

Thank you very much

Format long texts of parallel coordinates plot axes ticks

Problem

Parallel coordinates plot doesn't handle long texts

Screen Shot 2021-01-25 at 3 52 28 PM

Solution

Format long texts in parallel coordinates plot axes ticks:

  • Replace svg text elements with divs(wrapped in foreignObject) here
  • Apply css styles to render an ellipsis ("...") representing the clipped text

Specify -h HOST for aim up

Would it be possible to add a host option to aim up in addition to the port one? I'd like to run the server on 0.0.0.0 instead of localhost, like this:

aim up -p 5000 -h 0.0.0.0

Maybe it would be possible to take advantage of create_host_config or something along those lines...

How to install docker for aim?

Hi, I installed aim using pip and type aim up, it returns me:

Oops! You don't have docker installed. Aim UI runs on docker. Please install docker

I have followed https://docs.docker.com/engine/install/ubuntu/ to install docker engine on my machine. But the above errors are still there. What should I do for that?

Multiple runs in same Python script without using Session object.

Is there a way to record multiple runs, belonging to the same experiment, in the same Python script without explicitly using the Session object? So far, I was able to do it with the Session object, but it seems to me that it would be cleaner if it was possible by adding an extra argument in the aim.set_params function.

Restarting an experiment programmatically

Hi and thanks for the great work here :D So I am using the Session api, and I was wondering if there is any way to programmatically restart the experiment. I was thinking perhaps something like

sess = aim.Session(experiment = 'some_name')
sess.clear()

add subplot?

Hi!

it would be very helpful if the metric can be add in one tag like the tensorboard?

eg:

writer.add_scalar(f'train_detail_loss/WholeLoss' %
                      , WholeLoss, epoch)

writer.add_scalar(f'train_detail_loss/OriginalLoss' %
                     OriginalLoss, epoch)

besides, if the number of params is too big, the web page is all blank after ' aim up ' command?

thank you!!

Show "Loading" instead of an error

Sometimes aim takes too long to load the experiments in explore view.
This is okay, but while the browser is pending for the response, it tells that there are no such experiments.
After the response is received/loaded, it goes back to normal.

Screen Shot 2021-01-21 at 18 56 05

Deleting runs/experiments in a bulk.

Currently, the only way to delete the run(s) is by using the CLI. The thing is that as far as I understand you have to delete the runs 1 by 1, i.e. :

 aim experiment rm -n experiment_name

It would be very nice to have functionality like

 aim experiment rm -rf *

or

 aim experiment rm -n run1* (For specific batches of runs)

Taking regexp like expressions to find the runs and delete/archive them.

It would also be very helpful to have a "delete by date" type of functionality. I think it should also be easy to use AimQL to find experiments and delete them in the same line. I.E.

aim experiment rm ("metric-loss" , score_1, l_reg_1 if dataset.dataset_run==FB15K)

Of course, I would love to see a simple deletion/archive button in frontend as well.

New Session does not start at step 0

I'm trying to track multiple experiments within a Jupyter notebook by reinitializing a session like so:

import aim                                                                    
session = aim.Session()                                                               
session.track(1, name='val')                                                          
session.track(2, name='val')                                                          
session.close()                                                                       

session = aim.Session()                                                               
session.track(3, name='val')                                                          
session.track(4, name='val')                                                          
session.close()  

The runs show correctly in the dashboard but for the second run, the metric starts showing at step 2 instead of 0. This prevents me from comparing two runs in the Explore tab as they are on different time scales. Is this the expected behaviour? If so, how can I track multiple runs within a single process?

Dashboard:
Screenshot from 2020-11-13 13-42-17

Run:
Screenshot from 2020-11-13 13-43-31

Explore:
Screenshot from 2020-11-13 13-49-14

Listing and finding runs in CLI

It would be great to be able to list all of your runs in CLI as well as frontend. I.E:

aim experiment list

and filter runs in CLI as well.

Pytorch Lighting arguments marked as Optional but needed

Hello!
I am looking at your logger internal, and here I see that the args are optional.

class _PytorchLightningLogger(LightningLoggerBase):
            def __init__(
                self,
                repo: Optional[str] = None,
                experiment: Optional[str] = None,
                train_metric_prefix: Optional[str] = "train_",
                val_metric_prefix: Optional[str] = "val_",
                test_metric_prefix: Optional[str] = "test_",
                flush_frequency: int = DEFAULT_FLUSH_FREQUENCY,
            ):

Nevertheless, if I do not fill anything inside and try to run a model, I get the following error when running model.fit

  File "/home/gem/.pyenv/versions/3.8.6/lib/python3.8/posixpath.py", line 90, in join
    genericpath._check_arg_types('join', a, *p)
  File "/home/gem/.pyenv/versions/3.8.6/lib/python3.8/genericpath.py", line 152, in _check_arg_types
    raise TypeError(f'{funcname}() argument must be str, bytes, or '
TypeError: join() argument must be str, bytes, or os.PathLike object, not 'NoneType'

Aim UI not showing all metrics

Hi,
can I get some help regarding some situation probably regarding Aim UI?
I have many (48-64) runs in each experiment and also there exist almost 90 metrics.
At the early stages of the training, I could use Aim UI as usual.

But at some point (not sure when exactly), this issue appeared.
While Dashboard lists all the runs, they only show the run parameters set by set_params(), but not the metrics.
In Dashboard, I can click on each run to view its details.
However, Aim UI does not display anything but the message Experiment "[experiment name]" does not exist although the drop-down menu above still shows [experinemt name], for some of the runs.
I also saw some cases that in the run information page, only a subset of the metrics was shown.
Plus, in Explore, I can query only some subset of the metrics, as well.

Fortunately, it looks like the saving part is fine; I can see all of them (runs and metrics) and that the disk consumptions are consistent.
Is there any workaround that might help me resolve the issue, please?

failed in windows

failed in windows,there was no output after i type" aim up"
i have docker pull aimhubio/aim-board

Highlight all metric plots from the same run together in charts

It would be super helpful if highlighting one metric plot (e.g. mouse hovering on the plot) causes all the other metrics from that specific run in every chart to be highlighted, too!

I believe it is a common scenario to compare hyperparameter sets with more than one evaluation metrics at the same time.

Thank you.

refold graphs together

Hi, can aim fold several graphs together? As in medical image segmentaion field, we ofter need to try k-fold validation. so if aim can organize the metrics of same fold together, it would be great!! Actually, right now, i use the following code under tensorboard to realize this:

    ## fold can be in [fold1,fold2,...fold_n]
    writer.add_scalar(f'%s/train_detail/Loss' %fold, train_loss, epoch)
    writer.add_scalar(f'%s/train_detail/Acc' %fold, train_acc, epoch)

    writer.add_scalar(f'%s/val_detail/Loss' % fold, val_loss, epoch)
    writer.add_scalar(f'%s/val_detail/Acc' %fold, val_acc, epoch)

so the page will be like πŸ‘:

image

Moreover, can aim track several metrics at one time ? like < add_scalars > in tensorboard?

Huge thanks !!

Archive bug.

Thanks for your awesome product!

I found that archived runs reappear after a while.

I'm using the latest version (2.1.5).

Is there any debug log that I can provide to you? How to find these logs?

Ability to overwrite/manage runs and experiments from SDK

Sometimes, especially for testing purposes, I want to run a multitude of experiments/runs without having to manually delete the runs/experiments from the previous test script. Is there a way to manager runs and experiments from the SDK for this purpose?

Colab integration

It might be useful to have some native colab integration. I am wondering what are your plans in that direction?

Error running aim up

Hello, I am facing this issues trying to get aim up and running:

File "/conda/envs/torch/lib/python3.8/site-packages/aim/engine/container/container.py", line 50, in image_exist
images = cls._get_docker_client().images.list()
AttributeError: 'function' object has no attribute 'list'

I dug a little further, solving the issues below and found that docker_client.containers is an empty list. I am using a third-party code that uses aim, and I'm wondering if these issues are related to a different version used to create the logging files or if it is a docker issue.

Thanks in advance.

File "/conda/envs/torch/lib/python3.8/site-packages/aim/engine/container/container.py", line 52, in image_exist
for t in i.attrs['RepoTags']:
AttributeError: 'dict' object has no attribute 'attrs'

File "/conda/envs/torch/lib/python3.8/site-packages/aim/engine/container/container.py", line 100, in get_container
containers = docker_client.containers.list(all=~running_only,
AttributeError: 'function' object has no attribute 'list'

File "/conda/envs/torch/lib/python3.8/site-packages/aim/engine/container/container.py", line 112, in kill
aim_containers = docker_client.containers.list(all=True, filters={
AttributeError: 'function' object has no attribute 'list'

File "/conda/envs/torch/lib/python3.8/site-packages/aim/engine/container/container.py", line 83, in up
container = docker_client.containers.run(image_name,
AttributeError: 'function' object has no attribute 'run'

File "/conda/envs/torch/lib/python3.8/site-packages/aim/engine/container/container.py", line 83, in up
container = docker_client.containers(image_name,
TypeError: containers() got an unexpected keyword argument 'name'

Support for alternative x-axis

It might be a nice addition to the aim UI to support selecting one of the existing metrics and using it for the x-axis instead of the epochs.

Thank you.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.