azure/datasciencevm

Tools and Docs on the Azure Data Science Virtual Machine (http://aka.ms/dsvm)

Data Science Virtual Machine

The Data Science Virtual Machine (DSVM) is a customized VM image on Microsoft’s Azure cloud built specifically for doing data science. It has many popular data science and other tools pre-installed and pre-configured to jump-start building intelligent applications for advanced analytics. It is available for Windows Server 2019 and Ubuntu 18.04 LTS.

You can try the Data Science VM for free for 30 days (with $200 credits) with a free Azure Trial. The Ubuntu DSVM also provides a free trial through the Azure Test Driver. The Test Drive will provide full access to you own instance of the VM with just a free Microsoft account - No Azure subscription or credit card needed.

About this Repo

This repo features tools, tips and extensions (see below) to the Data Science VM. We invite the DSVM user community to contribute any useful tools, scripts, or extensions you may have written to enhance the user experience on the DSVM.

About Extensions

Azure Resource manager (ARM) provides a capability to define extensions for resources like Virtual Machines. VM Extensions are scripts that are run during the deployment of a VM to install additional pieces of software or reconfigure the VM to your needs or to comply with specific IT policies your company may have.

Contributing

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

datasciencevm's Issues

Cannot access Azure Blob Store via Spark nor Drill

Hi, I'm trying to access an Azure Blob Store file via Spark or Drill. In both cases I have the following error (here shown in Spark):

sc.textFile("wasb://[email protected]/TestQuery/InputMonitor/0_331b3d2e8ecd428398283b5d40d0e443_1.json").count()
java.io.IOException: No FileSystem for scheme: wasb
  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
  at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:258)
  at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
  at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
  at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
  at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2092)
  at org.apache.spark.rdd.RDD.count(RDD.scala:1162)
  ... 49 elided

I've checked the configuration and in both cases the azure-storage and hadoop-azure jars are loaded.

In both cases I've configured my site_config.xml to be something like:

<?xml version="1.0" encoding="UTF-8" ?>
<configuration>
  <property>
    <name>fs.AbstractFileSystem.wasb.Impl</name>
    <value>org.apache.hadoop.fs.azure.Wasb</value>
  </property>
  <property>
    <name>fs.azure.account.key.dmstore1.blob.core.windows.net</name>
    <value>{my key here}</value>
  </property>
</configuration>

Any idea on how to solve the problem? Thanks!

Can't create a new conda env: InvalidArchiveError

The DSVM is broken, I can't create a new conda env with this command

conda create -n py38 -c conda-forge python=3.8 toolz hyp3_sdk rioxarray geopandas pandas

It returns

ryan@vmName:~$ conda create -n py38 -c conda-forge python=3.8 toolz hyp3_sdk rioxarray geopandas pandas
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /anaconda/envs/py38

  added / updated specs:
    - geopandas
    - hyp3_sdk
    - pandas
    - python=3.8
    - rioxarray
    - toolz


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    python-dateutil-2.8.1      |             py_0         220 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         220 KB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-1_gnu
  affine             conda-forge/noarch::affine-2.3.0-py_0
  attrs              conda-forge/noarch::attrs-20.3.0-pyhd3deb0d_0
  boost-cpp          conda-forge/linux-64::boost-cpp-1.74.0-hc6e9bd1_2
  brotlipy           conda-forge/linux-64::brotlipy-0.7.0-py38h497a2fe_1001
  bzip2              conda-forge/linux-64::bzip2-1.0.8-h7f98852_4
  c-ares             conda-forge/linux-64::c-ares-1.17.1-h7f98852_1
  ca-certificates    conda-forge/linux-64::ca-certificates-2020.12.5-ha878542_0
  cairo              conda-forge/linux-64::cairo-1.16.0-h7979940_1007
  certifi            conda-forge/linux-64::certifi-2020.12.5-py38h578d9bd_1
  cffi               conda-forge/linux-64::cffi-1.14.5-py38ha65f79e_0
  cfitsio            conda-forge/linux-64::cfitsio-3.470-hb418390_7
  chardet            conda-forge/linux-64::chardet-4.0.0-py38h578d9bd_1
  click              conda-forge/noarch::click-7.1.2-pyh9f0ad1d_0
  click-plugins      conda-forge/noarch::click-plugins-1.1.1-py_0
  cligj              conda-forge/noarch::cligj-0.7.1-pyhd8ed1ab_0
  cryptography       conda-forge/linux-64::cryptography-3.4.4-py38h3e25421_0
  curl               conda-forge/linux-64::curl-7.75.0-h979ede3_0
  expat              conda-forge/linux-64::expat-2.2.10-h9c3ff4c_0
  fiona              conda-forge/linux-64::fiona-1.8.18-py38h58f84aa_1
  fontconfig         conda-forge/linux-64::fontconfig-2.13.1-hba837de_1004
  freetype           conda-forge/linux-64::freetype-2.10.4-h0708190_1
  freexl             conda-forge/linux-64::freexl-1.0.5-h516909a_1002
  gdal               conda-forge/linux-64::gdal-3.2.1-py38hc0b2d6b_7
  geopandas          conda-forge/noarch::geopandas-0.9.0-pyhd8ed1ab_0
  geos               conda-forge/linux-64::geos-3.9.1-h9c3ff4c_2
  geotiff            conda-forge/linux-64::geotiff-1.6.0-h2b14fbe_4
  gettext            conda-forge/linux-64::gettext-0.19.8.1-h0b5b191_1005
  giflib             conda-forge/linux-64::giflib-5.2.1-h36c2ea0_2
  hdf4               conda-forge/linux-64::hdf4-4.2.13-h10796ff_1004
  hdf5               conda-forge/linux-64::hdf5-1.10.6-nompi_h6a2412b_1114
  hyp3_sdk           conda-forge/noarch::hyp3_sdk-0.5.0-pyhd8ed1ab_0
  icu                conda-forge/linux-64::icu-68.1-h58526e2_0
  idna               conda-forge/noarch::idna-2.10-pyh9f0ad1d_0
  jpeg               conda-forge/linux-64::jpeg-9d-h36c2ea0_0
  json-c             conda-forge/linux-64::json-c-0.15-h98cffda_0
  kealib             conda-forge/linux-64::kealib-1.4.14-hcc255d8_2
  krb5               conda-forge/linux-64::krb5-1.17.2-h926e7f8_0
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.35.1-hea4e1c9_2
  libblas            conda-forge/linux-64::libblas-3.9.0-8_openblas
  libcblas           conda-forge/linux-64::libcblas-3.9.0-8_openblas
  libcurl            conda-forge/linux-64::libcurl-7.75.0-hc4aaa36_0
  libdap4            conda-forge/linux-64::libdap4-3.20.6-hd7c4107_1
  libedit            conda-forge/linux-64::libedit-3.1.20191231-he28a2e2_2
  libev              conda-forge/linux-64::libev-4.33-h516909a_1
  libffi             conda-forge/linux-64::libffi-3.3-h58526e2_2
  libgcc-ng          conda-forge/linux-64::libgcc-ng-9.3.0-h2828fa1_18
  libgdal            conda-forge/linux-64::libgdal-3.2.1-h38ff51b_7
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-9.3.0-hff62375_18
  libgfortran5       conda-forge/linux-64::libgfortran5-9.3.0-hff62375_18
  libglib            conda-forge/linux-64::libglib-2.66.7-h3e27bee_1
  libgomp            conda-forge/linux-64::libgomp-9.3.0-h2828fa1_18
  libiconv           conda-forge/linux-64::libiconv-1.16-h516909a_0
  libkml             conda-forge/linux-64::libkml-1.3.0-h02e6976_1012
  liblapack          conda-forge/linux-64::liblapack-3.9.0-8_openblas
  libnetcdf          conda-forge/linux-64::libnetcdf-4.7.4-nompi_h56d31a8_107
  libnghttp2         conda-forge/linux-64::libnghttp2-1.43.0-h812cca2_0
  libopenblas        conda-forge/linux-64::libopenblas-0.3.12-pthreads_h4812303_1
  libpng             conda-forge/linux-64::libpng-1.6.37-h21135ba_2
  libpq              conda-forge/linux-64::libpq-13.1-hfd2b0eb_2
  librttopo          conda-forge/linux-64::librttopo-1.1.0-h1185371_6
  libspatialindex    conda-forge/linux-64::libspatialindex-1.9.3-h9c3ff4c_3
  libspatialite      conda-forge/linux-64::libspatialite-5.0.1-he52d314_3
  libssh2            conda-forge/linux-64::libssh2-1.9.0-ha56f1ee_6
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-9.3.0-h6de172a_18
  libtiff            conda-forge/linux-64::libtiff-4.2.0-hdc55705_0
  libuuid            conda-forge/linux-64::libuuid-2.32.1-h7f98852_1000
  libwebp-base       conda-forge/linux-64::libwebp-base-1.2.0-h7f98852_0
  libxcb             conda-forge/linux-64::libxcb-1.13-h7f98852_1003
  libxml2            conda-forge/linux-64::libxml2-2.9.10-h72842e0_3
  lz4-c              conda-forge/linux-64::lz4-c-1.9.3-h9c3ff4c_0
  munch              conda-forge/noarch::munch-2.5.0-py_0
  ncurses            conda-forge/linux-64::ncurses-6.2-h58526e2_4
  numpy              conda-forge/linux-64::numpy-1.20.1-py38h18fd61f_0
  openjpeg           conda-forge/linux-64::openjpeg-2.4.0-hf7af979_0
  openssl            conda-forge/linux-64::openssl-1.1.1j-h7f98852_0
  pandas             conda-forge/linux-64::pandas-1.2.3-py38h51da96c_0
  pcre               conda-forge/linux-64::pcre-8.44-he1b5a44_0
  pip                conda-forge/noarch::pip-21.0.1-pyhd8ed1ab_0
  pixman             conda-forge/linux-64::pixman-0.40.0-h36c2ea0_0
  poppler            conda-forge/linux-64::poppler-0.89.0-h2de54a5_5
  poppler-data       conda-forge/noarch::poppler-data-0.4.10-0
  postgresql         conda-forge/linux-64::postgresql-13.1-h6303168_2
  proj               conda-forge/linux-64::proj-7.2.0-h277dcde_2
  pthread-stubs      conda-forge/linux-64::pthread-stubs-0.4-h36c2ea0_1001
  pycparser          conda-forge/noarch::pycparser-2.20-pyh9f0ad1d_2
  pyopenssl          conda-forge/noarch::pyopenssl-20.0.1-pyhd8ed1ab_0
  pyparsing          conda-forge/noarch::pyparsing-2.4.7-pyh9f0ad1d_0
  pyproj             conda-forge/linux-64::pyproj-3.0.1-py38h16ecdd7_0
  pysocks            conda-forge/linux-64::pysocks-1.7.1-py38h578d9bd_3
  python             conda-forge/linux-64::python-3.8.8-hffdb5ce_0_cpython
  python-dateutil    conda-forge/noarch::python-dateutil-2.8.1-py_0
  python_abi         conda-forge/linux-64::python_abi-3.8-1_cp38
  pytz               conda-forge/noarch::pytz-2021.1-pyhd8ed1ab_0
  rasterio           conda-forge/linux-64::rasterio-1.2.1-py38h57accd2_2
  readline           conda-forge/linux-64::readline-8.0-he28a2e2_2
  requests           conda-forge/noarch::requests-2.25.1-pyhd3deb0d_0
  rioxarray          conda-forge/noarch::rioxarray-0.3.1-pyhd8ed1ab_0
  rtree              conda-forge/linux-64::rtree-0.9.7-py38h02d302b_1
  scipy              conda-forge/linux-64::scipy-1.6.0-py38hb2138dd_0
  setuptools         conda-forge/linux-64::setuptools-49.6.0-py38h578d9bd_3
  shapely            conda-forge/linux-64::shapely-1.7.1-py38h4fc1155_4
  six                conda-forge/noarch::six-1.15.0-pyh9f0ad1d_0
  snuggs             conda-forge/noarch::snuggs-1.4.7-py_0
  sqlite             conda-forge/linux-64::sqlite-3.34.0-h74cdb3f_0
  tiledb             conda-forge/linux-64::tiledb-2.2.4-hb9a9e87_1
  tk                 conda-forge/linux-64::tk-8.6.10-h21135ba_1
  toolz              conda-forge/noarch::toolz-0.11.1-py_0
  tqdm               conda-forge/noarch::tqdm-4.59.0-pyhd8ed1ab_0
  tzcode             conda-forge/linux-64::tzcode-2021a-h7f98852_1
  tzdata             conda-forge/noarch::tzdata-2021a-he74cb21_0
  urllib3            conda-forge/noarch::urllib3-1.26.3-pyhd8ed1ab_0
  wheel              conda-forge/noarch::wheel-0.36.2-pyhd3deb0d_0
  xarray             conda-forge/noarch::xarray-0.17.0-pyhd8ed1ab_0
  xerces-c           conda-forge/linux-64::xerces-c-3.2.3-h9d8b166_2
  xorg-kbproto       conda-forge/linux-64::xorg-kbproto-1.0.7-h7f98852_1002
  xorg-libice        conda-forge/linux-64::xorg-libice-1.0.10-h7f98852_0
  xorg-libsm         conda-forge/linux-64::xorg-libsm-1.2.3-hd9c2040_1000
  xorg-libx11        conda-forge/linux-64::xorg-libx11-1.7.0-h7f98852_0
  xorg-libxau        conda-forge/linux-64::xorg-libxau-1.0.9-h7f98852_0
  xorg-libxdmcp      conda-forge/linux-64::xorg-libxdmcp-1.1.3-h7f98852_0
  xorg-libxext       conda-forge/linux-64::xorg-libxext-1.3.4-h7f98852_1
  xorg-libxrender    conda-forge/linux-64::xorg-libxrender-0.9.10-h7f98852_1003
  xorg-renderproto   conda-forge/linux-64::xorg-renderproto-0.11.1-h7f98852_1002
  xorg-xextproto     conda-forge/linux-64::xorg-xextproto-7.3.0-h7f98852_1002
  xorg-xproto        conda-forge/linux-64::xorg-xproto-7.0.31-h7f98852_1007
  xz                 conda-forge/linux-64::xz-5.2.5-h516909a_1
  zlib               conda-forge/linux-64::zlib-1.2.11-h516909a_1010
  zstd               conda-forge/linux-64::zstd-1.4.9-ha95c52a_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
python-dateutil-2.8. | 220 KB    |                                                                                                                                                                   |   0% WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/python_dateutil-2.8.1.dist-info/RECORD.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/python_dateutil-2.8.1.dist-info/zip-safe.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/python_dateutil-2.8.1.dist-info/top_level.txt.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/python_dateutil-2.8.1.dist-info/WHEEL.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/python_dateutil-2.8.1.dist-info/INSTALLER.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/python_dateutil-2.8.1.dist-info/METADATA.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/python_dateutil-2.8.1.dist-info/LICENSE.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/tz/tz.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/tz/__init__.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/tz/win.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/tz/_common.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/tz/_factories.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/parser/_parser.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/parser/__init__.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/parser/isoparser.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/zoneinfo/__init__.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/zoneinfo/dateutil-zoneinfo.tar.gz.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/zoneinfo/rebuild.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/tzwin.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/utils.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/__init__.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/relativedelta.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/easter.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/_common.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/_version.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/rrule.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/licenses/LICENSE.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/test/run_test.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/recipe/recipe_log.txt.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/recipe/conda_build_config.yaml.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/recipe/meta.yaml.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/recipe/meta.yaml.template.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/recipe/run_test.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/git.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/hash_input.json.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/link.json.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/index.json.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/about.json.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/paths.json.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/files.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/repodata_record.json.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/python_dateutil-2.8.1.dist-info/RECORD.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/python_dateutil-2.8.1.dist-info/zip-safe.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/python_dateutil-2.8.1.dist-info/top_level.txt.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/python_dateutil-2.8.1.dist-info/WHEEL.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/python_dateutil-2.8.1.dist-info/INSTALLER.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/python_dateutil-2.8.1.dist-info/METADATA.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/python_dateutil-2.8.1.dist-info/LICENSE.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/tz/tz.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/tz/__init__.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/tz/win.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/tz/_common.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/tz/_factories.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/parser/_parser.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/parser/__init__.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/parser/isoparser.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/zoneinfo/__init__.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/zoneinfo/dateutil-zoneinfo.tar.gz.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/zoneinfo/rebuild.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/tzwin.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/utils.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/__init__.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/relativedelta.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/easter.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/_common.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/_version.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/site-packages/dateutil/rrule.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/licenses/LICENSE.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/test/run_test.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/recipe/recipe_log.txt.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/recipe/conda_build_config.yaml.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/recipe/meta.yaml.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/recipe/meta.yaml.template.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/recipe/run_test.py.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/git.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/hash_input.json.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/link.json.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/index.json.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/about.json.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/paths.json.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/files.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /anaconda/pkgs/python-dateutil-2.8.1-py_0/info/repodata_record.json.  Please remove this file manually (you may need to reboot to free file handles)
python-dateutil-2.8. | 220 KB    | ################################################################################################################################################################# | 100% 

InvalidArchiveError('Error with archive /anaconda/pkgs/python-dateutil-2.8.1-py_0.tar.bz2.  You probably need to delete and re-download or re-create this file.  Message from libarchive was:\n\nCould not unlink')

Ubuntu deployment fails with linking error

Hello,

the deployment of Scripts/CreateDSVM/Ubuntu/dsvm-vmss-cluster.json (as mentioned in the official docs) fails with

ln: failed to create symbolic link '/home/<USER>/notebooks/remote': No such file or directory"

which is triggered in Extensions/General/mountazurefiles.sh because the notebooks directory doesn't exist at the stage where the script is run.

For debugging I tried the following:

• Disable the linking command: deployment works and the notebooks directory does exist, with autopopulated notebooks
• Create the directory before linking: deployment works, but the directory is not autopopulated with notebooks

So it seems that some process is creating that directory after mountazurefiles.sh is run.

Anyway, option 2 is probably the cleanest and can be achieved by running

test -d /home/$5/notebooks || mkdir -p /home/$5/notebooks

before the ln line in mountazurefiles.sh. The only disadvantage is that the notebooks directory is then empty / not autopopulated. So this requires a proper fix. The best would be to run the mount script, after whatever process created ~/notebooks

Andreas

Better error handling when json paramter are missing.

It took me a while to figure out that
'malformed node or string: <_ast.Attribute object at 0x7fe291a33518>' means I had just mistyped the path to my json template.

Can I have a Windows like Desktop view for Linux DSVM.

Can I have a Windows like Desktop view for Linux DSVM. I have created a Windows DSVM which feels same as windows desktop,Is there something like this in Linux DSVM so that i can directly connect using rdp file.

ps:I'm using a Linux DSVM for fastai course.

Stray commas in templates

Messing up the parsing in R:

DataScienceVM/Scripts/CreateDSVM/Ubuntu/azuredeploysshkey.json

Line 39 in 41c71ea

"Standard_NC24",

DataScienceVM/Scripts/CreateDSVM/Ubuntu/multiazuredeploy.json

Line 46 in 41c71ea

"Standard_NC24",

DataScienceVM/Scripts/CreateDSVM/Ubuntu/azuredeploywithext.json

Line 39 in 41c71ea

"Standard_NC24",

Open RStudio Server port on Linux images

Hi,

I had a user try to use RStudio Server after deploying a DSVM. They found that they had to manually open the port before they could connect. Would it be worthwhile to open this port automatically for the Linux-based images?

Default Network Security Group configuration insecure

When deploying DSVM through the Azure Portal the NSG configuration allows inbound connections on port 22 for any source IP address. After deploying such a DSVM in this manner it was not long before Security Center detected a brute force attack on that Virtual Machine with an attacker attempting to gain access via SSH. It may be worth considering starting with a more secure configuration accompanied by documentation on how to configure an inbound rule.

installing pdfminer.six

Hey,

I wanted to install a python package called pdfminer.six
i tried pip install the package into the data science vm. i ran into issues.

GPU Not Available using PyTorch in azureml_py38_PT_and_TF kernel on DSVM

Is it only me?

$ conda activate azureml_py38_PT_and_TF
(azureml_py38_PT_and_TF) $ python
Python 3.8.5 (default, Sep  4 2020, 07:30:14)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
False

$ nvidia-smi
Fri Apr 15 04:43:40 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01   Driver Version: 470.103.01   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  On   | 00000001:00:00.0 Off |                  Off |
| N/A   29C    P0    29W / 250W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Looks like pytorch installed is built as CPU-only?

~$ conda list | grep pytorch
pytorch                   1.10.0              py3.8_cpu_0    pytorch

Leaving from azureml_py38_PT_and_TF and after I installed pytorch using conda onto newly created conda env from scratch, torch.cuda.is_availabel() returned True. So the problem should not sit around CUDA.

My VM's image details:

"imageReference": {
      "communityGalleryImageId": null,
      "exactVersion": "22.04.01",
      "id": null,
      "offer": "ubuntu-1804",
      "publisher": "microsoft-dsvm",
      "sharedGalleryImageId": null,
      "sku": "1804-gen2",
      "version": "latest"
    },

It was succeeded when I was using old version of VM and previous conda env named azureml_py36_pytorch, which has been deleted recently according to the document.
BTW, in this env, PyTorch is built as cuda version.

$ conda list | grep pytorch
# packages in environment at /anaconda/envs/azureml_py36_pytorch:
pytorch                   1.10.0          py3.6_cuda11.1_cudnn8.0.5_0    pytorch

Connect from R Studio to DataBricks

I am currently using R Studio for a project and I am trying to connect to my databricks cluster via sparklyr.

For this I am following the process as described here: https://spark.rstudio.com/examples/databricks-cluster-remote/

However when I try to do databrick-connect test I get the following error

Incompatible Jackson version: 2.10.0 databricks connect

Is there any way to get a concrete way of connecting dsvm R environment to databricks?

update jupyterlab on image to 1.0

The 1.0.1 version is now out on conda and pip and should be officialy publicized soon: https://github.com/jupyterlab/jupyterlab/milestone/2

Where is ArcGIS desktop?

where is the ArcGIS desktop installed? I can not find it in the start menu (can't find it by search either). I installed the "Geo Artificial Intelligence Data Science Virtual Machine" hoping I can use both ArcGIS desktop and Anaconda. Can we only use ArcGIS through arcpy through python on "Geo Artificial Intelligence Data Science Virtual Machine" ?

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Script is not working: resource is currently not available in location 'westus2' zones

The setup process described here https://course.fast.ai/start_azure_dsvm is not working.

If fails with the following error:

Creating Azure Data Science VM fastai2...
It is recommended to use parameter "--public-ip-sku Standard" to create new VM with Standard public IP. Please note that the default public IP used for VM creation will be changed from Basic to Standardin the future.
{"error":{"code":"InvalidTemplateDeployment","message":"The template deployment failed with error: 'The resource with id: '/subscriptions/98d6eedf-a667-411b-8be4-5c537c646ada/resourceGroups/fastai2/providers/Microsoft.Compute/virtualMachines/fastai2' failed validation with message: 'The requested size for resource '/subscriptions/98d6eedf-a667-411b-8be4-5c537c646ada/resourceGroups/fastai2/providers/Microsoft.Compute/virtualMachines/fastai2' is currently not available in location 'westus2' zones '' for subscription '98d6eedf-a667-411b-8be4-5c537c646ada'. Please try another size or deploy to a differentlocation or zones. See https://aka.ms/azureskunotavailable for details.'.'."}}
(ResourceNotFound) The Resource 'Microsoft.Compute/virtualMachines/fastai2' under resource group 'fastai2' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Installing  fastai v2 and notebooks...
(ResourceNotFound) The Resource 'Microsoft.Compute/virtualMachines/fastai2' under resource group 'fastai2' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
ERROR: (ResourceNotFound) The Resource 'Microsoft.Compute/virtualMachines/fastai2' under resource group 'fastai2' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
You can now login to VM with SSH or use Jupyterhub by visiting https://:8000/ (Ignore self signed cert warnings)
Login userid is 'fastuser' with password you entered above

R kernels not working

Hi,
In any newly created DSVM (or new ML Notebook VM which I think is based on the DSVM), the R kernels are not working in Jupyter.
I've tried with multiple, freshly deployed VMs.
The kernel tries to start up and then dies.

Any examples/ resources for using DSVM with R

Hi Author,

Could you please share few examples, on how to use DSVM with R inside Azure batch jobs ? Your help is highly appreciated.

Thanks
Arun Rajesh B.

h2o on Windows 2019 DSVM

Add a conda environment for h2o

http://h2o-release.s3.amazonaws.com/h2o/rel-zipf/2/index.html

Create Ubuntu cluster with ssh key login

It doesn't look like this is a supported configuration. Any reason why? If not, I can submit a PR to enable it.

Linux DSVM images not using PTP for timesync

Hi everyone,

It seems that the DSVM images running Linux are not following the recommendations for timesync in Azure, described here:
https://learn.microsoft.com/en-us/azure/virtual-machines/linux/time-sync

The issue can be corrected by:

$ sudo apt install chrony
$ sudo mv /etc/chrony/chrony.conf /etc/chrony.conf.old
$ echo "keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
logdir /var/log/chrony
maxupdateskew 100.0
rtcsync
makestep 1.0 -1
local stratum 2
refclock PHC /dev/ptp_hyperv poll 3 dpoll -2 offset 0" > /etc/chrony/chrony.conf

$ sudo systemctl restart chronyd.service
$ sudo systemctl disable --now systemd-timesyncd
$ sudo systemctl enable chrony

We can then check it with:

$ sudo chronyc sources

Is there a way we can fix this in the current images so they come up correctly?

We are also updating the main reference about timesync on Azure with a few other remarks.

Publish a Windows Server 2019 Datacenter edition image to the Azure Marketplace

As an admin of an Azure subscription, I would like a DataScienceVM image created on the Windows Server 2019 Datacenter edition platform to enable users to login using Azure Active Directory authentication.

Recently it was announced (and still in preview as of 10/28/2019) the ability to sign into a Windows virtual machine using Azure Active Directory. This is only available for Windows Server 2019 Datacenter edition and later. However, currently the only available image for a Windows machine in the Marketplace is a Windows 2016 edition.

Please create a new DataScienceVM on the Windows Server 2019 Datacenter edition platform so we may take advantage of this new feature.

R Studio von Azure

Hi there!

What is the best way of programmatically spinning up an R Studio environment on demand with specific dependencies, e.g. R version, CPU, packages, ... ?

Azure ML CI is a managed VM. Thus, you can't choose the R version you want. One may fire up another CI within R Studio for using a different R environments but that seems to decrease rapid experimentation.

Is DSVM as an unmanaged VM the way to go here? One may create DSVM environments via Azure CLI by using ARM (templates). Perhaps there is something that allows to specify the R version?

AzureVM provides the vm_config function which allows to specify the image to deploy.

Any other ideas?

Thank you!

fastaiv1 Linux has dependency issues

When using the template $/Extensions/fastaiv1/azuredeployLinux.json there are dependency issues when running throught the first lesson of the fastai coursev3.

The issue is with Pillow and pytorch and torchvision.

Typo 'descsription' is spelled wrong in AzureML Template ... it should be description

'descsription' is spelled wrong in AzureML Template ... it should be description

DataScienceVM/Scripts/AzureML/CreateAzureMLWSWithDsvm.json

Line 104 in f24feb0

"descsription": "[parameters('description')]",

the last line of funtion build_model()

the last line of funtion build_model() is
'''
model.add(Activation('softmax'))

# compile and fit model, reduce epochs if you want a result faster
# the validation set is used to identify parameter settings (epoch) that achieves 
# the highest classification accuracy

return model

'''
should it be
'''
model.add(Dense(num_classes))
model.add(Activation('softmax'))

# compile and fit model, reduce epochs if you want a result faster
# the validation set is used to identify parameter settings (epoch) that achieves 
# the highest classification accuracy

return model

''

dsvm should come with nb_conda_kernels installed

so that new user environments can automatically be selected as kernels in jupyter

GPU Not Available in py37_tensorflow kernel on DSVM

I'm trying to use tensorflow with a gpu in a DSVM. I know that I have a gpu and it is enabled becasue I am able to use it with the azureml_py36_tensorflow kernel. However, I need the more recent version of tensorflow to use Hugging Face transformers library.

Output for print("GPU is", "available" if tf.config.experimental.list_physical_devices("GPU") else "NOT AVAILABLE"):

GPU is NOT AVAILABLE

However, output for from tensorflow.python.client import device_lib print(device_lib.list_local_devices()):

[name: "/device:CPU:0" device_type: "CPU" memory_limit: 268435456 locality { } incarnation: 943600624690846185 , name: "/device:XLA_CPU:0" device_type: "XLA_CPU" memory_limit: 17179869184 locality { } incarnation: 6190304407145073975 physical_device_desc: "device: XLA_CPU device" , name: "/device:XLA_GPU:0" device_type: "XLA_GPU" memory_limit: 17179869184 locality { } incarnation: 6446102091114248389 physical_device_desc: "device: XLA_GPU device" ]

Further, when I run !nvidia-smi, I get:

`Fri Dec 4 16:33:10 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.45.01 Driver Version: 455.45.01 CUDA Version: 11.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla K80 On | 00001DD8:00:00.0 Off | 0 |
| N/A 48C P8 25W / 149W | 3MiB / 11441MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+`

So I'm getting conflicting messages about GPU availability. That being said, when I train a model, it is definitely not training on a GPU.

Please help. I've been banging my head against the wall for hours on this.

Firmware bug on NC24 SKU

I'm seeing the following error when running the DSVM (Ubuntu 20.04.5 LTS (GNU/Linux 5.15.0-1019-azure x86_64)) on the NC24 SKU in Azure:

$ sudo  journalctl -p 3 -xb
-- Logs begin at Wed 2022-09-14 17:11:34 UTC, end at Tue 2022-11-15 12:20:04 UTC. --
Nov 12 00:34:13 localhost.localdomain kernel: [Firmware Bug]: CPU12: APIC id mismatch. Firmware: c APIC: 10
Nov 12 00:34:13 localhost.localdomain kernel: [Firmware Bug]: CPU13: APIC id mismatch. Firmware: d APIC: 11
Nov 12 00:34:13 localhost.localdomain kernel: [Firmware Bug]: CPU14: APIC id mismatch. Firmware: e APIC: 12
Nov 12 00:34:13 localhost.localdomain kernel: [Firmware Bug]: CPU15: APIC id mismatch. Firmware: f APIC: 13
Nov 12 00:34:13 localhost.localdomain kernel: [Firmware Bug]: CPU16: APIC id mismatch. Firmware: 10 APIC: 14
Nov 12 00:34:13 localhost.localdomain kernel: [Firmware Bug]: CPU17: APIC id mismatch. Firmware: 11 APIC: 15
Nov 12 00:34:13 localhost.localdomain kernel: [Firmware Bug]: CPU18: APIC id mismatch. Firmware: 12 APIC: 16
Nov 12 00:34:13 localhost.localdomain kernel: [Firmware Bug]: CPU19: APIC id mismatch. Firmware: 13 APIC: 17
Nov 12 00:34:13 localhost.localdomain kernel: [Firmware Bug]: CPU20: APIC id mismatch. Firmware: 14 APIC: 18
Nov 12 00:34:13 localhost.localdomain kernel: [Firmware Bug]: CPU21: APIC id mismatch. Firmware: 15 APIC: 19
Nov 12 00:34:13 localhost.localdomain kernel: [Firmware Bug]: CPU22: APIC id mismatch. Firmware: 16 APIC: 1a
Nov 12 00:34:13 localhost.localdomain kernel: [Firmware Bug]: CPU23: APIC id mismatch. Firmware: 17 APIC: 1b

How to extend the DSVM image

We would like to pre-install a conda env into this image by extending it. Where is the base image located? What are the options for customizing the baes image?