microsoft / oms-docker Goto Github PK

Docker image for OMS (Operations Management Suite) Linux agent.

License: Other

Shell 88.29% Dockerfile 11.71%

oms-docker's Introduction

Trying the container solution for Microsoft Operations Management Suite

The Microsoft Operations Management Suite (OMS) is a software-as-a-service offering from Microsoft that allows Enterprise IT to manage any hybrid cloud. It offers log analytics, automation, backup and recovery, and security and compliance. Sign up for a free account at http://mms.microsoft.com or read more about here: https://www.microsoft.com/en-us/server-cloud/operations-management-suite/overview.aspx

This container solution will generate a container which will runs OMS agent within. This is for Linux OS which has restriction in installing the Operations Management Suite Agent directly. However, it can also be used with other support Linux OS as well.

Supported Linux Operating Systems, Docker, and ACS Mesosphere DC/OS:

Docker 1.11 thru 1.13
Docker CE and EE v17.06+
An x64 version of Linux OS
- Ubuntu 14.04 LTS, 16.04 LTS
- CoreOS(stable)
- Amazon Linux 2016.09.0
- openSUSE 13.2
- openSUSE LEAP 42.2
- CentOS 7.2, 7.3
- SLES 12
- RHEL 7.2, 7.3
ACS Mesosphere DC/OS 1.7.3, 1.8.8, 1.9
ACS Kubernetes 1.4.5, 1.6+
ACS Docker Swarm
Redhat OpenShift (OCP) 3.4, 3.5

Support for Windows Operation Systems

This site is only for Linux.

For any information about Windows Operating System, please go here.

Release Note

Update Information are here.

Setting up

As a pre-requisite, docker must be running prior to this installation. If you have installed before running docker, please re-install OMS Agent. For more information about docker, please go to https://www.docker.com/.

This set up is not for ACS Mesosphere DC/OS or ACS Kubernetes.

For more information on Mesosphere DC/OS, please see here.
For Kubernetes, please see here. yaml file for the daemon-set is here.
For Docker Swarm, please see here.
For Redhat OpenShift, please see here. This set up provides a containerized Container Solution Agent (OMS Agent for Linux). If you are interested in a full OMS Agent for linux with Container Solution, please go here.

To use OMS for all containers on a container host

Start the OMS container:

$>sudo docker run --privileged -d -v /var/run/docker.sock:/var/run/docker.sock -v /var/log:/var/log -v /var/lib/docker/containers:/var/lib/docker/containers -e WSID="your workspace id" -e KEY="your key" -p 127.0.0.1:25225:25225 -p 127.0.0.1:25224:25224/udp --name="omsagent" -h=`hostname` --restart=always mcr.microsoft.com/azuremonitor/containerinsights/ciprod:microsoft-oms-latest

To use OMS for all containers on a container host for FairFax OMS Workspace

Start the OMS container on FairFax OMS workspace:

$>sudo docker run --privileged -d -v /var/run/docker.sock:/var/run/docker.sock -v /var/log:/var/log -v /var/lib/docker/containers:/var/lib/docker/containers -e WSID="your workspace id" -e KEY="your key" -e DOMAIN="opinsights.azure.us" -p 127.0.0.1:25225:25225 -p 127.0.0.1:25224:25224/udp --name="omsagent" -h=`hostname` --restart=always mcr.microsoft.com/azuremonitor/containerinsights/ciprod:microsoft-oms-latest

If you are switching from the installed agent to the container

If you previously used the directly installed agent and want to switch to using the container, you must remove the omsagent. See Steps to install the OMS Agent for Linux

If you have an older version of Docker and would want to still use OMS Container Solution to monitor your data go here.

Upgrade

You can upgrade to a newer version of the agent. See here.

What now?

Once you're set up, we'd like you to try the following scenarios and play around with the system.

More Container Management Scenarios

Let us know!!!

What works? What is missing? What else do you need for this to be useful for you? Let us know at [email protected].

Code of Conduct

This project has adopted the [Microsoft Open Source Code of Conduct] (https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ] (https://opensource.microsoft.com/codeofconduct/faq/) or contact [email protected] with any additional questions or comments.

oms-docker's People

Contributors

Stargazers

Watchers

Forkers

henryrawas liuxiangniuniu omsexpert lastcoolnameleft sasukeh spencerx jeffmcaffer mallikharjunrao anushkaee squawell mahmoudhatem 0x7c13 vikaskulwal gardlt fredrkl jobyms urwithrajesh hatemalsum rashmichandrashekar vladkashuba cnadolny sunilake vyta althafuddin jacobliscom-rackspace susunola rewt ganga1980 bmcder kant r-dilip arthuersundar dalavancloud tobiaskohlbau lukenny kingsleyck omerlh alexeldeib ryanmaclean rafalkasa vijayraavi pujithaz sewci0 mzeeswart timja gracewehner thebitstreamer blackbaud-brandonstirnaman delqn charlesakalugwu lucaslccs jetiba-ms eshack japollac vidyakukke abyatsyshyn hbiarge purushotham526 maurieezzz prasunjoshi bansal12 monubamb eldarsagirov bluescion umarmohamedusman quentinbisson mickstevenson monojit18 supreethbks26 clouda2z-code saaror kjdeepthi rishabhsaha bhvnsang1 mykolapelyp-softserve erjosito steinom-no shinoyk skynetrulz msftkenneth cjf-fuller internetgdl z0n3d100 bharathnimmala-msft anantakjena vad1mo fchavatte ondrejfiala djsly austonli bhaskers-blu-org2 gungorugur daweim0 robertoborges atcastel joergjo taffywrinkle claudiusgonzo xiaojing123456 wiolaswies

oms-docker's Issues

How to setup HTTP proxy server setting for OMS docker container

I'm using kubernetes cluster and deploying omsagent as daemonset. How do I configure HTTP proxy server setting for oms container to use ?

https://github.com/Microsoft/OMS-Agent-for-Linux/blob/master/docs/OMS-Agent-for-Linux.md#configuring-the-agent-for-use-with-an-http-proxy-server

script doesnt work

customer mentioned script doesn't work, feel free to contact me over teams

Custom Log support for Docker OMS Agent

Any plan to make custom log support for Docker OMS Agent? I have deployed OMS Agent daemon set into my ACS Kubenetes cluster. I would like to get some log output from containers.

Thanks

OC Commands is not updated

odam command are no longer exists

Docker container logs stop reporting with OMS agent

I have two VM's and 4 docker containers being monitored by OMS using the OMS agent for Linux.

When I start a container, it sends containerlog data to OMS for about a day or so and then stops sending it, or at least nothing is being collected in the OMS portal

The containers are still running fine and the VM hosts continue to report their stats to OMS.

Restarting the containers restarts the container log collection, but eventually it stops.

How can I troubleshoot/fix this issue?

fluent-gem not working on latest release 1.6.0-163

On latest version 1.6.0-163 it's not possible to install additional fluent addons with fluent-gem. Looks like an issue within https://github.com/Microsoft/OMS-Agent-for-Linux itself as the latest version states in it's release information:

Fix for broken gem and fluent-gem commands in previous release

I've created a change to incorporate 1.6.1-3 in this repository, PR is incoming.

Version number?

What is the best way to discover what version of the OMS agent you're running? Currently there is no --version flag on the agent cli so you have to do a lot of work just to confirm you're using the latest version. Any plans on adding that?

oms agent inside dokcer troubleshooting

Hello DEVs,

I would suggest to use Docker Volumes https://docs.docker.com/storage/volumes/ or other mechanism to store OMS Agent logs /var/opt/microsoft/omsagent/log /var/opt/microsoft/omsconfig/log and for omi service to HOST filesystem so we can have them for TS issues

How to get Kubernetes (ACS-Engine) logs for pods to Log Analytics

Hi, I have created an Kubernetes cluster using the ACS-Engine and created the OMSAgent daemonset using this Docker image. Is there an way to get newly created pods to send logs to Log Analytics? I see ContainerLogs for certain K8s services, but not any newly created pods. Not sure what the requirement is (Labels? Namespace?) for OMSAgent to pickup STDOUT/STDERR and push it to Log Analytics via OMSAgent..

Application Logs not getting forwarded to OMS

Here's the steps I ran:

NOTE: I'm running on Minikube, which should not matter, but noting just in case.

Create Minikube cluster
kubectl create -f oms-daemonset.yaml
kubectl run nginx --image nginx
minkube service list
- Get IP:Port
Curl IP:port from minikube service list
kubectl logs po/nginx
- Validate logs are running being generated
Search "ContainerLog", but access logs are not found.
- Check 1 hour later. Still no logs

Support for Application level metrics

Hi,
Is there any support for application level metrics or custom metrics using Container Monitoring solution. I have checked this "HTTP Data Collector API" but i'm confused if it's the right API for custom metrics or not ?
Like in Prometheus, you can send custom application level metrics using client libraries provided by Prometheus.

Segfaults of omiagent when running on Kubernetes/CoreOS

Originally filed at microsoft/OMS-Agent-for-Linux#585 by @edevil

           PID: 33069 (omiagent)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Mon 2017-10-02 15:20:23 UTC (19h ago)
  Command Line: /opt/omi/bin/omiagent 9 12 --destdir / --providerdir /opt/omi/lib --loglevel WARNING
    Executable: /opt/omi/bin/omiagent
 Control Group: /kubepods/besteffort/podca6c5d89-a503-11e7-8e02-000d3a2709aa/27e2a4b7b2cb2a3a32c6cdb928d1be5ed5d74e2ff727d43fd4125c6ed8bf694a
         Slice: -.slice
       Boot ID: 32449a510934451eb8a0cc253dceabe8
    Machine ID: 9a13f6cf2ea8409dbc456655fa12ca04
      Hostname: node-1-vm
       Storage: /var/lib/systemd/coredump/core.omiagent.0.32449a510934451eb8a0cc253dceabe8.33069.1506957623000000.lz4
       Message: Process 33069 (omiagent) of user 0 dumped core.

           PID: 37755 (omiagent)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Mon 2017-10-02 15:21:36 UTC (19h ago)
  Command Line: /opt/omi/bin/omiagent 9 11 --destdir / --providerdir /opt/omi/lib --loglevel WARNING
    Executable: /opt/omi/bin/omiagent
 Control Group: /kubepods/besteffort/podca6c5d89-a503-11e7-8e02-000d3a2709aa/27e2a4b7b2cb2a3a32c6cdb928d1be5ed5d74e2ff727d43fd4125c6ed8bf694a
         Slice: -.slice
       Boot ID: 32449a510934451eb8a0cc253dceabe8
    Machine ID: 9a13f6cf2ea8409dbc456655fa12ca04
      Hostname: node-1-vm
       Storage: /var/lib/systemd/coredump/core.omiagent.0.32449a510934451eb8a0cc253dceabe8.37755.1506957696000000.lz4
       Message: Process 37755 (omiagent) of user 0 dumped core.

Container version: "microsoft/oms:1.4.1-45"
OS: Container Linux by CoreOS stable (1465.8.0)

I can e-mail the core files but I'm not inclined to share them publicly.

@edevil Please email the core files to [email protected]

reaching free tier limit

after reaching free tier limit, Azure Monitor For Container (Insights) stop displaying metrics without clear reason why.

Only while visiting the log analytics namespace users might realize they were being throttled.

Update labels "agentVersion:" and "dockerProviderVersion:" in Openshift

OMS-docker/OpenShift/ocp-omsagent.yaml hasn't been updated in while, using old agentVersion & dockerProviderVersion

template:
metadata:
labels:
name: omsagent
agentVersion: 1.4.0-45
dockerProviderVersion: 10.0.0-25

Daemonset on Master Node

I was able to create a OMSagent as a Daemonset and they are working fine on worker nodes. But on master nodes i see below log in /var/log/waagent.log. But I see agent is running with workspace inside the pod.
2018/06/12 17:14:32.455868 INFO [HTTP Delay] Delay 15 seconds for Status Code 503
2018/06/12 17:14:32.477553 INFO [HTTP Retry] Attempt 2 of 3: [HTTP Retry] HTTP GET Status Code 503
2018/06/12 17:29:30.591233 INFO Event: name=WALinuxAgent, op=HeartBeat, message=, duration=0
2018/06/12 17:37:48.650153 INFO [HTTP Delay] Delay 15 seconds for Status Code 503
2018/06/12 17:37:48.670617 INFO [HTTP Retry] Attempt 2 of 3: [HTTP Retry] HTTP GET Status Code 503
2018/06/12 17:59:31.930307 INFO Event: name=WALinuxAgent, op=HeartBeat, message=, duration=0
2018/06/12 18:29:33.368940 INFO Event: name=WALinuxAgent, op=HeartBeat, message=, duration=0
2018/06/12 18:32:27.264432 INFO [HTTP Delay] Delay 15 seconds for Status Code 503
2018/06/12 18:32:27.268107 INFO [HTTP Retry] Attempt 2 of 3: [HTTP Retry] HTTP GET Status Code 503
2018/06/12 18:59:35.889614 INFO Event: name=WALinuxAgent, op=HeartBeat, message=, duration=0
Any help is appreciated. Thanks

Release new image for 1.3.1-15a

I've noticed that there are already placeholders for the 1.3.1-15a release, but the Dockerfile still references an older version.

OMS Docker doesn't onboard itself.

Hi,
After running the OMS-Docker on a VM, and navigating to azure portal to see the logs give or take around 1 hr. I see that I need to onboard the VM and then I am able to see the logs after sometime.
Doesn't it onboard itself automatically ?
Also I dont see syslog, is this expected ?

it did onboard and running fine as a Docker container on a VM.

Let me know if additional information is needed.

Issues with Docker Redeploys

Re-raised from microsoft/OMS-Agent-for-Linux#579

Really hoping someone can point me in the right direction here, every time we redeploy our docker instances our SCX logs at (/var/opt/microsoft/scx/log/scx.log) beging to fill very rapidly with the following messages:

2017-09-29T15:02:39,523Z Error [scx.core.common.pal.system.disk.statisticallogicaldiskinstance:############] statvfs() failed for /var/lib/docker/overlay/######################/merged; errno = 2

Systemctl restart omsagent##### seems to take care of this, but we were expecting the agent to be aware when a container went away and to stop trying to stat the directory it used to be mounted to.

So far we have tried removing and reinstalling the OMS bundle. But curious if there is something else we are doing wrong here?

@kevi5702

Issue on CentOS 7.4.1708 - omsagent service failing to start due to filter_docker_log?

So, OMSAgent seemed to fill up my /var with core dumps since I installed docker a while back.
It seems the prior version of omsagent instead of failing to start, was simply segfaulting every few minutes (looks remarkably like microsoft/OMS-Agent-for-Linux#315 ).

I purged a bunch of the core files to free up some space, and then I uninstalled OMSagent, and reinstalled. That at least stopped the core dumps.

I can see OMSAgent showing "connected in OMS/log analytics portal now allegedly, but when I look at my VM, it claims the service is failing.:

[root@testVM1 log]# tail -12 /var/opt/microsoft/omsagent/log/omsagent.log
2017-12-02 16:29:35 +0000 [info]: reading config file path="/etc/opt/microsoft/omsagent/3f0519f3-d261-4e86-bad4-0e341eed4ac6/conf/omsagent.conf"
2017-12-02 16:29:35 +0000 [info]: starting fluentd-0.12.24 without supervision
2017-12-02 16:29:35 +0000 [info]: gem 'fluentd' version '0.12.24'
2017-12-02 16:29:35 +0000 [info]: adding filter pattern="oms.changetracking.package" type="filter_changetracking"
2017-12-02 16:29:35 +0000 [info]: adding filter pattern="docker." type="filter_docker_log"
2017-12-02 16:29:35 +0000 [error]: config error file="/etc/opt/microsoft/omsagent/3f0519f3-d261-4e86-bad4-0e341eed4ac6/conf/omsagent.conf" error="Unknown filter plugin 'filter_docker_log'. Run 'gem search -rd fluent-plugin' to find plugins"
2017-12-02 16:34:49 +0000 [info]: reading config file path="/etc/opt/microsoft/omsagent/3f0519f3-d261-4e86-bad4-0e341eed4ac6/conf/omsagent.conf"
2017-12-02 16:34:49 +0000 [info]: starting fluentd-0.12.24 without supervision
2017-12-02 16:34:50 +0000 [info]: gem 'fluentd' version '0.12.24'
2017-12-02 16:34:50 +0000 [info]: adding filter pattern="oms.changetracking.package" type="filter_changetracking"
2017-12-02 16:34:50 +0000 [info]: adding filter pattern="docker." type="filter_docker_log"
2017-12-02 16:34:50 +0000 [error]: config error file="/etc/opt/microsoft/omsagent/3f0519f3-d261-4e86-bad4-0e341eed4ac6/conf/omsagent.conf" error="Unknown filter plugin 'filter_docker_log'. Run 'gem search -rd fluent-plugin' to find plugins"

And output from systemctl:

[root@testVM1 log]# systemctl status omsagent-3f0519f3-d261-4e86-bad4-0e341eed4ac6.service -l
● omsagent-3f0519f3-d261-4e86-bad4-0e341eed4ac6.service - Operations Management Suite agent
Loaded: loaded (/usr/lib/systemd/system/omsagent-3f0519f3-d261-4e86-bad4-0e341eed4ac6.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sat 2017-12-02 16:34:50 UTC; 8min ago
Process: 30436 ExecStop=/bin/rm -f /var/opt/microsoft/omsagent/3f0519f3-d261-4e86-bad4-0e341eed4ac6/run/omsagent.pid (code=exited, status=0/SUCCESS)
Process: 30427 ExecStart=/opt/microsoft/omsagent/bin/omsagent -d /var/opt/microsoft/omsagent/3f0519f3-d261-4e86-bad4-0e341eed4ac6/run/omsagent.pid -o /var/opt/microsoft/omsagent/3f0519f3-d261-4e86-bad4-0e341eed4ac6/log/omsagent.log -c /etc/opt/microsoft/omsagent/3f0519f3-d261-4e86-bad4-0e341eed4ac6/conf/omsagent.conf --no-supervisor (code=exited, status=1/FAILURE)
Main PID: 30427 (code=exited, status=1/FAILURE)

Dec 02 16:34:49 testVM1 systemd[1]: Started Operations Management Suite agent.
Dec 02 16:34:49 testVM1 systemd[1]: Starting Operations Management Suite agent...
Dec 02 16:34:50 testVM1 omsagent[30427]: WARN The ChangeTracking inventory xml file does not exists
Dec 02 16:34:50 testVM1 omsagent[30427]: {}
Dec 02 16:34:50 testVM1 omsagent[30427]: 2017-12-02 16:34:50 +0000 [error]: fluent/supervisor.rb:367:rescue in main_process: config error file="/etc/opt/microsoft/omsagent/3f0519f3-d261-4e86-bad4-0e341eed4ac6/conf/omsagent.conf" error="Unknown filter plugin 'filter_docker_log'. Run 'gem search -rd fluent-plugin' to find plugins"
Dec 02 16:34:50 testVM1 systemd[1]: omsagent-3f0519f3-d261-4e86-bad4-0e341eed4ac6.service: main process exited, code=exited, status=1/FAILURE
Dec 02 16:34:50 testVM1 systemd[1]: Unit omsagent-3f0519f3-d261-4e86-bad4-0e341eed4ac6.service entered failed state.
Dec 02 16:34:50 testVM1 systemd[1]: omsagent-3f0519f3-d261-4e86-bad4-0e341eed4ac6.service failed.

Any ideas?
Relevant details maybe:

[root@testVM1 log]# rpm -qa | grep oms
auoms-1.0.0-2.x86_64
omsconfig-1.1.1-560.x86_64
omsagent-1.4.2-124.x86_64

[root@testVM1 log]# uname -a
Linux testVM1 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[root@testVM1 log]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

OMS Agent does not detect Kubernetes orchestrator

I run the OMS Agent in all vms (masters and nodes) but the agent log shows that it doesn't detect that it is running inside Kubernetes:

instance of Container_HostInventory
{
    [Key] InstanceID=52b9541c-8123-481e-8b70-d31f2dc258af
    Computer=node-3-vm
    DockerVersion=1.12.6
    OperatingSystem=Container Linux by CoreOS 1465.8.0 (Ladybug)
    Volume=local
    Network=null host bridge overlay
    NodeRole=Not Orchestrated
    OrchestratorType=None
}
Primary Workspace: XXX    Status: Onboarded(OMSAgent Running)
omsagent 1.4.1.45

Is there some configuration variable for this?

OMS portal shows errors for doke

Hi I am trying to connect my swarm cluster with OMS, I followed the instruction and ran the swarm service (global) - but I receive error messages on OMS portal like following:

Failed to resolve entity 'DockerVersion_s'.
Failed to resolve entity 'Name_s'.

What's going wrong here?

Is it possible to exclude namespaces from the osmagent using wildcards?

Would it be possible to use some sort of wildcards in the configmap when excluding namspaces from the loggin? E.g. would be great if we can exclude namspaces with a certain prefix.

https://github.com/microsoft/OMS-docker/blob/ci_feature_prod/Kubernetes/container-azm-ms-agentconfig.yaml

OMS agent dies within docker container, change main.sh

I have been running OMS agent inside of docker as described in the docs. I soon realised that the OMS would die very frequently, even when upgraded to the latest version. The logs would say nothing. This is OMS agent running on ubuntu 16.04 on an Azure VM started via systemd. Right now the Dockerfile runs a sleep inf && wait:

https://github.com/Microsoft/OMS-docker/blob/master/1.4.4-210/main.sh#L80

My systemd job looks as follows:

[Unit]
Description=OMS Agent for Linux Docker
Requires=docker.service
After=docker.service

[Service]
Restart=always
RestartSec=10
RestartPreventExitStatus=5
TimeoutStartSec=20
TimeoutStopSec=15
SyslogIdentifier=oms-agent
ExecStop=/usr/bin/docker rm -f omsagent
ExecStartPre=-/usr/bin/docker kill omsagent
ExecStartPre=-/usr/bin/docker rm -f omsagent
ExecStartPre=/usr/bin/docker pull microsoft/oms:1.4.4-210
ExecStart=/usr/bin/docker run --privileged \
    -v /etc/omsagent/main.sh:/opt/main.sh:ro \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /var/log:/var/log \
    -e WSID="XXX" \
    -e KEY="XXX" \
    -e SERVICE_25225_CHECK_TCP=true \
    -e SERVICE_25225_CHECK_INTERVAL=15s \
    -e SERVICE_25225_CHECK_TIMEOUT=3s \
    -h="XXX" \
    --net=host \
    --restart=always \
    --publish 10.x.x.x:25225:25225 \
    --publish 10.x.x.x:25224:25224/udp \
    --name="omsagent" \
    microsoft/oms:1.4.4-210

[Install]
WantedBy=multi-user.target

Right now if the OMS service stops responding, the docker container will keep running. I am suggesting that instead of sleep infinity we change the main.sh to something like this:

ERROR_CODE=1
until [ $ERROR_CODE == '0' ] ; do /opt/microsoft/omsagent/bin/service_control is-running; ERROR_CODE=$?; echo "OMSAgent Running"; sleep 15; done

Notice that the exit codes are not standard unix. 0 is false and 1 is success.

Custom logs support

Hi,

I am using containerized OMS agent running on a Docker for Azure Swarm cluster. I tried to configure Custom log forwarding using this documentation: https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-data-sources-custom-logs
Unfortunately, it is not working. Should this be supported by containerized OMS agent, or only by "native" OMS agent or by Azure VM OMS extension?

Thank you.

MCR repository?

Are you publishing an image to MCR yet? I am working on moving images into MCR that are needed to support AKS and OMS is one of those images.

no containerlogs found after launching kubernetes daemonset

I applied the oms yaml file as described in the readme. I have a connection for each of the nodes. I can also query for the node information such as amount of memory available.

However I cannot get any of the container specific logs. I'd especially like to be able to see the stdout logs of the individual containers. But if I do a query like Type=ContainerLog I get 0 results.

Do I need to configure something else? Is there anyplace I can see what might be going wrong?

The pod logs look like this:

2017-05-02T11:53:39.659989573Z + sed -i -e 's/bind 127.0.0.1/bind 0.0.0.0/g' /etc/opt/microsoft/omsagent/sysconf/omsagent.d/container.conf
2017-05-02T11:53:39.661263985Z + sed -i -e 's/^exit 101$/exit 0/g' /usr/sbin/policy-rc.d
2017-05-02T11:53:39.662258880Z + sed -i.bak 's/record\["Host"\] = hostname/record\["Host"\] = OMS::Common.get_hostname/' /opt/microsoft/omsagent/plugin/filter_syslog.rb
2017-05-02T11:53:39.724063342Z + /opt/omi/bin/service_control start
2017-05-02T11:53:39.730663074Z invoke-rc.d: could not determine current runlevel
2017-05-02T11:53:39.738063698Z  * Starting Microsoft OMI Server: 
2017-05-02T11:53:39.762201512Z    ...done.
2017-05-02T11:53:39.762694714Z + '[' -z ']'
2017-05-02T11:53:39.762705603Z + /opt/microsoft/omsagent/bin/omsadmin.sh -w <redacted> -s <redacted>
2017-05-02T11:53:40.067206382Z info	Generating certificate ...
2017-05-02T11:53:40.772758980Z -e info	Onboarding success
2017-05-02T11:53:40.813631111Z rm: cannot remove '/var/opt/microsoft/omsagent/state/containerhostname': Device or resource busy
2017-05-02T11:53:40.826291250Z Configuring rsyslog for OMS logging
2017-05-02T11:53:40.827954045Z Restarting service: rsyslog
2017-05-02T11:53:40.835891777Z invoke-rc.d: could not determine current runlevel
2017-05-02T11:53:40.844586396Z  * Stopping enhanced syslogd rsyslogd
2017-05-02T11:53:40.845137432Z    ...done.
2017-05-02T11:53:40.848593366Z  * Starting enhanced syslogd rsyslogd
2017-05-02T11:53:40.859470654Z    ...done.
2017-05-02T11:53:40.869103142Z Configuring OMS agent service ...
2017-05-02T11:53:40.896602693Z invoke-rc.d: could not determine current runlevel
2017-05-02T11:53:40.905504363Z  * Starting Operations Management Suite agent (ff8a4d0d-951e-47c6-a1c6-f22ce09ed910): 
2017-05-02T11:53:41.324327424Z    ...done.
2017-05-02T11:53:41.853948411Z -e info	Configured omsconfig
2017-05-02T11:53:41.856226666Z + /opt/microsoft/omsagent/bin/service_control start
2017-05-02T11:53:41.865615177Z + trap shutdown SIGTERM
2017-05-02T11:53:41.865628762Z + wait
2017-05-02T11:53:41.865632258Z + sleep inf

Support for K8s Windows Containers

Please add support for Windows Containers to monitor ACS K8s

Bad Argument Error when deploying OMSAgent

When deploying OMS as described here:
https://docs.microsoft.com/en-us/azure/aks/tutorial-kubernetes-monitor

I get the following error:
{"error":{"message":"The request had some invalid properties","code":"BadArgumentError","innererror":{"code":"SyntaxError","message":"Syntax Error","information":{"details":"{"error":{"code":"Bad request","message":"Request is invalid and cannot be executed.","innererror":{"code":"SEM0001","message":"Failed to resolve entity 'KubePodInventory_CL'","@context":{"activityId":"3949f942-5757-4176-89d5-1252ebbb0626"},"@errorcode":"SEM0001","@errorMessage":"Failed to resolve entity 'KubePodInventory_CL'"}}}"}}}}

Updating the omsagent to agentVersion: 1.4.1-45 did not help.

Instructions for docker swarm

I used the command on a swarm manager create with Docker for Azure template:

swarm-exec  docker run --privileged -d -v /var/run/docker.sock:/var/run/docker.sock -e WSID="MY_ID" -e KEY="MY_KEY" -h=`hostname` -p 127.0.0.1:25225:25225 --restart=always microsoft/oms

Seems to work fine (all nodes show up on OMS portal). If it is correct could you please add this the the docs?

Add Volume to run commands to retain token/configuration between containers

I am using the docker image to run a containerized oms agent.

When the image is updated, I have to re-run the container with the new image.

Apparently, the agent seems to redo the whole onboarding process and OMS seems to see two agents instead of one.

It would be interesting that all the initialisation data after the first start be stored in a volume so that it can be reused between containers.

Logs not making it up to Container Monitoring Solution

Hello,

I'm having issues getting any of my logs to reach the Container Logs section of Container Monitoring Solution. Im guessing this is because the agent is sending the logs using the ContainerLogs data type? I can see some logs coming thru on syslog but all of the docker information is lost.

I'm running the OpenLogic CentOS 7.3 image, and have selinux disabled. I am not doing any fancy orchestration, simply launching containers using the run command. I do see logs from my containers coming on standard out. Agent 1.4.0 is running locally alongside docker 1.12.6.

Any help would be greatly appreciated!

Cheers,
Nick

Possible typecheck failures when parsing agent prometheus configmap

There is an issue with how the log analytics agent image ciprod:ciprod07092019 parses the prometheus data collection configmap container-azm-ms-agentconfig.

Fields which are defined as array of strings cause the parsing to fail if specified as an empty array e.g. fieldpass = [].

****************Start Prometheus Config Processing********************
config::configmap container-azm-ms-agentconfig for settings mounted, parsing values for prometheus config map
config::Successfully parsed mounted prometheus config map
config::Typecheck failed for prometheus config settings for replicaset, using defaults
****************End Prometheus Config Processing********************

The workaround is to comment out those fields - exclude_namespaces, fieldpass, fielddrop, urls and kubernetes_services - if there are no concrete values to specify for them.

/kind bug
/cc @rashmichandrashekar

Please provide examples for Windows Containers

Need more examples about deploying and utilizing containers with OMS on Windows.
I have Windows swarm running with Windows containers. It's 100% Microsoft+docker environment. Not even scent of Linux. And here we are asking Microsoft to provide documentation how to support their oldest baby.

Docker container logs not being collected

1st case. Standalone Ubuntu server

Docker installed via apt.
Few containers are running.
OMS docker container running via command

docker run --privileged -d -v /var/run/docker.sock:/var/run/docker.sock -v /var/log:/var/log e WSID="..." -e KEY="..." -p 127.0.0.1:25225:25225 -p 127.0.0.1:25224:25224/udp --name="omsagent" -h=test --restart=always microsoft/oms

just as described here https://github.com/Microsoft/OMS-docker

Result: no container logs are collected, when looking inside omsagent container's /var/opt/microsoft/omsagent/e54f2b16-0fc5-4b80-a635-260e70a8dd31/log/omsagent.log

a few warnings found

2018-01-25 09:10:21 +0000 [warn]: /var/lib/docker/containers/b8d84dcc623538481e0450fe72919b9df3283a801f2eac3e3e0360a0e747295a/b8d84dcc623538481e0450fe72919b9df3283a801f2eac3e3e0360a0e747295a-json.log is not readable. Cannot tail the file.
2018-01-25 09:10:21 +0000 [warn]: /var/lib/docker/containers/b1e55eec38746cb8d5c93bbff24c62552ffc0630c6cccc20ad4d6d3c84760fad/b1e55eec38746cb8d5c93bbff24c62552ffc0630c6cccc20ad4d6d3c84760fad-json.log is not readable. Cannot tail the file.
2018-01-25 09:10:21 +0000 [warn]: /var/lib/docker/containers/22411faf09a0933cc69a7a4567d01b3a6b9795b3b4d1c3b7e15b3a5ed0f77a00/22411faf09a0933cc69a7a4567d01b3a6b9795b3b4d1c3b7e15b3a5ed0f77a00-json.log is not readable. Cannot tail the file.
2018-01-25 09:10:21 +0000 [warn]: /var/lib/docker/containers/16ef7c7345513decddb7b42d965c56b68b7e7055f4c8d0e91aea7c57a9b2d95f/16ef7c7345513decddb7b42d965c56b68b7e7055f4c8d0e91aea7c57a9b2d95f-json.log is not readable. Cannot tail the file.
2018-01-25 09:10:21 +0000 [warn]: /var/lib/docker/containers/0632dceb120ff7053d3a7a3df59892a0016c8ce4aab01f69991fd8aad4b604c5/0632dceb120ff7053d3a7a3df59892a0016c8ce4aab01f69991fd8aad4b604c5-json.log is not readable. Cannot tail the file.
2018-01-25 09:10:21 +0000 [warn]: /var/lib/docker/containers/16185e569412c438bd4fa5c23089d9127dd85954799b87acddd83c8b1af7f13d/16185e569412c438bd4fa5c23089d9127dd85954799b87acddd83c8b1af7f13d-json.log is not readable. Cannot tail the file.

Adding this to the command:

-v /var/lib/docker/containers:/var/lib/docker/containers

solves the issue.

2nd case. Docker swarm.

Running Docker Swarm on Azure (https://docs.docker.com/docker-for-azure/)
OMS service started globally inside swarm as described here: https://github.com/Microsoft/OMS-docker/tree/master/Swarmmode

docker service create --name omsagent --mode global --mount type=bind,source=/var/run/docker.sock,destination=/var/run/docker.sock -e WSID="<WORKSPACE ID>" -e KEY="<PRIMARY KEY>" -p 25225:25225 -p 25224:25224/udp --restart-condition=on-failure microsoft/oms

Same result as above: no container logs are collected, when looking inside oms container same warnings are present. But the workaround above cannot be applied for Swarm.

Expected: container logs are collected in both cases.
`

Errors in omsagent-ds-secrets.yaml

Hi,

Having trouble with deploying latest OMS agent to Kubernetes with omsagent-ds-secrets.yaml

Git commit: cd9103e (HEAD -> master, origin/master, origin/HEAD)

First error:
C:\repo\oms-docker\Kubernetes>kubectl apply -f omsagent-ds-secrets.yaml
error: error converting YAML to JSON: yaml: line 31: found a tab character that violate indentation

Fixed that then ran the command again:

C:\repo\oms-docker\Kubernetes>kubectl apply -f omsagent-ds-secrets.yaml
The DaemonSet "omsagent" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"agentVersion":"1.4.3-174", "app":"omsagent", "dockerProviderVersion":"1.0.0-30"}: selector does not match template labels

Any chance this could be fixed?

Thanks.
Andrew.

ContainerLog not collected in OMS (swarm mode)

I will provide some more details related to the issue I'm facing.
See also the same issue opened in a different project due to error message "Two successive configuration applications from OMS Settings failed – please report issue to github.com/Microsoft/PowerShell-DSC-for-Linux/issues" microsoft/PowerShell-DSC-for-Linux#415

Issue: I'm not able to collect "ContainerLog" data even if my container (Swarm service) produces log rows.

Environment:

Ubuntu Linux as below:
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial

Docker CE as below:
docker --version
Docker version 17.12.1-ce, build 7390fc6

Configuration: 1 node swarm cluster as below:
docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
l3yzhsv1e32rbaa7gpcn4wczf * ubuntu001 Ready Active Leader

Hardware/virtualization platform: Microsoft Azure

Running services as below:
docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
pb5k9j7bbzvq local_reader replicated 1/1 tiian/reader:1.3
l7v9r5c9ekm6 local_writer replicated 1/1 tiian/writer:1.3
gvjxzasi2w9b omsagent global 1/1 microsoft/oms:latest :25225->25225/tcp,:25224->25224/udp

Running containers:
docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b7a32f0f0190 tiian/writer:1.3 "/bin/sh -c '/bin/sh…" 39 minutes ago Up 39 minutes local_writer.1.lsl1ob0lm1vilg8rf6iwujdn9
d0d45a2c53ff tiian/reader:1.3 "/bin/sh -c '/bin/sh…" 39 minutes ago Up 38 minutes local_reader.1.6p93rmj18zyohobfsfor9g1zx
55fe8a0ac945 microsoft/oms:latest "/opt/main.sh" 39 minutes ago Up 38 minutes omsagent.l3yzhsv1e32rbaa7gpcn4wczf.vd08221gz40l67sq2ms67y049

Logs produced by the "local_reader" service:
[...]
local_reader.1.6p93rmj18zyo@ubuntu001 | Thu Jul 5 10:08:52 UTC 2018 number of lines in file are 4109 /v4/date.txt
local_reader.1.6p93rmj18zyo@ubuntu001 | Thu Jul 5 10:08:53 UTC 2018 number of lines in file are 4109 /v4/date.txt
local_reader.1.6p93rmj18zyo@ubuntu001 | Thu Jul 5 10:08:54 UTC 2018 number of lines in file are 4110 /v4/date.txt
local_reader.1.6p93rmj18zyo@ubuntu001 | Thu Jul 5 10:08:55 UTC 2018 number of lines in file are 4110 /v4/date.txt
local_reader.1.6p93rmj18zyo@ubuntu001 | Thu Jul 5 10:08:56 UTC 2018 number of lines in file are 4111 /v4/date.txt
local_reader.1.6p93rmj18zyo@ubuntu001 | Thu Jul 5 10:08:57 UTC 2018 number of lines in file are 4111 /v4/date.txt
local_reader.1.6p93rmj18zyo@ubuntu001 | Thu Jul 5 10:08:58 UTC 2018 number of lines in file are 4112 /v4/date.txt
local_reader.1.6p93rmj18zyo@ubuntu001 | Thu Jul 5 10:08:59 UTC 2018 number of lines in file are 4112 /v4/date.txt
local_reader.1.6p93rmj18zyo@ubuntu001 | Thu Jul 5 10:09:00 UTC 2018 number of lines in file are 4113 /v4/date.txt
local_reader.1.6p93rmj18zyo@ubuntu001 | Thu Jul 5 10:09:01 UTC 2018 number of lines in file are 4113 /v4/date.txt
local_reader.1.6p93rmj18zyo@ubuntu001 | Thu Jul 5 10:09:02 UTC 2018 number of lines in file are 4114 /v4/date.txt

Logs produced by the "local_writer" service:
[...]
local_writer.1.lsl1ob0lm1vi@ubuntu001 | Writing date Thu Jul 5 10:09:15 UTC 2018
local_writer.1.lsl1ob0lm1vi@ubuntu001 | Writing date Thu Jul 5 10:09:17 UTC 2018
local_writer.1.lsl1ob0lm1vi@ubuntu001 | Writing date Thu Jul 5 10:09:19 UTC 2018
local_writer.1.lsl1ob0lm1vi@ubuntu001 | Writing date Thu Jul 5 10:09:21 UTC 2018
local_writer.1.lsl1ob0lm1vi@ubuntu001 | Writing date Thu Jul 5 10:09:23 UTC 2018
local_writer.1.lsl1ob0lm1vi@ubuntu001 | Writing date Thu Jul 5 10:09:25 UTC 2018
local_writer.1.lsl1ob0lm1vi@ubuntu001 | Writing date Thu Jul 5 10:09:27 UTC 2018
local_writer.1.lsl1ob0lm1vi@ubuntu001 | Writing date Thu Jul 5 10:09:29 UTC 2018

OMS agent (swarm service configuration):
docker service inspect omsagent
[
{
"ID": "gvjxzasi2w9buuveqzbdo5ugc",
"Version": {
"Index": 18023
},
"CreatedAt": "2018-06-29T10:26:35.619912541Z",
"UpdatedAt": "2018-07-05T09:29:04.054650027Z",
"Spec": {
"Name": "omsagent",
"Labels": {},
"TaskTemplate": {
"ContainerSpec": {
"Image": "microsoft/oms:latest@sha256:6844e0798d7c943a4f95e545fffd05f43917ef72e31a53b3fcdfeb5b067b5202",
"Mounts": [
{
"Type": "bind",
"Source": "/var/run/docker.sock",
"Target": "/var/run/docker.sock"
}
],
"StopGracePeriod": 10000000000,
"DNSConfig": {},
"Secrets": [
{
"File": {
"Name": "WSID",
"UID": "0",
"GID": "0",
"Mode": 292
},
"SecretID": "yberlz1cp9hubeaos0rxbzxh7",
"SecretName": "WSID"
},
{
"File": {
"Name": "KEY",
"UID": "0",
"GID": "0",
"Mode": 292
},
"SecretID": "nh5seunun4ly2ib794terckdm",
"SecretName": "KEY"
}
],
"Isolation": "default"
},
"Resources": {
"Limits": {},
"Reservations": {}
},
"RestartPolicy": {
"Condition": "on-failure",
"Delay": 5000000000,
"MaxAttempts": 0
},
"Placement": {
"Platforms": [
{
"Architecture": "amd64",
"OS": "linux"
}
]
},
"ForceUpdate": 1,
"Runtime": "container"
},
"Mode": {
"Global": {}
},
"UpdateConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"RollbackConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"EndpointSpec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 25225,
"PublishedPort": 25225,
"PublishMode": "ingress"
},
{
"Protocol": "udp",
"TargetPort": 25224,
"PublishedPort": 25224,
"PublishMode": "ingress"
}
]
}
},
"PreviousSpec": {
"Name": "omsagent",
"Labels": {},
"TaskTemplate": {
"ContainerSpec": {
"Image": "microsoft/oms:latest@sha256:6844e0798d7c943a4f95e545fffd05f43917ef72e31a53b3fcdfeb5b067b5202",
"Mounts": [
{
"Type": "bind",
"Source": "/var/run/docker.sock",
"Target": "/var/run/docker.sock"
}
],
"DNSConfig": {},
"Secrets": [
{
"File": {
"Name": "WSID",
"UID": "0",
"GID": "0",
"Mode": 292
},
"SecretID": "yberlz1cp9hubeaos0rxbzxh7",
"SecretName": "WSID"
},
{
"File": {
"Name": "KEY",
"UID": "0",
"GID": "0",
"Mode": 292
},
"SecretID": "nh5seunun4ly2ib794terckdm",
"SecretName": "KEY"
}
],
"Isolation": "default"
},
"Resources": {
"Limits": {},
"Reservations": {}
},
"RestartPolicy": {
"Condition": "on-failure",
"Delay": 5000000000,
"MaxAttempts": 0
},
"Placement": {
"Platforms": [
{
"Architecture": "amd64",
"OS": "linux"
}
]
},
"ForceUpdate": 0,
"Runtime": "container"
},
"Mode": {
"Global": {}
},
"EndpointSpec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 25225,
"PublishedPort": 25225,
"PublishMode": "ingress"
},
{
"Protocol": "udp",
"TargetPort": 25224,
"PublishedPort": 25224,
"PublishMode": "ingress"
}
]
}
},
"Endpoint": {
"Spec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 25225,
"PublishedPort": 25225,
"PublishMode": "ingress"
},
{
"Protocol": "udp",
"TargetPort": 25224,
"PublishedPort": 25224,
"PublishMode": "ingress"
}
]
},
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 25225,
"PublishedPort": 25225,
"PublishMode": "ingress"
},
{
"Protocol": "udp",
"TargetPort": 25224,
"PublishedPort": 25224,
"PublishMode": "ingress"
}
],
"VirtualIPs": [
{
"NetworkID": "m0x6kvjcgc563uv81zqyz1260",
"Addr": "10.255.0.5/16"
}
]
},
"UpdateStatus": {
"State": "completed",
"StartedAt": "2018-07-04T12:19:34.301759207Z",
"CompletedAt": "2018-07-04T12:19:47.081110853Z",
"Message": "update completed"
}
}
]

Why "ContainerLog" are not collected?
How can I troubleshoot the issue and provide more info?

Kind Regards
Ch.F.

Running and Stopped container instances duplicated in Dashboard

I'm running OMS-Docker on Minikube and see incorrect counts for Running and Stopped containers:

NOTE: I'm using Minikube; however, I don't think that should matter.

Create a new MiniKube instance
Install Nginx
Install oms-docker.yml
Verify Container status is updated
kubectl delete po/omsagent (Some logs were not going to OMS, so I tried to restart to see if that fixed it)
Search for: "ContainerInventory | where ContainerState == "Running" | summarize AggregatedValue = count() by Computer, Image"
Should see two instances.

8/24/2017 7:16:55.443 PM | ContainerInventory
...TimeGenerated:8/24/2017 7:16:55.443 PM
...Computer:minikube
...Name:k8s_nginx_nginx-4217019353-fh659_default_3b3d418a-891c-11e7-85c8-0800277bef1c_0
...ContainerHostname:nginx-4217019353-fh659
...Image:nginx
...ImageTag:latest
...ContainerState:Running
...Ports:null
...ExitCode:0
...EnvironmentVar:["KUBERNETES_SERVICE_PORT=443", "KUBERNETES_SERVICE_PORT_HTTPS=443", "KUBERNETES_PORT=tcp://10.0.0.1:443", "KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443", "KUBERNETES_PORT_443_TCP_PROTO=tcp", "KUBERNETES_PORT_443_TCP_PORT=443", "KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1", "KUBERNETES_SERVICE_HOST=10.0.0.1", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "NGINX_VERSION=1.13.3-1~~stretch", "NJS_VERSION=1.13.3.0.1.11-1~~stretch"]
...Command:["nginx", "-g", "daemon off "]
...CreatedTime:8/24/2017 5:33:32.114 PM
...StartedTime:8/24/2017 5:33:32.200 PM
...FinishedTime:12/31/0000 6:00:00.000 PM
...Type:ContainerInventory
...TenantId:ada812eb-67e3-44e2-8da8-11c31fd741a1
...SourceSystem:Containers
...ContainerID:d14a9111af4ff912fa2befdd028ef65c6491e02849a39e3c60860829ac91423c
...ImageID:sha256:b8efb18f159bd948486f18bd8940b56fd2298b438229f5bd2bcf4cedcf037448
[-] show less
8/24/2017 8:09:55.007 PM | ContainerInventory
...TimeGenerated:8/24/2017 8:09:55.007 PM
...Computer:minikube
...Name:k8s_nginx_nginx-4217019353-fh659_default_3b3d418a-891c-11e7-85c8-0800277bef1c_0
...ContainerHostname:nginx-4217019353-fh659
...Image:nginx
...ImageTag:latest
...ContainerState:Running
...Ports:null
...ExitCode:0
...EnvironmentVar:["KUBERNETES_SERVICE_PORT=443", "KUBERNETES_SERVICE_PORT_HTTPS=443", "KUBERNETES_PORT=tcp://10.0.0.1:443", "KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443", "KUBERNETES_PORT_443_TCP_PROTO=tcp", "KUBERNETES_PORT_443_TCP_PORT=443", "KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1", "KUBERNETES_SERVICE_HOST=10.0.0.1", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "NGINX_VERSION=1.13.3-1~~stretch", "NJS_VERSION=1.13.3.0.1.11-1~~stretch"]
...Command:["nginx", "-g", "daemon off "]
...CreatedTime:8/24/2017 5:33:32.114 PM
...StartedTime:8/24/2017 5:33:32.200 PM
...FinishedTime:12/31/0000 6:00:00.000 PM
...Type:ContainerInventory
...TenantId:ada812eb-67e3-44e2-8da8-11c31fd741a1
...SourceSystem:Containers
...ContainerID:d14a9111af4ff912fa2befdd028ef65c6491e02849a39e3c60860829ac91423c
...ImageID:sha256:b8efb18f159bd948486f18bd8940b56fd2298b438229f5bd2bcf4cedcf037448
[-] show less

Configuring what to log in AKS

I followed the installation guide (link below) to get Log Analytics working in a AKS instance using the Azure Portal (the automated, GUI approach).

https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-onboard

Everything appears to be working. However, I'd like to configure what gets logged. Currently, all of our containers' stdout/stderr is being forwarded to the workspace. We have a few services that are very chatty, for no good reason, and can't be adjusted.

Is it possible to configure the agent to not log stdout but continue to log stderr for specific container instances? If not, is it possible to configure the agent to not log anything for specific container instances?

Azure Monitor Agent support for Openshift

Can Azure Monitor be used to collect logs from custom deployments of Openshift (i.e. not managed by Microsoft) on Azure? I’d like to run Azure monitor agents on Openshift nodes and have them to send container logs to Azure Monitor. Is this approach only limited to the Microsoft managed Openshift?

OMSagent as a daemonset on Kubernetes cluster

When we deploy a OMSagent as a daemonset it should turn green in the security center showing as VM is monitored by the OMSagent. And we are deploying OMSagent on a Kubernetes cluster with COREOS VM.
we are provisioning the vm with omsagent as extension and as extension doesn't work on coreos vm. we are deploying agent as a daemonset.
When we are having the extension and deploying the daemonset it is turning green in the security center.
We decided to disable the extension as extension doesn't work on COREOS and we deployed the agent as a daemonset. But is not turning into green. I don't see any difference between the VM's other than the extension enabled on one vm and disabled on the other.
I don't understand what is the relation between the extension and daemonset. Can anyone here help me understand as I don't see any log why it is not turning into green?. Any help is appreciated. Thanks!

OMSAgent + K8S + deallocation issue

I have a sandbox ACS K8S cluster that I stop and deallocate to save money.

The issue is that when I start the cluster up again, the cluster comes up fine and the OMSAgent starts; however, OMS does not show any connected sources. (Even after hours of waiting)

To fix the issue, I must delete the OMS Daemon Set and then re-deploy.

Here's the logs from one of the OMS Agent containers:
2017-05-05T15:23:05.341544301Z + sed -i -e 's/bind 127.0.0.1/bind 0.0.0.0/g' /etc/opt/microsoft/omsagent/sysconf/omsagent.d/container.conf
2017-05-05T15:23:05.534342875Z + sed -i -e 's/^exit 101$/exit 0/g' /usr/sbin/policy-rc.d
2017-05-05T15:23:05.985113531Z + sed -i.bak 's/record["Host"] = hostname/record["Host"] = OMS::Common.get_hostname/' /opt/microsoft/omsagent/plugin/filter_syslog.rb
2017-05-05T15:23:06.356117939Z + /opt/omi/bin/service_control start
2017-05-05T15:23:07.704065749Z invoke-rc.d: could not determine current runlevel
2017-05-05T15:23:07.829577472Z * Starting Microsoft OMI Server:
2017-05-05T15:23:08.859949752Z ...done.
2017-05-05T15:23:08.860477001Z + '[' -z ']'
2017-05-05T15:23:08.860487359Z + /opt/microsoft/omsagent/bin/omsadmin.sh -w -s
2017-05-05T15:23:14.962488370Z info Generating certificate ...
2017-05-05T15:23:37.207734685Z -e error Error during the onboarding request. Check the correctness of the workspace ID and shared key or run omsadmin.sh with '-v'
2017-05-05T15:23:37.733051283Z + /opt/microsoft/omsagent/bin/service_control start
2017-05-05T15:23:37.745968208Z Warning: Specified workspace doesn't exist
2017-05-05T15:23:37.746132029Z + trap shutdown SIGTERM
2017-05-05T15:23:37.746206855Z + wait
2017-05-05T15:23:37.746447541Z + sleep inf

OMS Agent will not pick up the new Docker CE and DC editions with new versions

Last week Docker released new versions of their engine. When this new version 1.17* is installed however the OMS agent does not recognize that version and exits...

Network request and response

Hi,

Is the OMS agent support capturing the network request and response b/w different containers running in the same cluster ?
for example, i have deployed multiple micro-services using containers in a Kubernetes cluster and now i want to collect data related to network request, response and failure b/w those micro-services. Do OMS agent support that ?

container log contains sensitive information

Docker container logs print out workspace key in plain text:

invoke-rc.d: could not determine current runlevel
* Starting Microsoft OMI Server:
  ...done.
+ '[' -z ']'
+ '[' -a /etc/omsagent-secret/DOMAIN ']'
+ '[' -a /etc/omsagent-secret/WSID ']'
+ '[' -a /run/secrets/DOMAIN ']'
+ '[' -a /run/secrets/WSID ']'
+ '[' -z ']'
+ /opt/microsoft/omsagent/bin/omsadmin.sh -w a4b3b..... -s 0U.....uHQ==
info    Generating certificate ...
-e info Agent GUID is 5fd6....5c84
-e info Onboarding success

OMS agent kubernetes daemonset is not collecting host journald logs

I setup the daemonset as per the instructions but the daemon is not collecting the kubernetes' hosts journald logs. Is there a way to collect these as well?

Ignore logs from certain containers (kube-svc-redirect)

In Azure Container Service (AKS), the built in service kube-svc-redirect generates a huge amount of logs, more than 500k daily. Which fills up mos the free quota of Azure Log Analytics.

Since those logs are of no particular value to us, I would like a way to be able to specify that a specific container should not have their logs collected. Possibly using kubernetes annotations on kube-svc-redirect if possible.

VM not being monitored by OMS agent

@here Can someone help me with the below issue ?
I am installing the omsagent as a daemonset kubernetes hosts. I was able to install successfully with no error at the time of installation but it is not turning in green(green tick mark in the portal stating it is being monitored). And I see below error

Error:Unsupported Operation system: COREOS 1632.2.1.
I am installing the OMSagent version(1.6.0-42) on COREOS version VERSION=1576.5.0
VERSION_ID=1576.5.0
BUILD_ID=2018-01-05-1121
PRETTY_NAME="Container Linux by CoreOS 1576.5.0 (Ladybug)"
Any help is appreciated. Thanks in advance

omsagent on kubernetes in restart loop because of failed Liveness probe

Hello,

I'm running the omsagent on a kubernetes cluster. Unfortunately the pod keeps getting restarted because the Liveness probe fails. Here is the message

Liveness probe failed: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "read /proc/588/status: bad address"

I just used the template from this repository without secretes and replaced the two variables, no other changes.
I see values in the Azure Health Dashboard. So it seems like the agent is working fine, but the Liveness probe has an issue.

Can you please help me?

Error response in Docker swarm in Azure container Service

Hi,
when i try to run OMS agent in Azure Container Service swarm cluster master node.
command: docker service create --name omsagent --mode global --mount type=bind,source=/var/run/docker.sock,destination=/var/run/docker.sock -e WSID="myid" -e KEY="mykey" -p 25225:25225 -p 25224:25224/udp --restart-condition=on-failure microsoft/oms

it displays an error message: Error response from daemon: 404 page not found.
What is the reason for it ?

ContainerLog not captured for OMSAgent configured in Kubernetes Cluster.

I am trying to run OmsAgent in a Kubernetes cluster. The OmsAgent is configured to send logs to Azure Oms portal. I am able to see logs of the type

Perf
ContainerProcess_CL
ContainerNodeInventory_CL
Heartbeat
Usage
Operation
KubePodInventory_CL
KubeEvents_CL

Was expecting the agent to also pull the container logs (which has application logs). But not able to see logs of the type "ContainerLog" in the azure portal. Not sure if this is a bug. What could be wrong ?