A Newbie’s Guide to ESXi and VM Log Files

 

A VMware ESXi instance will generate a ton of events throughout its lifetime. These events, depending on type, will be written to one or more log files, the bulk of which are found under /var/log in the ESXi filesystem. As with any log file, the idea here is to help you and others troubleshoot issues and keep an eye on things by perhaps forwarding important events to a syslog server and such. Log files also make for an excellent audit trail so you can determine, for instance, who or what is accessing your hosts, VMs, etc.

As per VMware’s documentation, I’ve listed eight of the most frequently used log files on ESXi.

 

Note: If you are experiencing problems with ESXi you may consider patching to ensure you have the latest bugs fixed.

 

VM Log Files

---

In addition to the above, every Virtual Machine hosted on ESXi will have corresponding VM log files called vmware.log recording events related to machine activity, system failures, hardware changes, migrations, status and so on. To view the log file, SSH to the ESXi host and navigate to the datastore on which the VM folder resides.

The path to the log file should be similar to /vmfs/volumes/<datastore>/<virtual machine>/vwmare.log as shown in the next screenshot.

 

There are some adjustments you can make to the log rotation and logging options for a VM. I’ve summarized these as follows:

• Disable or enable logging: Set the logging value in the VM’s VMX file to false or true respectively. You can do this by editing the VM’s settings (VM Options -> Advanced -> Edit Configuration) in vSphere Web client as shown.

 

Alternatively, logging can be enabled or disabled via the Enable Logging option. It is also possible to include diagnostic information via the Debugging and statistics drop down box.

 

• Logfile size: Similarly, you can control the log file size by adding the log.rotateSize to VM’s VMX file. Ex: log.rotateSize=1024000 sets the log file maximum size to 1 MB.

• Logfile retention: To set the level of log file rotation, add the log.keepOld value to the VM’s VMX file. Ex: log.keepOld=5 sets a maximum of 5 log files at any one instance, with the oldest being overwritten when a new log file is created. You’ll find these listed as vmware.log, vmware-1.log … vmware-n.log.

• Logfile name: To change the log file filename and/or location, add the log.fileName to the VM’s VMX file. Ex: log.fileName=vmLog.txt or  log.fileName=/vmfs/volume/VMLogFiles/vm1Log.txt.

Note: The above should work for most pre-ESXi 6.5 versions as per this KB. I’ve also tested this on ESXi 6.5 and the only parameter that seems to be unsupported is log.fileName in that the vmware.log filename is retained regardless of the value set.

 

Viewing ESXi log file contents

---

From Shell

An easy way to view logfiles on VMware ESXi is to SSH to the host and use old fashioned Linux commands such as cat, more, less, tail and head with a little bit of grep thrown in for filtering. Here are a couple of examples.

In this first one, I’m displaying the last 15 lines from the vmkwarning.log using the command tail -n 15 <filename>. This is marked (1) in the next screen screenshot. The text labeled (2), tells us that the host is failing to connect to host 192.168.20.20. As it turns out, 192.168.20.20 happens to be the IP address of a replication server I had set up on a second vCenter instance, which was powered off at the time.

 

Here’s another example, where I use the auth.log log file to determine if connections are being established from subnet 192.168.11.0 and by whom. To do this, cat auth.log and pipe it into grep filtering by the string 192.168.11 as shown. The output shows a number of successfully established SSH connections via the root account from the 192.168.11.45.

 

Using the ESXi host client

The ESXi host client makes it even easier to view the contents of a select number of log files. Navigate to Monitor -> Logs to view the list of logfiles available for viewing. Highlighting a log file, displays the contents in the underlying pane which can be copied to the clipboard and exported to file.

 

You can also generate a log bundle (see next section) using the Generate support bundle button labeled (4) in the above screenshot. When the task finishes, you are prompted to download the bundle to a folder on your computer.

 

Generating an ESXi Log Bundle

---

There may be times where you simply cannot solve an issue. Calling VMware support is one way to go about it and if you do, generating a log or host support file bundle is one thing you’ll be asked to do. This is then uploaded to VMware for further troubleshooting and diagnostics.

We’ve already seen how the bundle is generated via the ESXi host client. There are a couple more methods you can use.

The first is to run /usr/bin/vm-support from within an SSH session while logged as root. Once the bundle file is generated, you can copy it using scp or similar.

 

The second method is easier. Just point a browser to http://<ESXi IP address/cgi-bin/vm-support.cgi. You are promoted for the host’s credentials after which, the vm-support script is executed on ESXi. The generated bundle is then downloaded as a compressed tar file (TGZ). The process may take a while depending on the size of the logs, host utilization, uptime, etc.

 

You can then upload the bundle to VMware’s site using the vSphere Web client. Just navigate to Administration -> Support -> Upload File to Service Request and click on the Upload File to Service Request button. Finish off by selecting the bundle file with the Choose File button and press OK.

 

This post should have given you a basic introduction to some of the log files generated by VMware ESXi and VMs and how they are used to troubleshoot and diagnose problems. We’ve also had a look at ways by which a support bundle is generated whenever VMware support is called for.

For further details, have a look at the System Log Files section on the vSphere 6.5 documentation website.

[the_ad id=”4738″][the_ad id=”4796″]