Each Pythomnic cage does its own logging into its own log file. This is convenient and simple to use, mostly for keeping low-level logs, only readable by a developer or aware administrator. Also, when there are dozens of cages, you have dozens of separate log files and the state of the entire application cannot be estimated at a glance. The pmnc_log facility is therefore supposed to be used in cage-level logging only.

To deal with application-level messages, Pythomnic cages use the health monitor facility. Health monitor cage serves as a centralized message collector. Regular cages use the following constructs to send messages to health monitor:

pmnc_info("Everything's fine")
pmnc_warning("Hm-m-m-m, something's wrong")
pmnc_error("I told you")
pmnc_alert("Contact the administrator at once")

No matter which one is used, the message is delivered to the health monitor using identical queued RPC call to the health_monitor cage. The health monitor messages are thus delivered reliably. Even if health_monitor cage is down, the regular cages will accumulate messages locally, and whenever health_monitor appears again, they are going to be delivered.

For this reason, health monitor is really an important part of Pythomnic application. As you can see from cage's default config/startup.py, health monitor support is not started by default, but you are encouraged to behing your Pythomnic application development with rolling put a health_monitor cage. Once health monitor cage is started, all the regular cages could have their health monitor support enabled.

By default, all health_monitor cage does to the messages it receives is to log them to its own log file, prefixing with an appropriate level, for example, a message initiated by source_cage with

pmnc_info("how about a little logging ?")

will appear in health_monitor's log file (likely on a different node) as

HH:MM:SS.NN [ThreadName] [INFO] source_node.source_cage: how about a little logging ? \
(YYYY-MM-DD HH:MM:SS) # health_monitor_receiver.py:109 in notify()

This typical behaviour is informative but not very convenient and so you may want to override it in _static\health_monitor_receiver.py. With a little modification, it proves useful to store the incoming messages to a database and then have a simplistic web-page built on top of http_interface to be able to quickly estimate the state of the individual application cages (up/down) and display messages logged by a particular cage. With a little effort you can equip the health monitor database with a full-text search and have the displayed logs hyperlinked for greater ease of use.

Here is a couple of screenshots of an actual a health monitor web interface, implemented in the most simple and straightforward fashion for the administrators to browse the cages status and logs.