Living

General/Installation

DISCLAIMER: I am NOT a Programmer. I am a sysadmin who just happens to be able to code his way out of a paper bag. This Blackboard building block written (badly) in JSP, client example monitoring script and scenario with Bash and Zabbix. Canary is a portable way of monitoring blackboard app servers, which is designed be uniform between schools and app servers. This makes monitoring large Bb farms such as the one we have at $wageslavery, 12 concurrent 24/7 Bb implementations — doing it manually was not feasible, so Canary was born.

1. Install the plugin from the canary-datestamp.zip from source. Making it inactive would be smart, this stems from me being a lazy, hubris loving sysadmin.
2. Create a user named “ucwebmonitor” (user configurable in the top of the canary.jsp)
3. Import the course into your BB system. You’ll probably want to do this on your test box at first. Make the course ID something simple like UCWEBMONITOR.
4. Enroll the user user (ucwebmonitor) into your monitor course (UCWEBMONITOR)
5. Edit canary.jsp and specify your shared secret password, monitoring user (ucwebmonitor) and course_id (UCWEBMONITOR).
6. Visit the course and read the documentation (haha) that comes with it.

• Provides a test course in an archive
• Checks that plugin can connect to that app server’s database pool (by using hidden API in Bb)
• Checks announcements (is mod perl taking to the DB?)
• Checks course file download (is the app server attached to the content server properly?)

Examples

Example client script written using Bash and CURL: canary.sh

Practical demonstration of canary.sh
[root@vzabbix ~]# ssh vuc1 -C /usr/local/blackboard/content/scripts/canary.sh
SIGNAL: database_connection_success
SIGNAL: session_authenticated_success
Wed, Oct 01, 2008 -- announcement_success success! modperl appears to be able to read from the database to get annoucements. Posted by: Blackboard Administrator     
course_content_successsuccess! Tomcat/httpd are able to connect to the database as normal.
download_file_success


Use in Zabbix (and easily adapted for Nagios): add an item system.run[/path/to/canary.sh | wc -l], alert “CANARY has died on {HOST}” with condition system.run[/path/to/canary.sh | wc -l] != 5

Signals (more to be written here)

SIGNAL: database_connection_success : building block successfully opened a connection to the app server’s connection pool (praise hidden APIs)
SIGNAL: session_authenticated_success : building block successfully ran an UPDATE statement against the BB_BB60.SESSIONS table to authenticate the session. You might have to do GRANT UPDATE TO BB_BB60 ON BB_BB60.SESSIONS or the equivalent MSSQL to get this part to work. Bb uses Javascript in their login form, and I didn’t have the patience to try to figure out how to authenticate the session that the curl example in canary.sh uses, so I took the shortcut – I highjack the anonymous session and force it to the correct user by manipulating the database. I’d love to see a curl example that would allow me to remove this particular hack.

Overall Design Thoughts

Canary sticks its fingers in behind the scenes, but only to figure out WHAT it should click on. From there, Canary redirects (302 MOVED TEMPORARILY) the monitoring request to whatever URL the user would actually be clicking on, so that the client will see the same thing as a REAL client would see, errors and all. The name, of course, comes from the application of canaries in mines before electronic detectors were available; if the caged canary in the mine died, there was something very bad and the miners should get out… or in the case of a BB sysadmin, kick the app server in the head and restart the services. Incidentally, Zabbix does this automatically for us in our implementation. If the automatic restart succeeds, no action on the part of the 24/7 on call system admin is needed… meaning we can actually get a decent nights sleep for once!

Source Code

Course Archive, Building Block and canary.sh example available in the source archives.