Rants with "Actual Informational Content"(c)

You are in the potential jeopardy of learning something if you stay here and look at these.  Caveat Lector.

Obscurity Divers and their search for knowledge
September 15th 2016 to...

obscurity divers

exploring the trackless depths of the net where the wild hidden things are

By Chris Koeritz

i've been using a search engine a lot these days.  i think we all have.  and i've noticed an odd thing; some questions are harder to get good answers for than others.  perhaps that seems really obvious, since some topics are harder to comprehend than others, and search engines don't have flawless language parsing skills yet, and so on.  but the weaknesses i'm seeing in the search results don't quite seem grounded in how difficult the question was, or in the knowledge base assumed by the question.  it seems instead that there are occasionally topics that are just harder to answer by automated means.

if that were true, why would it be so? 

using a search engine can still be a frustrating experience for some questions, for potentially numerous reasons...  the information may simply not be visible to the search engine, but still exist hidden away in the dark parts of the net.  or the search could be blocked by mundane issues, such as when terms in the query are way more popular in other usages, so that one's search is hijacked by pages and pages of trivia before the result that might usefully answer the search.  or maybe the answer is not even there, anywhere on the internet, despite the subject matter still being known to humans, some of whom even know the answer (that is, the internet doesn't know everything yet, and never will).

of course there are topics still unknowable to us tiny humans, like "where did the first atom come from and how do i get there on a map?", or "how many intelligent species have existed in the universe so far?".  and of course some questions cannot be answered because they really don't make a lot of sense, such as "how blue is my turnip brain tomorrow yesterday?".  but i posit that there are whole classes of questions that are totally sensible but still don't yield good search results; i think everyone has seen an example of this.

so, i thought to myself, i thought, why not try to take more notes when this breakdown in a search happens, and write down what i'm looking for and how it's missing (what came up instead, what i actually needed to see, etc)...  and then what if i post this information in some kind of spooling "couldn't find it" blog.  it also seems really important to update this record if i do find the information later, via whatever means, even if i needed to consult the "real" world.  if one posts this "missing data" record up on a blog or website, and the record is indexed by search engines, then my "hard to search for" question is also documented.

if people really did this, the process wouldn't necessarily create useful reading material, nor even usable search results.  i probably wouldn't want to read someone else's failed search terms... not unless they were into some freaky stuff, anyway.  but this process is one method for filling in between the cracks in our knowledge with some electronic spackle, and
it's a way expand our understanding to cover more topics, reaching everywhere the mind traces.  if we keep a record of the blank places in our online knowledge, maybe we can eventually fill them in.

there, that's my silly obscurity divers meme.  thank you very much for being willing to rummage around in my old notes.  come back any time.
and yes, i admit the title of this article made it seem more exciting than it really was.  that's marketing!

* maybe a special tag for this spooling page could provide the semantic information needed to keep these things (spooled failure blogs) from showing up as the results of searches.  we don't after all want the search engine to substitute someone's failed search for an answer to the search.  the spooled failures are initially an absence of results, and then maybe eventually a sign-post to the results (when people actually go back and fill in the things they found out.  also unicorns!).  so the semantic tag embedded in a spooled failure page would indicate that the information there is purely connectivity, not a result in itself.  search engines could avoid showing any of these search failure pages as actual search results, but the engine could still build a neural network of associations from the topics listed on the page, in effect using the page as a hint about understanding more specifics of the question or even what an expected result would be from such a query.  and then the usual problems of accurately weighting the informational value of these pages would ensue.  perhaps community-ranked afficionados would jump into their meta search engine managers and rate the value of these interstitial pages, thus providing some feedback about the search failure page's "quality".

** hmmm, a lot of this work could be done by the search engines themselves, if they offered a very flexible rating system for how well they answered the search.  if the search engine itself were to take the comments about what was really sought and how it was missing, it could chew on these things and come back at a later time with other proposals for the search, and ask the user if these are any better.  if the users would put up with it, and rate their searches somewhat honestly, then the search engine could actually get smarter all the time, or at least get to know the tricksy humans a lot better.  so, google, where's the "rate this search" button on your search interface, which feeds human responses to a giant AI who tries to figure out what results are missing from the overall goog database, eh?

[posted September 15 2016]

Installshield Intro -- Bring the Pain
September 17th 2010 to July 22 2012

Installshield Intro -- Bring the Pain

By Chris Koeritz


    I am co-opting this section for talking about the "design" of Installshield installations, if there can be said to be one.  Obviously, there is a lot of structure buried under the Installshield program, the design of most of which Microsoft can be held responsible for when they produced the Windows Installer (or MSI--MicroSoft Installer).  MSI is a complex system that often works, but in the end has some very annoying complications that the Setup Developer is left holding the bag on.  The remainder of the section will mostly be opinion-free, but I can only liken my role in introducing you to Installshield to that of Cerberus introducing souls to hell.

    Forget what you know about Install programs as dynamic, goal-directed programs with flashy interfaces.  Microsoft has boiled down some of the essential nuggets of an installation into the MSI format, which is essentially interfaceless.  It's up to vendors like Wise and Installshield to add glitz back in, if desired.  Installshield supports extending the Basic MSI type of installer with a User Interface front-end, such as that used by LightLink.

    The main product of an MSI installation package building tool is the MSI file itself.  This file is crafted differently according to the Windows Installer version that it supports, but it is in essence just a database.  The MSI file is a set of tables with content for the different phases of an Installation.  So, anything that an MSI installer does (or can do) must be something that can be represented as a row in a database table.

    There are many activities that Microsoft has envisioned an installation program may need to perform, and a lot of operations are provided for these tasks.  These include things like registering services, copying files into place, deleting files, etc.  Each of these types of tasks lives as a line in a table somewhere.  Hopefully you will never need to know more detail about the MSI tables than that, except for one important table which has the installation properties stored in it.  The properties are the set of values used by the Installation for directory locations and so forth.  These can be reconfigured by a knowledgeable system admin to make the installation perform differently.

    It is important to have a basic understanding of the installation process, because it helps fit the random pieces of the Installshield interface together into a somewhat sensible scheme.  Each installation follows a sequence of activities.  In general, the majority of actions in a sequence are handed to you when you create the basic framework of the install with a wizard.  At some point in the sequence, choices are made about which features to install.  This can either be automatic, based on parameters passed to the MSI, or it can be user driven, via a set of dialogs.  Each feature is a collection of components that will be installed, but that does not mean each component is merely a collection of files.  A component can also be one of the provided types of actions, such as writing a registry entry.

    Since even Microsoft realizes that they cannot think of everything, there is a special type of action called a custom action.  These are actions that the setup developer creates to perform more general tasks than might be provided in the default toolkit.  For example, LightLink has a custom action that performs a large number of configuration steps by running some script files.  Custom actions are still just lines in a database somewhere, but they provide a way to arbitrarily link to other programs rather than being limited to features directly provided by MSI.

    The actions listed in a sequence will not always necessarily be operated on.  Each action is dependent on a set of conditions that dictate whether it can fire or not.  Some conditions are simple, such as those based on the choices of components made in the UI.  Others are more complex and may themselves even need to be set via a custom action, especially if the condition is needed for a UI decision.

    At the time the MSI package is being installed, Windows passes the package over to the msiexec process.  This is a system process that runs as a service (under the LocalSystem, aka SYSTEM, account).  The msiexec application simply iterates across the tables in the MSI in a pre-defined way until it hits an error or concludes the installation.

    For some unknown reason, an MSI package is not allowed to invoke another MSI package while the installation is running.  This leads to numerous hilarious problems and is the reason that merge modules are created.  The merge module allows what is basically an entire, separate installation program to be stuffed into your own installer such that the MSI constraints against running two MSIs are not violated.  The LightLink installer uses a wide variety of merge modules to plug in third-party features we depend on, or even "first-party" features like extensions to Windows written by Microsoft but not shipped in their default OS.  The above is a really rough overview of Windows Installer and Installshield, but hopefully it provided some guidance on how things flow inside an MSI file when it is running on the target system.


Action: The fundamental unit of processing in an installation.

Component: A quantum of installable material.  This can be a file, multiple files, or even an action provided by the Windows Installer support.

Condition: A boolean statement involving properties which is used to guide whether actions are taken or not.

Custom Action: An action created by the setup developer for a purpose which is either not available through MSI or which could be better implemented outside MSI.

Dialog: A user interface component that displays or gathers info (or both) for the user.

Execute Sequence: The set of actions taken once the UI is put away and the installation is given the go ahead.

Feature: A major division of the installation's deliverables.  Features can be made visible to the user as choices they are given about the installation payload and installed configuration.  During the UI sequence, users select which features will be installed.

Key File: A component can have a key file that represents the presence or absence of the component.  If the key is absent or out of date, the component must be installed.

Property: A string or numeric value with a name that can be used to store locations, settings, and other facts about the installation.

Sequence: The ordered stream of activities that make up the install process from initial execution to completion.

UI Sequence: The set of actions taken to interact with a user.  Generally involves dialogs and fact checking.


    Other operating systems had actually already created general installation packaging formats that perform really well.  Many of these existed long before Windows Installer was released into the wild.  Most Linux distributions religiously use a packaging format to deliver their OS itself as well as updates to the OS.  In Redhat Linux, there is the RPM format, and in Debian Linux, there is the DEB format.

    Each of these formats is widely used by other distributions also.  I'm afraid that if you expect MSI to live up to either of these types of packaging methods, then you will be disappointed.  For example, where a Debian Linux install can be said to be completely determined by the set of DEB packages that are installed, there is no such promise for Windows.  A Windows user may not even know of some MSI packages that got installed, whereas others will be visible in Add/Remove.

    Unlike Debian, a Windows user can easily break a program by uninstalling something it depends on; the Windows user is not always warned about interdependencies between packages.  And Windows itself is definitely not packaged into discrete, installable MSI chunks that can be switched out for better versions at any time.  Debian Linux suffers from none of those disadvantages; the set of installed packages and their interdependencies are clear, and installing a new kernel is nothing more than getting the package and installing it like any other package.

    Further, each of the distributions will generally have a huge number of free packages available for a variety of purposes, and only a small number of these are installed by default.  Microsoft provides no centralized repository of MSIs for customer's needs, nor would those packages be free (as in beer) if there was such a repository.
[posted September 17 2010]

Simulation Environments for Business
August 20th 2010 to September 16th 2010

Simulation Environments for Business

July 4 2010 –By Chris Koeritz

virtual operating room Graphical 3D simulators could easily be mistaken for video games, but these environments provide a number of benefits for business, chiefly because the simulated environment is much cheaper and more predictable than many real-world business scenarios. For education and training, simulators are superb because the students can work with virtual versions of real devices that are either too expensive or too dangerous to provide. Scientists can use the 3D simulation environment to stage experiments and look for potential problems before moving a single test tube around in the real laboratory. Medical students in training can use surgical instruments in a virtual operating room on a virtual patient, providing a valuable opportunity to explore the human body for less expense. The virtual operating room can allow the educator to reset the simulator in seconds for the next student.

After the early splash of Second Life on the gaming and business simulation scene, many other companies and groups have developed simulators. Linden Labs, the creators of Second Life, may have accelerated this activity by providing their viewer (the Second Life client program) as tibetan classroom an open source application that can be freely modified and extended. The OpenSimulator group has now developed a free server that is compatible with the Second Life viewer and they have released the server's source code as well. This has spawned a number of other simulation companies that provide OpenSim grids. The biggest of these grids is called the “osgrid”, which has roughly 4,000 simulator regions online. The osgrid is small compared to Second Life, but is steadily growing.

Many educators are looking to virtual environments as being the next phase in distance learning, and they value the ability to give lectures to large numbers of students without needing an actual huge room. One might suspect that they are sneakily attempting to draw in the digitally intoxicated youth into a learning environment without the students quite noticing. The virtual tools for learning can be tailored for particular types of teaching environments, such as in the Tibetan Classroom in the figure. Students can learn the Tibetan characters and can read short phrases from the Wall of Tibetan Tiles. A “Cube of Tibetan Consonants” is also available, where a Tibetan Lama pronounces the sound for each of the characters.

The most useful feature of the osgrid for businesses is probably the virtual meeting room. Businesses can have mt. tamalpais conferences with a full multimedia onslaught, using the built-in ability of the viewer to show web pages and play movies and sound files. Since the room is virtual, there is no bad seat in the house; any attendee can zoom in and peer at the presentation as closely as they wish. And because the simulator can be run over the internet or a VPN, attendees do not need to be in the same town or even in the same country. Collaboration during the meeting can be a breeze with any number of useful meeting tools that have already been implemented, such as virtual whiteboards, live chat, webcam broadcasting, and others.

The military is also very interested in simulated battlefield environments, given the cost and danger of training with live ammunition in a sufficiently realistic environment. Nothing can replace that type of training (or the rude awakening of the real battlefield), but simulators can help soldiers prepare in advance and learn the techniques that will protect them during combat. And when real tanks, airplanes and helicopters can cost in the hundreds of thousands of dollars, a simulator can provide early maze island training on the control panels for those vehicles for mere hundreds.

Those managing Geographic Information Systems (GIS) have made a lot of progress simulating real-world places in the virtual environment. Here is an example GIS system depicting Mount Tamalpais in California. This is a 49 “region” simulation, meaning that there are 49 separate plots of virtual land that represent the mountain. Each simulator is a 256x256 meter segment in virtual space, which may impact whether the GIS simulation will be to full scale in the virtual world. There are also plenty of whimsical GIS simulations that have been developed. These may bear little relation to the real world, instead being figments of the terraformer's imagination. An example of imaginary geography is shown here, depicting Maze Island.

City planners might also find it compelling to move into simulators for their early designs. A simulated version of that new city block might be even cheaper in osgrid than building a more traditional physical model (even using popsicle sticks and construction paper, for example). The simulator version can be refactored very easily using the viewer's built-in CAD tools, whereas the physical model city planning may need to be ripped apart multiple times to achieve the same results, which is possibly a slower and more annoying process.

Architects can test out their designs and allow their clients to “walk through” the simulation before the project has even picked out a real world location. Exact versions of the proposed plans can be rendered in the simulator in order to test out utility connections, room dynamics, feng shui concerns, visibility, space utilization, and even the lighting of the facility at different times of day. Some architects might find it more convenient to design the building using the CAD tools in the Second Life viewer directly, rather than starting with a hand drawing or entering the details into AutoCAD. Since OpenSim is free source and the database is available to the owner of a simulator, the design could be extracted from OpenSim and converted into other formats as necessary.

At Inova Solutions we set up many call centers with our high-visibility OnTrack displays and LightLink server software. Often when planning the installation, we create two dimensional mock-ups of the intended display locations. We architecture island could actually get a much more realistic view of how the system will work by creating a virtual call center that models the installation environment. This simulation could be shown to the customer prior to the install to allow them to test out how their new system will work and to ensure the necessary visibility. After creating simulation materials for a few different install sites, a collection of templates could be developed to make it much easier to build future simulated call centers.

For the museum curator, the simulator can offer a cheaper alternative than collecting priceless antiques while still allowing visitors to see a bit of history. The example in the figure shows an Argand whale-oil-based lighthouse lamp that is historically accurate, but which may be out of the financial reach of a coastal museum. However, visitors could take a virtual tour and see this and other rare items, even if the museum itself doesn't run the simulation. Museums can each contribute models and pictures from their own inventory to be shared across the virtual “museum space”.argand lamp

Simulations are only limited by the power of our imaginations. The simulated environment lets us show things from perspectives we can never achieve in the real world, such as an amoeba's eye view of a protein construction (if amoebae had eyes). Things that are not physically possible can be depicted (like traveling into a black hole), and very difficult environments from the real world can be simulated much more easily (such as the fluid flow of the wind during a tornado). Mathematicians can travel to a very realistic version of flatland if they so choose. Astronauts can try out the dynamics of zero gravity, or even negative gravity, if desired. The possibilities are “virtually” unlimited.

About the Author

Chris KoeritzChris Koeritz is a Research Scientist with the University of Virginia who works on grid computing infrastructure software with the GenesisII project.  He has achieved some small amount of notoriety in Second Life by creating a search engine device called “searchbert” that looks for virtual objects.  In his spare time, he builds simulators and scripted objects in the osgrid.


1. The Operating Room picture is from http://perspectives.3ds.com/2010/06/03/3dvia-for-operating-room-designs-that-work/
2. The Tibetan Classroom is from http://billmagee.wordpress.com/
3. The GIS picture of Mount Tamalpais is from http://opensimulator.org/wiki/Image:Kentfield_20071029.jpg
4. The picture of maze island is from http://opensimulator.org/wiki/Image:Maze_island.jpg
5. The futuristic city design is from the NuAthens sims mentioned at http://www.sluniverse.com/php/vb/project-development/16014-nu-athens-interactive-story-opensim-2.html
6. The picture of the futuristic building is from http://opensimulator.org/wiki/Image:ArchitectureIsland1.jpg
7. The Argand Oil Lamp is from http://opensimulator.org/wiki/Image:Argand_lamp_(OLG)_002.jpg
[posted August 20 2010]