JIRA Subscriptions

I'd like to see everyone become more aware of development issues or helpdesk requests as they arise to help us collect input and respond to issues more rapidly.  I'd like the same for the forum but the forum is NOT the subject of this post. JIRA is the subject of this post.


So I guess I have to first remind everyone that opt in communication is a basic tenet of these systems. This is because the opposite of opt in is the well known phenomena called spam.   If you don't choose to listen to these issues then your feedback cannot be collected and your help cannot be solicited.   In other words, failing to opt in is a choice to skip participation and self censor your opinion.

So to keep abreast of changes in JIRA tickets one has to opt-in to its alert systems. There are many ways to do this and this post is my summary of the options including a new one I just figured out myself.


The options as I see them are:

1. Watching tickets. For any JIRA ticket, login and "watch" the ticket.   This should mean that every change to that ticket triggers an email alert to you.  I've done this for many tickets but a problem is keeping up with new tickets and failing to watch new tickets or track ones that suddenly become active.

2. RSS for tickets/changes.  Using RSS means you think email is for personal communication not news.  There are RSS options for all kinds of JIRA activities; however, RSS on VAO JIRA does not work. I do not know why this does not work and it bugs me. I suspect is has to do with the fact that we close off JIRA to the outside world and one needs login cred to pull the posts. Anyway, RSS should work but I don't have time to figure it out.  Let me end by saying that RSS is NOT the subject of this post.
3. "Subscribe" to a JIRA Filter.  This is the issue I'd like to alert you to right now.

The sidebar of a  JIRA
filters page. This is where
you edit, save or subscribe
to a filter results.

As you may or may not know, one can write SQL like filters in JIRA (they call it JQL, haha) that capture some subset of tickets.  Following Portal activity from within JIRA is made easier by using one of the filters Tom Donaldson created for us:  "Portal 1.1" or "Portal open tickets".  These currently have counts of 58 and 151 tickets repectively.

There is another option in JIRA to receive emails about these ticket filters. This is called "subscribing" to a filter. You may have experienced and quickly deleted the "all bugs" JIRA subscription that we all started getting right after JIRA was setup.  The problem is either you think email is bad for this purpose (I am in this camp) or you fail to achieve any new information from these posts. The latter issue exists because these filters and their auto email alerts were not about giving you any new information.

The gist of this post is this:  one can create filters that capture only tickets that have "changed" within a certain amount of time. If you care the JQL is (updatedDate > "-1d" OR createdDate > "-1d"). This means that if you subscribe to these kinds of filters you receive email digests that alert you to the most recent ticket changes. For the portal this reduces the list from an email of 151 open tickets down to a list of typically 1-5 new or recently updated portal open tickets.

You can find, after logging into JIRA, the following such updating filters:

• portal open updated recently

• Help Desk Web Tickets Updated Recently

So how do you opt in?  To subscribe you look at the sidebar on either of the linked filters above and then click "Subscriptions" at the bottom.  You then add yourself as a subscriber to the issue. There are many options for how often you want these digests but the obvious one for a Daily filter is the "Daily" frequency.  Screen grabs of these subscription pages are below.

Coda:  In case I lost you, all these steps, from creating filters to subscribing to them, are actions you do while in JIRA.  You. Do. So if you have only a Monday per week to work on VAO then one might simply create a filter that grabs all the changes in a week long period (">-7d") and subscribe to a 2am Monday morning email you will be all set.

Subscription page for a filter.

Personal Subscription options.

USVAO Web site overview

With the imminent delivery of VAO science services, the organization of the VAO web site is an urgent issue.  We are going to make decisions in the next few weeks as to where users will see the VAO portal, SED, cross-correlation and time series services.  Recently Sarah asked if we should create a new iris.usvao.org virtual host to provide a location for us to serve the IRIS service.  My suggestion was to wait a few days so that we might think about this in the context of what we want the VAO web site to look like.

First my quick summary of where we currently stand.  This is intended to list the web sites that people are charging the VAO to support.  I'm sure there are inaccuracies.

Services currently using USVAO web address:

• www.usvao.org    Primary Web site (at CalTech) 
• help.usvao.org     JIRA (at NOAO)
• dev.usvao.org      TRAC (at NCSA?)
• wiki.usvao.org     Wiki (at CalTech)
• portal.usvao.org  Portal GUI (at ST ScI)

VAO supported services not using USVAO Web address:

• nvo.stsci.edu/vor10                            VAO registry
• cxc.cfa.harvard.edu/csc1/temp/sed    IRIS help and download
• heasarc.gsfc.nasa.gov/vo/...               Validation, monitoring and notification
• voservices.net/nvolog                         Logging
• nvo.ncsa.uiuc.edu/dalvalidate/...        Validators
• iraf-nvo.noao.edu/vo-cli                     VO Client services (site?)
• sso.us-vo.org                                       Single Sign on service (NCSA/NOAO)
• astrobabel.com                                   Forum

Services supported on social media web sites.

• usvao.blogspot.com    VAO Blog
• twitter.com/usvao       Twitter
• facebook.com/usvao    Facebook
• calendar.google.com    Calendars 

External services supported

• ivoa.net           IVOA website/reg. of registries (rofr.ivoa.net)
• skyalert.org     Not sure this belongs or perhaps its moving to  being a core function...

Not yet released (I don't know the full URLs here)

• ipac.caltech.edu Cross-correlation service, time series service
• jhu.edu               Cross-correlation service, TAP server
• cfa.harvard.edu  TAP Client
• stsci.edu              EPO pages

This does not include services that are intended primarily for non-interactive consumption (e.g., the inventory and DataScope web services used in the Data Discovery Service).  These are the addresses that a user trying to find/use a VAO supported capability might need to know about.

My belief is that simply adding new virtual web sites  willy nilly like
   iris.usvao.org
and
  timeseries.usvao.org
will lead to a cluttered and potentially confusing web site. In fact I think we're already getting there.

So here's a suggested strawman organization for the VAO web site.  I'm not particularly wed to the specific
tags or this structure but I do think that having some structure will make it easier for us and our users to follow.


www.usvao.org                           (current)
    science.usvao.org                     (new -- directed towards scientists)
       /discovery       (current portal.usvao.org)
       /iris                 (currently at Harvard)
       /tapexplorer    (TAP Client)
       /xcorr             (integrate TBR x-corr interfaces)
       /time               (TBR timeseries services)

   support.usvao.org                     (new -- directed towards developers/institutions)
       /software                               (new)
          /clientTools                         (Voclient and such)
          /serverTools                       (DALServer and TAPserver)
          /svn                                   (Link to SVN or its replacement)
       /docs                                    (new probably has lots of custom links underneath)
          /repository                         (documentation repository)
          /wiki                                  (wiki if visible to public)
          /devel                                (trac if visible to public)
       /status                                   (operations validation/stuff currently at HEASARC)

     help.usvao.org                     (reuse name.  Pages that help users.)
          /request                             (Form for users to submit requests to user input)
          /internal                             (the current top level page. JIRA for VAO users only)
          /forum                               (the current forum)
          /staff                                  (pages to help users identify/contact staff)

    epo.usvao.org                          (new, directed towards teachers, students, public)
  
    internal.usvao.org                     (new, directed to VAO staff)
        /resources                            Resources that must be hosted outside our web site
        /staffGuide                           Documentation to help employees understand VAO.    
        /dev                                     Non-public development pages (current dev.usavo.org)
        /ops                                     Non-public operations pages
        /logs                                     Logging        


Please feel free to comment.  We may try to discuss this at the Ops telecon next Monday but this clearly affects all of us not just ops.

One technical issue does need to be kept in mind.  Given a remote site it is possible to link to it as both

      remote.usvao.org
or
      usvao.org/remote

but the mechanisms are different and we may have access to only one or the other.  So at least in the short term we may not always be able to get the site we want.  However I think we should first design the site we'd want to see and then recognize that we may need to have deviations.

What is the Portal?

As you might know, many of us are meeting next week in Baltimore (physically and virtually) to discuss portal development in Year 2 (which we plan to dive into right after the second beta release of the Data Discovery Tool at the end of September). This meeting is part of an effort across all our science initiatives to ensure that our development is science-driven. Consequently, the big focus of this meeting will be science use cases. Gretchen Greene has assembled a great set of use cases that were developed by a number of people reaching back to the NVO days and spanning to more recent examples from the VAO and the IVOA. This week I spent some time with these use cases to try to pull out some common features and needs. It was an inspiring and worthwhile exercise--I highly recommend it.

Doug Tody made the comment this week that considering these use cases as they are laid out on that page makes this meeting more of one about the VAO architecture. I believe he was making reference to the fact that our first year's effort for the portal was about data discovery, resulting in a nicely focused tool that includes some new techniques for quickly finding and selecting datasets. The use cases, on the other hand, go beyond discovery. So, yes, I have to agree with him: addressing these use cases does mean addressing the VAO architecture.

He also said in the same breath, "I'm not sure what the Portal is." As we take a step back to consider what we've accomplished in the first year and what we'll aim for in the second, it's a good time to try re-answering that question. We should do it in terms of science use cases--what it is we want astronomers to accomplish--and, yes, this means addressing some of the essentials of the VAO architecture.

I want to suggest that we see the Portal as a web-based entryway to a collection of tools that work together. What you can do in the portal goes beyond data discovery; that is, the Data Discovery Tool is just one of those tools. The Cross Comparison Tool should be another (actually quite a critical one, as I hope to discuss in a later post). The TAP client should have a presence there as well. And what can you accomplish in the portal? Those use cases, of course.

Updated 9/6: Given the discussion below, I should note that I meant to say that the Portal--in my definition--is explicitly browser-based (with programmatic web service interfaces possible, too).

In this vision, the Portal is not a monolithic tool that does everything; each tool specializes in some task, and over time we can add more tools. It's worth noting though that like all good use cases, ours say nothing about "tools." Clearly, to solve those problems, one will have to use several tools together. Thus, a key aim of the Portal is allowing the tools to work together: the user should be able to easily discover and create lists of datasets and sources with the Discovery Tool and join them together with the Cross-Comparison Tool. It would be great if that collaboration of tools were so quick and smooth, that the user may not realize that two different tools were used. Of course, this kind of collaboration extends to the desktop: the user should be able to send selected catalog records to Topcat and a list of SEDs to Iris.

A key question, then, we need to answer in the design of the Portal is what is needed to allow these tools to work together? This, some of you will recall, was a key problem that had to be tackled when we were developing the NVO portal. Fortunately, we have a few more technological options available to us today than we had back then.

The other key question that comes out of this vision is what tools do we need to put into the Portal? To answer this, we need to look carefully at the use cases. I would like to see us break down each of these into a sequence of scientific steps. Then we can ask, which of these tasks can be answered with the Discovery Tool? Where can we call in the Cross-Comparison Tool? And more importantly, what tools are missing? We're also likely to discover that our current tools are not quite tuned to these tasks and need some adjusting.

Actually, having gone through these use cases once, dissecting them can be a bit challenging (but interesting). I did glean, I think, some important and common features that we need to support which I hope to share in my next post. Next week, we'll get more eyes on these use cases, and I hope we can pick out a few exemplary ones which we will use to guide further portal development. So what is the Portal, or rather, what will it become? Ultimately, it is--and I know this is cheap to say--a place to do science. We need to be able tell users what science they can do with the portal; we should be able to describe at least a few of these use cases and have it sound compelling and useful to them.

Are we asking the right questions?

You already have a list of desired functionality, so complete your list and download and review each of the top 3-5 OpenSource CMS. Questions that interest me:

Ease of installation.
Ease of upgrades (especially if you add a bunch of 3rd party plugins).
Security - how does the CMS rank (PHP has a security history).
System resource requirements (small or big footprint).
Learning curve (for system admin and users).

What kind of web pages do we need to support in a CMS

In building our Web site there are a variety of different kinds of pages that we need to support.  Here's a first cut at what we want to be able to do:

• Static pages with standard headers, banners, logos, etc. E.g.,  documentation, home pages for the VAO and various subelements, some tutorials...
• Lists of personnel associated with the VAO or particular subprojects.  Could be derived from personnel databases.
• Calendars.  Dynamically updated from some master calendar.
• Document repostory.  Easily updated with version tracking.
• Blogs (including comments from users)
• User forum (or we may simply continue to use Gus's software).
• Polls (including easy access to summary of results and database management)
• Simple web submission/order/registration forms (including access to summary of results and database management)
• Slideshows (often on home pages of sites that want to highlight more than one item

Some of the capabilities we want overall are:

• Easy access to pages outside of CMS control (e.g., science forms like the portal).
• Analytics of usage
• Ability to export templates or elements thereof for use outside of the CMS system.
• Ability to distribute responsibility for updates to different individuals and teams
• Import/export to Wiki/Trac
• Ability to mirror
• Integration with SVN including automated updates

Not all of these are critical, but they were the kinds of features that I think would make our life easier.  Does anyone else have thoughts on this?

CMS: The Big Three

A quick review of web content management systems suggests that there are three free systems that have substantially more market share than others.  We may want to think if it would be appropriate for us to limit our consideration to these three systems: WordPress, Drupal and Joomla.  A very recent comparison with some graphical statistics is at:
   http://www.networkworld.com/community/blog/drupal-joomla-wordpress-smackdown

A PDF which compares the 'top' 20 open CMS systems is available at:
   http://www.waterandstone.com/sites/default/files/2010%20OSCMS%20Report.pdf

There are plenty of other CMS systems out there, including just the free ones.  However unless we have some strong advocate for a particular system, I don't see any other natural dividing line and my inclination would be to start with these to see if one of them can meet our requirements.  Sarah's listed many of the requirements in here previous post (I think a choice of mayo or mustard on the sandwich is needed too) but we'll want to delineate these more completely and clearly and see how these tools match up.

I did a bit of searching on how difficult it is to copy web sites from one of these systems to another.  Generally it seems to be non-trivial but not a horrible task so there is some ability to transition.

The overall impression I get in reading many discussions of these is:

Similarities:

All have a free core system but significatn use of any of these typically involves getting plugins that address specific needs.  Some of these plugins may be commercial, but are typically not very expensive.

Differences:

WordPress, Joomla and Drupal go from easiest/least powerful to hardest/most powerful.  However new releases in all of these systems are reducing the differences on all sides.  WordPress has moved from being primarily a blogging platform to a full-featured CMS.  Drupal is providing more out of the box capability to users.


If there are members of the team that can comment from experience on comparisons of these systems that would be very helpful.  Since these are evolving rapidly mentioning when you used them would be helpful.

to cms or not to cms

I hope this is the right place to put this...

As VAO document coordinator and webmistress and wiki manager etc. I love the idea of something that can "do it all". That is some web framework that can:

• manage the public website, including epo
• allow for different users to have different viewing/writing/editing/approval permissions for different sections of the site
• keep a blog
• manage a document repository
• create web forms for registration/polls/surveys/summer school applications/research proposal
• provide a template to remote sites for use in web tools hosted other than at caltech
• create a way to edit, review, and publish user documentation
• make me a ham sandwich

If you were to ask me directly what I think we need, I would probably answer "some kind of CMS". My only reservation with this is the initial effort it takes to learn a new tool and get it configured all exactly right, especially with the limited time I have.

When we were first trying to get the VAO website going last year I had thoughts of using Drupal or Wordpress to manage the content. Drupal I have a little experience with, Wordpress a bit more (but I don't think WP does what we need). But with so little time, and the need to get something ready before AAS/Seattle, there was no time for this kind of discussion with the group as to what CMS to use, not to mention time for me to get up to speed with Drupal or any other CMS sufficiently to create a good-looking and well-organized website for the VAO. I just had to get it done with the tools I had at hand, thus HTML and ColdFusion we have now. I don't say it's the best way, but it was the only way I could get it done quickly.

I wonder, if we can agree on a content management system that will do all the things we need, would it be a crazy idea to hire a contract developer/designer to create the initial framework for us that we can then maintain ourselves? Do we have a budget for this, and is it allowed?