JIRA Subscriptions

I'd like to see everyone become more aware of development issues or helpdesk requests as they arise to help us collect input and respond to issues more rapidly.  I'd like the same for the forum but the forum is NOT the subject of this post. JIRA is the subject of this post.


So I guess I have to first remind everyone that opt in communication is a basic tenet of these systems. This is because the opposite of opt in is the well known phenomena called spam.   If you don't choose to listen to these issues then your feedback cannot be collected and your help cannot be solicited.   In other words, failing to opt in is a choice to skip participation and self censor your opinion.

So to keep abreast of changes in JIRA tickets one has to opt-in to its alert systems. There are many ways to do this and this post is my summary of the options including a new one I just figured out myself.


The options as I see them are:

1. Watching tickets. For any JIRA ticket, login and "watch" the ticket.   This should mean that every change to that ticket triggers an email alert to you.  I've done this for many tickets but a problem is keeping up with new tickets and failing to watch new tickets or track ones that suddenly become active.

2. RSS for tickets/changes.  Using RSS means you think email is for personal communication not news.  There are RSS options for all kinds of JIRA activities; however, RSS on VAO JIRA does not work. I do not know why this does not work and it bugs me. I suspect is has to do with the fact that we close off JIRA to the outside world and one needs login cred to pull the posts. Anyway, RSS should work but I don't have time to figure it out.  Let me end by saying that RSS is NOT the subject of this post.
3. "Subscribe" to a JIRA Filter.  This is the issue I'd like to alert you to right now.

The sidebar of a  JIRA
filters page. This is where
you edit, save or subscribe
to a filter results.

As you may or may not know, one can write SQL like filters in JIRA (they call it JQL, haha) that capture some subset of tickets.  Following Portal activity from within JIRA is made easier by using one of the filters Tom Donaldson created for us:  "Portal 1.1" or "Portal open tickets".  These currently have counts of 58 and 151 tickets repectively.

There is another option in JIRA to receive emails about these ticket filters. This is called "subscribing" to a filter. You may have experienced and quickly deleted the "all bugs" JIRA subscription that we all started getting right after JIRA was setup.  The problem is either you think email is bad for this purpose (I am in this camp) or you fail to achieve any new information from these posts. The latter issue exists because these filters and their auto email alerts were not about giving you any new information.

The gist of this post is this:  one can create filters that capture only tickets that have "changed" within a certain amount of time. If you care the JQL is (updatedDate > "-1d" OR createdDate > "-1d"). This means that if you subscribe to these kinds of filters you receive email digests that alert you to the most recent ticket changes. For the portal this reduces the list from an email of 151 open tickets down to a list of typically 1-5 new or recently updated portal open tickets.

You can find, after logging into JIRA, the following such updating filters:

• portal open updated recently

• Help Desk Web Tickets Updated Recently

So how do you opt in?  To subscribe you look at the sidebar on either of the linked filters above and then click "Subscriptions" at the bottom.  You then add yourself as a subscriber to the issue. There are many options for how often you want these digests but the obvious one for a Daily filter is the "Daily" frequency.  Screen grabs of these subscription pages are below.

Coda:  In case I lost you, all these steps, from creating filters to subscribing to them, are actions you do while in JIRA.  You. Do. So if you have only a Monday per week to work on VAO then one might simply create a filter that grabs all the changes in a week long period (">-7d") and subscribe to a 2am Monday morning email you will be all set.

Subscription page for a filter.

Personal Subscription options.

USVAO Web site overview

With the imminent delivery of VAO science services, the organization of the VAO web site is an urgent issue.  We are going to make decisions in the next few weeks as to where users will see the VAO portal, SED, cross-correlation and time series services.  Recently Sarah asked if we should create a new iris.usvao.org virtual host to provide a location for us to serve the IRIS service.  My suggestion was to wait a few days so that we might think about this in the context of what we want the VAO web site to look like.

First my quick summary of where we currently stand.  This is intended to list the web sites that people are charging the VAO to support.  I'm sure there are inaccuracies.

Services currently using USVAO web address:

• www.usvao.org    Primary Web site (at CalTech) 
• help.usvao.org     JIRA (at NOAO)
• dev.usvao.org      TRAC (at NCSA?)
• wiki.usvao.org     Wiki (at CalTech)
• portal.usvao.org  Portal GUI (at ST ScI)

VAO supported services not using USVAO Web address:

• nvo.stsci.edu/vor10                            VAO registry
• cxc.cfa.harvard.edu/csc1/temp/sed    IRIS help and download
• heasarc.gsfc.nasa.gov/vo/...               Validation, monitoring and notification
• voservices.net/nvolog                         Logging
• nvo.ncsa.uiuc.edu/dalvalidate/...        Validators
• iraf-nvo.noao.edu/vo-cli                     VO Client services (site?)
• sso.us-vo.org                                       Single Sign on service (NCSA/NOAO)
• astrobabel.com                                   Forum

Services supported on social media web sites.

• usvao.blogspot.com    VAO Blog
• twitter.com/usvao       Twitter
• facebook.com/usvao    Facebook
• calendar.google.com    Calendars 

External services supported

• ivoa.net           IVOA website/reg. of registries (rofr.ivoa.net)
• skyalert.org     Not sure this belongs or perhaps its moving to  being a core function...

Not yet released (I don't know the full URLs here)

• ipac.caltech.edu Cross-correlation service, time series service
• jhu.edu               Cross-correlation service, TAP server
• cfa.harvard.edu  TAP Client
• stsci.edu              EPO pages

This does not include services that are intended primarily for non-interactive consumption (e.g., the inventory and DataScope web services used in the Data Discovery Service).  These are the addresses that a user trying to find/use a VAO supported capability might need to know about.

My belief is that simply adding new virtual web sites  willy nilly like
   iris.usvao.org
and
  timeseries.usvao.org
will lead to a cluttered and potentially confusing web site. In fact I think we're already getting there.

So here's a suggested strawman organization for the VAO web site.  I'm not particularly wed to the specific
tags or this structure but I do think that having some structure will make it easier for us and our users to follow.


www.usvao.org                           (current)
    science.usvao.org                     (new -- directed towards scientists)
       /discovery       (current portal.usvao.org)
       /iris                 (currently at Harvard)
       /tapexplorer    (TAP Client)
       /xcorr             (integrate TBR x-corr interfaces)
       /time               (TBR timeseries services)

   support.usvao.org                     (new -- directed towards developers/institutions)
       /software                               (new)
          /clientTools                         (Voclient and such)
          /serverTools                       (DALServer and TAPserver)
          /svn                                   (Link to SVN or its replacement)
       /docs                                    (new probably has lots of custom links underneath)
          /repository                         (documentation repository)
          /wiki                                  (wiki if visible to public)
          /devel                                (trac if visible to public)
       /status                                   (operations validation/stuff currently at HEASARC)

     help.usvao.org                     (reuse name.  Pages that help users.)
          /request                             (Form for users to submit requests to user input)
          /internal                             (the current top level page. JIRA for VAO users only)
          /forum                               (the current forum)
          /staff                                  (pages to help users identify/contact staff)

    epo.usvao.org                          (new, directed towards teachers, students, public)
  
    internal.usvao.org                     (new, directed to VAO staff)
        /resources                            Resources that must be hosted outside our web site
        /staffGuide                           Documentation to help employees understand VAO.    
        /dev                                     Non-public development pages (current dev.usavo.org)
        /ops                                     Non-public operations pages
        /logs                                     Logging        


Please feel free to comment.  We may try to discuss this at the Ops telecon next Monday but this clearly affects all of us not just ops.

One technical issue does need to be kept in mind.  Given a remote site it is possible to link to it as both

      remote.usvao.org
or
      usvao.org/remote

but the mechanisms are different and we may have access to only one or the other.  So at least in the short term we may not always be able to get the site we want.  However I think we should first design the site we'd want to see and then recognize that we may need to have deviations.

What is the Portal?

As you might know, many of us are meeting next week in Baltimore (physically and virtually) to discuss portal development in Year 2 (which we plan to dive into right after the second beta release of the Data Discovery Tool at the end of September). This meeting is part of an effort across all our science initiatives to ensure that our development is science-driven. Consequently, the big focus of this meeting will be science use cases. Gretchen Greene has assembled a great set of use cases that were developed by a number of people reaching back to the NVO days and spanning to more recent examples from the VAO and the IVOA. This week I spent some time with these use cases to try to pull out some common features and needs. It was an inspiring and worthwhile exercise--I highly recommend it.

Doug Tody made the comment this week that considering these use cases as they are laid out on that page makes this meeting more of one about the VAO architecture. I believe he was making reference to the fact that our first year's effort for the portal was about data discovery, resulting in a nicely focused tool that includes some new techniques for quickly finding and selecting datasets. The use cases, on the other hand, go beyond discovery. So, yes, I have to agree with him: addressing these use cases does mean addressing the VAO architecture.

He also said in the same breath, "I'm not sure what the Portal is." As we take a step back to consider what we've accomplished in the first year and what we'll aim for in the second, it's a good time to try re-answering that question. We should do it in terms of science use cases--what it is we want astronomers to accomplish--and, yes, this means addressing some of the essentials of the VAO architecture.

I want to suggest that we see the Portal as a web-based entryway to a collection of tools that work together. What you can do in the portal goes beyond data discovery; that is, the Data Discovery Tool is just one of those tools. The Cross Comparison Tool should be another (actually quite a critical one, as I hope to discuss in a later post). The TAP client should have a presence there as well. And what can you accomplish in the portal? Those use cases, of course.

Updated 9/6: Given the discussion below, I should note that I meant to say that the Portal--in my definition--is explicitly browser-based (with programmatic web service interfaces possible, too).

In this vision, the Portal is not a monolithic tool that does everything; each tool specializes in some task, and over time we can add more tools. It's worth noting though that like all good use cases, ours say nothing about "tools." Clearly, to solve those problems, one will have to use several tools together. Thus, a key aim of the Portal is allowing the tools to work together: the user should be able to easily discover and create lists of datasets and sources with the Discovery Tool and join them together with the Cross-Comparison Tool. It would be great if that collaboration of tools were so quick and smooth, that the user may not realize that two different tools were used. Of course, this kind of collaboration extends to the desktop: the user should be able to send selected catalog records to Topcat and a list of SEDs to Iris.

A key question, then, we need to answer in the design of the Portal is what is needed to allow these tools to work together? This, some of you will recall, was a key problem that had to be tackled when we were developing the NVO portal. Fortunately, we have a few more technological options available to us today than we had back then.

The other key question that comes out of this vision is what tools do we need to put into the Portal? To answer this, we need to look carefully at the use cases. I would like to see us break down each of these into a sequence of scientific steps. Then we can ask, which of these tasks can be answered with the Discovery Tool? Where can we call in the Cross-Comparison Tool? And more importantly, what tools are missing? We're also likely to discover that our current tools are not quite tuned to these tasks and need some adjusting.

Actually, having gone through these use cases once, dissecting them can be a bit challenging (but interesting). I did glean, I think, some important and common features that we need to support which I hope to share in my next post. Next week, we'll get more eyes on these use cases, and I hope we can pick out a few exemplary ones which we will use to guide further portal development. So what is the Portal, or rather, what will it become? Ultimately, it is--and I know this is cheap to say--a place to do science. We need to be able tell users what science they can do with the portal; we should be able to describe at least a few of these use cases and have it sound compelling and useful to them.

Are we asking the right questions?

You already have a list of desired functionality, so complete your list and download and review each of the top 3-5 OpenSource CMS. Questions that interest me:

Ease of installation.
Ease of upgrades (especially if you add a bunch of 3rd party plugins).
Security - how does the CMS rank (PHP has a security history).
System resource requirements (small or big footprint).
Learning curve (for system admin and users).

What kind of web pages do we need to support in a CMS

In building our Web site there are a variety of different kinds of pages that we need to support.  Here's a first cut at what we want to be able to do:

• Static pages with standard headers, banners, logos, etc. E.g.,  documentation, home pages for the VAO and various subelements, some tutorials...
• Lists of personnel associated with the VAO or particular subprojects.  Could be derived from personnel databases.
• Calendars.  Dynamically updated from some master calendar.
• Document repostory.  Easily updated with version tracking.
• Blogs (including comments from users)
• User forum (or we may simply continue to use Gus's software).
• Polls (including easy access to summary of results and database management)
• Simple web submission/order/registration forms (including access to summary of results and database management)
• Slideshows (often on home pages of sites that want to highlight more than one item

Some of the capabilities we want overall are:

• Easy access to pages outside of CMS control (e.g., science forms like the portal).
• Analytics of usage
• Ability to export templates or elements thereof for use outside of the CMS system.
• Ability to distribute responsibility for updates to different individuals and teams
• Import/export to Wiki/Trac
• Ability to mirror
• Integration with SVN including automated updates

Not all of these are critical, but they were the kinds of features that I think would make our life easier.  Does anyone else have thoughts on this?

CMS: The Big Three

A quick review of web content management systems suggests that there are three free systems that have substantially more market share than others.  We may want to think if it would be appropriate for us to limit our consideration to these three systems: WordPress, Drupal and Joomla.  A very recent comparison with some graphical statistics is at:
   http://www.networkworld.com/community/blog/drupal-joomla-wordpress-smackdown

A PDF which compares the 'top' 20 open CMS systems is available at:
   http://www.waterandstone.com/sites/default/files/2010%20OSCMS%20Report.pdf

There are plenty of other CMS systems out there, including just the free ones.  However unless we have some strong advocate for a particular system, I don't see any other natural dividing line and my inclination would be to start with these to see if one of them can meet our requirements.  Sarah's listed many of the requirements in here previous post (I think a choice of mayo or mustard on the sandwich is needed too) but we'll want to delineate these more completely and clearly and see how these tools match up.

I did a bit of searching on how difficult it is to copy web sites from one of these systems to another.  Generally it seems to be non-trivial but not a horrible task so there is some ability to transition.

The overall impression I get in reading many discussions of these is:

Similarities:

All have a free core system but significatn use of any of these typically involves getting plugins that address specific needs.  Some of these plugins may be commercial, but are typically not very expensive.

Differences:

WordPress, Joomla and Drupal go from easiest/least powerful to hardest/most powerful.  However new releases in all of these systems are reducing the differences on all sides.  WordPress has moved from being primarily a blogging platform to a full-featured CMS.  Drupal is providing more out of the box capability to users.


If there are members of the team that can comment from experience on comparisons of these systems that would be very helpful.  Since these are evolving rapidly mentioning when you used them would be helpful.

to cms or not to cms

I hope this is the right place to put this...

As VAO document coordinator and webmistress and wiki manager etc. I love the idea of something that can "do it all". That is some web framework that can:

• manage the public website, including epo
• allow for different users to have different viewing/writing/editing/approval permissions for different sections of the site
• keep a blog
• manage a document repository
• create web forms for registration/polls/surveys/summer school applications/research proposal
• provide a template to remote sites for use in web tools hosted other than at caltech
• create a way to edit, review, and publish user documentation
• make me a ham sandwich

If you were to ask me directly what I think we need, I would probably answer "some kind of CMS". My only reservation with this is the initial effort it takes to learn a new tool and get it configured all exactly right, especially with the limited time I have.

When we were first trying to get the VAO website going last year I had thoughts of using Drupal or Wordpress to manage the content. Drupal I have a little experience with, Wordpress a bit more (but I don't think WP does what we need). But with so little time, and the need to get something ready before AAS/Seattle, there was no time for this kind of discussion with the group as to what CMS to use, not to mention time for me to get up to speed with Drupal or any other CMS sufficiently to create a good-looking and well-organized website for the VAO. I just had to get it done with the tools I had at hand, thus HTML and ColdFusion we have now. I don't say it's the best way, but it was the only way I could get it done quickly.

I wonder, if we can agree on a content management system that will do all the things we need, would it be a crazy idea to hire a contract developer/designer to create the initial framework for us that we can then maintain ourselves? Do we have a budget for this, and is it allowed?

Documents in the VAO

We now lots of different ways that a document can be published in the VAO: the wiki, forum, blog, JIRA issue, mailing lists, public website or the document repository.  This post suggests how these elements might work together.

The wiki (including Trac) serves as the internal web site.  It provides an easy way to publish and organize information that we want to use in running the VAO.  Though there may be exceptions, information here is generally not visible to the public.  Limited effort are made to enforce style guidelines.  Obsolete information is deleted (though available through the internal CM capabilities of the wiki).  Organization is according to the functional needs of the VAO, notably through WBS elements and software development projects.

The document repository is intended to provide a standard location for finding all formal documentation in the VAO.  Any document that have a version number (and some that don't) should be available in the repository. All versions of a document re available.     My recollection is that the current plan is to have the repository basically be a wiki page with lots of attachments, but I may be out of date.  If this is still the approach the documents in the repository would be an exception to the visibility rules for the wiki since they would be visible to the public.

The forum provides an area where questions and comments may be asked and discussed.  The post initiating a forum discussion may require a fairly long prologue setting up a context or giving some background information, but an archetypical post would boil down to a fairly short question or comment:  "How do I get the list of sources I extracted from the portal into the  cross-correlation engine?"  "Wouldn't it be nice if I could see the positions in galactic coordinates?"  "The SED service crashes on my Linux xxx box."

A JIRA ticket implicitly describes both a situation and a resolution.  A ticket is the appropriate document to initiate and monitor a specific action by the VAO team.   If no specific action is intended then a JIRA ticket is not the appropriate vehicle.

Mailing lists are the only documentation framework outside JIRA which demands attention from the user.  Thus they are used to make sure that people are informed when meetings are to take place or when critical documents are to be reviewed.  Generally other formats are better for structured discussion of issues, especially the Wiki and forums.

The web site is what we present to our science and EPO communities.  The web site may reference the document repository for documentation of some user services but probably does not have much visibility into the other documentation classes discussed here.   Ray has suggested that we might use Trac pages for software documentation.  Personally I'm a little dubious here.  Good documentation for a service as seen by users is rather different from that for developers.  I also think that it's important that documentation look like it belongs to the service its documenting.  Typically a web page will use lots of internal anchors and such.  So my suspicion is that on-line documentation will tend to be either complete documents from the repository or specially crafted using the overall web site guidelines for the VAO.

Finally the blog is used for essays: what does a member or team from the VAO think about something.  The blog can be documentation but tends to be developmental rather than descriptive.   E.g., suppose the portal effort goes off and tries a couple of different techniques for coupling to the cross-correlation service.  SAMP for some reason doesn't work and a VOSpace-based approach is adopted.   A blog entry describing this process could be invaluable.  It doesn't really belong in the documentation of the service as developed, but a blog post provides a place to say why a given approach was adopted.  Similarly the blog can be used to talk at length about how we might do something complex (say mesh our various documentation frameworks to pick a random one:) in a non-authoritative way.

To give a concrete example of how all of these might work together consider the current  discussion of how we will build web pages.  I tried to kick off the discussion with a blog post trying to summarize what we think we agree on and what questions need to be addressed.  Other's are free to comment on or edit the post.  If they have some substantial thoughts themselves, a parallel post can be provided.  If we've more than a trickle we'll want to add tags to discussions to group them.  Over the course of the next month, as we begin to understand the boundaries of what we are trying to do, we may start a wiki page with an outline for the policy document.  A post to the team mailing list lets everyone know what's happening so this can be reviewed by and edited by the team and a version can be agreed upon.  At that point the document should be copied in some fashion, TBD, to the document repository.  It may require purchase of specific software by one or more members of the VAO and a JIRA ticket can be issued for each such requirement.  This discussion manifestly affects the appearance of the public web pages we have, but no reference to it is necessarily visible in those pages.  Once the process begins a VAO member with a question about how to do a specific step in the documentation process may enter a query in the forum (especially if we have an inward facing version of the forum) and other members of the team can respond with suggestions.

Building Web Pages for the VAO

Many of us in the VAO will be involved in writing web pages.  This post discusses how we work together to build a coherent and consistent web environment, discussing what has been agreed and trying to identify questions that we need to address.

Classes of Web pages.

Per the discussion of the July team meeting, we define three classes of web pages according to the anticipated audience.

•  Internal web pages are intended only for the eyes of members of the VAO.  This may include the VAO wiki and raw Trac and JIRA web pages along with this blog.
• Public web pages are web pages intended to serve our role in the science community.  This includes our  science services, documentation, help pages, newsletters, forum....
• Outreach web pages are pages specific to our EPO efforts which are intended for non-science users. 

The recommendations of this memo are intended primarily for public web pages. Practice for outreach web pages will extend and modify the public web page policy.

Where do web pages go: physical?

Where feasible web pages should be included on the VAO's primary Web site (hosted at Caltech).  Documentation, software downloads, and simple forms can all be easily accommodated by the main web site.  Science capabilities, e.g., the VAO portal and cross-match services may need to be hosted on other machines.  This should be discussed in the operations plan for the service.

Where do web pages go: virtual?

All VAO web pages should appear with in the usvao.org web space.  This will naturally follow when web pages are included on the Caltech host.  If a service must be hosted remotely, then a web-address of the form xxx.usvao.org shall be aliased to the appropriate site.

How do we define the 'xxx's?

Currently we have help.usvao.org, and dev.usvao.org (which both link to internal sites).

How do we build web sites?

Do we want a content management system?  Do we build web sites as an operation on the SVN repository?

Do we want a standard form support system (a la ColdFusion)?

How do we ensure uniformity among our Web sites?

This would be one goal for a content management system.
Alternatively we could have standard CSS and HTML templates that all web sites are required to integrate possibly using different techniques.

What are the design elements that appear on VAO web pages?


Logo's, institutional references, style, ...

A general web design framework would be nice.  Probably User Support's role to answer.  One role of the content management system is to separate the design elements from the content, but that is unlikely to be entirely successful.

Do we need to identify logical cross-references?


One suggestion during the team meeting was that we should use logical names for links within the VAO (i.e., from the portal to the cross-match service) and have these resolved at some point.  This would allow use to each change locations for given services without breaking links.  It would also allow us to provide some support for handling references to services that are not yet available.

Do we want to do this?  If so how and at what level do we define these links?


Who builds web sites?

Can't place all of the burden on Sarah.

Automated creation by content management system in fashion similar to Jenkins testing framework?

Multiple authorized users at Caltech site?

Who authorizes creation and modification of web site?

At the team meeting we discussed this as a CMB responsibility and that is certainly the case for significant changes.  However if we want a responsive we site we probably need to allow much more freedom for at least some areas of the web site (e.g., latest news, personnel, faq).  Nor do we wish typo fixes to required CCB approval.


Added:
Sarah brought up some issues in the telecon including:

• What is the relationship with the document repository?
• We need to identify actual people involved at various steps.

Ani noted in the comments that we will want to have an expedited process by which the web master can update the page with the CMB's blessing without having to go through an actual meeting. 

Purpose of this blog

This blog is intended to be used to facilitate discussions of the VAO team