New Expression Data

Genome-wide RNA_seq expression data for D. melanogaster have been added.  This data was produced by the modENCODE project (www.modencode.org) and analysed by FlyBase, who calculated a single value for the expression level of each gene at each developmental stage or tissue (see http://flybase.org/reports/FBrf0212041.html for more details).  This data is available on report pages for each gene and can searched through template queries (for example: www.flymine.org/query/template.do?name=Gene_RNA_seq and www.flymine.org/query/template.do?name=RNA_seq_Genes).

Improved RNAi Data

RNAi data for D. melanogaster is now loaded from www.genomernai.org.  This data provides a number of new RNAi experiments and improved phenotype descriptions for both new and existing RNAi experiments.  See the RNAi templates to learn more about this data (for example: www.flymine.org/query/template.do?name=Gene_RNAi&scope=all and www.flymine.org/query/template.do?name=RNAiScreen_genes&scope=all)

Yeast Interactions are back

Due to popular demand, we have put the yeast interaction data back into FlyMine.

Website

List Analysis

• Your list is automatically mapped to other organisms via orthologs loaded from Ensembl and TreeFam
• Orthologue and Interactions widgets have been restored (after a user request)

Region Search

• Extend regions at both sides by entering a value in the new input box
• Download links are icons
• Added the option to download chromosome sequence

Export

• GFF3 export correctly storing parents’ relationship
• “Hidden” export menu now direct links on results tables
• FIX – Export works correctly when all columns of an outer join are removed
• FIX – Missing attributes restored in fasta format

List Upload

• Specific object types can have a different example list on Bag Upload page, eg. protein accessions displayed if a user selected proteins, gene identifiers displayed if a user selected gene.
• Go back in history to the filled in form, preserves the input values.
• “Match on case” – checkbox added to list upload form to toggle case sensitivity

Report page

• Delay fixed – caching intelligently now
• Templates displayed in summary also displayed in “Other” category
• Displayers show nice “No Results” message if data fails to load
• Displayers by default loaded through AJAX to speed up page loading. Can be set to display immediately without waiting for the DOM to render.
• Left aligned displayers to make more readable
• FIX – counts in overlapping features displayer are now correct
• Protein sequence – added a new displayer
• Protein Structure ATM: turned into a popup window
• FIX – links to Reactome correct now

QueryBuilder

• Add toggle to show/hide empty fields in the QueryBuilder tree
• Added a button to save/update then run template
• FIX – if a user clicks [SHOW] in the query builder twice really fast, the field only gets added once
• FIX – displaying messages too late when the constaint logic is edited
• FIX – can now build templates with Chrome
• FIX – deleting a query from query history doesn’t forward user to my mine
• FIX – prevented name of last saved query becoming the default name in the text box

Query results

• The values the user selected on the template form are displayed on the results page
• Updated results table with an exact count of results when available

Template form

• Show OR logic when appropriate
• Fixed hyphen breaking on auto-complete in template search on template list page

MyMine

• Timestamps are now client specific and handle timezones correctly
• Prompt users if they want to delete a list that’s being used by a query
• Login/create account page improvements
• Cookie saved remembering the list order preference(s).
• Show the new size of a list after upgrade.
• MyMine table sorting correctly
• Renaming lists now works as expected

Other

• FIX – Importing templates with ONE OF constraints works correctly
• FIX – 4 year old bug which prevented LOOKUP constraints for non-bioentities from working correctly with wildcards
• FIX – If you remove columns when exporting number of rows returned is not changed

External Links

• NEW – more and better links to YeastMine and RatMine
• FIX – Links to FlyBase from the list anlaysis page
• FIX – Links to Reactome from pathway report pages

Other

• FIX – bug preventing renaming lists and queries twice
• NEW – timestamps are now client specific
• FIX – query results with outer joins are easier to read
• UPDATE – template queries are easier to build
• FIX – bug preventing the import of queries with ONE OF constraint
• NEW – downloading widget results on list analysis pages is easier and faster
• FIX – sorting of lists and queries is fixed in MyMine
• FIX -  hyphens no longer break auto-completion in queries
• NEW – list tagging now supported by web services
• FIX – LOOKUP constraints querying non bioentities with wildcards now work correctly
• NEW – list upload examples now suggest identifiers based on the type you select.

Human Readable Labels

The labels for the data in FlyMine come from the field and table names from the database.

We have updated this to use user-friendly labels instead in an effort to make the labels more human readable.

Open ID

You can now log in to FlyMine using your Gmail or Yahoo account.  Simply click on the “Log in” link and follow the intructions.

Region Search

• Additional parameter allows users to extend regions at both sides
• Can create lists by feature types for all results or for an individual region
• Results show gene symbols as well as DB identifier
• Fixed by when exporting a long list of objects to galaxy caused a “Request-URI Too Large” error
• Now works correctly with subclasses

Web Services

• NEW: region search The ability to search for genomic features by interval (the Regions tab of the web-app) has been exposed as a web-service resource. Region data can be converted to lists, or downloaded directly as GFF3, FASTA or BED.
• NEW: token authentication A new authentication mechanism has been introduced which has advantages over the current system of sending username and password details. The use of tokens means that users can authenticate requests without having to send their username and password, thus minimising chances of these being compromised. Tokens can be replaced without having to change login details, or indeed disabled entirely, meaning web-service access to user data can be forbidden on a user-by-user basis. This authentication mechanism is to be preferred to the current mechanism. One important benefit is the ability to make authenticated requests (for dealing with lists for instance) from the browser.
• NEW: Ruby client Our commitment to supporting the range of languages used by bio-informatians has been extended to Ruby, which has a growing community based around the Bio-Ruby project. The Ruby client libraries are available from [https://rubygems.org/gems/intermine rubygems.org]. For documentation please visit [http://www.intermine.org/docs/ruby-docs].
• NEW: Exporting BED from webservice queries In addition to GFF3 and FASTA, the BED biological standard format is supported from web-service queries.
• NEW: Column summary data The column summary data (visible when clicking on the ? icon on a results table) is now available via web-services when passing the ”’summaryPath”’ parameter to a query or template results request.
• NEW: Code generation The ability to generate code to run queries is now available programmatically via a web-service. Please see the WebService wiki page for details.

Export

• New export format – BED
• Fixed bugs that caused export to fail when:
  • feature didn’t have location info
  • sequence feature was not the root class, eg. Pathways.genes
  • all columns of an outer join are removed

Report Pages

We have added new widgets to the report pages:

• Publications displayer orders by number of genes mentioned
• New Pathways displayer shows pathways from other Mines for orthologues of current gene
• New RGD disease displayer shows disease ontology terms from RatMine for orthologues of current gene
• New viewer shows interactions, GO terms etc on publication report page

Data

• New species from OMIM: human, rat and mouse.
• YeastMine is now live, so we have removed yeast data from FlyMine, including publication, pathways, GO and interaction data.

Network Displayer

• Added help text for different file types
• Large datasets are now handled correctly. If there are too many interactions (> 2000) a link to download the data is shown instead of the displayer:

  The network contains more than 2000 elements, the interaction can become sluggish in the displayer.
  Please [Download the network data] and import it to [Cytoscape desktop] and use Force-Directed Layout
  to view or [view interaction data] in a table.

Templates

• Radio buttons stay checked when selecting/deselecting list constraint
• Clicking on the “query >” breadcrumb to get back to the template form works correctly
• Editing constraints for nested classes (eg. Gene > homologue > organism) doesn’t break model browser
• In query builder, “Show results” button greyed out and not clickable anymore
• After editing a template, if the user clicked on a template on the home page they would be directed to the previous template they had been editing.
• Can now import template queries containing “ONE OF” constraints
• Added option to delete trackers to the template import form. This allows superusers to export/import templates from other Mines, but save the usage information.
• Fixed error when running a template with a bag constraint in an attribute and a lookup constraint in its parent

Miscellaneous

• FIX – keyword search now handles booleans correctly
• FIX – pathway widget correctly filters for KEGG pathways
• UPDATE – widget data easier to download
• UPDATE – list upload form has default organism selected, leading to less issues when uploading lists
• FIX – links to other mines is cleverer, so results appear faster
• FIX – report pages are now cached speeding up loading times

After each FlyBase release, we build a new FlyMine database and release a new version of FlyMine.  It is possible that genes saved in a user’s list have been merged or deleted from the new FlyBase database.  Previously, the FlyMine team upgraded everyone’s lists for them automatically.  This has the drawback of being time-consuming and opaque to the users.

So we have updated our system to require the user to manually resolve any conflicts.  Usually there will be no changes and no action will need to be taken.  However, if a user’s list has changed in FlyBase, users must go to the “My Mine” section of FlyMine and upgrade their lists manually before the list can be used.

You can check the status of your lists by visiting the “Lists” page in the “My Mine” section of FlyMine at http://www.flymine.org/query/mymine.do.  In the last column will be the list’s status:

1. Current – The list is current, no action is needed
2. Not current – The list is in the queue to be upgraded, you may have to wait a minute or so.
3. Upgrade – The list contains genes that have been merged or deleted and needs to be upgraded.  Click on the “Upgrade” link and follow the instructions.

We hope the new system will be an improvement and make it more clear if list contents change over time.

UPDATES AND BUG FIXES

• NEW – Sequence features can be exported as a file or sent to Galaxy in UCSC BED format
• NEW – We now use human readable names for data labels and column names; previously we used the database field name
• NEW – The region search has an additional parameter that allows you to extend your regions at both sides
• NEW – Expression enrichment widget for Probesets
• FIX – Some yeast and worm genes had not merged properly, the duplicates have now been removed
• NEW – Better link outs to more external databases

New Report Pages

Report pages have had a major design and technical overhaul. Based on a lot of feedback we’ve redesigned the header section of the page, added section headers for easy navigation in the page and added a new information section on the right hand side. The main changes:

• Data is presented in compact inline lists where appropriate, e.g. synonyms, for greater readability.
• New graphical data displayers have been added for homologues, expression data.
• New section headers appear as tab bar that docks to the top of the screen as you scroll, the relevant header is highlighted when the section is in view.
• Query results are greyed out if they do not have any results.
• The various tables and lists of data have expanders, togglers and collapsers.
• Related collections have been grouped together, for example now you would find CRMs and TFBindingSites under Regulation.
• You can quickly move across categories of content through a sticky menu. This is a compromise that lets us show you all of the content we have for a given object at once, while allowing you to skip to the things you really want to see.

Search by Region

We have added a new search the Genomic Region Search which provides users the tool to query a huge amount of genomic sequence features such as genes, exons, etc. hosted in Flymine, with a list of chromosomal regions/intervals of users’ interests as input.  Located on the “Region” tab on menu bar to start the options page on which users can select species and feature types, and input their chromosomal regions (please read the help text on top of the page for more format information). After submitting the query, the results page will display all the features that overlap with the user’s regions. It’s also easy to export the results in different formats.

List Upload page redesign

• List analysis is visually split into the stages of list upload, verifying identifier matches and the analysis itself.
• A default list name (based on type of items, date and time) is generated
• List upload verification stage now shows you how many items and additional matches you have in a list, allowing you to add groups of them while informing you if you have more matches to add to the list.

Other Layout changes

• Removed breadcrumbs on right side
• Tab behaviour: no tab is active on results and report pages
• In the query builder page, changed links so they look like the ones on template form and added the green button Show Result in the top too.
• Moved Contact Us link, removed contextual help.

Web Services

In addition the changes below, all our client libraries now have corresponding code generators accessible from the web interface to help you start writing your code.

We have added:

• List functionality
  • Retrieval of details and inspection. Logged in users can retrieve their own list details.
  • Queries against lists (including private lists, if logged in).
  • Renaming
  • Deletion
  • Creation, from lists of identifiers, and queries.
  • Appending.
  • Union/Intersection/Symmetric Difference
  • Subtraction (Asymmetric Difference)
  • List membership (which lists is an object in) Many services support tagging.
• Biological webservices:
  • Getting results as GFF3 and Fasta is now supported.
• Possible Path Values:
  • Users can get summaries of the values a path may represent, along with frequency details.
• Template and Query upload services:
• Templates and queries can be saved to a user’s profile.
• Resource reporting – users are able to check if an optional service has been enabled.
• Schemata for result formats (xsd for XML and json-schema for JSON)

See WebService for details.  Later this summer we are holidng a webinar in which we review the available web services in detail.  If you are interested in attending, please let us know!  info[at]flymine.org

Python Library

There is a new client library for our webservices API written in Python. This offers access to a very broad range of webservice functions, including the new list webservices. There are complete API docs available at our docs section.  See PythonClient

JavaScript library

This has been made more flexible and extensible, and includes helpers for accessing services beyond query results. The main changes regard content presentation in the default table display, with a number of rendering bug fixes and UI improvements. The latest version is 0.2.  See JavaScriptClient

Perl Library

Now with access to all webservice features, including lists and possible path values. The major addition here however is character safe result parsing using structured message formats. Also there have been a number of method additions and improvements to make common jobs easy, such as shortcuts for adding all attributes on a class, or iterating over a query’s result-set. Lists and queries are broadly interchangable (both can be used as sources of results, or used in list operations, or in list constraints). All constraint types are now supported, including list and loop constraints.

In addition, a separate optional module, Webservice::InterMine::Bio provides access to the new biological webservices.  See PerlWebServiceAPI

Java Library

The Java webservice client now has paging support in its requests, along with better, more thorough documentation.  See JavaClient

Data

• New data from OMIM
• Data from UniProt for all Drosophilas, not just melanogaster
• Introns collection added to gene
• We’ve also removed several datasets – Tiffin, protein structures, InParanoid orthologues, all bee and pombe data.

Keyword Search

• Keyword search correctly handles special characters
• You can now use a wildcard as the first character of a search string

Updates and Bug Fixes

• UPDATE – Add LIKE and NOT LIKE to input boxes of constraints
• UPDATE – The dropdowns of lists in the template form are in alphabetical order
• FIX – Query Builder constraint logic is case insensitive and displays A and B instead of A,B
• FIX – Template export works when select single or multiple templates on MyMine and Templates tab
• FIX – Forgotten password link sends a message with the URL
• FIX – Input a filter value or select a category in Templates (or List) page , select a template (or a list), click ‘Back’: the page still displays the filter edited and/or the category selected
• FIX – When copying one or more lists the name is automatically generated
• UPDATE – Javascript and CSS are now minified
• DELETED – Removed Excel export
• NEW – Display different content on homepage if new user
• NEW – Users can change column header descriptions
• FIX – Error generated when clicking on summary button twice
• FIX – Bug which caused templates to not appear on the correct report page
• FIX – Search box toggle on homepage displays correct help text
• FIX – Nav trail now displayed in the query builder
• FIX – Multi-Value template parameters now comma-safe
• UPDATE – Entrez gene identifier, pathway and protein domain names are now links in results tables, making it easier to find the data you want.
• UPDATE – Ontology name is now a dropdown in queries so you don’t have to guess the exact name of the ontology you need.
• UPDATE – Pathways have been added to the keyword search so you can filter your search results by pathway.

Data

In this release 27.0 we have updated data from FlyBase to release FB2011_01 and updated several data sets to their most recent versions. Please see the data section of FlyMine for more details.

New Features

There are lots of new features we are really excited about:

• Improvements to the list upload procedure

  The list upload procedure is easier to use, having been redesigned to provide more help to the user

• Linking In

  There is now better support for linking in to FlyMine. As well as linking to report pages,
  you can link directly to query result sets, list analysis pages, or a choice of the two.

• Embedding Results on Your Pages

  In addition to the old method of using IFrames, you can now embed tables of data from FlyMine directly into your own pages using JavaScript.
  There is a full tutorial on this exploring some of its potential here. Among its benefits, is the
  ability to have full stylistic integration with your own page using native css styling.

• Meaningful Order for Templates

  The list of templates, which was previously simply ordered by template name,
  now shows templates according to popularity, so the most popular (and thus most useful, hopefully) templates will appear at the top,
  making it easier to discover helpful tools.

Updates

As well as new features, there are a few tweaks to the existing functionality:

• Stricter test correction on enrichment widgets
• More helpful filter options on enrichment widgets
• New “delete” button on lists page
• Report pages load faster (ajaxy magic means the interactions displayer doesn’t hold things up now)
• The QueryBuilder is more intuitive: selecting an item in the list will take you back to the original item after page refresh/reload
• Distinct values are easier to spot in query results – we’ve added a link to the column summary to help

Bug Fixes

And we haven’t stopped working to keep FlyMine bug-free…

• “Score” images now display correctly on search results
• Login bug concerning mixed case usernames fixed
• Fixed widget bug when viewing after exporting widget data
• Data isn’t repeated in the “miscellaneous” section on report pages

New, More Powerful Keyword Search

Our simple search has been improved using Lucene to provide comprehensive faceted results over all FlyMine data. It is now faster, more intuitive, and returns more informative results, with the ability to divide the search results into their data categories. This allows you to quickly zoom in on what you are looking for even when you are starting from just a few letters.

Redesigned Homepage

FlyMine’s homepage has been reworked to get you to the data you need faster and more easily.  You can now upload a list of identifiers directly from the homepage, and we’ve separated queries into their different data categories to make it easier to find the query you need.  Template queries are listed in order popularity so the most useful template queries are listed first.

Data Model Update

We’ve done a major overhaul on the FlyMine data model.  The Sequence Ontology, which underpins our data model, has been updated to version 2.4.2.  We’ve removed unused data types and renamed others for easier querying.  Please see here for the entire list of changes.

Exporting Results – Faster and to More Places

Back-end optimisations mean that you can get your results much faster when exporting a set of results to various file formats. Specifically, we now cache your query better, so that we don’t have to repeat it to do the export step.

An addition to our export options is Galaxy, a collection of data analysis tools in the cloud. You can now tie up the huge data stores of FlyMine with Galaxy’s flexible data-processing tools to really make your research easier.

Queries and Templates

We have overhauled the XML format we use, to provide one that is much easier to read – we are still keeping backwards compatibility with pre-0.94 version XML, so if you have any queries floating about in XML, they will still work, see XMLFormatChanges0.94.

We have also added a new constraint type to the searches you can make, called “MultiValue” constraints.  These let you choose, as the name suggests, more than one value for a match, using the operators “ONE OF/NONE OF”.  Subclass constraints (an advanced constraint feature that allows very fine-grained searching) are now much simpler to add.

Templates have seen some major new features too, the most prominent of which is the “optional” constraints, which allow you to build and use more flexible, templates. The template form now has a new, clearer layout.  More customisable too: You can also specify the order constraints are presented to the user of your template simply by dragging and dropping.

Building queries has seen improvements too. Reverse references are now made more obvious to the user, and you will no longer get lost in an endless loop of self-referencing objects, as the links back to identical objects have been disabled. Subclasses are now hopefully much easier to discover and constrain by using the graphical interface, allowing you the precision and control you need to build sophisticated queries. We have also made various changes to the layout and design of this feature focussed on making the process of building a query easier and more intuitive, such as a simple “remove all” button, when you would just like to start over. Like may features of the site,  there is more customisation here too, with custom column headers meaning you can get the data named the way you want.

Gene and Protein Interactions Visualised

We have a new gene visual analysis tool – Cytoscape Web – which allows network analysis of interaction between genes and proteins on their report pages. This uses flash to provide a simple but compelling look at a complex area. The report pages have seen general performance increases, meaning faster more responsive page views.

Web Services API – Automate your queries

Access to our web-services is now even easier. We have a new section of our site dedicated to helping you get started with the query API. Using the API means you can automate repetitive queries and data processing steps, and quickly gather and analyse massive amounts of data. Use of the API does require a little programming knowledge, in either Java or Perl, so we have provided examples for every query you build or run on the site – clicking the links to ‘Perl’ or ‘Java’ source code will provide you with source code you can run now to get the results you want.

Perl Web Service Client – Simpler but More Powerful

Our Perl client modules have been rewritten to provide a simpler interface for you. They are also much easier to download and install, and they have several powerful new features.

The ease of download is thanks to the CPAN, which means you can download, test and install in one simple command, and have your dependencies managed to boot. The new distribution (Webservice::InterMine) also has a set of tutorial how-to’s that guide you through writing and executing webservice queries.

Boilerplate has been stripped away, meaning you only need to write code that actually does something. Writing a real, functioning query can be done in as little as four or five lines!

The new features that we hope you will make use of include:

• Access to the template service,  so you can query your favourite templates automatically.
• Automatic parsing of results into Perl data structures, so you don’t have to. No more splits and regexes when you can use array indices and hash keys. This means you can not just get the results, but more easily do something with them.
• You can extend the webservice client dynamically, using Roles to provide new functionality and behaviour on a first-class basis.
• The ability to handle the sub-class and multi-value constraints, means you can write more complex, more powerful queries.

http://www.biomed.cam.ac.uk/gradschool/skills/intermine-users.html