GSoC 2011: Week 13

This week mostly I worked on fixing bugs, improving some features and documenting my work. The panning feature required much attention as mentioned in my previous report. So I worked around this by changing the equations used for panning. One issue that we faced was that the existing panning feature had a zoom associated with it. On exploring the Highcharts reference manual we found out that Highchart rounds off to nearest tick in the plot. So on disabling that option the panning feature worked fine.

Apart from that I worked on providing a 'Reset zoom' link to get the original display(if zoomed/panned), added a FAQ in the official documentation about 'How to use the zoom search feature?' (FAQ 6.32), added some code documentation, fixed issue with browsing foreign values and handled the notice that used to be displayed in case data label not selected.

GSoC 2011: Week 12

This week I continued my work on the issues of plotting support for date-time fields. After looking around for a suitable distance criteria, I decided upon the timestamp values of each date-time (milliseconds passed since 1970-01-01 00:00:00). Date, time, datetime field values were converted into timestamps and plotted with these values to give a meaningful distance measure. Time fields were measured by considering the time values on the day 1970-01-01. The date library at http://www.mattkruse.com/javascript/date/source.html was helpful in converting values to and from timestamps.

The second issue was implementing zoom and pan based on the code snippet at
http://jsfiddle.net/HXUmK/5/. Although the logic of zoom and pan is right the system response time is not good. Also, the equations used for implementing zoom by readjusting extremes are not correct. The equations used in the snippet were:           
newXMin = xMin + (1 - zoomRatio) * xMax                                                            newXMax =  Max * zoomRatio 
(zoomRatio varying from 1 to .6, xMin and xMax being the datamin and datamax)

I replaced the equation with:                                                                                    newXMin  = xMin + (xMax - xMin) * (1 - zoomRatio) / 2;
newXMax = xMax - (xMax - xMin) * (1 - zoomRatio) / 2;
(zoomRatio varying from 1 to .6, xMin and xMax being the axis-min and axis-max)

This improved the mousewheel zoom to some extent. The panning feature was taken from the snippet but its very difficult to use. The logic seems to be right, readjusting the axis extremes based on the position of mouse drag but it is very sensitive. Even a little drag sometimes shifts the axis to a much larger value. I'll look to improve on this but I still think that using the inbuilt zoom, where we can select an area and zoom into it is much powerful and faster way of implementing the zoom/pan.

GSoC 2011: Week 11

Things worked on this week:

• Using AJAX to get data row on point click. 
• Better support for date/time objects
• Panning feature

Using ajax to get the data row on click reduces the amount of data being passed on to the javascript layer. On data point click the where clause of the point is passed on to the server in request and the data row is returned in response. 

Support for date/time turned out to be rather tricky in Highcharts. The different date/time formats have to be converted into javascript 'date' type to have correct distance between them. Somehow using new Date(dateString) does not produce correct plots. Next I tried code snippet at http://www.mattkruse.com/javascript/date/source.html , which a getDateFromFormat function which return the getTime of the date. So the difference of those getTime() can be a measure of distance between two dates. Will continue on on working on it this week.

Highcharts has an inbuilt zoom feature but not a panning feature. Also, the zoom feature utilizes the mouse click and drag events, so a pan cannot be used with those events. The code snippet at http://jsfiddle.net/HXUmK/4/ provides a method of implementing zoom(mousewheel) and pan(click and drag) manually. Currently I'm trying to integrate it within my code, there are some unresolved issues with use of  setExtremes() functions. 

GSoC 2011: Week 10

I made a lot of changes to the edit feature this week. Main objective was to avoid replot of charts on edit and to replot only the points which gets changed. Progess this week:

• Optimizations to edit feature.
• Edit support when using strings as fields (only replot of chart , query generation remains the same)
• Minor changes like validations, some bugs.

Also I was working on using ajax to get a selected point's data row, rather than posting the whole data on submission of input criteria. The edit feature also changes accordingly along with some events, so its getting a little complicated. This was in my to-do list of 10th week but it stalled. I'll try to finish it in this week and continue on to work on more optimizations.

GSoC 2011: Week 9

This week I worked on two major issues:

• Changing the display of the data point content  from a form based interface to a dialog based display.
• Providing support for plotting of other data-types like strings.

Using the jQuery-UI plugin the first issue was rather straight forward. On point click/select just open a dialog which will contain the HTML of the dataDisplay form. This helped in cleaning up the interface. It adjusts well to different resolutions which the previous work did not.

The second issue was a little complex. Direct plotting of strings is not well supported in Highcharts, it is usually through 'categories' option of xAxis/yAxis. The data points are plotted corresponding to each of category values in this case. In our case, a field may have duplicates so in categories, the labels are repeated on the axes. So instead of using categories I assigned each distinct string (for a field) an integer value (eg. 1,2,3...), mapped the corresponding string values to integer values and formatted the labels to show the string values instead of the integer map. This helped to solve the following issues:

• Duplicate labels in the axes are removed (as opposed to categories)
• Can now plot them as simple integer values
• In addition to above sorting the array containing the distinct values (inbuilt sort for now) solves the issue where we have a large number of labels and their text get mixed up. What happens now is like integer labels some divisions are created on the axes and when we zoom in the divisions get smaller and labels within that division are show. 

For the next week, compatibility issues still need to be fixed for different browsers. Also, the update feature needs to be modified to support the replotting of strings.

GSoC 2011: Week 8

This week's work was to fix bugs and test the interface. Some pending issues were fixed like:

• The display field of a table is the defualt selected value for DataLabel and if not selected it defaults to the display field also.
• The edit functionality was malfunctioning, there were issues with null values on fileds. The Datapoint display form now includes a null checkbox to indicate null value. 

  I ran some tests on the interface to check response timings and also asked some users to try and give suggestions. Some of the things that came up were:

• The input phase, when we select the two fields; on selection of a field the data-type,operators and value field change and it takes some time. I was suggested to improve upon the response time there.
• The plot generation process also slow according to some users. If we are accessing the database locally (on the same system) it is relatively fine but when accessed on a remote server, it slows down. 
• Depending on the resolution, the plot and the Data point display form can appear adjacent to each other or one below the other (the plot up and the form below it)
• One of the user reported that his chrome browser crashed once on the plot generation phase.
• Some users suggested that the divs are not alligned properly. For example, for a higher resolution when the plot and form appear adjacent to each other, the plot div is slightly higher than the form div. So it did not look good.
• Some users were more interested in have the plot of text fields( fields that are strings and not only numeric), which is not currently available in the interface.

So looking at the response I got from these tests, I have to focus on response timings more and include the plot functions of strings next. Also, the positioning of the Data-point form and plot is a problem. It could well be fixed as Marc suggested, by having a dialogue box type form which appears at the center of the screen when we click a point and disappears after we edit/view and not always have it on the screen.  The input phase can be improved by passing the entire table information(field type, collation, operators and value field) as json and using jQuery to update it onChange.

Row Edit feature: Null checkbox does not gets checked back on null value

GSoC 2011: Week 7

I spent this week implementing the edit feature in zoom-search. The edit feature included updating the plot(client side) on edit and also updating the database(server side).

On the client side, first the data row is updated on submit. If either of the column values or the data label is edited, it should be reflected in the plot as well. The margins also had to be updated/resized if required. For highcharts, I just needed to update the series data and redraw the chart. It was mentioned in option reference that this is done automatically(on data/axis extreme modification) or we can use chart.redraw() function, which redraws parts of the chart which have changed and not the entire plot.

Next part was to update the database. To avoid reloading the form each time an edit is done, the database update is done asynchronously using AJAX. In my original php cone(tbl_zoom_select.php) I included a unique condition (where clause) for every data row in the query result of the input criteria. This might be helpful in future if we want to speed-up the interface by generating the query not like SELECT * from table but SELECT xColumn,yColumn from table and the rest of data about the row can be fetched by using this unique condition (where clause).

For the edit feature, I use this unique condition as where clause in generating the UPDATE query for the row under consideration. After generating the UPDATE query, i post it to sql.php which executes this query and sends back the data and message. I append these responses to the <sqlqueryresults> div to show the UPDATE query executed and some statistics.  There are many areas of this code which can be optimized as this is a basic/raw implementation.

Apart from these I fixed some small issues and bugs like changing the 'How to use?' (help) link text, making it disappear on click, removed the mode feature. This week I will work on optimizing the code for better response time, bug fixes and hopefully panning feature (if released from Highcharts).

GSoC 2011: Week 6

This past week I spent most of my time reorganizing my code and tried to use ajax in some parts. I redesigned my interface into two parts (forms), one form for taking in the input criteria(zoom_select_form) for generating the plot and the other form where the user can view/edit the data points(zoom_display_result). 

By doing so, once the input criteria is submitted I can show/hide the form just like the table search page. So more of code reuse and the interface also looks similar to the table search. Next, I tried to use ajax to post the details from the zoom_display_result so that an update query can be issued without interference. I got stuck here for some time but I've stated fresh now.

I did some more fixes like functioning of the modes: In browse mode the zoom_display_result fields are disabled so they cannot be edited and active again on edit mode, added the link for 'How to use', renaming of configuration directives.

Tasks for this week: Fix the default label feature, handle submit of zoom_display_result form.

GSoC 2011: Week 5

This week I worked on some of the suggestions I had received on the mailing list. Mostly I had to clean up my code, explore more of Highcharts option reference page and started work on edit mode for zoom-search.

The page now has option of selecting modes, where you could either select browse mode or edit mode. The cursor changes accordingly (pointer to crosshair) as Highcharts provides options of cursor change in case of clickable data points.

First I though of creating a popup with jQuery where I would present the form containing the fields and current values, which could be edited but now I've decided to divide my interface into two parts, one for input and the other contains the plot on the right and the datapoint content on the left. Next task was to design the form to display the current values of the datapoint. To avoid duplication of code/functionality:

• Use edit row functionality from table browse mode or
• Use table search page form

I had earlier organized the table search code into libraries so I went through with using it instead of table browse edit. The form looks like the table search page minus the field type, collations and operator columns. The values are updated on clicking data points. The task that remains is updating the database upon submit. I also have to optimize my code such that I don't end up redrawing the plot for every change made. I do feel that in the past week I did not do a lot of work, so this week I'll try to catch up on it. I'll try to finish the edit mode quickly and work on the feedbacks.

GSoC 2011: Week 4

This week I implemented a basic model of the functionalites that my zoom-search will offer. Previous week I had created the plot using SVG and this week I used Highcharts (which Tyron suggested as he has replaced pChart with it) to generate the charts.

So basically this week I looked into Highcharts and explored its options reference manual. I used the scatter plot chart type with point objects. It offers zooming functionality ( in x, y and xy), although the panning feature is still missing. Next I also included the clickable point feature to show the data row when the point is clicked. It also offers datalabel feature which I take as input from user or use the table display field otherwise.

The charts produced by Highcharts are good visually and it also has a lot of inbuilt functionality but one problem I was found was that: For tables with large number of rows, the response time goes up. For some cases in my firefox 4.0.1 it alerts to stop script or stops responding. So maybe we could use AJAX to improve on the response time (as Marc suggested).

Apart from this I also did some fixes in my code and added resizing chart functionality (by dragging a corner), provided a 'Directions to use' link. I also received suggestions on the mailing-list so I will work towards them next. This includes improving the search criteria, using AJAX to improve the response time, panning feature for the plot.

GSoC 2011: Week 3

This week I have implemented the plotting feature using SVG. My task is to try and test out different plotting libraries for my requirements. So the things I did this week:

• Implemented classes for the plot based on svg.
• Classes for data objects.
• Generating the plot for different data types.
• Fixing some bugs and issues in the code.
• Testing and changing the plot parameters to find some patterns on a database(sakila).
• Reviewed tyrons work with about replacing pChart with highcharts.
• Started work with highcharts for the plot.

The idea was to generate a plot such that some patterns could be revealed in the process. The user then drills down to search in detail. I also tried to look into the clickable points feature of plot in SVG, with <a> tags and jQuery, trying to implement something similar to image maps.

This week I'll try to implement the plot feature using highcharts and try out its features: zoom, image map and other features and decide upon the plotting library, SVG, highcharts or some other.

GSoC 2011: Week 2

This week's task was to get back in touch with SVG and other SVG manipulation libraries. I spent most of this week looking at the current 'Display Chart' functionality and following others related work using SVG like  Madhura recently created a GIS Visualization feature using SVG, so I went through his work. There are some doubts over SVG support by different browsers but I think by the time this feature will be available for release, SVG support should get better.

I also found out some design errors I made while designing the interface for input and the query generation phase and corrected them. As the 'Display Chart' feature works for specific queries (http://demo.phpmyadmin.net/gsoc-madhura/Documentation.html#faq6_29), I am hoping that this interface would restrict the user to generate only these type of queries.

I then used SVG to generate a basic plot for two columns with numeric values. Just a basic framework to include other plot features. I will be working on the same this week. My tasks would be to improve the plot, plotting data points based on string field and other field types, deciding over the labels of the data points ( trying whether to take as input from user or based on some index ).

GSoC 2011: Week 1

So this was the official first week of coding. My first task was to design an interface where the user will provide his search criteria, which will then be used to generate a plot.

Firstly, we had to look around and see where to put the interface. There are two places where users can search:

• The multi-table query generator: It looked a bit complicated to add the interface/link there. Also its not a relatively popular place to search.
• The table-search page: It looks simple, just displays the fields of a table in query-by-example manner and user can search on it. It is better suited as the task requires search criteria over a single table. 

After discussing it further on the list, we went ahead with developing it on the table-search page. Within the table-search page, two sub-tabs were provided, one for normal table search and the other for zoom-search.

Since this interface requires search criteria on just two columns and the user can put two different search criteria on a single column. This case could not be handled in the table-search page so we made slight modifications to the design. A user first selects the two columns from two drop down lists. The column type,collation, operations and criteria fields of the column are displayed upon selection. It also saves a lot of space on the interface. It uses a little less of jQuery and has a different query generation process. Instead on one search-query spanning all columns, two queries (one for each criteria) is generated. This will also help in future, if we decide on letting the search span multiple tables.


While coding for this I tried coding both the interface in the table-search page(tbl_select.php) but realized that it will complicate future work. So I created a separate component for the zoom-search interface. In doing so I have ended up duplicating a lot of the code, for which I have to refactor the common code into functions and put it in the library part.

My immediate task is to complete this interface asap by refactoring the code and discussing the design for further modifications.

Here are my project details:
Title: Zoom Search

The basic functionality of system can be described as:

• user selects a table, ie movies
• user picks two columns, ie popularity and year
• a visual grid is built, showing year on the X axis and popularity on Y; each movie is represented by a dot on the grid
• user can select a part of the grid to zoom the specific portion; then dots become more spaced and hovering over them shows for example the movie's title
• user clicks on a dot to browse this data row

The system can be divided into three components:
1) Plot criteria/Interface:
This component will consist of an interface which allows users to select the two columns, constraints and data point labels that they want to be represented in the plot. To avoid duplication of functionality the table search page, multi-table query generator page from the phpMyAdmin project along with some additions can provide us with a suitable form to fulfill the input criteria.

2) Plotting using a charting library:
This component calculates the coordinates of each data point based on a predefined distance measures while handling cases like overlapping data points. Also, each data point will consist of a label and a clickable link which can display the contents of the data row.

The plot can be created using charting libraries like pChart, JpGraph, SVG, jquery-SVG. pChart has the advantage that it is currently being used in phpMyAdmin so it saves us from including another charting library but it does not supports clickable links. JpGraph allows clickable links at the cost of inclusion of another library. SVG provides vector images compared to bitmaps of pChart/JpGraph which suits the quality of zoom feature and also saves us from including another charting library. jQuery-SVG is a jQuery plugin that lets us interact with an SVG canvas. With jQuery-SVG it will also be easier to include other jQuery plugins(zoom/pan for example). Looking at the pros and cons I will prefer using SVG/jQuery-SVG.

3) Zoom-In/Out:
A jQuery script will allow the plot to be zoomed-in/out on demand. Zoom can either be gradual or in increments. As per the the system requirements a zoom and pan image or zoom and crop image tool will work well.  Another way could be a simple + and - zoom bar (just like google maps) over the plot image, it will also look familiar to users. The focus will be on having a fast response time so we can try different plugins or write our own code and based on usability tests we can fix which one suits our system better.

Absolutely delighted for being selected into GSoC 2011 for phpMyAdmin.
Looking forward to "Zoom search" :)  I am sure it will be a wonderful opportunity and also, its my first step towards working as an open source contributor. I hope this journey continues .
Thanks phpMyAdmin team and Google.