After formulating an experimental plan, we needed to have some datasets to run those experiments on. There are some requirements that classify a good dataset for this task: it must be non-trivial in size, needs to include a protective attribute (such as gender or race), and it must have some true ranking. While it is possible to find a datasets that satisfies two out of the three requirements, it becomes difficult to satisfy all of the requirements.
Encompassing a dataset from Rankit, list of Fifa 2018 players was added as a possibility. It contained 17981 players, has a protective attribute (age and nationality), and potentially has a true ranking. The true ranking can be based on goals scored, or money earned by the player. Some ranking data can also be found over this datasets, although it does not encompass all of the players.
Hospitals and doctors also have a significant amount of data, both attribute and entity-vise. However, finding a true ranking might prove to be impossible.
For last week we aimed to arrange an informal interview sessions with faculty from school of business and the CS department to get some input on our tool.
In order to accomplish that tasks, we created a list of tasks that the participant will go through. After each task is completed, the participant would be asked a set of questions to encourage constructive and aimed feedback.
From this questionnaire, we focused on several major improvements.
The machine learning tool gave reversed ranking. Therefore, we decided to run a couple of sanity checks on the algorithm to further make sure that the results are as expected.
For the list comparison view, ranking more than two objects requires the user to scroll down and reveal the ranked box. To fix this, the dataset box will be moved to the side of the screen rather than the top and the ranked box will be set to the left of, as opposed to below the dataset box. Furthermore, in order to accept the dropped object, the ranked box …
With the Rankit paper submitted, it was time for me to change gears and dive into the research with MaryAnn.
For me, this week revolved around getting up to speed with the fair ranking research. I read over the current in progress fair ranking paper and attended meetings where MaryAnn and Caitlin helped familiarize me with the code base and research that they've completed.