DSAI Challenge Evaluation - next steps

Please note

•   The date and location for the Challenge Event has not yet determined
  

•   Travel cannot be covered by the data challenge team
  

•   Please talk to your manager if travel can be supported, in particular if you belong to the finalists and potential awardees
  

•   Details will be shared as soon as possible

View post 0

DSAI Challenge Evaluation - next steps

Dear all,

We wanted to share the upcoming milestones regarding challenge evaluation:

1. Next 2 weeks - Core challenge team to shortlist top solutions for subjective categories and validate top performing solutions from the leaderboard:

•      Technical evaluation of top solutions – robustness, generalizability
  

•   	Evaluation of innovative approaches and business value for the respective categories
  

2.  Broader evaluation committee to confirm final selections – potentially reaching out during this process. 
   

3.  Finalists to be notified mid-March and prepped to present their solutions at the Challenge event in front of the committee, business, and community – and compete for best presentation!
   

4.  All participants are invited to the Challenge Event in Q2 to exchange learnings on everything from PoS, to platform feedback, to “what’s next” for future challenges at Novartis. 
   

The team will be reaching out with any questions or requests regarding your solution. We look forward to future communications around the event and seeing you there!

Best,
Nick and core challenge team

View post 1

Urgent: Request asap if you need access to your vm this week - vm's start being deleted Tuesday morning

Dear all,

You will have access to your workspaces throughout this week.

If you need continued access to your VM - which is launched from the workspace- you can request this.

We will delete all other VM’s starting tomorrow but we can make exceptions where needed.

Please request below by replying to this thread.

In either case, please make sure you have followed directions to transfer your VM files to the workspace so you do not lose your VM data/files once the VM’s are deleted.

Best,
Nick

View post 0

DSAI Challenge UPDATE: decommissioning workspaces, final presentations, leaderboard

Hi Shravan,

I had to read your question more carefully - but yes you have access to the code etc via gitlab and can use your local machine to prepare, and push back to gitlab. If you have trouble here - don’t stress.

Best,
Nick

View post 0

REMINDER: data in the VM's will be deleted starting Monday 13th

Hi all,

Reminder that we plan to start the process permanently deleting the VM’s Monday 13th.

The workspaces they are launched from will remain operational through the end of next week.

Please reach out to Ned with any questions regarding the workspaces, and see this post regarding where to place data/files that you would like to keep.

View post 0

DSAI Challenge UPDATE: decommissioning workspaces, final presentations, leaderboard

Hi - some quick answers:

Yes on 1. and 2.

Yes you can utilize time until 13th to clean up code etc

The hard deadline is when the VM’s will start being deleted - but we will have a call monday morning before this in case difficulties reported.

The workspaces will be available next week, but the VM’s launched from them will not be. I’ll post again more detail on the presentations. (but it will be certainly be possible to modify them next week)

View post 0

DSAI Challenge UPDATE: decommissioning workspaces, final presentations, leaderboard

Dear all,

Based on feedback from teams as well as the evaluation team and process, we are happy to extend the presentation deadline and keep the workspaces open through the end of next week.

• Presentations submitted by end of week Jan 17.
• Code and models due Friday 10th – hard deadline “before January 13th” as the evaluation team will start

(note: VM’s will still go down)

Best,
Nick

View post 0

DSAI Challenge Update: Presentations can be submitted through week Jan 17

Dear all,

Based on feedback from teams as well as the evaluation team and process, we are happy to extend the presentation deadline and keep the workspaces open through the end of next week.

• Presentations submitted by end of week Jan 17.
• Code and models due Friday 10th – hard deadline “before January 13th” as the evaluation team will start

(note: VM’s will still go down)

Best,
Nick

View post 0

DSAI Challenge UPDATE: decommissioning workspaces, final presentations, leaderboard

Dear all,

Happy new year and welcome back!

As the Solutioning phase of the DSAI Challenge comes to a close – I wanted to make sure we were aware of the process – and if there existed any questions, concerns or general panic:

Virtual Machines will be brought down starting Jan 13. All valuable files and data must be copied to the appropriate shared location or it will be lost forever.

• See here
• Please reach out now on the forums if there are concerns!

Leaderboard:

• Public view locked on 10th – EoB (6pm) EST
• Submissions will be re-run including the hold-out data. Your best submission will count – based on monitoring and discussions with AI Crowd we have not seen “gaming” with frequent submissions
• Therefore, Winning solutions will be under some scrutiny – i.e. if it looks like a statistical fluctuation from many submissions gave an edge, this will be taken into account and adjusted.
• And will consider other factors as well to differentiate between and acknowledge multiple top solutions.

Presentations should be ideally be placed into GIT by EoB Friday.

• Please reach out if you are having trouble or if you have questions with presentations or some extenuating circumstance.
• Please email me directly – some short grace period could be possible if we know in advance.
• Also we can post about getting files into git – don’t stress
• In general - are teams now ready with their presentations??

Hope everyone is feeling good about their solutions and presentation – looking through the code and discussions, it seems there are quite some interesting approaches!

Best,
Nick and team

View post 1

UPDATE / EXTENSION: DSAI Challenge: Leaderboard & Presentation deadlines

Hi Bin,

I am sending an update now - as suggested in the last update - we are no longer limited by last submission, so do not worry there. We have not observed gaming and have ability to rerun on multiple now - see the coming announcement.

Jan 10th EoB EST is what we had planned - but open to feedback. (will give precise time)

View post 0

IMPORTANT INFORMATION - Presentation work in workspaces

Hi Shravan -
I’m sending an update now. In theory we had specified the submitted presentation timeline was also 10th. Please email me if you have concerns about the deadline, open to feedback.
Best,
Nick
nicholas.kelley@novartis.com

View post 0

Reminder: All code must be checked into GIT to validate Leaderboard submissions

Hi all,

First off - for the most part repositories and reproducibility of solutions are looking good!

Just a reminder that to be eligible - all code surrounding the various aspects of what was needed to enable your model’s Leaderboard predictions must be submitted to GIT. Top performing models will need to be validated/interrogated by team (information leak, etc) as well as ensuring we have reproducible solutions.

“Special circumstances”, uncertainty or questions if having trouble - just reach out to Shivam and the AI Crowd team - they will be looking to help!

Thanks!
Nick

View post 1

UPDATE / EXTENSION: DSAI Challenge: Leaderboard & Presentation deadlines

Dear all,

IMPORTANT UPDATE: We have made some necessary changes to the test data set which will allow for better balance and stability as we transition to the final leaderboard, and may cause the current leaderboard to shuffle. (models will be re-run)

As a consequence of this – in addition to some general requests - we assume it is only fair to allow teams an additional non-holiday week to work on finalizing their submission.

1. Leaderboard submissions will be frozen on Jan 10th
2. The organizing team will decide if an additional grace period will be given for presentation submissions.
3. Machines are accessible from now until Jan 10th

As always, thanks for your continued engagement and support of the organizing team’s efforts around your feedback throughout the initiative.

Thanks and happy holidays

View post 2

Challenge timelines updates/clarifications

Dear all,

We just wanted to clarify a few things on the upcoming Challenge process:

1. Leaderboard submissions will be frozen on Dec 20th – no further submissions will be possible

   • Your final submission will then be used on the holdout data – please reach out to Satya if this causes problem and we will adjust

2. Final presentations can be submitted the week of Jan 6th. Due to perceived need based on questions, guidance coming in a separate mail tomorrow morning.

3. Workspaces will be unavailable starting Dec 23rd until Jan 6th but will be accessible JAN 6th UNTIL JAN 10TH. (thanks Aridhia).

   • This will allow you to get data and relevant files off of them, as well as to upload your final presentation to git if you have not already done so.

4. Once the workstations are decommissioned Jan 10th all data and code on them will disappear EXCEPT:

   • SOLUTIONS AND PRESENTATION MUST BE IN AICROWD GIT REPOSITORY
     i. code/notebooks should happen somewhat automatically if you participated with the leaderboard – PLEASE REACH OUT IF YOU ARE UNSURE.

   • ALL NEW WRANGLED DATA OR FILES SHOULD BE TRANSFERRED TO THE COMMON SPACE – /home/workspace/files/teams/your_team_name – otherwise it will be lost. Copy it over so we can all benefit and you’ll be eligible for wrangling recognition.

Please let Satya and/or myself know if there are major concerns.

View post 1

Presentation guidance/suggestions (due Jan)

Some guidance on presentations due in January – this will live in a live version on the forums should details/suggestions need to be added.

Here are some general tips for what to communicate:

1. Explain what you did for a general audience - clarity, transparency, scientific integrity of information

2. “Confidence in the model”: Convince us that your model performs and is generalizable and robust – especially if you used additional data so that we are sure your performance was not due to information leak.

3. “Model transparency”: To the extent possible, provide insight or transparency on how your model is making predictions – top features, how the features are related or their context, etc

4. “Innovation”: Your top most innovative aspects you focused on either Data Wrangling or Novel methodology / approach

5. “Insights”: What did we learn at a high level about the risks and success factors of trials, etc? (possibly covered in other sections)

6. “Enable business decisions”: Make sure to connect any actual insights or tools/methods that a clinical project team might be able to act on or learn from – or simply see the problem through a different lens.

For example: consider how your model could be used in the context of various decisions – in the context of a portfolio for example, where we might want to understand the degree of correlation multiple assets might be in their outcomes.

Please review the full list of proposed insights to provide:
https://www.aicrowd.com/organizers/novartis/challenges/novartis-dsai-challenge#detailed-evaluation-criteria

View post 0

Final submission selection

Hi all,

We will use the last submission.

This will allow users to decide which submission they feel is their best.

If this becomes problematic based on feedback, we could also give the option to use the submission with the best score. Email incoming today as well about the challenge end - but please comment below.

Best,
Nick

View post 0

New data available - structured phase 3 information

Thanks so much Ned!

And thanks to the Hyderabad team who generously provided this wrangled data set in order to help encourage additional explorations into this portion of the insights challenges!

Reminder to particpants:

YOU MAY NOT USE PHASE 3 DATA IN THE LEADERBOARD PREDICTIONS.

The simple existence of a phase 3 being launched carries information - as well as much of it not being available at the time of the phase 3 investment decision being made.

HOWEVER - PLEAES DO EXPLORE THE INFORMATION IT CARRIES FOR POST-PHASE 2 PREDICTIONS FOR THE INSIGHTS SECTION!

good luck, nick and team

View post 0

Submission of only final predictions file

It would help to go back to the underlying motivation:

• We wanted to reduce the chance of visibly fooling ourselves with top solutions including leaked information, rendering them irrelevant for real world decision making.

• We wanted all top solutions able to be re-run by the evaluation & project team, to be interrogated for generalizability etc. By design, the kubernetes cluster and git combo enables this.

That said, we also want the best solutions possible for the larger initiative at the end of the event - which is why we were trying to ease some of the frustrations which were blocking some teams.

I discussed with the team, and we would highly encourage to continue to predict on the original test data in the evaluation clusters rather than provide a table of solutions. Especially for the final solution.

However, do what you feel you need to do as a team in order to come up with your optimal solution. But keep in mind, the final leaderboard will change when we add in the hold out test data, and winners will need their model to be validated by the evaluation team, so please make it clear how one would load and interrogate your model.

View post 0

Test data matrix available

Please always feel free to reach out to me if you would like to discuss.

View post 0

Test data matrix available

Hi Bjoern,

It would help to go back to the underlying motivation:

• We wanted to reduce the chance of visibly fooling ourselves with top solutions including leaked information, rendering them irrelevant for real world decision making.
• We wanted all top solutions able to be re-run by the evaluation & project team, to be interrogated for generalizability etc. By design, the kubernetes cluster and git combo enables this.

That said, we also want the best solutions possible for the larger initiative at the end of the event - which is why we were trying to ease some of the frustrations which were blocking some teams.

I discussed with the team, and we would highly encourage to continue to predict on the original test data in the evaluation clusters rather than provide a table of solutions. Especially for the final solution.

However, do what you feel you need to do as a team in order to come up with your optimal solution. But keep in mind, the final leaderboard will change when we add in the hold out test data, and winners will need their model to be validated by the evaluation team, so please make it clear how one would load and interrogate your model.

View post 1