for a couple of days now, the website hosting the prize challenge and the bonus exercise is almost completely inoperative for me.
In the prize challenge most of my solutions I upload to the platform are not recognized. I have uploaded roughly 4000 solutions (there may be some duplicates since the upload fails and duplicates aren’t really handled), but I know that at least 3000 solutions are unique. However, only 1370 end up in the prize challenge.
What makes me a little nervous, though, is that I cannot publish any solutions for the bonus exercise, either. I have 358 valid solutions listed in my submission overview for the challenge. However, only 338 are published and I can’t seem to publish the rest. This effectively puts me under the threshold for the grade bonus.
Is there anything I can do until Friday?
Sorry to hear you are still having problems. We’ve had some issues previously with the website upload which have (hopefully) been fixed. At least for the other forum users, the problem with “pending” upload was resolved. Maybe you could help me understand the problem(s) a little more, so we can help you:
prize challenge: The scenarios which are valid for the prize challenge are listed in the “SCENARIOS” Tab when you open the prize challenge on the website. They are about 2000 scenarios which can be downloaded via the link provided in the description of the challenge on the website. Which scenarios are you using?
bonus exercise: Are the missing solutions still stuck in “pending” or can’t they be uploaded at all? Have you checked if these (20 scenarios) in your case are contained in the “SCENARIOS” of the bonus exercise challenge?
First of all, as the Website Team, we would like to apologize for the inconvenience caused. We were trying to fix a performance-related problem in our system for the last couple of days. But good progress has been made on improving it and now the evaluation of the solution files is much faster.
Additionally, can you share your username, so that we can check the issues you mentioned in detail?
Regarding the prize challenge, I ran the 2000 scenarios with GBFS once and, after uploading my solutions, everything worked fine. Then I did some tweaks to my heuristic function and switched to A* Search and ran the same scenarios again. Now the solutions from A* I uploaded only show up as valid under all submissions:
Is there some logic in the server that prevents duplicate solutions in a challenge? Maybe, by chance, the A* search found identical solutions (which feels unlikely but you never know )
I see the 352 solutions I have also uploaded to moodle under the challenge tab, however no matter how often I try to publish all or even individual ones, they don’t appear in the ranking but still show up as unpublished in the list.
My username is tigru.
I didn’t want to reupload all of my solutions since I thought this would cause unnecessary strain on the server. Is that something I should try?
Thanks again, for all your efforts!
Best,
Till
PS: Sorry for spamming, as a new user I can only post one image per post so I split up my posts.
Exactly. If a user uploads duplicate solutions for a scenario in a challenge, then our system picks the best one by comparing their costs. In other words, the solution with the lowest cost is considered the best.
So when we investigated the issues you mentioned, the calculated costs are the same for all duplicate solutions you uploaded. Therefore, the system picks one of those solutions and evaluates it as the only valid solution for that specific scenario.
We also provide some basic statistics regarding your solution entries:
Prize Challenge:
You have made 4441 submissions in total.
The number of unique solutions with success status is 1370. Therefore, you observe this value under the Prize Challenge tab. Since, you published all of them, you also observe the same amount under Solved Scenarios column of the leaderboard.
Bonus Challenge:
You have made 1363 submissions in total.
The number of unique solutions with success status is 352. But 338 of them were published.
Here are the benchmark_ids of the solutions that are not published yet (For Bonus Challenge):
KS2:SM1:RUS_Bicycle-10_1_T-1:2020a
KS2:SM1:RUS_Bicycle-11_1_T-1:2020a
KS2:SM1:RUS_Bicycle-12_1_T-1:2020a
KS2:SM1:RUS_Bicycle-13_1_T-1:2020a
KS2:SM1:RUS_Bicycle-14_1_T-1:2020a
KS2:SM1:RUS_Bicycle-1_1_T-1:2020a
KS2:SM1:RUS_Bicycle-2_1_T-1:2020a
KS2:SM1:RUS_Bicycle-3_1_T-1:2020a
KS2:SM1:RUS_Bicycle-4_2_T-1:2020a
KS2:SM1:RUS_Bicycle-5_1_T-1:2020a
KS2:SM1:RUS_Bicycle-6_1_T-1:2020a
KS2:SM1:RUS_Bicycle-7_1_T-1:2020a
KS2:SM1:RUS_Bicycle-8_1_T-1:2020a
KS2:SM1:RUS_Bicycle-9_1_T-1:2020a
Can you try to publish them? And if you encounter any issues, please report it in this thread
I have now successfully published the scenarios you have listed and they indeed now show that they’re public.
And they even show up in the leaderboard!
)
